watercolour-grpo-hps-only

A LoRA adapter for Qwen/Qwen3.5-35B-A3B, trained with GRPO to paint watercolours by writing p5.brush sketches. The reward is almost entirely one aesthetic preference model, with the pairwise judge switched off.

This is the first run in the project whose reward curve holds up over its full length.

Loading it, because the obvious way fails silently

Qwen3.5-35B-A3B declares Qwen3_5MoeForConditionalGeneration and carries a vision tower, so its layers live at model.language_model.layers. AutoModelForCausalLM resolves to the text-only variant, whose layers sit at model.layers, and 700 of the adapter's 920 tensors then fail to match. PEFT reports that as a UserWarning, not an error, so you get the base model back and nothing tells you.

from transformers import Qwen3_5MoeForConditionalGeneration, AutoTokenizer
from peft import PeftModel

base = Qwen3_5MoeForConditionalGeneration.from_pretrained(
    "Qwen/Qwen3.5-35B-A3B", dtype="bfloat16", device_map="auto"
)
model = PeftModel.from_pretrained(base, "HuggingEnvs/watercolour-grpo-hps-only")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-35B-A3B")

If you see Found missing adapter keys while loading the checkpoint, the adapter did not load and you are running the base model.

Training

60 steps in 17h46m on one H200. train_loss 0.0127, epoch 0.25.

reward	gate 0.05 + length 0.05 + pairwise judge 0.00 + HPSv3 0.90
learning rate	5e-5, constant_with_warmup, 5 warmup steps
scale_rewards	none
LoRA	all-linear, r16, alpha 32. 30,431,360 trainable, 0.0866%
batch	8 generations per step, per_device_batch_size 1, grad_accum 8
sampling	top_p 0.95, top_k 20, max_completion_length 8192

all-linear reaches the linear-attention projections in all 40 layers and the shared expert, but not the 256 routed experts: those are a fused 3D nn.Parameter and PEFT only selects nn.Linear. Verified after the fact from the published adapter: every lora_B in the language model has a non-zero norm, and the 110 zero-norm tensors are all in the vision tower, which a text-only run was never going to touch.

hf jobs uv run examples/watercolour_grpo.py --flavor h200 --timeout 24h --secrets HF_TOKEN -- \
  --env-url https://sergiopaniego-watercolour-env-v20.hf.space \
  --model Qwen/Qwen3.5-35B-A3B --lora --all-linear --bf16 --gradient-checkpointing \
  --subject 'a peach hibiscus' --references 4 \
  --top-p 0.95 --top-k 20 \
  --lr 5e-5 --lr-scheduler constant_with_warmup --warmup-steps 5 \
  --scale-rewards none \
  --steps 60 --n-episodes 240 --num-generations 8 \
  --per-device-batch-size 1 --gradient-accumulation-steps 8 \
  --max-completion-length 8192 --probe-samples 40 --film

The uv header pins no versions (trl, peft, transformers, torch), so a run today will resolve different ones. That is a real reproducibility gap, stated rather than hidden.

Results

	first third	second	third	slope t
reward	0.579	0.637	0.710	+6.41
HPSv3	0.571	0.633	0.714	+6.47
paint coverage	0.107	0.115	0.143	+4.64
reward std	0.262	0.213	0.153	−5.00
entropy	0.315	0.313	0.295	−2.85

Best group mean 0.811, at step 57, and the best single rollout of the run is 0.869. frac_reward_zero_std stayed at 0.000 for all 60 steps, so no step ever lost its gradient. The last 15 steps have a steeper slope (+0.0084/step) than the run as a whole, so it was cut by the step counter rather than by running out of progress.

What it learned is to stop producing bad paintings, not to paint better ones. Decomposing the rise: +0.0290 comes from failing less often, +0.0017 from the paintings that did render being better. Rollouts scoring under 0.3 fall from 33 to 11 across the run while the best of each group barely moves. Every number here is recomputable from watercolour-rollouts-hps-only.

The after-probe is not reported, on purpose

The run's own post-training probe measured the base model, because the script reloaded the adapter through AutoModelForCausalLM and hit exactly the bug described at the top. Its numbers were indistinguishable from the before-probe and they are not published here. The training curve is unaffected: those 60 steps were measured on the live weights. A corrected generalisation probe is pending.

Siblings

run	judge	HPSv3
hps-only	0.00	0.90
judge-led	0.60	0.30
hps-led	0.30	0.60

Limitations

• One subject, a peach hibiscus, and one library. It does not generalise to other drawing tasks.
• 60 steps is short. The mechanism above projects to exhausting bad rollouts near step 94.
• Reward ceiling is 0.901 by construction, not 1.0: that would need an infinite HPSv3 score.
• Reproducing it needs an H200, an a100-large Space for HPSv3 and inference quota for the judge. It is not cheap.

Method reproduced from Surya Narreddi's "RL'ing Qwen to paint with code". Internally this run is v20, HF job 6a936642984507d9db4ec2a9.

Where this comes from

Part of Paint with Code, a complete recipe: the environment, the pool that defines the reward, the trainer, the curves and every rollout.

the recipe, and how to reproduce it	02-watercolour/
the environment	envs/watercolour/
the trainer	train/watercolour_grpo.py
the reference pool	watercolour-reference-pool
the trained adapter	watercolour-grpo-hps-only
every rollout	watercolour-rollouts-hps-only