watercolour-grpo-hps-led

A LoRA adapter for Qwen/Qwen3.5-35B-A3B, trained with GRPO to paint watercolours by writing p5.brush sketches. This is the middle point of the project's three reward mixes: the generic aesthetic preference model (HPSv3) holds most of the weight, and the pairwise judge, the term that carries the hand-rated reference pool, holds the rest.

Loading it, because the obvious way fails silently

Qwen3.5-35B-A3B declares Qwen3_5MoeForConditionalGeneration and carries a vision tower, so its layers live at model.language_model.layers. AutoModelForCausalLM resolves to the text-only variant, whose layers sit at model.layers, and 700 of the adapter's 920 tensors then fail to match. PEFT reports that as a UserWarning, not an error, so you get the base model back and nothing tells you.

from transformers import Qwen3_5MoeForConditionalGeneration, AutoTokenizer
from peft import PeftModel

base = Qwen3_5MoeForConditionalGeneration.from_pretrained(
    "Qwen/Qwen3.5-35B-A3B", dtype="bfloat16", device_map="auto"
)
model = PeftModel.from_pretrained(base, "HuggingEnvs/watercolour-grpo-hps-led")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-35B-A3B")

If you see Found missing adapter keys while loading the checkpoint, the adapter did not load and you are running the base model.

Training

110 steps in 32h15m on one H200. Launched for 200 steps; the job ran a few steps past 110 before it was cancelled, and this adapter is the step-110 checkpoint, the last one saved. The published rollouts dataset is trimmed to the same 110 steps so the numbers match.

reward	gate 0.05 + length 0.05 + pairwise judge 0.30 + HPSv3 0.60
learning rate	5e-5, constant_with_warmup, 5 warmup steps
scale_rewards	none
LoRA	all-linear, r16, alpha 32. 30,431,360 trainable, 0.0866%
batch	8 generations per step, per_device_batch_size 1, grad_accum 8
sampling	top_p 0.95, top_k 20, max_completion_length 8192

WATERCOLOUR_JUDGE_WEIGHT=0.30 WATERCOLOUR_QUALITY_WEIGHT=0.60 \
hf jobs uv run examples/watercolour_grpo.py --flavor h200 --timeout 96h --secrets HF_TOKEN -- \
  --env-url https://YOURORG-watercolour-env.hf.space \
  --model Qwen/Qwen3.5-35B-A3B --lora --all-linear --bf16 --gradient-checkpointing \
  --subject 'a peach hibiscus' --references 4 \
  --top-p 0.95 --top-k 20 \
  --lr 5e-5 --lr-scheduler constant_with_warmup --warmup-steps 5 \
  --scale-rewards none \
  --steps 200 --num-generations 8 \
  --per-device-batch-size 1 --gradient-accumulation-steps 8 \
  --max-completion-length 8192 --probe-samples 40 --film

The uv header pins no versions (trl, peft, transformers, torch), so a run today will resolve different ones. That is a real reproducibility gap, stated rather than hidden.

Results

	first third	second	third	slope t
reward	0.573	0.740	0.815	+15.6
pairwise judge term	0.43		0.83
HPSv3 term	0.63		0.82
paint coverage	0.128		0.298

Absolute rewards are not comparable across reward mixes: each run optimises a different blend. This was the smoothest climb of the three runs, still inching upward when it was stopped (+0.0023/step over the last 30 steps).

Best group mean 0.885, at step 72, and the best single rollout of the run is 0.91. The pairwise judge term climbed from 0.43 to 0.83 even at 0.30 weight, and paint coverage more than doubled, from 0.128 to 0.298, where the judge-free hps-only run barely moved it.

The base model's probe before training: reward 0.464, judge term 0.285, paint coverage 0.083, over 40 samples. Every training number here is recomputable from watercolour-rollouts-hps-led.

Siblings

run	judge	HPSv3
judge-led	0.60	0.30
hps-led	0.30	0.60
hps-only	0.00	0.90

Limitations

• One subject, a peach hibiscus, and one library. It does not generalise to other drawing tasks.
• The pairwise judge is the noisiest reward term, and its consistency (scoring the same image twice) has not been tested.
• Reproducing it needs an H200, an a100-large Space for HPSv3, a cpu-upgrade Space for the environment and inference quota for the judge. It is not cheap.

Method reproduced from Surya Narreddi's "RL'ing Qwen to paint with code". Internally this run is v22c, HF job 6a95468c0718b0f6d890881b, adapter at revision 39f9fa0f32 of the training repo.

Where this comes from

Part of Paint with Code, a complete recipe: the environment, the pool that defines the reward, the trainer, the curves and every rollout.

the recipe, and how to reproduce it	02-watercolour/
the environment	envs/watercolour/
the trainer	train/watercolour_grpo.py
the reference pool	watercolour-reference-pool
the trained adapter	watercolour-grpo-hps-led
every rollout	watercolour-rollouts-hps-led