andreayhchen/sherlock-qwen2.5-3b-multiobj-v1

Sherlock Qwen2.5-3B, multi-objective GRPO (week 3)

LoRA adapter (r=16) trained with GRPO on top of the merged SFT model, not on base Qwen. To reproduce the base: merge andreayhchen/sherlock-qwen2.5-3b-sft-v1 into Qwen/Qwen2.5-3B-Instruct, then apply this adapter.

Reward

Behind a format gate (final-answer line, exactly one \boxed{}, length band, no markdown), a tiered combination of an LLM judge (RLAIF) and math_verify (RLVR):

correct:  0.5 + 0.5 * judge/10     in [0.5, 1.0]
wrong:    0.2 * judge/10           in [0.0, 0.2]

Every correct rollout outranks every wrong one, but an all-wrong group still has gradient. A strict correct x judge gate would leave 21% of prompt groups with zero spread (19% have every rollout wrong); this leaves 0.2%.

The judge is Claude Sonnet 5 (thinking disabled) scoring persona 1-10. The name "Watson" is stripped from each rollout before scoring, because it was measured to be worth about +1.25 on its own.

Stopped at step 54 of a planned 100; this is the step-50 checkpoint.

Results (200 held-out MATH levels 1-3, greedy)

Model	Accuracy	Persona	Persona, names stripped	"Watson"
SFT v1	59.0%	5.28	-	12%
RLVR only, step 100	68%	-	-	-
This model, step 50	57.5%	5.65	5.36	12%

Paired vs SFT (bootstrap 95% CI): accuracy -1.5 pts (-7.5 to +4.5), persona +0.37 (+0.14 to +0.60).

Persona and capability compete under RL. Optimising correctness alone moved accuracy 59% -> 68%. Adding the persona term consumed that gain entirely, while buying roughly the persona improvement a persona-only reward buys.

Stripping the name stopped the exploit being learned. "Watson" usage held at the SFT rate of 12%, against 33% for an unstripped GRPO run and 74% for DPO, so the +0.37 here is not the name.

Code: https://github.com/harvard-cs2881f26/hw1-angafor-chen