test-1-lora (experimental, not a release)

This is my scratch repo for an RL experiment on XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B. The experiment didn't work. I'm leaving the adapters up for transparency. Please don't use them for anything real. This is a write-up of what I did, what went wrong, and what I'm doing next.

TL;DR: I spent about a month running GRPO and then several DPO/SFT "recovery" passes. When I finally ran a proper side-by-side eval, the untouched base model beat every one of my adapters at the thing I cared about most: writing three.js and Blender code that actually runs on the first try. The one place DPO clearly helped was SVG quality. I'm building a better RL environment, and this model won't be released for some time.

What I was trying to do

I wanted a 9B model that can reliably produce code that runs, plus decent SVGs and 3D scenes (three.js and Blender bpy), both in plain chat and inside agent loops. Basically the floor other models in this size class already hit.

What I did

1. GRPO (200 steps)

• LoRA rank 64 / alpha 128 on the language-model linear layers. Vision tower untouched, bf16 frozen base.
• Xiaomi's verl fork in async colocate mode with partial rollouts, on a single GPU.
• 16 prompts per step (later a minimum of 14 groups), 4 rollouts each. LR 2e-5 (briefly 1e-5). Low-variance KL at 0.001. Up to ~49k response tokens per episode.
• One mixed run over a lot of environments:
  • general agent tasks
  • multi-language code repair in Docker
  • web dev
  • Blender and Godot
  • freeform three.js and Blender
  • one-shot SVG
  • hallucination and overthinking (math) environments
• The freeform 3D/web/SVG tasks were graded by actually running the code (page loads, renders, no console errors, bpy builds) plus an LLM vision judge on the render. The chat versions had a repair round where the model is shown the error and has to fix it.

Early on it looked promising. At step 25, held-out hallucination reward went from -0.525 → -0.012. 3D rollouts were passing at 42% (Blender) and 33% (three.js), though that was on the easier, non-freeform tasks.

Then it got messy:

• Grading and harness bugs. Timeouts that failed working three.js pages, a probe that turned its own crashes into notes, loop detection gaps, and a few others. These rewarded and punished the wrong things for long stretches.
• Damaged weights. By step ~125 the model had degraded: word salad late in episodes, format violations in 9–19% of rollouts. I had to roll back to step 75 and redo that whole section.
• Hardware. Driver wedges, a host memory stall that killed the rollout engine, and reboots all cost time.

The run did finish all 200 steps, and I picked step 189 as the final GRPO checkpoint.

2. Post-GRPO "polish" (DPO and SFT)

Testing step 189 by hand, freeform three.js was still full of errors and it almost never one-shot anything. So I collected teacher demonstrations from a stronger model, had them executed and repaired so the "chosen" side actually ran, and tried to polish:

pass	data	focused checks	notes
DPO62 (from 189)	494 train / 47 dev pairs, LR 5e-7, β 0.1	3/12	felt overfit
SFT96 (after DPO62)	384 train / 30 dev rows, 2 epochs	5/12, 2 loops	noticeably worse in real use
RL8 (after SFT96)	8 more GRPO updates	2/12	severe SVG/code loops
DPO54 (fresh from 189)	430 train / 63 dev pairs, 273 tasks	3/12, 6 cut-offs	branch grpo189-dpo54
expanded DPO117 (from 189)	743 train / 79 dev pairs, LR 1e-7	failed its own gate at update 26	stopped at update 117; this is main

Those "focused checks" were only 12 prompts. In hindsight that's way too small to decide anything on, and I was making calls based on noise. The DPO logs also showed the classic displacement pattern: the chosen response's likelihood barely moved (+1) while the rejected side got pushed way down (−7 to −14). The model was learning what not to say rather than how to write working code.

The eval that settled it

I finally ran a matched comparison. All models were merged to q8_0 and served the same way, with the same sampling (temp 0.6, top-p 0.95, top-k 20) and a 32k token cap per answer. The tasks were never seen in training: held-out chat tasks plus new three.js/Blender prompts. Grading is execution-based, with the same vision judge for every model. One sample per task.

First answer actually works (runs/renders without errors):

family (n tasks)	base	GRPO189	DPO62	DPO54
three.js, new prompts (20)	15%	5%	0%	0%
three.js, held-out (3)	33%	0%	33%	0%
Blender, new prompts (12)	0%	0%	25%	17%
Blender, held-out (4)	25%	0%	0%	50%
SVG, held-out (8)	100%	75%	75%	75%

Visual quality of the first answer (vision judge, 0–1, only scored when it rendered):

family	base	GRPO189	DPO62	DPO54
SVG	0.31	0.29	0.22	0.36

Answers that hit the 32k cap before finishing (three.js, 20 tasks):

base	GRPO189	DPO62	DPO54
8	13	12	13

The median three.js answer from every trained adapter is exactly 32,768 tokens, i.e. it ran out. The base model's median is ~21.7k. Several adapter answers never even got to the code. They just kept reasoning until they were cut off.

The samples are small, so I wouldn't read much into any single cell. But the overall picture is consistent:

• GRPO made the model worse at my main target. Three.js went from 15% to 5%, and SVG reliability dropped too.
• The biggest regression is the model not knowing when to stop. It's not just buggy code.
• DPO helped SVG quality (best visual score, and it's visibly better in my own testing), and maybe Blender a bit. It didn't fix three.js or the run-on reasoning.

Why I think it failed

1. Not enough learning signal on the hard tasks. Chat three.js was passing around 9% during training. With 4 rollouts per prompt, most groups were all-fail, which gives GRPO zero advantage to learn from. The only gradient left on those tasks came from penalties, not from examples of success.
2. Too many environments at once on a 9B LoRA. I mixed agent, code repair, hallucination, overthinking, web, 3D and SVG into one run. The easier families got most of the improvement, and the hard ones I actually cared about barely moved.
3. Reward bugs early and often. Every grading bug I fixed mid-run meant some number of steps were trained on a wrong signal. The step-125 damage and rollback were a direct result.
4. Length was never really controlled. The penalties targeted loops and degenerate repetition, not "you thought for 30k tokens and never wrote the file". The eval shows that's now the dominant failure.
5. Long-session recovery was never really trained. Repairs and follow-ups were rare and often cut short, so the model never learned to stop, step back, and fix things across a long session. In practice it hits one error after another.
6. The polish passes were judged on 12-prompt checks. That's how I ended up stacking DPO and SFT passes that felt like progress but weren't.

What's next

I'm working on a better RL environment before touching this model again:

• tasks curriculum-matched so pass rates sit in a learnable range
• explicit control over reasoning length
• reward checks that are tested before a run, not fixed during it
• multi-turn repair training that actually happens
• a fixed, decent-sized held-out eval from day one, so every checkpoint gets measured the same way

I'll likely start again from the base model rather than build on these adapters.

This model will not be released for some time. If you came here looking for a usable checkpoint, the base MiMo-V2.6-Distill-Qwen-9B is honestly the better choice right now.

Branches

Every branch is one cumulative LoRA (r64 / alpha 128). The name is its lineage, oldest stage first:

branch	what it is
main	GRPO step 189 → DPO 117 updates (the last thing I trained; not in the eval above)
grpo189-dpo117	same weights as main, pinned
grpo189-dpo54	GRPO 189 → DPO 54 updates (the "DPO54" column above)
grpo189-dpo62	GRPO 189 → DPO 62 updates (the "DPO62" column above)
grpo189-dpo62-sft96	→ SFT 96 updates (noticeably worse)
grpo189-dpo62-sft96-rl8	→ 8 more GRPO updates (severe SVG/code loops)

Each adapter applies once to the original XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B. Don't stack them.

from peft import PeftModel
model = PeftModel.from_pretrained(base, "armand0e/test-1-lora", revision="grpo189-dpo54")

merge.py merges an adapter into the base in fp32: python merge.py --adapter armand0e/test-1-lora --revision main --out ./merged