sn120-exp019-S2: MiMo-9B GRPO with unresolved rollouts trained as failures

A controlled 29-step replication of the Affine SN120 MiMo GRPO run (affine-mimo9b-grpo-20261013), started from the same initial checkpoint. It uses the official trainer and settings unchanged except for one switch: cap_policy = affine-unresolved-zero-reward-active-v1. Unresolved rollouts (hit the 8192-token cap, or no gradeable final answer) get reward 0 and stay in the loss, instead of being masked.

A control arm, S0, is identical except it uses the live run's original affine-unresolved-zero-reward-mask-all-v1. S0 reproduces the live run's rise-then-decline on held-out data and its entropy collapse. S2 shows neither.

Recipe

Initial checkpoint	87de8304 = HF XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B (sha256 of all 12 files checked against the signed manifest)
Trainer	subnet/mimo_grpo/training.py::train_update, optimizer.py::CPUAdamW, upstream.py (pinned verl losses), called unmodified from AffineFoundation/affine @ f07c54e
Config	TrainerConfig(max_tasks=32, cap_policy=TRAIN_UNRESOLVED); everything else default = the live run's logged config: group size 8, unnormalized group advantages (no std), prompt-mean aggregation, dual-clip PPO (0.2 / 3.0), 1 PPO epoch, lr 1e-6, AdamW β=(0.9, 0.999), eps 1e-8, wd 0.01 (mcore bias/1-D exclusion), CPU fp32 master weights, fp32 gradient accumulation, grad-norm clip 1.0, KL 0, entropy 0, T=1 for OLD/current logprobs
Runtime	HF Qwen3_5ForConditionalGeneration, bf16, eager attention, torch fallback for linear attention (no fla / causal-conv1d), layer checkpointing; transformers 5.14.1, torch 2.13.0 (the live run uses 2.14.0)
Sampling	vLLM 0.31.0, T=1, top-k 20 → top-p 0.95, max 8192 output tokens, 12,288 context, thinking on, official chat template (token-identical to the live training prompts, 1,248/1,248). On the same texts, vLLM prompt logprobs match HF eager teacher-forcing within 0.0021 per sequence.
Grading	the subnet's v1 rules copied one for one: mimo_answer_projection final channel (including the $$ + bare \boxed{} rejection), completed_answer, affine_math_v1/verify.py (math-verify 0.9.0). Matches the official classification on 1,248 training rollouts and 896 fixed32 rows.
Task selection	uniformly random problems from the 6,746 mining indices, 8 rollouts each, kept if ≥1 correct and ≥1 completed wrong (the v1 gate, min_completed_failure=1); first 32 qualifying groups per step, at most 3 sampling rounds of ≤256 problems. Late steps found only 12–24 qualifying groups (qualifying rate fell to 2–3%). No miner-style task reuse. Seeds depend only on (step, index, nonce), so S0 and S2 draw the same problems per step.
Steps	29 updates. Stopped by a deadline; steps 27–29 are an exact resume from state027 (Adam moments and step count restored, master projection verified).

S2 is the live run's new training policy without the new qualification gate. Our reading of the v2 switch (e000039+: outcome policy ...-unresolved-failure-v2, 128 groups/step, loss moving from about −0.13 to about +0.02) is that unresolved rollouts count as failures both in training and for qualification. Arm S4 of the same experiment replicates that (v2 gate, 32 groups/step) and is still running.

Results (8192 cap; fixed32 × 8 samples, 300 held-out problems × 1 sample; same seeds at every step)

	step 0	10	20	end
S2 heldout300	62.0%	72.7%	77.7%	75.7% (step 29)
S0 heldout300	64.3%	73.3%	71.0%	65.3% (step 30)
S2 fixed32	64.1%	69.1%	71.9%	69.1%
S0 fixed32	59.0%	75.0%	69.9%	66.4%
S2 heldout300 capped at 8192	13.7%	14.3%	16.3%	20.0%
S0 heldout300 capped at 8192	14.7%	13.7%	17.0%	24.7%

• S2 (step 29) − S0 (step 30): heldout300 +10.3 pp (p=0.0001, paired by problem, sign-flip test), fixed32 +2.7 pp (p=0.60). On heldout, the samples the two arms disagree on are mostly S0 truncated → S2 correct (26), then S0 format-rejected → S2 correct (19). The reverse cases are 7 truncated and 6 format-rejected.
• S0 declines from its step-10 peak by −8.0 pp on heldout300 (p=0.01), the same shape as the live run.
• Median training-batch logprob of the sampled tokens under the current policy: S0 −0.245 → −0.028 (live run −0.248 → −0.045 by step 20); S2 stays at −0.25 to −0.30 for all 29 steps.
• Limits: S2 does not stop length growth. Median output length on heldout goes 806 → 2,406 tokens (step 0 → 29), and gains level off after step 20. Under the v1 gate, 23% of sampled problems at step 26 had a correct answer and capped failures but no completed wrong, so they never entered training. Part of S2's gain is format: format-rejected answers are trained as failures. Single seed, 8192-cap evaluation only (no 24576 eval of these checkpoints yet).

Files

• step010/, step020/, step027/, step029/: bf16 weights, loadable with transformers ≥5.14 (Qwen3_5ForConditionalGeneration) or vLLM 0.31. step020 is the best held-out checkpoint (77.7%); step029 is the last. Our save routine copied the base model's README.md / .gitattributes into step010/. They are the base card, not this one.
• state027/, state029/: full optimizer state from the official CPUAdamW.save_local (fp32 masters + exp_avg + exp_avg_sq, 256 MB shards, descriptor.json with per-shard sha256) plus meta.json (step, checkpoint id, descriptor_sha256, task-selection state). Use it to resume exactly: python recipe/train_b.py --arm S2 --model <base dir> --out <dir> --resume <dir containing step029/ and state029/> --steps 40.
• recipe/: experiment scripts. train_b.py drives sampling → grading → group selection → official train_update; gen.py is the vLLM worker; common.py holds prompt + grading; runtime.py is the HF runtime; s3.py / s4.py are the other arms. It also expects the official subnet/mimo_grpo/{config,data,optimizer,training,upstream}.py and the two pinned verl files from affine @ f07c54e under recipe/affine/, plus the task snapshot original-math7496.tasks.json (sha256 77a4524a…).
• results/S2, results/S0: per-step training metrics (steps.jsonl) and per-sample eval grades (eval_stepNNN.jsonl).