MMR-RL-only: S3 (MMR + Steep Softmax + TIES) merged checkpoint

Merged model of Qwen3-8B GRPO RL checkpoints trained on AppWorld, produced by the S3 recipe: Maximal Marginal Relevance (MMR) checkpoint selection + steep softmax weighting + TIES sign-consensus merge, all in float32.

Recipe

component	value
pool	11 checkpoints (step 200–300) of a single GRPO run from the raw pretrained Qwen3-8B (no SFT)
behavior vector	per-task score vector (57 AppWorld dev tasks, T=0.5 n=1)
selection	MMR, λ=0.7, K=5 → {250, 270, 280, 290, 300}
weights	softmax(q/τ=0.02) → 270/300 ≈ 0.456 each, 280 0.079, 250 0.006, 290 0.002
operator	TIES (task vector sign election + magnitude aggregation)
base (task-vector zero point)	raw pretrained Qwen-8B
dtype	float32 (compute + storage)

Results

AppWorld official eval stack (same code path as training validation), T=0.5, n=1, fp32 engine:

eval	score
dev pass@1 (57 tasks)	0.7544 (43/57)
test_normal pass@1 (168 tasks)	0.4583 (77/168)

Best single checkpoint (270/300): dev 0.6667 → +5 tasks over the best single checkpoint.

Key ablation: pruning the bank to the two strongest members ({270, 300} equal weight) drops dev to 0.5439 (−12 tasks). The gain comes from near-zero-weight members participating in the TIES sign election — "wide bank + steep weights" are jointly necessary, not winner-take-all.

⚠️ Note: dev@1 (T=0.5, n=1) has a measured noise band of ±6/57 tasks between two identical-model runs; the +5-task margin is at the edge of this band and awaits T=0.9 n=8 confirmation.

The same recipe transferred to an SFT+RL pool gives +13 tasks over the best single checkpoint on test_normal (0.5238 vs 0.4464), confirming cross-pool generalization of the recipe.

Reproducibility

Full pipeline (weight generation, mergekit YAML, merge driver, eval protocol, paired statistics, frozen inputs and checksums): https://github.com/YimingZhou2002/s3-mmr-merge

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("GaryZhou/MMR-RL-only", torch_dtype="float32")
tokenizer = AutoTokenizer.from_pretrained("GaryZhou/MMR-RL-only")

Weights are stored in float32 (31 GB). If you cast to bfloat16 for inference, be aware that merge-induced deltas partially live below bf16 resolution (~80–88% of small-magnitude updates are rounded away); for faithful reproduction of the numbers above, keep fp32 end-to-end.