JugnuLM-110M-R3 (value residuals + Muon + data blend)

What "R3" means. This is rung 3 of the JugnuLM phase-2 ablation ladder. We take rung R2 — JugnuLM-110M-R2 (value residuals + Muon) — and change one thing: the training data. Instead of single-source FineWeb-Edu, R3 trains on a blend (per-sequence weighted mix), holding architecture, optimizer, schedule, tokens, batch, and seed fixed.

What changed: the data blend

source	weight	role
FineWeb-Edu (sample-10BT)	55%	language + general knowledge
DCLM-baseline	35%	diversity → broader distribution
FineMath-4plus	10%	light math reasoning

Sequences are drawn per-step from a weighted-random source (a shuffled blend, not a staged curriculum). Validation stayed a fixed FineWeb-Edu held-out set for comparable curves.

Result — an honest negative for ARC

110M rung	BLiMP ↑	ARC-Easy ↑	WikiText-2 byte-ppl ↓
R0 — baseline	81.25	52.48	1.95
R1 — + value residuals	81.10	54.67	1.94
R2 — + Muon	80.78	56.10	1.932
R3 — + data blend	81.79	53.62	1.9092

R3 has the best BLiMP and best perplexity of any rung — but ARC-Easy dropped −2.48 vs R2 (56.10 → 53.62). Since ARC-Easy is our binding constraint, R3 is not kept in the ladder (it does not beat R2 on the metric that gates leaderboard rank; its EFF is marginally below R2's). We publish it anyway, as the ablation result and as our strongest grammar/perplexity checkpoint.

Why the blend helped BLiMP/perplexity but hurt ARC

FineWeb-Edu's educational-quality filtering is closely aligned with ARC-Easy's grade-school science questions. Diluting it 100% → 55% with general web text (DCLM) and math (FineMath) improved broad language modeling (grammar, perplexity) but removed ARC-relevant educational signal — DCLM is diverse but less science-dense, and FineMath targets math reasoning, not ARC science. The lesson: at this scale, for ARC-Easy the educational distribution matters more than raw diversity. Evaluated with EleutherAI lm-evaluation-harness.

⚠️ Loading — requires trust_remote_code

Value residuals are a custom attention pathway, so this model must be loaded with the included modeling file. Loading it as a stock Qwen3 silently drops the residual and understates quality.

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("altslate/JugnuLM-110M-R3")
model = AutoModelForCausalLM.from_pretrained("altslate/JugnuLM-110M-R3", trust_remote_code=True)
ids = tok("The theory of relativity was developed by", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=30, repetition_penalty=1.3)[0], skip_special_tokens=True))

Architecture & training

Identical architecture/optimizer to R2 (value residuals + Muon): Qwen3-arch, 23 layers × 576, GQA 9/3, RoPE, SwiGLU, RMSNorm, QK-Norm, tied embeddings, SmolLM2 tokenizer (49,152), z-loss 1e-4. Trained from scratch on ~8.4B tokens of the 55/35/10 blend above, Muon (peak 2e-2) + AdamW (1.5e-3) on a shared cosine, ~0.5M-token global batch, bf16, DDP on 4× RTX PRO 4500 Blackwell. Only the data differs from R2.

Intended use & limitations

Research base model (not instruction-tuned). Best-in-family English fluency/grammar and lowest perplexity, but weaker ARC-Easy reasoning than R1/R2. Occasional repetition, English-only. Not for production.

Links

• Previous rung (best ARC): altslate/JugnuLM-110M-R2
• Baseline: altslate/JugnuLM-110M
• Training code & ablation ladder: https://github.com/AltSlate-Labs/jugnu