BossCalabrese/guido-sft-10b

Guido-0.5B — Final SFT (10B tokens): complete training card

Full, reproducible record of the final Supervised Fine-Tuning of Guido-0.5B: every data source and count, the cleaning/decontamination pipeline, the exact curriculum schedule, every hyperparameter, the multi-node launch, the checkpoints, and the measured results. Run on CINECA Leonardo (A100). SLURM job 48442615, launched 2026-07-03.

---

1. Model

• Base: Paerle/Guido-1-0.5B-Base, checkpoint step_148000_final.pt — 553.4M params, vocab 32768, Route-2 PiCOFormer (d_model 1280, 24 layers, 20 heads, gated-MHA, qk_norm, RoPE base 1e6, ffn_mult 3, logit_softcap 30, tied embeddings, smear-gate). Warm-start: loads missing=0 unexpected=0.
• Tokenizer: Mathstral-7B-v0.1 (SentencePiece, 32768).
• SFT is a warm continuation of this pretrained base.

---

2. Data — sources, counts, tiers

The SFT pool is 7.497 B unique tokens / 9,151,399 examples across 25 sources, tokenized into per-(tier, band) buckets (sft_pretok_FINAL). Sampled to 10.0 B tokens over the run (~1.33 average passes; per-tier passes in §4). Tiers are assigned from measured teacher solve-rate (easy ≥65%, med 40–65%, hard <40%). is_teacher = solution was generated by the Qwen2.5-Math teacher; otherwise it is a dataset's own reference/worked solution.

source	examples	tier	schema	teacher-gen
tg_PromptCoT	1,300,830	med	prob→sol	✅
or_caco (Caco-1.3M)	1,229,008	easy	instr→out	—
or_hard80 (hard80easy20)	925,821	med	prob→sol	—
or_scalequest (ScaleQuest-Math)	925,512	med	q→r	—
or_stage1 (Paerle stage1_sft)	897,591	med	prob→sol	—
tg_Eurus	793,006	med	prob→sol	✅
or_rephrase (Paerle rephrase_aug)	670,244	easy	prob→sol	—
or_aops (AoPS-Instruct)	557,600	hard	messages	—
or_dart (DART-Math-Hard)	376,976	hard	q→r	—
or_deepmath_r1 (DeepMath-103K)	290,438	hard	deepmath-R1	—
tg_Skywork	241,575	hard	prob→sol	✅
or_mathinstr (MathInstruct)	239,474	med	instr	—
tg_DeepMath	191,895	med	prob→sol	✅
openr1_R1 (OpenR1-220k)	146,503	med	openr1 first-verified gen	—
or_sciinstr (SciInstruct)	83,810	med	content/summary	—
or_mathcode (MathCodeInstruct)	69,266	easy	messages	—
tg_DeepScaleR	65,870	hard	prob→sol	✅
tg_MiroMind	52,149	easy	prob→sol	✅
tg_MathInstr	40,979	hard	prob→sol	✅
tg_SimpleRL	25,942	easy	prob→sol	✅
tg_DAPO	17,870	hard	prob→sol	✅
or_jee (jee_math)	4,353	med	jee	—
or_deepscaler (DeepScaleR)	3,390	hard	prob→sol	—
tg_MathOdyssey	934	hard	prob→sol	✅
or_odyssey (MathOdyssey)	363	hard	odyssey	—

Per-tier: easy 2,046,609 ex / 1.234 B tok · med 5,508,795 ex / 3.872 B tok · hard 1,595,995 ex / 2.391 B tok. Per-band (by sequence length): band0 (≤2048) 5.333 B · band1 (≤4096) 0.578 B · band2 (≤8192) 1.587 B. The long chain-of-thought lives in band2 (hard deepmath_r1 + med openr1_R1).

---

3. Data pipeline (three cleaning passes)

(a) Verification filter — only correct. For every teacher-gen source we load ONLY the *_correct.jsonl split (verification == "correct", 100% of rows in that file). The upstream 3-way split is correct / wrong / unverifiable; we drop wrong AND unverifiable (e.g. PromptCoT −986k wrong / −150k unverifiable; DeepMath −108k / −61k). For openr1_R1 we pick the first verified-correct generation (correctness_math_verify). Non-teacher sources use their own reference solutions as-is.

(b) 7-eval decontamination (decontam2026/). Every source is decontaminated against MATH-500, GSM8K, AIME24, AMC, AIME25, HARP, OlympiadBench-EN using English word-level 13-gram + LCS lexical matching and BAAI/bge-large-en-v1.5 semantic matching. Removes train/eval leakage on the benchmarks Guido is judged on.

(c) English-only filter (sft_curriculum_final.py::is_english, load-time). The decontam ran English-only methods on any-language text (no language filter), so we added one:

• Dropped or_mathnet entirely (international-olympiad data, ~30% French/Romanian/Russian).
• Per-row is_english: drops non-Latin scripts (Cyrillic/CJK/Arabic/Hebrew) and high foreign-function- word density; deliberately conservative so symbol-heavy English math is never over-dropped. Catches the Qwen teacher's Chinese code-switching (e.g. 一步步分析) — ~3-12% of teacher sources (DeepMath ~12%, DAPO/MiroMind ~6%, SimpleRL/DeepScaleR/MathInstr/Skywork ~3-5%; or_* ~0%). Net −2-3%. Full method: ENGLISH_FILTER_METHOD.md.

(d) GRPO hold-out. A deterministic 2% (by problem-hash, per source, on rows with a verifiable gold answer) is reserved OUT of SFT before tokenization, so downstream RL never trains on SFT data (+ a pre-existing 1% slice).

Tokenization (pretokenize_sft.py) writes BFD-packed (tier, band) buckets (~99.9% fill) with a per-token loss mask (\n<output>...</output>\n blocks masked — harmless for non-TIR data, all solution tokens loss=1).

---

4. Curriculum — 4 stages

Length-and-difficulty curriculum. Each stage packs only examples that fit its sequence length (band0 ≤2048, band1 ≤4096, band2 ≤8192), weighted by tier. Global batch is constant at TOKENS_PER_STEP = 262,144 at every stage (micro-batch × grad-accum adjust so tokens/step is identical) — this keeps the LR schedule valid across sequence lengths and world size.

stage	seq	tiers easy:med:hard	eff-batch (seqs)	steps	tokens	checkpoint
1	2048	0.5 : 0.3 : 0.2	128 (bs4×accum4×8gpu)	8,901	2.33 B	CP1
2	4096	0.2 : 0.4 : 0.4	64 (bs2×accum4×8)	8,901	2.33 B	CP2
3	8192	0.1 : 0.45 : 0.45	32 (bs1×accum4×8)	8,901	2.33 B	CP3 (= 7B)
4 (continuation)	8192	0.1 : 0.45 : 0.45	32	11,444	3.00 B	CP4 (= 10B)

Total 38,120 steps = 10.0 B tokens. Stages 1-3 = the 7B curriculum (equal 33/33/33 token split); stage 4 = a 3B continuation on the stage-3 config, which doubles as the LR warm-down tail.

Per-tier passes over the unique pool: easy 1.76×, med 1.04×, hard 1.59×. The .1/.45/.45 weighting on stages 3-4 was chosen so the richest pool (med — includes the proven hard80easy20) gets ≥1 full pass. The long-CoT band2 data is only eligible in stages 3-4 and is seen ~once there. Sampling is BFD-bucket with replacement (bins re-shuffle when a stage exhausts them) — not literal epochs.

---

5. Hyperparameters (every one)

	value	notes
optimizer	Muon (matrix params) + Adam (embeddings, scalars)	Mario's split
learning rate (div5)	matrix 2.0e-3, embed 1.0e-3, scalar 8.0e-3	= base LR / 5
LR schedule	single cosine over all 38,120 steps	one continuous decay
warmup	3% ≈ 1,144 steps	linear
min_lr_ratio	0.0	decay-to-zero (D2Z); the 3B continuation is the low-LR tail
Muon momentum	0.95 (constant, muon_warmup_steps=0)
Adam betas / eps	(0.9, 0.95) / 1e-8
weight decay	0.0
grad clip	1.0 (global norm)
Muon backend steps	5 (Newton-Schulz)
precision	fp32 master weights + bf16 autocast	mixed precision
loss	fused cut-cross-entropy, packed, response-masked	loss only on solution tokens
tokens/step (global batch)	262,144	constant across stages & world size
per-micro token budget	8,192	OOM-safe
torch.compile	on (per-stage; stage-4 is a cache hit of stage-3)
KEEP_LAST	4	retains CP1-CP4
SAVE_EVERY	100 steps (~26 min)	rolling resume.pt for manual resubmit
seed	0
total	38,120 steps = 10.0 B tokens

Why div5 (not div3). A LR grid on the earlier 1B run found div3 best at <1 epoch. The 10B run makes ~1.3–2 passes, so the more conservative div5 avoids over-fitting/memorization across the extra epochs.

Why min_lr_ratio = 0.0 (D2Z). The cosine anneals LR to ~0 across the run so CP4 is a fully-annealed checkpoint — the best final model and the clean base for downstream RL/tool-use. Decay-to-zero beats a fixed floor (WSD schedules decay to ~0 over the final ~10%; Why Linearly Decaying the LR to Zero Works, arXiv 2502.15938). Code-safe: mult = min_lr_ratio + (1-min_lr_ratio)·cosine → 0.0 reduces to cosine, no NaN; the last real step runs at a tiny non-zero LR (observed 8.11e-7).

---

6. Multi-node launch (Mario's proven recipe)

2 nodes × 4× A100-64GB = 8 GPU, boost_usr_prod, qos normal, 24 h wall. sft_train_final.slurm:

#SBATCH --nodes=2 --ntasks-per-node=1 --gres=gpu:4 --cpus-per-task=32 --time=24:00:00
export NCCL_SOCKET_IFNAME=ib0            # auto-detect hangs the NCCL watchdog at >=3 nodes on Leonardo
export NCCL_IB_TIMEOUT=22 ; export NCCL_DEBUG=WARN     # do NOT set NCCL_IB_HCA (mlx5 ambiguity)
export SLURM_EXPORT_ENV=ALL
export TORCHINDUCTOR_CACHE_DIR=${TMPDIR:-/tmp}/inductor_${SLURM_JOB_ID}   # node-local per-job
export TRITON_CACHE_DIR=${TMPDIR:-/tmp}/triton_${SLURM_JOB_ID}
head_ip=$(srun --nodes=1 --ntasks=1 -w "$head" hostname --ip-address)     # IP, not hostname (c10d needs it)
srun --cpus-per-task=$SLURM_CPUS_PER_TASK python3 -m torch.distributed.run \
  --nnodes=$SLURM_NNODES --nproc-per-node=4 \
  --rdzv-id=$SLURM_JOB_ID --rdzv-backend=c10d --rdzv-endpoint=${head_ip}:29513 sft_train_fast.py

Notes: torchrun is not on PATH (torch is on PYTHONPATH) → python3 -m torch.distributed.run. Compute nodes are air-gapped → HF_HUB_OFFLINE=1 + cached base checkpoint. Global batch is invariant to world size (grad-accum halves for 8 GPU vs 4), so div5 LR is unchanged. Multi-node gives ~1.5× wall speedup (Muon does a full-model cross-node all-reduce each step → ~78% efficient) — enough to fit 10B in one 24 h job. A 30-min dbg smoke test (job 48439644) validated the full path before launch.

---

7. Checkpoints

OUT = /leonardo_scratch/.../models/sft_final/. KEEP_LAST=4 keeps all four stage checkpoints:

• CP1 sft_stage0_step8901.pt — end of stage 1 (seq2048).
• CP2 sft_stage1_step17802.pt — end of stage 2 (seq4096).
• CP3 sft_stage2_step26703.pt — end of stage 3 (= 7B).
• CP4 sft_stage3_step38120.pt + sft_final.pt — end of stage 4 (= 10B, fully annealed) — the primary deliverable and the base for the calc/check TIR stage (CP5) and GRPO. Plus a rolling resume.pt (every 100 steps) for a manual resubmit if the 24 h wall is hit. Each ≈ 2.79 GB (model + Muon/Adam optimizer state).

---

8. Measured results (this run)

• Throughput (8× A100, fp32-master/bf16): seq2048 ~1.87 s/step, seq4096 ~1.50, seq8192 ~1.5–1.6. Full run ≈ 17 h wall (≈ 135 A100-hours) — fits the 24 h job with margin.
• Health: 0 OOM, 0 NaN, 0 non-finite grads across the run; GPU mem steady ~19.9 GB / 64.
• Per-stage training loss (first→last, response-masked): stage1 0.389→0.213 (↓), stage2 0.340→0.272 (↓), stage3 0.457→0.329 (↓) — each stage decreases; loss stays low/stable (a warm-started base refining, not a pretraining descent). LR followed the single cosine: warmup→2e-3 peak by ~step 1200, decaying to ~0.

---

9. Difference vs the previous SFT (1B tokens)

	1B run (sftprod, div3)	this 10B run
tokens	1.034 B (3,944 steps)	10.0 B (38,120 steps) — ~9.7×
hardware	1 node × 4 GPU	2 nodes × 4 GPU = 8 GPU
stages	3	4 (+3B continuation → CP4)
data	MATH-500/GSM8K-only decontam, no language filter, tg10 slice	7-eval decontam + English-only + 2% GRPO hold-out, full pool
LR	div3	div5 (safer at ~2 epochs)
min_lr_ratio	0.05	0.0 (D2Z)
stage-3 tiers	.1/.3/.6	.1/.45/.45 (med ≥1 pass)
KEEP_LAST	3	4

Unchanged: base checkpoint, TOKENS_PER_STEP 262,144, per-micro budget 8192, Muon momentum 0.95, warmup 3%, betas/eps/weight-decay/grad-clip, the 2048→4096→8192 sequence ramp, Muon+Adam split, response-masking.

In one line: same base and same training shape, but ~10× more data that is much cleaner and more diverse (7-eval decontaminated + English-only), +1 continuation stage, a more conservative LR (div5), and a decay-to-zero warm-down for a fully-annealed final checkpoint.

---

Files: sft_train_fast.py (trainer) · sft_train_final.slurm (launcher) · sft_curriculum_final.py (registry + is_english) · pretokenize_sft.py · ENGLISH_FILTER_METHOD.md · FINAL_SFT_RUNBOOK.md · pretok SUMMARY.json. Generated 2026-07-04 for SLURM job 48442615.