Paerle/guido-grpo-runbook

GRPO on Guido — RUNBOOK (everything needed to run it)

Self-contained operating guide for running GRPO (RLVR) on Guido (Mario's PiCOFormer, ~195M, math-SFT) on the Lagrange / Castelnuovo H100 cluster. Written so a fresh agent can reproduce a run end-to-end without reading the whole history. Companion docs: GRPO_HANDOVER.md (experiment log + diagnoses), smoke_phases/REPORT_smoke_phases.md (budget).

Last updated: 2026-06-24 · Author: Opus agent.

---

0. What this is / goal

De-risk the RL part of the math-LLM thesis on a fast toy (Guido, 195M) before doing it on the 1B model. Concretely: a working, efficient GRPO loop with a verifiable outcome reward, and a clean before/after measurement on MATH-500 (eval-only). Team split: Gabriele + Alessio = SFT + RL (this work); Mario = pretraining/architecture (do NOT make architecture calls for him).

---

1. Hard operational constraints (DO NOT violate)

• ASK-FIRST before any GPU action (sbatch / scancel / srun). A one-time "go ahead" covers that single action only.
• Never run torch / vLLM / sympy / heavy compute on the login node (lagrangectl, 3.8 GB RAM, no GPU). Use Slurm batch jobs on partition mat. python -m py_compile (parse-only) is fine.
• QOS caps (enforced): per-user cpu=12, gres/gpu=1, mem=150G. Only ONE GPU job runs at a time; the rest queue. Partition mat = MaxTime=UNLIMITED (no time-kill — but you still set --time, and Slurm WILL kill at that wall, see §9 job 2312).
• Compute nodes are air-gapped: run with HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1; pre-cache any HF asset on the login node first. Never print/echo HF tokens.
• /home is shared GlusterFS: do NOT launch multi-worker HF downloads on the login node (freezes the H100 + other users). Download on a compute node, single worker.

---

2. Environment & paths

Thing	Path
Project root	/home/prignano/tesi/grpo_h100
Python venv (use this)	/home/prignano/tesi/.venv/ (.venv/bin/python)
Training code	grpo_h100/code/
Vathos model lib (Mario's, our edits)	grpo_h100/Aplos/ (on PYTHONPATH)
Data	grpo_h100/data/
Checkpoints	grpo_h100/ckpt_grpo_*/
Logs	grpo_h100/logs/
HF cache	~/.cache/huggingface (HF_HOME)

Required env for every GPU job (already baked into grpo_caco.sbatch):

export PATH="/home/prignano/tesi/.venv/bin:$PATH"
export PYTHONPATH="/home/prignano/tesi/grpo_h100/Aplos:/home/prignano/tesi/grpo_h100/code:$PYTHONPATH"
export HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

⚠️ code/grpo_train.py contains two sys.path.insert("/leonardo_work/IscrC_YENDRI/...") lines — those are dead Leonardo/CINECA paths on Lagrange (harmless). The imports verifier, grpo_step_example actually resolve from code/ via PYTHONPATH. Keep code/ on PYTHONPATH.

⚠️ vLLM is unusable on these nodes (llguidance needs GLIBC 2.30, node has 2.28). Generation uses transformers + a custom compiled-decode path (see §4). ~2800 tok/s is the practical wall.

---

3. Model & token

• SFT model: Paerle/GUIDO_test_200M_SFT (private, Mario's). Loaded clean (0 missing/unexpected) via guido_common.load_guido(dev, dtype, model_dir=SFT_DIR).
• Prompt format: ### MATH\n\nProblem: {q}\nSolution: → final answer in \boxed{}.
• HF tokens (files in project root, NEVER echo):
  • tokenHF_write.txt — write token, user Paerle (models). Use for uploads.
  • tokenHF.txt, token_guido.txt, tokenBossTesi.txt — read / gated-dataset tokens.
  • On Leonardo the SFT-scoped token was …/HF_TOKEN_SFT.txt (different cluster).

---

4. The pieces (built & validated)

4a. Fast generation (prereq)

policy.generate_grpo(..., compile_decode=True, cache_len=CACHE_LEN) uses a CUDA-graph decode path (5–8× decode) that is token-exact vs eager. cache_len sizes the KV buffer / decode-graph to a constant length across ragged prompts → ONE captured graph (no per-prompt recompile). CACHE_LEN = MAXLEN_PROMPT + T. Edits live in Aplos/Vathos/blocks.py + _spatials.py.

4b. GRPO loop — code/grpo_train.py

• Online dynamic sampling (DAPO-style): stream prompts, generate G rollouts/prompt, skip degenerate groups (reward sum 0 or G → zero gradient), accumulate LOCAL_GROUPS usable groups per rank, then one optimizer step.
• Group-normalized advantage; PPO-clipped surrogate + β·KL(π_θ‖π_ref) (k3 estimator); ref = frozen deepcopy of the SFT policy.
• old_lp = new_lp.detach() → ratio≡1 at the step (on-policy single-epoch; removes bf16 noise).
• Score-pass micro-batching (MICRO): the [G, L, V] logits OOM at T=2048, so forward+backward MICRO rollouts at a time; normalizing by the full-group mask makes it bit-exact to one big loss.backward().
• OOM guard: torch.cuda.empty_cache() each step; cudagraph_skip_dynamic_graphs=True prevents the per-call cudagraph-pool explosion (~90 GB) that the compiled decode would otherwise cause.
• Single-GPU or DDP (python3 -m torch.distributed.run --standalone --nproc_per_node=N; torchrun is NOT on PATH). On Lagrange we run single H100.

4c. Reward / verifier — code/verifier.py (REBUILT, validated)

Use verify_answer(pred, gold) for BOTH reward AND eval (so deltas are clean): = numnorm string-match OR Math-Verify(\boxed-wrap) OR calc_verifier(+preprocess).

• preprocess() strips $, \dfrac→\frac, \left\right, ^\circ, %, thousands commas, style wrappers, \cdot→*; extracts \boxed{}; canonicalizes pi/2pi→\pi; trig degrees→radians.
• numnorm() float-normalizes without int-truncation (fixes the 0.5==0.05 false-positive).
• Length-guard skips parsing >80-char junk (also stops NCCL-timeout hangs).
• Validated: test_verifier2.py 45/45 recall + 0/17 FP; test_verifier_deep.py 43/43 + 0/15 FP; Gabriele re-audit test_verifier_gabriele.py 0 FP on 2573 golds. One real bug fixed (integers

  2^53 routed through float64 collided with gold±1 → pure-int fast path added).

• ⚠️ Residual noise still seen in logs: Timeout during comparison and a 'set' object is not callable SyntaxWarning — occasional reward mislabel; on the fix-list, not yet resolved.

4d. Checkpointing — crash/timeout-safe (added 2026-06-24)

grpo_train.py now writes a rolling checkpoint every CKPT_EVERY opt-steps (default 25; 5 in SMOKE). save_ckpt() is atomic: torch.save → *.tmp then os.replace, so a kill mid-write can't corrupt the .pt. It also drops a *.json sidecar (step/usable/raw_seen/min).

• Rolling file: ${OUT}/grpo_policy_latest.pt (+ .json).
• Final file (after the AFTER-eval): ${OUT}/grpo_policy.pt (+ sidecar with before/after MATH-500).
• Effect: a walltime kill now loses ≤ ~50 min, never the whole run (see §9 — why this was added).
• ⚠️ Resume-from-checkpoint is NOT wired yet: a relaunch starts again from the SFT weights; the latest ckpt is at least eval-able / usable as the result. True resume (load latest + optimizer + step counters) is a TODO.

---

5. Data

• Training (current run): data/caco_grpo.jsonl — Caco-1.3M reformatted (1,348,795 rows; problem, gold_answer). ⚠️ NOT decontaminated (user-directed). Seeded from MATH/BigMath/DeepScaleR → MATH-500 contamination risk + Big-Math provenance flag ⇒ MATH-500 delta may be inflated; treat GSM8K as the clean readout (GSM eval not wired yet — see §10).
• Eval (ONLY, never training): data/math500.jsonl (500 rows: problem, answer, level) + a held-out olympiad slice taken from the training pool (indicative only — shares unverified golds).
• Other data files mentioned across the project: PromptCoT shards (Leonardo), Alessio-filtered olympiad filter/clean_olympiad_alessio.jsonl (2573, leaks bad rows — needs a stricter gate).

---

6. Launch a run (ASK GABRIELE FIRST)

Production launcher: grpo_caco.sbatch. It exports the env (§2), then runs code/grpo_train.py with the config via env knobs. Current config:

DATA_FILE=.../data/caco_grpo.jsonl \
G=32 T=2048 MAXLEN_PROMPT=512 LOCAL_GROUPS=8 BETA=0.04 LR=5e-6 CLIP=0.2 MICRO=8 \
TARGET_USABLE=3400 EVAL_N=500 BEFORE_ACC=34.9 CKPT_EVERY=25 \
OUT=.../ckpt_grpo_caco \
python -u code/grpo_train.py

Submit (only after explicit GPU go-ahead):

cd /home/prignano/tesi/grpo_h100
sbatch grpo_caco.sbatch          # one GPU job at a time (QOS); the rest queue
squeue -u prignano

Env knobs (all read by grpo_train.py)

Knob	Meaning	Current
G	rollouts per prompt (group size)	32
T	max new tokens per rollout	2048
MAXLEN_PROMPT	prompt token cap (Caco fits in 512; loss ≈ 0%)	512
CACHE_LEN	KV/decode-graph const length (=MAXLEN_PROMPT+T if unset)	2560
LOCAL_GROUPS	usable groups accumulated per optimizer step	8
TARGET_USABLE	stop after this many usable groups. opt-steps = TARGET_USABLE ÷ LOCAL_GROUPS	3400 (→425 steps)
LR	learning rate (Adam, wd=0 ≡ AdamW)	5e-6
BETA	KL coefficient	0.04
CLIP	PPO clip	0.2
MICRO	score-pass micro-batch	8
EVAL_N	MATH-500 eval size (after-eval)	500
BEFORE_ACC	reuse a known SFT baseline % to skip the ~35 min before-eval	34.9
CKPT_EVERY	rolling-save period in opt-steps (0=off)	25
OUT	checkpoint dir	ckpt_grpo_caco
SMOKE	1 = tiny sanity run	0

Decided/locked: G=32 · online dynamic sampling (skip degenerate) · reward = verifier.py:verify_answer (PRM later) · MATH-500 = eval ONLY · report string-match and Math-Verify side by side.

Abort: scancel <jobid>. (No afternotok chain on this cluster.)

---

7. Monitoring

squeue -u prignano
JID=<jobid>
tail -f logs/grpo_caco_${JID}.out
grep -E "step .* usable|\[CKPT\]|\[AFTER\]|\[DELTA\]|\[SAVED\]|CANCELLED" logs/grpo_caco_${JID}.out

Per-step lines log: usable k/TARGET | raw_seen | mean_reward | win5_reward | loss | kl | peak GB | min.

• mean_reward is logged only over usable groups → pinned by construction; it is an artifact, not a clean learning signal. Real signal = in-distribution eval (+ skip-rate, currently untracked).
• Watch peak GB for drift (OOM guard) and kl for collapse.

---

8. Eval

• In-loop AFTER-eval = greedy pass@1 on MATH-500 (EVAL_N) + held-out olympiad → [AFTER]/[DELTA].
• Standalone: code/eval_mv.py (re-score saved checkpoints with the good verifier; report string-match AND Math-Verify both, as Gabriele requires).
• MATH-500 baseline for Guido-SFT at T=2048 greedy: 34.9% (job 2277). Older A100 measure: 25.2%.

---

9. War story: job 2312 (why §4d exists)

Overnight GRPO-on-Caco (lr 5e-6, T2048, G32, LG8, β0.04, TARGET_USABLE=3400, --time=15:00:00). Result: TIMEOUT, killed 2026-06-24 10:15 after the full 15 h, ckpt_grpo_caco/ EMPTY — zero artifacts. It reached step 422 of 425 (99%), usable 3360/3400 at 891 min, then hit the wall. Because save + after-eval only ran after the while usable_global < TARGET_USABLE loop, nothing was saved. ~15 H100-h lost. Throughput ran ~10–15% slower than the (job-2294-scaled) estimate, so the 15 h wall had no margin. Fixes: periodic atomic checkpoint (§4d, DONE). Still TODO: give the wall margin (--time up, or TARGET_USABLE down) and add resume-from-latest.

---

10. Open items / next steps (in order)

1. Relaunch GRPO-on-Caco with the checkpoint fix + a wall margin (e.g. --time=20:00:00 or TARGET_USABLE≈3000). This finally yields a saved policy + a MATH-500 delta.
2. Wire resume-from-latest (load grpo_policy_latest.pt + optimizer + step/usable) so a kill is fully recoverable, not just eval-able.
3. Fix residual verifier noise (Timeout during comparison, 'set' not callable).
4. Add GSM8K (in-distribution, clean) eval — MATH-500 is OOD for Caco's GSM-seeded mix and is contamination-risky; GSM is the trustworthy readout.
5. Data gate: Caco is not decontaminated; a verified/decontaminated subset is the clean baseline.
6. PRM axis (later): wire Qwen2.5-Math-PRM-7B as a dense reward and compare vs outcome — Gabriele wants the no-PRM outcome experiment done thoroughly FIRST.

---

11. Key files

Path	What
code/grpo_train.py	the GRPO loop (+ checkpointing §4d)
code/verifier.py	reward + eval verifier (verify_answer, numnorm, extract_boxed_answer)
code/guido_common.py	load_guido, load_tokenizer, SFT_DIR
code/grpo_step_example.py	group_normalized_advantages
code/eval_mv.py	standalone re-eval
code/test_verifier*.py	verifier batteries (recall/FP)
Aplos/Vathos/	model lib (compiled-decode edits)
grpo_caco.sbatch	production launcher
data/caco_grpo.jsonl	training data (Caco-1.3M, not decontaminated)
data/math500.jsonl	MATH-500 eval (eval ONLY)
docs/GRPO_HANDOVER.md	full experiment log + diagnoses
smoke_phases/REPORT_smoke_phases.md	H100-hour budget (~163 h for SFT+RL plan)