Guido-1-0.5B-Base
A ~553M-parameter math-focused base language model (PiCOFormer, dense decoder-only), pre-trained from scratch on ~77.6B tokens of a math-dominant corpus. Tokenizer: Mathstral-32k. This is a base model — not instruction-tuned, not aligned.
Architecture
- 24 layers · d_model 1280 · 20 heads (head_dim 64) · FFN 3840 · ~553M params
- Channel mixer: SwiGLU (GLU + SiLU)
- Attention: gated multi-head (sparse per-head output gate,
gate_input_dim=128), RoPE base 1e6, QK-norm - Tied input/output embeddings · logit soft-cap 30 · smear-gate
- fp32 master weights, bf16 autocast
Training data — Corpus v5 (QA-dominant + full math, ~77.6B tokens)
Mixture during the stable phase (QA/CoT 0.65 / math-web 0.35):
| dataset | weight | type |
|---|---|---|
| OpenMathReasoning (CoT) | 0.18 | QA/CoT |
| OpenMathInstruct-2 | 0.16 | QA |
| TinyGSM | 0.07 | QA (code) |
| Nemotron-Math | 0.06 | QA |
| GLM-reasoning | 0.06 | CoT |
| NuminaMath-CoT | 0.05 | QA/CoT |
| NuminaMath-1.5 | 0.04 | QA |
| Orca-Math | 0.03 | QA |
| FineMath-4+ | 0.13 | math-web |
| FineMath-3+ | 0.12 | math-web |
| Nemotron-CC-Math-4+ | 0.10 | math-web |
Data annealing: the final ~12% of training switches to a decay mixture that reinforces the boxed/code answer format (more OpenMathInstruct/TinyGSM/Orca), which keeps the answer-format healthy through warmdown.
Training recipe
- Optimizer: Muon (Polar-Express orthogonalizer, matrix LR 0.01) for 2-D weights; Adam for embeddings (LR 5e-3) and scalars (LR 4e-2).
- Schedule: WSD — warmup 2000 · plateau · linear warmdown 18000 to ~0 (decay-to-zero).
- Batch / compute: 524,288 tokens/step · 148,000 steps · 4 nodes × 4× A100-64GB.
Prompt format (matches pre-training — use as-is, zero-shot)
- MATH:
### MATH\n\nProblem: {problem}\nSolution: - GSM (code):
### TinyGSM\n\nQuestion: {question}\nAnswer:\n
Results — full benchmarks, post-warmdown
| Benchmark | pass@1 (greedy) | maj@16 | pass@16 |
|---|---|---|---|
| GSM8k (n=1319) | — | 67.0% | 78.2% |
| MATH-500 (n=500) | 29.2% | 39.0% | 55.2% |
maj@16 = self-consistency (T=0.6, top_p 0.95, top_k 40). GSM is solved by generating
executable Python; MATH answers are \boxed{{}}. MATH pass@1 (greedy) is a slight
under-estimate (~16% of greedy completions truncate at 1024 new tokens).
Loading (via Aplos)
from Vathos.guido import load_guido
# this checkpoint uses the SwiGLU channel (separate expand/up projections + SiLU):
model, cfg = load_guido("Paerle/Guido-1-0.5B-Base", channel="torch_swiglu")
Requires the Aplos repo (https://github.com/MarioPaerle/Aplos). The torch_swiglu channel
loads the checkpoint 1:1 (no weight fusion) and runs anywhere PyTorch runs. SFT/GRPO
examples + docs: examples/guido_{sft,grpo}.py, docs/GUIDO_GRPO.md.
Limitations
Base, math-domain, English-only. Not instruction-tuned / not aligned. GSM is solved via generated Python code — a fraction (~20%) of generated programs fail to execute.