Atharva1232/tiny-math-1m

tiny-math

How small can a model be and still learn arithmetic? — answered.

	smallest that passes (≥95% on 4-digit addition)	accuracy
From-scratch (digit vocab)	805,632 params	100.0% (5000/5000)
Real pretrained LLM (fine-tuned, minimal tokenizer fix)	TinyStories-1M, 3.75M params	99.7% (n=2000)
Real pretrained LLM (native BPE, no tokenizer change)	pythia-14m, 14.1M params	99.8% (n=2000)

The closing experiment: a one-rule tokenizer ablation moved TinyStories-1M from 0.2% → 99.7% — same weights, same data, same recipe; only change: number runs tokenize per digit instead of inconsistent chunks (1842 → ['18','42'] native vs ['1','8','4','2'] fixed), same 50,257 vocab, no new tokens. The BPE floor was never about capacity — it was the tokenizer's chop points. Zero-shot, every real LLM up to 134.5M still scores 0%. Full story: VERDICT.md.

Three structural findings: (1) capacity buys sample efficiency and the transition is a cliff (0.81M: 0.2% → 100% between 357k and 1.78M problems); (2) in-distribution perfection ≠ algorithm learning — 100% at the trained digit-length, 0% at every other length, with example answers showing fixed-slot positional choreography; (3) pythia-14m's 99.8% trained only its 1.19M transformer blocks (vocab frozen) — nearly the same trainable capacity as the from-scratch winner.

Model + full results on Hugging Face

Headline from-scratch results

model (actual params)	71k problems	357k problems	1.78M problems
0.81M	0.0%	0.2%	100.0%
3.58M	0.0%	95.7%	100.0%
6.34M	0.0%	98.8%	100.0%

Baselines (bigram count model, untrained model): 0.0% — the floor is zero; everything above it is learned.

Layout

VERDICT.md              # the final answer: tables, chart, disclosures
train.py                # tokenizer + data gen (3 formats) + TinyGPT + train/eval loop
baselines.py            # bigram count model + untrained-model baselines
verdict.py              # combines ALL results -> verdict.json + verdict_chart.png
make_kernel.py          # builds single-file Kaggle kernels (no sibling imports)
nan_autopsy.py          # fp16/AdamW NaN forensics (the pythia-14m saga)
MODEL_CARD.md           # honest report for the from-scratch model
STATUS.md               # full session log: bugs found, decisions, pre-committed rules
kernels/                # every GPU run: bakeoff, sweep, extrapol, quiz, stageb2-6
results/                # all raw results + charts

Methodology notes (the parts that matter)

• Contamination prevented by construction: eval sets built before training streams; streams reject any eval pair; extrapolation control excludes train-seen pairs by exact RNG replay.
• Exact match only, never training loss or shaped rewards.
• Format chosen by measurement (3-format bake-off), not taste.
• Independent tuning vs final eval sets with overlap asserts.
• Eval sizes stated per table (n=5000 from-scratch, n=2000 LLMs).
• One seed per cell; disclosed. Six training rounds for the LLM arm — every null result, bug, and fix is in STATUS.md. Nothing was dropped for being inconvenient.

Citation

@misc{atharvagupta123-tiny-math,
  title = {tiny-math: the smallest model that can do arithmetic (0.81M from scratch; 14.1M as a pretrained LLM)},
  author = {Atharva Gupta},
  year = {2026},
  howpublished = {\url{https://github.com/Its-Atharva-Gupta/tiny-math}}
}