alpha-nanochat-d24

nanochat's d24 base model (1,384,122,122 parameters), trained for the whole of Andrej Karpathy's nanochat speedrun record recipe (leaderboard Run 6, commit a825e63): 5,568 steps of 2²⁰ tokens (5.84 billion tokens of ClimbMix), FP8 matrix products, Muon + AdamW. Every step ran in a trainer compiled by the Alpha compiler straight to sm_90 machine code, with no CUDA runtime, NCCL or PyTorch on the training path.

This is a base (pretrained) model. Try it: Space: alpha-nanochat-d24-base (text completion). Its chat version, after nanochat's own SFT, is ajaxdavis/alpha-nanochat-d24-sft (chat Space).

The run

Hardware	2× H100 SXM (NVLink NV18), RunPod, driver 595.91.07
Steps	5,568 (base_train's derived horizon at data:params ratio 8)
Tokens	5,838,471,168
Training time	23,555 s (6 h 32 min), one uninterrupted invocation
Throughput	247,865 tokens/s (nanochat a825e63 on two H100s: 247,886)
Ranks	2, one a card; optimizer state sharded as nanochat's DistMuonAdamW; both ranks' final weights identical
Data	nanochat's own: 170 ClimbMix shards, its trained tokenizer, its loader as each rank of 2
Initialization	nanochat's own d24 init

Training loss (mean of each step's 32 micro-batches, each measured before the step's update):

Step	1	10	100	500	1,000	2,000	3,000	4,000	5,000	5,568
Loss	10.398	6.969	4.452	3.097	2.922	2.799	2.677	2.573	2.496	2.457

The last 100 steps average 2.478. The live record of the run is at lsalpha.com/training/speedrun.

Evaluation

nanochat's own scripts.base_eval (a825e63), on 2× H100:

Metric	This model	nanochat leaderboard notes
train bits per byte	0.7554	—
val bits per byte	0.7544	0.718 (Runs 4–5, same ClimbMix data)
CORE	0.2089	record Run 6: 0.2626; GPT-2 1.6B: 0.2565

This model does not reach the record's gate. CORE is 0.2089 against GPT-2 1.6B's 0.2565, and its validation bits per byte are about 5% above what nanochat's leaderboard reports for its d24 runs on this data. The run reproduces the recipe's schedule, data and kernels but not its result yet; the gap is being investigated (candidates: the two-rank data order against the record's eight, and the FP8 rounding paths that differ from nanochat's after the first steps).

CORE by task (nanochat's base_eval, 22 tasks, centered accuracy averaged):

Task	Shots	Accuracy	Centered
hellaswag_zeroshot	0-shot	0.4885	0.3181
jeopardy	10-shot	0.0227	0.0227
bigbench_qa_wikidata	10-shot	0.4006	0.4006
arc_easy	10-shot	0.6566	0.5421
arc_challenge	10-shot	0.3643	0.1524
copa	0-shot	0.6600	0.3200
commonsense_qa	10-shot	0.2154	0.0192
piqa	10-shot	0.7280	0.4559
openbook_qa	0-shot	0.3940	0.1920
lambada_openai	0-shot	0.3681	0.3681
hellaswag	10-shot	0.4877	0.3169
winograd	0-shot	0.6154	0.2308
winogrande	0-shot	0.5233	0.0466
bigbench_dyck_languages	10-shot	0.0690	0.0690
agi_eval_lsat_ar	3-shot	0.2435	0.0543
bigbench_cs_algorithms	10-shot	0.3848	0.3848
bigbench_operators	10-shot	0.1286	0.1286
bigbench_repeat_copy_logic	10-shot	0.0312	0.0312
squad	10-shot	0.2571	0.2571
coqa	0-shot	0.2366	0.2366
boolq	10-shot	0.5709	-0.1291
bigbench_language_identification	10-shot	0.2533	0.1785

Files

base_checkpoints/d24/model_005568.pt   nanochat checkpoint (state dict), sha256 e0539e86...aa310b
base_checkpoints/d24/meta_005568.json  nanochat's model config for the checkpoint
tokenizer/tokenizer.pkl                nanochat's tokenizer trained for this run, sha256 387cfc08...a06a6c
tokenizer/token_bytes.pt               its token byte table, sha256 398732da...b54b7
run/README.md                          the run record's summary
run/events                             the run script's timeline, with every artifact's hash
run/ranks-full.elf                     the Alpha-compiled trainer that ran (sm_90 + x86-64 host), sha256 ee64abbb...7c3c16
run/words-full.bin                     the schedule's per-step words the trainer read
run/losses-full.f32                    178,176 binary32 losses: 5,568 steps x 2 ranks x 16 micro-batches
run/stamps-full.bin                    per-step timestamps
run/batches-r0.u32.json, -r1           each rank's first batches' token ids, for checking data order against nanochat
run/base_eval.log                      nanochat base_eval (CORE) on this checkpoint
run/*.log, gpus.txt, topo.txt          setup, build and export logs; the pod's cards and topology

The token table and value tables are bfloat16 (as nanochat's init casts them), everything else binary32.

Use with nanochat

The layout is what nanochat's checkpoint_manager reads. With a nanochat checkout at a825e63:

export NANOCHAT_BASE_DIR=$PWD/nanochat-home
huggingface-cli download ajaxdavis/alpha-nanochat-d24 --local-dir $NANOCHAT_BASE_DIR
# CORE (the record's gate)
torchrun --standalone --nproc_per_node=8 -m scripts.base_eval -- --device-batch-size=16
# then midtraining and SFT as nanochat's runs/speedrun.sh does

How the weights were produced

The trainer, its kernels, its memory plan, the GPU command streams and the host program that drives the driver are all generated by the Alpha compiler from Alpha source (GPT2.SpeedrunBuild). The run script is ranks-full.sh. Rank 0's binary32 masters were exported with export-nanochat-checkpoint.py, whose self-check round-trips every tensor bit for bit through nanochat's loader.