ChrisMcCormick/decoderstack-d24

DecoderStack d24

A 1.38B-parameter d24 model pre-trained from scratch with DecoderStack — a single-file training stack with a handwritten forward/backward and a written-out optimizer: no autograd, no torch.optim, no nn.Module. bf16, varlen FlashAttention, 8xH100, 1h51m.

The architecture is the nanochat d24 architecture, so these weights load into nanochat once converted — see Loading into nanochat.

Results

Full horizon: 5,568 steps x 2^20 tokens = 5.84B tokens (data:param ratio 8).

step 05567/05568 (99.98%) | loss: 2.388180 | lrm: 0.05 | dt: 1198.08ms | tok/sec: 875,210 | bf16_mfu: 53.76 | total time: 110.74m
step:5568/5568 val_bpb:0.719042
CORE metric: 0.2517 | total CORE eval time: 80.46s
peak memory allocated: 71569 MiB reserved: 72736 MiB
total training time: 110.74m
minimum validation bpb: 0.719042

	value
parameters	1,384,122,122
min val bpb	0.719042 (varlen packing, 10.5M val tokens — see the caveat below)
CORE	0.2517
train time	110.74 min on 8xH100 80GB
throughput	1,212 ms/step, 864,856 tok/sec (steps 20–99)
precision	bf16 (no FP8)

val bpb curve: 0:3.175835 / 250:0.997472 / 500:0.904081 / 1000:0.846058 / 2000:0.809199 / 3000:0.773139 / 4000:0.747155 / 5000:0.726908 / 5568:0.719042

Against upstream nanochat, run on the same box on the same rebuilt validation shard, DecoderStack tracks 0.2–1.4% behind through step 1000 (where the upstream baseline was stopped) while running ~6% faster per step (1,212 vs 1,291 ms/step). Full head-to-head in logs/baseline_report.md.

⚠️ but 0.719042 is not comparable to an upstream nanochat val bpb, and reads better than a like-for-like number would. Upstream batches with best-fit cropping — it fills each row with largest-doc-that-fits and crops a document to finish the row, then attends causally across the whole row with no document isolation (cu_seqlens appears nowhere in upstream nanochat/gpt.py). DecoderStack packs varlen and isolates documents in validation as in training. Different truncation points, different conditioning context: the two are not scoring the same predictions. Measured below.

Measured 2026-08-01: what this checkpoint scores under upstream's packing

Varlen packing does not exist in upstream nanochat — it is this fork's own work, and every nanochat leaderboard entry is measured with karpathy's best-fit row packer. So a varlen val bpb cannot be read against the leaderboard at all. To get a comparable number, both checkpoints below were re-scored under upstream's row packer, driven with each model's own tokenizer over the same pinned val shard and unwound into varlen batches whose cu_seqlens sit on the row boundaries — the model predicts exactly the tokens upstream would ask for, with exactly the same context.

Both models, both packings, same code, same shard, 41.9M val tokens:

model (step 5568)	best-fit rows	varlen
this checkpoint (d24_decoderstack)	0.737183	0.724316
nanochat-varlen-d24-2026-03-22	0.737535	0.725014

At the 10.5M budget this run originally used: 0.735274 (best-fit) / 0.720717 (varlen). All on the legacy token_bytes basis, matching how the leaderboard numbers were computed; the regenerated basis reads ~0.05% higher — see ⚠️ Tokenizer.

What this says:

1. Against the leaderboard, the honest gap is ~0.019, not ~0.006. The nanochat record is 0.71808 (Run 5) / 0.71854 (Run 4), both best-fit. Like-for-like this checkpoint is 0.737183, i.e. +0.0191 — matching upstream's tokens moves away from the record, not toward it. The 0.719042 headline above invited the opposite impression.
2. The packing penalty is universal, not model-specific: +0.0129 here, +0.0125 for the sibling release. Best-fit truncation costs ~0.0127 bpb whoever runs it.
3. The two checkpoints are near-identical, this one marginally ahead under both packings (−0.00035 best-fit, −0.00070 varlen). Not a controlled comparison — the sibling used FP8 and a different 32k tokenizer — but the two pipelines land in the same place.
4. The 10.5M val budget reads optimistic, not merely noisier: ~0.002–0.004 bpb below the 41.9M number. Upstream's default is --eval-tokens 80*524288 = 41.9M.

Harness check: the sibling run self-reported 0.724772 and is measured here at 0.725014 varlen — 0.03% agreement, using the same loader that run used.

Reproduce: agent-ops/stacks/2026-08-01_1241pm_d24-val-bpb-recalibration/.

Throughput remains the clean comparison: 1,212 vs 1,291 ms/step, ~6% faster per step.

Both numbers also deserve error bars: Karpathy's re-runs of an identical d24 config span 0.0153 in CORE (5 runs, mean 0.261), so our 0.2517 and that 0.2614 are inside each other's noise. Sources in stacks/models/nanochat/reference/METRICS.md.

⚠️ Tokenizer

These weights are tied to the 32k vocabulary in ChrisMcCormick/climbmix_32k_8_170, mirrored here under tokenizer/. It is a different vocabulary from the one in ChrisMcCormick/nanochat-varlen-d24-2026-03-22 — 31,474 of 32,759 merge ids differ. Both are 32,768 tokens and both load without complaint, so nothing will warn you. Pairing these weights with the wrong tokenizer produces garbage, not slightly-worse text.

token_bytes.pt was regenerated 2026-08-01

bpb divides summed nats by summed BYTES, and the bytes-per-token table was built by decoding each id to a Python str and measuring .encode("utf-8") — which corrupts every token that is not valid standalone UTF-8 (a lone continuation byte round-trips through U+FFFD and measures 3 bytes instead of 1). 191 of 32,768 ids were affected here. Upstream fixed the calculation in 2ce972a, but the fix lives in tok_train.py and so only lands when a tokenizer is retrained — a released token_bytes.pt keeps the stale values forever.

tokenizer/token_bytes.pt here is now the corrected table; tokenizer/token_bytes_legacy.pt is the pre-fix one, kept so previously published numbers stay reproducible. Measured on this model, the correction moves bpb by +0.05% (0.724316 → 0.724741 varlen at 41.9M), matching upstream's estimate. tokenizer.pkl is untouched — encode/decode are unchanged, this is purely the byte-accounting used by the metric.

Contents

checkpoints/                        native DecoderStack captures (the source of record)
  model_step001950.pt   2.8 GB      {step, code, weights} -- bf16 live weights + fp32 scalars
  optim_step001950.pt  11.1 GB      {step, t_step, state} -- mantissas + Muon/AdamW moments
  model_step005568.pt   2.8 GB
  optim_step005568.pt  11.1 GB
base_checkpoints/d24_decoderstack/  READY-MADE nanochat checkpoint -- no conversion needed
  model_005568.pt       4.2 GB      fp32 masters + bf16 embeddings, 175 tensors
  optim_005568_rank0.pt 1.0 GB      ZeRO-2 optimizer shards for an 8-GPU SFT run
  ...    _rank7.pt      1.0 GB      (8 x 1.0 GB; see Optimizer state)
  meta_005568.json                  model_config + user_config + run metadata
  meta_001950.json                  metadata only -- step 1950 is not converted
code/run_full_d24_w8.py             the exact training script this run executed
logs/
  d24_decoderstack_20260801_023219.log   full training log
  run_full_d24_w8.sh                     launcher, with the config rationale
  baseline_report.md                     throughput + val bpb vs upstream nanochat
tokenizer/                          the 32k vocab these weights were trained on
convert_ckpt_to_nanochat.py         DecoderStack capture -> nanochat state_dict (+optimizer)
test_convert_ckpt_to_nanochat.py    its test, if you change the converter

Two capture points: 5568 is the end of the run, 1950 is the last uncooled state — the LR/momentum hold ends after update 1949 (N - round(0.65*N)), so 1950 is the capture to branch from if the horizon is ever extended. (Extending it needs a load path that does not exist yet in either codebase — see Optimizer state.)

DecoderStack writes code (the full training script, open(sys.argv[0]).read()) into every model capture, so each .pt carries its own exact source. code/run_full_d24_w8.py is that string extracted from model_step005568.pt.

Loading into nanochat

Step 5568 is already converted. base_checkpoints/d24_decoderstack/ is a nanochat checkpoint dir — download it and the tokenizer into a nanochat base dir and it loads. No conversion step, and the 11 GB optimizer capture is not part of it.

pip install huggingface_hub torch
python - <<'PY'
import os
from huggingface_hub import snapshot_download
snapshot_download("ChrisMcCormick/decoderstack-d24",
                  local_dir=os.path.expanduser("~/.cache/nanochat-ds"),
                  allow_patterns=["base_checkpoints/d24_decoderstack/*", "tokenizer/*"])
PY
export NANOCHAT_BASE_DIR=~/.cache/nanochat-ds

Use a base dir of its own, not a shared ~/.cache/nanochat. nanochat resolves the tokenizer from one global tokenizer/ per base dir, and this vocabulary is not the one any other d24 release uses (see ⚠️ Tokenizer). A base dir holding both this checkpoint and, say, nanochat-varlen-d24-2026-03-22 can only be right about one of them, and build_model only asserts the vocab size — which matches — so it will not warn you.

Then, on varlen-gsm8k (the branch whose SFT/RL pipeline this checkpoint targets; fa-varlen also works for eval):

import os, torch
from nanochat.checkpoint_manager import load_model
model, tokenizer, meta = load_model("base", torch.device("cuda"), phase="eval",
                                    model_tag="d24_decoderstack", step=5568)

Pass model_tag explicitly — with more than one d24* dir under base_checkpoints/, find_largest_model picks between them on depth alone and both are 24.

Verified on an H100 (2026-08-01): 175 tensors, 1,384,122,122 params, shapes and dtypes identical to a freshly built nanochat GPT at this config, strict=True load, and 2.4839 nats/token (0.7255 bpb) on held-out English — in line with the run's own 0.719 val bpb — with coherent greedy continuations (The capital city of France is → Paris.). A scrambled mapping scores 10.4.

Re-converting yourself

convert_ckpt_to_nanochat.py unbanks the per-layer matrix banks, renames tensors to nanochat's module paths, maps value-embedding bank slots back to their layers, and rebuilds the fp32 masters from the bf16 live weights plus the uint16 mantissa held in the optimizer file. You need it to convert step 1950, or to emit optimizer shards for a GPU count other than 8:

python convert_ckpt_to_nanochat.py \
    --model checkpoints/model_step005568.pt \
    --optim checkpoints/optim_step005568.pt \
    --meta  base_checkpoints/d24_decoderstack/meta_005568.json \
    --out   $NANOCHAT_BASE_DIR/base_checkpoints/d24_decoderstack \
    --world-size 8

--optim is optional for the weights. It only supplies the mantissas — the lower 16 bits of the fp32 masters. Without it the fp32 parameters carry bf16 precision, which costs about 0.0005 nats/token (2.4833 vs 2.4828 on a held-out paragraph); skip the 11 GB download unless you want the exact master, or the optimizer shards, which need it.

DecoderStack itself cannot load these back yet. The capture path is write-only by design; a load/resume path is future work.

Optimizer state

SFT works with the model alone. nanochat's chat_sft builds a fresh optimizer and only optionally warm-starts it; with no optimizer shard present it prints starting with fresh optimizer (slightly worse) and carries on.

The optim_005568_rank{0..7}.pt files here remove that "slightly worse". They are the ZeRO-2 shards DistMuonAdamW expects, cut for an 8-GPU run — the shard count must equal the GPU count of the SFT run, because rank r holds only its own slice. For any other world size, re-run the converter with --world-size N (see above); --world-size 1 emits a single rank0 file holding the full unsharded state.

Verified on an H100 (2026-08-01), all 8 shards: 10 groups / 175 params / 23 state entries each, shapes equal to what DistMuonAdamW allocates per rank at this config, accepted by load_state_dict, and reassembling bit-exactly back to the original capture — including the ragged 12-param ve_gate group, where chunk 2 × 8 ranks leaves ranks 6–7 holding only zero padding. End to end, chat_sft's own load block warm-starts from them (23 entries, step 5568) and steps.

meta_005568.json carries a user_config block, which is where chat_sft reads the pretraining LRs from — without it SFT silently falls back to unembedding_lr=0.004 instead of this run's 0.008.

Note that torch's load_state_dict replaces param-group dicts wholesale, so the emitted lr/betas/weight_decay become the optimizer's on load — they default to setup_optimizer()'s own values with weight_decay=0.0 (the SFT setting, and where DecoderStack's cosine-to-zero Muon decay lands), and are CLI-overridable. chat_sft restores its own LRs immediately after loading regardless.

Pre-training resume is out of reach, and it is the dataloader that closes the door, not the optimizer: nanochat's resume needs a dataloader_state_dict to put its tokenizing loader back in the stream, and DecoderStack's pre-tokenized binary loader has no equivalent state to hand over. The data order could not be continued no matter what the optimizer held.

Every buffer does have a counterpart, and the precisions line up on everything except the embedding tables:

DecoderStack	nanochat	precision
.frst_mntm	Muon momentum_buffer	fp32 both
.scnd_mntm	Muon second_momentum_buffer	fp32 both
.exp_avg / .exp_avg_sq	AdamW exp_avg / exp_avg_sq	fp32 both, except wte and value_embeds, which nanochat allocates zeros_like(p) on a bf16 param — so bf16 there, fp32 here
.mantissa (uint16)	—	nanochat's fp32 param is the master; consumed by the converter to rebuild it

DecoderStack is the more precise of the two on the embeddings, deliberately: only the gradients are bf16 for those tables (they are the model's largest tensors), while the moment math stays fp32 — the kernel upcasts on the way in. There is no bf16 AdamW variant; the two AdamW kernels differ in whether the param carries a mantissa, not in moment dtype.

W_O's NorMuon reduction axis differs — nanochat infers it from the shape and lands on -1 for a square c_proj, DecoderStack states residual_dim = -2 — but at d24 it costs nothing: polar express returns a ~orthonormal update, and a square orthonormal matrix has ~uniform neuron norms along either axis, so there is no variance to reduce. The converter mean-fills onto nanochat's axis, and asserts rather than converting if the bank is not square, where the two axes would carry genuinely different information. The explicit axis only earns its keep when n_heads * d_head != d_model (and, for ve_gate, when n_kv_head != d_ve_gate; both are equalities at d24).

The optimizer path is verified twice over: against a real nanochat GPT and MuonAdamW at a toy config (structure, per-rank sharding at world sizes 1–4, reassembly, a warm-started optimizer.step()), and against the full d24 capture at --world-size 8 on an H100 (2026-08-01) — see above.

Provenance

Trained by chrisjmccormick/stacks — the converter is committed at 5f3d29e (utils/convert_ckpt_to_nanochat.py). The training script that produced these weights is code/run_full_d24_w8.py in this repo, which is the run copy of the single-file d24 trainer with three launcher overrides (micro_batch_tokens 32768→65536, max_num_docs 96→192, NUM_TRAIN_SHARDS 20→80); the rationale for each is in logs/run_full_d24_w8.sh.

Training data: ChrisMcCormick/climbmix_32k_8_170 (ClimbMix, pre-tokenized to 32k binary shards). Validation is a rebuild of shard 06542 to match nanochat's pinned val split.

The nanochat-format checkpoint and its optimizer shards were produced and verified on 2026-08-01 against nanochat varlen-gsm8k d9dffe9, torch 2.9.1, 1×H100.