README.md

---
license: mit
language:
- en
tags:
- decoderstack
- nanochat
- pretraining
---

# DecoderStack d24

A 1.38B-parameter d24 model pre-trained from scratch with
[**DecoderStack**](https://github.com/chrisjmccormick/stacks) — a single-file training
stack with a handwritten forward/backward and a written-out optimizer: no autograd, no
`torch.optim`, no `nn.Module`. bf16, varlen FlashAttention, 8xH100, 1h51m.

The architecture is the nanochat d24 architecture, so these weights load into
[nanochat](https://github.com/karpathy/nanochat) once converted — see
[Loading into nanochat](#loading-into-nanochat).

## Results

Full horizon: 5,568 steps x 2^20 tokens = 5.84B tokens (data:param ratio 8).

```
step 05567/05568 (99.98%) | loss: 2.388180 | lrm: 0.05 | dt: 1198.08ms | tok/sec: 875,210 | bf16_mfu: 53.76 | total time: 110.74m
step:5568/5568 val_bpb:0.719042
CORE metric: 0.2517 | total CORE eval time: 80.46s
peak memory allocated: 71569 MiB reserved: 72736 MiB
total training time: 110.74m
minimum validation bpb: 0.719042
```

| | value |
|---|---|
| parameters | 1,384,122,122 |
| min val bpb | 0.719042 (varlen packing, 10.5M val tokens — see the caveat below) |
| CORE | 0.2517 |
| train time | 110.74 min on 8xH100 80GB |
| throughput | 1,212 ms/step, 864,856 tok/sec (steps 20–99) |
| precision | bf16 (no FP8) |

val bpb curve: 0:3.175835 / 250:0.997472 / 500:0.904081 / 1000:0.846058 / 2000:0.809199 /
3000:0.773139 / 4000:0.747155 / 5000:0.726908 / 5568:0.719042

**Against upstream nanochat**, run on the same box on the same rebuilt validation shard,
DecoderStack tracks 0.2–1.4% behind through step 1000 (where the upstream baseline was
stopped) while running ~6% faster per step (1,212 vs 1,291 ms/step). Full head-to-head in
[`logs/baseline_report.md`](logs/baseline_report.md).

⚠️ **but 0.719042 is not comparable to an upstream nanochat val bpb, and reads better than
a like-for-like number would.** Upstream batches with best-fit cropping — it fills each row
with largest-doc-that-fits and crops a document to finish the row, then attends causally
across the whole row with no document isolation (`cu_seqlens` appears nowhere in upstream
`nanochat/gpt.py`). DecoderStack packs varlen and isolates documents in validation as in
training. Different truncation points, different conditioning context: the two are not
scoring the same predictions. Measured below.

### Measured 2026-08-01: what this checkpoint scores under upstream's packing

Varlen packing does not exist in upstream nanochat — it is this fork's own work, and
**every nanochat leaderboard entry is measured with karpathy's best-fit row packer**. So a
varlen val bpb cannot be read against the leaderboard at all. To get a comparable number,
both checkpoints below were re-scored under upstream's row packer, driven with each
model's own tokenizer over the same pinned val shard and unwound into varlen batches whose
`cu_seqlens` sit on the row boundaries — the model predicts exactly the tokens upstream
would ask for, with exactly the same context.

Both models, both packings, same code, same shard, 41.9M val tokens:

| model (step 5568) | best-fit rows | varlen |
|---|---|---|
| **this checkpoint** (`d24_decoderstack`) | **0.737183** | 0.724316 |
| `nanochat-varlen-d24-2026-03-22` | 0.737535 | 0.725014 |

At the 10.5M budget this run originally used: 0.735274 (best-fit) / 0.720717 (varlen).
All on the legacy `token_bytes` basis, matching how the leaderboard numbers were computed;
the regenerated basis reads ~0.05% higher — see [⚠️ Tokenizer](#️-tokenizer).

What this says:

1. **Against the leaderboard, the honest gap is ~0.019, not ~0.006.** The nanochat record
   is 0.71808 (Run 5) / 0.71854 (Run 4), both best-fit. Like-for-like this checkpoint is
   **0.737183**, i.e. **+0.0191** — matching upstream's tokens moves *away* from the
   record, not toward it. The 0.719042 headline above invited the opposite impression.
2. **The packing penalty is universal, not model-specific**: +0.0129 here, +0.0125 for the
   sibling release. Best-fit truncation costs ~0.0127 bpb whoever runs it.
3. **The two checkpoints are near-identical**, this one marginally ahead under both
   packings (−0.00035 best-fit, −0.00070 varlen). Not a controlled comparison — the
   sibling used FP8 and a different 32k tokenizer — but the two pipelines land in the same
   place.
4. **The 10.5M val budget reads optimistic**, not merely noisier: ~0.002–0.004 bpb below
   the 41.9M number. Upstream's default is `--eval-tokens 80*524288` = 41.9M.

Harness check: the sibling run self-reported 0.724772 and is measured here at 0.725014
varlen — 0.03% agreement, using the same loader that run used.

Reproduce: `agent-ops/stacks/2026-08-01_1241pm_d24-val-bpb-recalibration/`.

Throughput remains the clean comparison: 1,212 vs 1,291 ms/step, ~6% faster per step.

Both numbers also deserve error bars: Karpathy's re-runs of an identical d24 config span
**0.0153 in CORE** (5 runs, mean 0.261), so our 0.2517 and that 0.2614 are inside each
other's noise. Sources in
[`stacks/models/nanochat/reference/METRICS.md`](https://github.com/chrisjmccormick/stacks/blob/main/models/nanochat/reference/METRICS.md).

## ⚠️ Tokenizer

These weights are tied to the 32k vocabulary in
[`ChrisMcCormick/climbmix_32k_8_170`](https://huggingface.co/datasets/ChrisMcCormick/climbmix_32k_8_170),
mirrored here under [`tokenizer/`](tokenizer). It is a **different vocabulary** from the
one in `ChrisMcCormick/nanochat-varlen-d24-2026-03-22` — 31,474 of 32,759 merge ids
differ. Both are 32,768 tokens and both load without complaint, so nothing will warn you.
Pairing these weights with the wrong tokenizer produces garbage, not slightly-worse text.

### `token_bytes.pt` was regenerated 2026-08-01

bpb divides summed nats by summed BYTES, and the bytes-per-token table was built by
decoding each id to a Python `str` and measuring `.encode("utf-8")` — which corrupts every
token that is not valid standalone UTF-8 (a lone continuation byte round-trips through
U+FFFD and measures 3 bytes instead of 1). **191 of 32,768 ids** were affected here.
Upstream fixed the calculation in
[`2ce972a`](https://github.com/karpathy/nanochat/commit/2ce972a), but the fix lives in
`tok_train.py` and so only lands when a tokenizer is retrained — a released `token_bytes.pt`
keeps the stale values forever.

`tokenizer/token_bytes.pt` here is now the corrected table; `tokenizer/token_bytes_legacy.pt`
is the pre-fix one, kept so previously published numbers stay reproducible. Measured on this
model, the correction moves bpb by **+0.05%** (0.724316 → 0.724741 varlen at 41.9M), matching
upstream's estimate. `tokenizer.pkl` is untouched — encode/decode are unchanged, this is
purely the byte-accounting used by the metric.

## Contents

```
checkpoints/                        native DecoderStack captures (the source of record)
  model_step001950.pt   2.8 GB      {step, code, weights} -- bf16 live weights + fp32 scalars
  optim_step001950.pt  11.1 GB      {step, t_step, state} -- mantissas + Muon/AdamW moments
  model_step005568.pt   2.8 GB
  optim_step005568.pt  11.1 GB
base_checkpoints/d24_decoderstack/  READY-MADE nanochat checkpoint -- no conversion needed
  model_005568.pt       4.2 GB      fp32 masters + bf16 embeddings, 175 tensors
  optim_005568_rank0.pt 1.0 GB      ZeRO-2 optimizer shards for an 8-GPU SFT run
  ...    _rank7.pt      1.0 GB      (8 x 1.0 GB; see Optimizer state)
  meta_005568.json                  model_config + user_config + run metadata
  meta_001950.json                  metadata only -- step 1950 is not converted
code/run_full_d24_w8.py             the exact training script this run executed
logs/
  d24_decoderstack_20260801_023219.log   full training log
  run_full_d24_w8.sh                     launcher, with the config rationale
  baseline_report.md                     throughput + val bpb vs upstream nanochat
tokenizer/                          the 32k vocab these weights were trained on
convert_ckpt_to_nanochat.py         DecoderStack capture -> nanochat state_dict (+optimizer)
test_convert_ckpt_to_nanochat.py    its test, if you change the converter
```

Two capture points: **5568** is the end of the run, **1950** is the last uncooled state —
the LR/momentum hold ends after update 1949 (`N - round(0.65*N)`), so 1950 is the capture
to branch from if the horizon is ever extended. (Extending it needs a load path that does
not exist yet in either codebase — see [Optimizer state](#optimizer-state).)

DecoderStack writes `code` (the full training script, `open(sys.argv[0]).read()`) into
every model capture, so each `.pt` carries its own exact source. `code/run_full_d24_w8.py`
is that string extracted from `model_step005568.pt`.

## Loading into nanochat

**Step 5568 is already converted.** `base_checkpoints/d24_decoderstack/` is a nanochat
checkpoint dir — download it and the tokenizer into a nanochat base dir and it loads. No
conversion step, and the 11 GB optimizer capture is not part of it.

```bash
pip install huggingface_hub torch
python - <<'PY'
import os
from huggingface_hub import snapshot_download
snapshot_download("ChrisMcCormick/decoderstack-d24",
                  local_dir=os.path.expanduser("~/.cache/nanochat-ds"),
                  allow_patterns=["base_checkpoints/d24_decoderstack/*", "tokenizer/*"])
PY
export NANOCHAT_BASE_DIR=~/.cache/nanochat-ds
```

**Use a base dir of its own, not a shared `~/.cache/nanochat`.** nanochat resolves the
tokenizer from one global `tokenizer/` per base dir, and this vocabulary is not the one any
other d24 release uses (see [⚠️ Tokenizer](#️-tokenizer)). A base dir holding both this
checkpoint and, say, `nanochat-varlen-d24-2026-03-22` can only be right about one of them,
and `build_model` only asserts the vocab *size* — which matches — so it will not warn you.

Then, on
[`varlen-gsm8k`](https://github.com/chrisjmccormick/nanochat/tree/varlen-gsm8k) (the branch
whose SFT/RL pipeline this checkpoint targets; `fa-varlen` also works for eval):

```python
import os, torch
from nanochat.checkpoint_manager import load_model
model, tokenizer, meta = load_model("base", torch.device("cuda"), phase="eval",
                                    model_tag="d24_decoderstack", step=5568)
```

Pass `model_tag` explicitly — with more than one `d24*` dir under `base_checkpoints/`,
`find_largest_model` picks between them on depth alone and both are 24.

Verified on an H100 (2026-08-01): 175 tensors, 1,384,122,122 params, shapes and dtypes
identical to a freshly built nanochat `GPT` at this config, `strict=True` load, and
**2.4839 nats/token (0.7255 bpb)** on held-out English — in line with the run's own 0.719
val bpb — with coherent greedy continuations (`The capital city of France is` → ` Paris.`).
A scrambled mapping scores 10.4.

### Re-converting yourself

`convert_ckpt_to_nanochat.py` unbanks the per-layer matrix banks, renames tensors to
nanochat's module paths, maps value-embedding bank slots back to their layers, and rebuilds
the fp32 masters from the bf16 live weights plus the uint16 mantissa held in the optimizer
file. You need it to convert **step 1950**, or to emit optimizer shards for a GPU count
other than 8:

```bash
python convert_ckpt_to_nanochat.py \
    --model checkpoints/model_step005568.pt \
    --optim checkpoints/optim_step005568.pt \
    --meta  base_checkpoints/d24_decoderstack/meta_005568.json \
    --out   $NANOCHAT_BASE_DIR/base_checkpoints/d24_decoderstack \
    --world-size 8
```

`--optim` is optional for the weights. It only supplies the mantissas — the lower 16 bits
of the fp32 masters. Without it the fp32 parameters carry bf16 precision, which costs about
0.0005 nats/token (2.4833 vs 2.4828 on a held-out paragraph); skip the 11 GB download
unless you want the exact master, or the optimizer shards, which need it.

**DecoderStack itself cannot load these back yet.** The capture path is write-only by
design; a load/resume path is future work.

### Optimizer state

**SFT works with the model alone.** nanochat's `chat_sft` builds a fresh optimizer and only
*optionally* warm-starts it; with no optimizer shard present it prints `starting with fresh
optimizer (slightly worse)` and carries on.

The `optim_005568_rank{0..7}.pt` files here remove that "slightly worse". They are the
ZeRO-2 shards `DistMuonAdamW` expects, **cut for an 8-GPU run** — the shard count must equal
the GPU count of the SFT run, because rank *r* holds only its own slice. For any other
world size, re-run the converter with `--world-size N` (see above); `--world-size 1` emits
a single rank0 file holding the full unsharded state.

Verified on an H100 (2026-08-01), all 8 shards: 10 groups / 175 params / 23 state entries
each, shapes equal to what `DistMuonAdamW` allocates per rank at this config, accepted by
`load_state_dict`, and **reassembling bit-exactly** back to the original capture — including
the ragged 12-param `ve_gate` group, where chunk 2 × 8 ranks leaves ranks 6–7 holding only
zero padding. End to end, `chat_sft`'s own load block warm-starts from them (23 entries,
step 5568) and steps.

`meta_005568.json` carries a `user_config` block, which is where `chat_sft` reads the
pretraining LRs from — without it SFT silently falls back to `unembedding_lr=0.004` instead
of this run's **0.008**.

Note that `torch`'s `load_state_dict` replaces param-group dicts wholesale, so the emitted
`lr`/`betas`/`weight_decay` become the optimizer's on load — they default to
`setup_optimizer()`'s own values with `weight_decay=0.0` (the SFT setting, and where
DecoderStack's cosine-to-zero Muon decay lands), and are CLI-overridable. `chat_sft`
restores its own LRs immediately after loading regardless.

**Pre-training resume is out of reach**, and it is the dataloader that closes the door, not
the optimizer: nanochat's resume needs a `dataloader_state_dict` to put its tokenizing
loader back in the stream, and DecoderStack's pre-tokenized binary loader has no equivalent
state to hand over. The data order could not be continued no matter what the optimizer held.

Every buffer does have a counterpart, and the precisions line up on everything except the
embedding tables:

| DecoderStack | nanochat | precision |
|---|---|---|
| `.frst_mntm` | Muon `momentum_buffer` | fp32 both |
| `.scnd_mntm` | Muon `second_momentum_buffer` | fp32 both |
| `.exp_avg` / `.exp_avg_sq` | AdamW `exp_avg` / `exp_avg_sq` | fp32 both, **except** `wte` and `value_embeds`, which nanochat allocates `zeros_like(p)` on a bf16 param — so bf16 there, fp32 here |
| `.mantissa` (uint16) | — | nanochat's fp32 param *is* the master; consumed by the converter to rebuild it |

DecoderStack is the more precise of the two on the embeddings, deliberately: only the
*gradients* are bf16 for those tables (they are the model's largest tensors), while the
moment math stays fp32 — the kernel upcasts on the way in. There is no bf16 AdamW variant;
the two AdamW kernels differ in whether the param carries a mantissa, not in moment dtype.

`W_O`'s NorMuon reduction axis differs — nanochat infers it from the shape and lands on
`-1` for a square `c_proj`, DecoderStack states `residual_dim = -2` — but at d24 it costs
nothing: polar express returns a ~orthonormal update, and a square orthonormal matrix has
~uniform neuron norms along either axis, so there is no variance to reduce. The converter
mean-fills onto nanochat's axis, and *asserts* rather than converting if the bank is not
square, where the two axes would carry genuinely different information. The explicit axis
only earns its keep when `n_heads * d_head != d_model` (and, for `ve_gate`, when
`n_kv_head != d_ve_gate`; both are equalities at d24).

The optimizer path is verified twice over: against a real nanochat `GPT` and `MuonAdamW` at
a toy config (structure, per-rank sharding at world sizes 1–4, reassembly, a warm-started
`optimizer.step()`), and against the full d24 capture at `--world-size 8` on an H100
(2026-08-01) — see above.

## Provenance

Trained by [`chrisjmccormick/stacks`](https://github.com/chrisjmccormick/stacks) — the
converter is committed at
[`5f3d29e`](https://github.com/chrisjmccormick/stacks/commit/5f3d29e)
(`utils/convert_ckpt_to_nanochat.py`). The training script that produced these weights is
`code/run_full_d24_w8.py` in this repo, which is the run copy of the single-file d24
trainer with three launcher overrides (`micro_batch_tokens` 32768→65536,
`max_num_docs` 96→192, `NUM_TRAIN_SHARDS` 20→80); the rationale for each is in
`logs/run_full_d24_w8.sh`.

Training data: [`ChrisMcCormick/climbmix_32k_8_170`](https://huggingface.co/datasets/ChrisMcCormick/climbmix_32k_8_170)
(ClimbMix, pre-tokenized to 32k binary shards). Validation is a rebuild of shard 06542 to
match nanochat's pinned val split.

The nanochat-format checkpoint and its optimizer shards were produced and verified on
2026-08-01 against nanochat
[`varlen-gsm8k`](https://github.com/chrisjmccormick/nanochat/tree/varlen-gsm8k) `d9dffe9`,
torch 2.9.1, 1×H100.