decoderstack-d12

Training-state checkpoints for the DecoderStack d12 Colab notebook line (stacks/decoder-medium, single-GPU 40GB-A100 configuration of the nanochat-style d12: 286,261,730 params, 12 layers x 768, FA3 varlen, Muon + AdamW with per-step schedule tables, bf16-live + uint16-mantissa fp32 masters).

Layout

checkpoints/<run_name>/state_stepNNNNNN.pt

Each state_step*.pt is the entire training state at that step, written by the walkthrough notebook's write_state and consumed by its load_state:

• params[name] — per-Param w (live weights), mantissa (uint16 low bits of the fp32 master, where applicable), first_mntm, scnd_mntm
• step, t_step — schedule position
• rng — torch CPU + CUDA generator states
• batch — the next unconsumed training micro-batch (inputs, targets, cu_seqlens), so the walkthrough dissects exactly the batch training would have seen next
• config — the full StackConfig (asserted on load)
• code — the notebook source that produced the state

Produced by the DecoderStack d12 Walkthrough notebook: Part 1 trains under the real 1680-step schedule, stops at cfg.walkthrough_step, saves + pushes here; the walkthrough part reloads and spells the last layer's forward/backward and one Muon step out flat.

Current states (run 40GB-A100_d12_walkthrough, 2026-08-31)

file	where in the run	val bpb
state_step000250.pt	step 250 of 1680 (past all warmups, lr at peak)	1.076657
state_step000015.pt	step 15 (mid lr/scalar/lm-head warmup)	1.905353

To walk through a different state, set cfg.walkthrough_step to its step number and run the notebook's walkthrough part.