nanochat-d24-base-champion

A 910 M-parameter nanochat base model pretrained entirely on one RTX 4070 12 GB.

This is the promoted base checkpoint from Bl4ckd09/nanochat-on-rtx4070 — a fork of karpathy/nanochat that asks how far the 8×H100 speedrun codebase goes on a single consumer GPU. It represents ~465 GPU-hours of pretraining on that one card, across three resumable continuation segments.

📊 Training runs on W&B · 📄 Full experiment log and negative results

Model details

Model tag	d24_asp48_track
Checkpoint step	820,230 (target ratio 32)
Parameters	910,691,760
Layers / heads / n_embd	24 / 9 / 1152
Vocab / max seq len	32,768 / 2,048
Dtype	bfloat16 weights; a few tensors stored fp32 (2.66 GB on disk)
Tokenizer	GPT-4-style BPE, 32,768 vocab — not bundled here, bring the one from the nanochat repo
Pretraining data	karpathy/fineweb-edu-100b-shuffle parquet shards
Trained on	1× NVIDIA RTX 4070 12 GB (Windows 11 + WSL2, i7-13700K, 64 GB RAM)

Evaluation

Metric	Value	Reference
Validation bits-per-byte	0.9199	lower is better
Train bits-per-byte	0.8014
CORE (DCLM)	0.1514	GPT-2 (1.6 B) = 0.2565; upstream 8×H100 speedrun ≈ 0.26

The CORE score comes from the fixed skip5120 path (--core-overflow-policy skip --core-max-seq-len 5120) with transparent skipped/evaluated counters. The bits-per-byte figures come from scripts/base_eval.py, which reads 20,971,520 tokens per split (--split-tokens). The trainer also logs a val/bpb value every eval tick from 524,288 tokens. That sample is 40x smaller, reads about 0.007 lower, and is not used for checkpoint selection.

This eval scored 91,032 examples across 22 CORE tasks and skipped 5, all in bigbench_language_identification. That is 0.0055%, so the overflow policy did not move the score. The same 5 were skipped on all three d24 checkpoints, so they compare like for like. Full log with per-task accuracy, centered score and counts: d24_asp48_track_s820230_base_eval_skip5120_2026-03-19_0456.log.

meta.json in this repo is the trainer's own checkpoint metadata. Its val_bpb field reads 0.9133, which is the 524,288-token inline sample, not the 0.9199 quoted above. Both paths call the same loader from the start of the same split, so the inline window is a prefix of the full one. The gap holds across every checkpoint: 0.9018 against 0.9081 for r24, 0.9133 against 0.9199 here, and 0.9177 against 0.9244 for r40.

This model does not match GPT-2. It reaches roughly 59% of GPT-2's CORE score. That is the honest result of the experiment, and it is the point: it maps what a 12 GB consumer card actually buys.

Why this checkpoint and not a later one

Training was continued past this point to target ratio 40 (step 1,025,288, another 133 GPU-hours). That run finished cleanly and improved training bpb to 0.7846 — but regressed both selection metrics (val bpb 0.9244, CORE 0.1440). Step 820,230 is promoted; r40 is archived as an ablation.

Checkpoint	Train bpb	Val bpb	CORE	Status
r24 @ 615,173	0.7992	0.9081	0.1494	best val bpb, published
r32 @ 820,230	0.8014	0.9199	0.1514	✅ promoted
r40 @ 1,025,288	0.7846	0.9244	0.1440	archived

Architecture

This is not a stock transformers model and will not load with AutoModelForCausalLM. It uses:

• rotary position embeddings
• QK-norm
• relu² MLP
• untied input/output embeddings
• grouped-query attention (GQA)
• ResFormer value embeddings
• per-layer resid_lambdas / x0_lambdas scalars

Load it with the nanochat GPT class.

Usage

This repo ships weights only. You need the nanochat GPT class from the fork on your PYTHONPATH, and a matching 32,768-vocab tokenizer for text in and out — the tokenizer is not bundled here.

git clone https://github.com/Bl4ckd09/nanochat-on-rtx4070
cd nanochat-on-rtx4070
pip install torch safetensors huggingface_hub

python -c "from huggingface_hub import snapshot_download as d; d('Marcolini/nanochat-d24-base-champion', local_dir='ckpt')"
cd ckpt && PYTHONPATH=.. python load_example.py

load_example.py reads config.json and model.safetensors from the working directory, rebuilds GPTConfig from the stored hyperparameters, and loads the state dict with assign=True after upcasting the bf16 tensors. As stored the checkpoint occupies about 2.7 GB; cast it to bf16 throughout and inference sits comfortably inside 12 GB.

Intended use and limitations

Intended for: research on consumer-GPU LLM training, as a base for further SFT or LoRA experiments, and as a reference point for what ~465 single-GPU hours produces.

Not intended for: production use, factual question answering, or deployment in any user-facing setting.

Limitations, stated plainly:

• It is a base model — no instruction tuning, no alignment, no safety training. It completes text; it does not follow instructions.
• CORE 0.1514 places it well below GPT-2. Expect frequent incoherence and confident fabrication.
• Trained on English FineWeb-Edu only. It inherits that corpus's biases and coverage gaps.
• Chat-adapted derivatives of this base reach only ~4.6% GSM8K pass@8 and ~27.4% MMLU (barely above the 25% random baseline). Reasoning is essentially absent at this scale.

Related artifacts

• Chat champion (partial fine-tune of this base, provisional): GitHub release
• Full experiment log, including 14 documented failed SFT recipes: report_v3.md
• The eight SFT data mixes that tried to beat that chat recipe and failed: nanochat-rtx4070-sft-mixes
• Training curves: W&B

Citation

Upstream codebase:

@misc{nanochat,
  author = {Andrej Karpathy},
  title  = {nanochat: The best ChatGPT that \$100 can buy},
  year   = {2025},
  publisher = {GitHub},
  url    = {https://github.com/karpathy/nanochat}
}

This checkpoint:

@misc{nanochat_rtx4070_2026,
  author = {Sun},
  title  = {nanochat on a single RTX 4070: consumer-GPU pretraining and SFT at 0.9B scale},
  year   = {2026},
  publisher = {Hugging Face},
  url    = {https://huggingface.co/Marcolini/nanochat-d24-base-champion}
}

License: MIT, as upstream.