AlxSp/nag-nanochat-d64

NAG nanochat d64 checkpoints

Raw PyTorch checkpoints from a controlled comparison between a 64-layer nanochat GPT baseline and a Norm-AGnostic (NAG) residual-stream model. The repository is private while the checkpoint descriptions and release details are reviewed.

These are research checkpoints, not transformers-native model packages. Loading them requires the matching model definitions from Alxsp/nag-nanochat, specifically nanochat/gpt.py for the baseline and nanochat/nag_gpt.py for NAG.

Checkpoints

Directory	Description	Step	Approx. tokens	Validation BPB
nag-d64-3e19-gatefix/	Final NAG run with the gate-training fix	9,425	9.883B	0.758923
gpt-d64-3e19-baseline/	Matched GPT baseline	9,474	9.934B	0.750289
nag-d64-3e19-collapsed/	Failed NAG run retained for gate-collapse analysis	9,425	9.883B	0.785782

The collapsed checkpoint is included to make the failure analysis reproducible. Only its final model_009425.pt and meta_009425.json are published. The known-corrupt intermediate model_003500.pt was intentionally excluded. Optimizer shards are excluded from every checkpoint.

Shared configuration

• Decoder-only causal language model
• 64 transformer blocks
• Hidden width: 640
• Attention heads: 5
• Head dimension: 128
• Vocabulary size: 32,768
• Training sequence length: 2,048
• Full causal attention in every layer
• Target training compute: 3e19 FLOPs
• Global training batch: 1,048,576 tokens
• Training data: NVIDIA ClimbMix as configured by nanochat

The complete per-run configuration is stored beside each model in its meta_*.json file.

Tokenizer

The tokenizer/ directory contains the tokenizer artifacts used for these runs:

• tokenizer.pkl
• token_bytes.pt

tokenizer.pkl is a Python pickle. Load it only from a trusted source and use the matching nanochat tokenizer code.

Loading

Download the desired model and metadata with huggingface_hub, instantiate the matching nanochat model class from meta["model_config"], and load the checkpoint as a PyTorch state dictionary. Checkpoint keys may carry an _orig_mod. prefix from compilation and should be stripped in the same way as the nanochat checkpoint loader.

Example download:

from huggingface_hub import hf_hub_download

repo_id = "AlxSp/nag-nanochat-d64"
model_path = hf_hub_download(
    repo_id,
    "nag-d64-3e19-gatefix/model_009425.pt",
)
meta_path = hf_hub_download(
    repo_id,
    "nag-d64-3e19-gatefix/meta_009425.json",
)

Intended use and limitations

These checkpoints are intended for architecture analysis and reproducibility, including residual-stream geometry, attention-sink, layer-gain, and gate collapse studies. They are base pretraining checkpoints and have not been instruction-tuned or safety-aligned. They should not be treated as production assistants. The collapsed checkpoint is a deliberately retained failed run.