loop-vanilla-d12

Vanilla, depth coordinate d12, from How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents.

This is the original final checkpoint used by the paper's FineWeb scaling ladder. It contains 494,272,512 stored parameters in FP32 (1.977 GB). This is a pretrained base language model, without instruction tuning. The checkpoint preserves the original training artifact and does not contain optimizer state for resuming training.

Setting	Value
Training corpus	FineWeb
Tokenizer	GPT-2 (tiktoken.get_encoding("gpt2"))
Vocabulary	50,257 tokens, padded to 50,304 model rows
Context length	2,048
Width	1536
Attention heads	12
Model depth mode	none
Configured core repetitions	1
Final evaluation repetitions	1
Pretraining validation NLL	3.029247 nats/token

The depth coordinate is the ladder's scaling coordinate; it need not equal the number of executed Transformer blocks. result.json provides the model configuration, parameter counts, training settings, and recorded validation metrics. Validation NLL is measured on the pretraining corpus and is distinct from CORE answer NLL.

Files

• final.pt: original PyTorch checkpoint, including learned weights and training arguments.
• result.json: portable training metadata for reconstructing the model.
• SHA256SUMS: checksum of the original checkpoint.

Evaluation

Use the paper's codebase at cue-engineering/loop, install its dependencies, and download this repository's files:

from huggingface_hub import snapshot_download
snapshot_download(repo_id="CharlieChen/loop-vanilla-d12", local_dir="checkpoints/loop-vanilla-d12")

From the codebase root, run a bounded GPU smoke evaluation:

CUDA_VISIBLE_DEVICES=0 python eval.py \
  --checkpoint checkpoints/loop-vanilla-d12/final.pt \
  --result-json checkpoints/loop-vanilla-d12/result.json \
  --max-per-task 10 --seeds 0 1 2 \
  --out checkpoints/loop-vanilla-d12/core_smoke.json

The paper uses H100 GPUs, FlashAttention-3, and bfloat16 autocast. The custom TransformerGPT model is reconstructed by the paper code; this artifact is not a Transformers AutoModel checkpoint. Full CORE evaluation uses all 22 tasks and seeds 0/1/2; omit --max-per-task to run it. Smoke scores are not full-suite paper results.