loop-fwe-vanilla-d18

FineWeb-Edu Vanilla d18, from How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents.

This pretrained base language model contains 1,378,418,688 stored parameters in FP32. The exported weights are bit-for-bit identical to the paper checkpoint. The public checkpoint contains only model tensors and the final evaluation recurrence. Model configuration and selected numeric training metrics are provided in result.json.

Setting	Value
Training corpus	FineWeb-Edu
Depth coordinate	d18
Width	2304
Attention heads	18
Final core repetitions	1
Tokenizer	GPT-2, via tiktoken
Context length	2,048
Vocabulary	50,257 tokens, padded to 50,304 model rows
Pretraining validation NLL	2.52351159 nats/token
Paper CORE accuracy	0.23780689
Paper CORE answer NLL	2.49055914 nats/token

CORE values above are the paper's archived means over seeds 0, 1, and 2, using all 91,037 examples across 22 tasks. CORE answer NLL differs from pretraining validation NLL. The depth coordinate is the ladder's scaling coordinate and can differ from the number of executed Transformer blocks.

Files and use

• final.pt: FP32 model tensors and final evaluation recurrence.
• result.json: architecture, training hyperparameters, and selected numeric metrics.
• SHA256SUMS: SHA-256 checksum of final.pt.

Install the paper codebase and its dependencies, then download the checkpoint:

from huggingface_hub import snapshot_download
snapshot_download("CharlieChen/loop-fwe-vanilla-d18", local_dir="checkpoints/loop-fwe-vanilla-d18")

From the codebase root, run a bounded evaluation:

CUDA_VISIBLE_DEVICES=0 python eval.py \
  --checkpoint checkpoints/loop-fwe-vanilla-d18/final.pt \
  --result-json checkpoints/loop-fwe-vanilla-d18/result.json \
  --max-per-task 10 --seeds 0 1 2 \
  --out checkpoints/loop-fwe-vanilla-d18/core_smoke.json

For the full paper protocol, omit --max-per-task. The paper uses H100 GPUs with FlashAttention-3 and bfloat16 autocast. The checkpoint uses the paper's custom TransformerGPT implementation and is not a Transformers AutoModel artifact. Optimizer state is not included.