loop-operator-1-d6

Operator 1, depth coordinate d6, from How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents.

This is the original final checkpoint used by the paper's FineWeb scaling ladder. It contains 119,734,272 stored parameters in FP32 (0.479 GB). This is a pretrained base language model, without instruction tuning. The checkpoint preserves the original training artifact and does not contain optimizer state for resuming training.

Setting	Value
Training corpus	FineWeb
Tokenizer	GPT-2 (tiktoken.get_encoding("gpt2"))
Vocabulary	50,257 tokens, padded to 50,304 model rows
Context length	2,048
Width	768
Attention heads	6
Model depth mode	loop
Configured core repetitions	1
Final evaluation repetitions	1
Pretraining validation NLL	3.507466 nats/token

The depth coordinate is the ladder's scaling coordinate; it need not equal the number of executed Transformer blocks. result.json provides the model configuration, parameter counts, training settings, and recorded validation metrics. Validation NLL is measured on the pretraining corpus and is distinct from CORE answer NLL.

Files

• final.pt: original PyTorch checkpoint, including learned weights and training arguments.
• result.json: portable training metadata for reconstructing the model.
• SHA256SUMS: checksum of the original checkpoint.

Evaluation

Use the paper's codebase at cue-engineering/loop, install its dependencies, and download this repository's files:

from huggingface_hub import snapshot_download
snapshot_download(repo_id="CharlieChen/loop-operator-1-d6", local_dir="checkpoints/loop-operator-1-d6")

From the codebase root, run a bounded GPU smoke evaluation:

CUDA_VISIBLE_DEVICES=0 python eval.py \
  --checkpoint checkpoints/loop-operator-1-d6/final.pt \
  --result-json checkpoints/loop-operator-1-d6/result.json \
  --max-per-task 10 --seeds 0 1 2 \
  --out checkpoints/loop-operator-1-d6/core_smoke.json

The paper uses H100 GPUs, FlashAttention-3, and bfloat16 autocast. The custom TransformerGPT model is reconstructed by the paper code; this artifact is not a Transformers AutoModel checkpoint. Full CORE evaluation uses all 22 tasks and seeds 0/1/2; omit --max-per-task to run it. Smoke scores are not full-suite paper results.