Mini Research on Looped Transformers 2026
This model repository contains the important final 100M-token checkpoints from the research codebase. The models are small causal language models whose shared Qwen3-style decoder core is reused recurrently. The release uses raw PyTorch training checkpoints so that optimizer and experiment metadata remain available.
Released checkpoints
| File | Architecture / training depth | Parameters | Robust best loop | Robust best CE / PPL |
|---|---|---|---|---|
checkpoints/curriculum_8_to_16_100m.pt |
BASE, 60% of tokens at T=8 then 40% at T=16 | 7,342,592 | 16 | 4.381245 / 79.937 |
checkpoints/curriculum_16_to_8_100m.pt |
BASE, matched reverse curriculum | 7,342,592 | 8 | 4.412476 / 82.473 |
checkpoints/base_fixed8_100m.pt |
BASE, fixed T=8 | 7,342,592 | 8 | 4.374509 / 79.400 |
checkpoints/far_h0_fixed8_100m.pt |
FAR-H0, fixed T=8 | 8,261,184 | 8 | 4.382841 / 80.064 |
All runs used 99,999,744 actual training tokens, seed 42, sequence length 512, global batch 8,192 tokens, and final-output next-token cross-entropy only. The FORWARD and REVERSE curricula have identical T=8/T=16 token counts and recurrent compute; only temporal order differs.
The primary checkpoint is curriculum_8_to_16_100m.pt. Its useful inference-depth
optimum moves to loop 16. The reverse control returns the optimum to loop 8.
Files required for evaluation
src/andscripts/evaluate.py: exact model and depth-sweep implementation.configs/: exact configuration for each released checkpoint.tokenizer/tokenizer.json: byte-level BPE tokenizer with vocabulary 16,384.results/: compact published metrics.SHA256SUMS.txt: integrity hashes for all checkpoint and tokenizer artifacts.
The FineWeb token streams are not distributed. Recreate them using
scripts/prepare_data.py or follow the GitHub reproduction guide.
Download and evaluate the primary checkpoint
hf download GeneralFY/mini-research-on-looped-transformers2026 \
--local-dir mini-research-on-looped-transformers2026
cd mini-research-on-looped-transformers2026
python -m venv .venv
source .venv/bin/activate
python -m pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cu128
python -m pip install -r requirements.txt
sha256sum -c SHA256SUMS.txt
python scripts/evaluate.py \
--config configs/curriculum_8_to_16_100m.yaml \
--checkpoint checkpoints/curriculum_8_to_16_100m.pt \
--data-dir /path/to/data \
--loops 1:32 \
--validation-tokens 65536 \
--diagnostic-tokens 65536 \
--kl-tokens 512 \
--output results/reproduced_100m_65k.csv
The evaluator constructs the model from the YAML configuration and loads its state
dict with strict=True. These files are research artifacts rather than a
transformers.AutoModel package.
Integrity
The primary checkpoint SHA256 is:
8bc102b584df978bd7619fb752ad84ca41ed1656aa403d406f1345c92f836d6b
Tokenizer SHA256:
5f2f0950743afdc9f338fc0d0d408e4214529940bc3f9d04e0218d4ec29760f8