Spidory: small looped nGPT models with experts

Weights of the 30 models in Table 2 of the paper draft Choose Again Each Loop: Expert Re-selection and the One-Loop Tax in a Small Looped Transformer (David Ternoposkiy, 2026). Code, paper and training scripts: https://github.com/Davidooff/Spidory

The models follow NVIDIA's Dory recipe at 26M parameters: a 4-layer input block, a 10-layer shared centre block run k times (s0 = x, u = N(s + x), s = N(F(u))) and a 4-layer output block, with nGPT layers, SwiGLU and RoPE. Some variants add four small experts to the centre block, a per-loop run index, or a block between loops. Each was trained on 1.2B tokens, on seeds 0, 1 and 2.

These are research checkpoints for reproducing the paper, not general-purpose language models. They were trained on TinyStories, generated arithmetic, short QA and a few small maths sets, with a 4,096-token vocabulary and a 256-token context.

Models

Test loss is the mean over 7 test sets. Arithmetic is exact-match accuracy on 500 held-out problems. Looped models are scored at k = 3, no-loop models at k = 1. Mean ± standard deviation over the 3 seeds; per-seed numbers are in each folder's results.json.

Model	Folders	Params	Test loss	Arithmetic (%)
No-loop, 10 layers (k=1)	noloop_s0 noloop_s1 noloop_s2	26.1M	1.710 ± 0.002	65.5 ± 7.3
No-loop, 30 layers (k=1)	noloop_deep_s0 noloop_deep_s1 noloop_deep_s2	53.6M	1.694 ± 0.010	70.6 ± 4.8
Plain	plain_s0 plain_s1 plain_s2	26.1M	1.708 ± 0.006	65.6 ± 4.6
Plain + run index	plain_signal_s0 plain_signal_s1 plain_signal_s2	26.1M	1.708 ± 0.009	66.7 ± 5.6
Plain + run index, no norm	plain_signal_nonorm_s0 plain_signal_nonorm_s1 plain_signal_nonorm_s2	26.1M	1.711 ± 0.004	69.3 ± 4.1
Per-sequence experts, re-selected each loop	perloop_s0 perloop_s1 perloop_s2	26.6M	1.768 ± 0.035	66.6 ± 10.6
Per-token experts, re-selected each loop	token_s0 token_s1 token_s2	26.6M	1.716 ± 0.005	69.9 ± 4.0
Per-token experts + run index, re-selected each loop	token_signal_s0 token_signal_s1 token_signal_s2	26.6M	1.711 ± 0.013	71.3 ± 10.0
Per-token experts + run index, chosen once	token_signal_once_s0 token_signal_once_s1 token_signal_once_s2	26.6M	1.711 ± 0.012	59.7 ± 8.4
Between-loop block + run index	between_signal_s0 between_signal_s1 between_signal_s2	34.9M	1.752 ± 0.012	64.1 ± 9.5

Files

• <model>_s<seed>/model.safetensors: fp32 weights, no optimizer state.
• <model>_s<seed>/config.json: model and training settings, and the final step.
• <model>_s<seed>/results.json: every test score at k = 1..4, as reported in the paper.
• tokenizer.json: the BPE tokenizer (4,096 tokens) shared by all models.
• manifest.json: parameters, test loss and the SHA-256 of the training checkpoint each export came from.

Each export was checked against its training checkpoint: every tensor is equal, and logits at k = 1 and k = 3 match exactly.

Use

pip install git+https://github.com/Davidooff/Spidory

from spidory.evaluate import generate_text
from spidory.hub import load

model, tok = load("token_signal_s0")
print(generate_text(model, tok, "Q: Calculate 12 + 7 * 3.\nA:", 8, k=3))

k is the number of passes through the centre block (1 to 4). model.infer(ids, k=3) returns the logits, and for models with experts also the expert and run count each token or sequence used.

License

Apache-2.0, for both the weights and the code.