carrybit checkpoints
Trained weights from carrybit, a small research project on tiny transformers learning exact integer arithmetic. The code, training configs, and result plots live in that repo; this repo only holds the final checkpoints so the analysis experiments can be run without retraining.
Every checkpoint is a plain PyTorch state_dict for carrybit.model.Transformer.
Each folder has the exact config.json it was trained with and the
metrics.csv logged during training.
import json, torch
from carrybit.config import load_config
from carrybit.model import Transformer
cfg = load_config("configs/addition.yaml", ["task.zero_pad=true", "task.positions=coupled"])
model = Transformer(16, cfg.model)
model.load_state_dict(torch.load("ladder/position_coupling_s0/step_50000.pt"))
Contents
modular_add/: one-layer transformer trained on a + b mod 113 (Nanda et al.
setup), 40k full-batch steps. Groks around step 2000.
ladder/: the length generalization ladder. 4-layer, width 256 decoder-only
transformers (3.4M parameters) trained for 50k steps on addition with 1 to 20
digit operands, three seeds per format. Exact match accuracy on operands of
exactly n digits:
| run | 20 | 25 | 30 | 40 | 50 |
|---|---|---|---|---|---|
| plain_s0 | 0.93 | 0 | 0 | 0 | 0 |
| plain_s1 | 0.93 | 0 | 0 | 0 | 0 |
| plain_s2 | 0.81 | 0 | 0 | 0 | 0 |
| reversed_s0, s1, s2 | 1.00 | 0 | 0 | 0 | 0 |
| reversed_zero_pad_s0, s1, s2 | 1.00 | 0 | 0 | 0 | 0 |
| abacus_s0 | 1.00 | 0 | 0 | 0 | 0 |
| abacus_s1 | 1.00 | 0 | 0 | 0 | 0 |
| abacus_s2 | 1.00 | 1.00 | 0.93 | 0.75 | 0.25 |
| position_coupling_s0 | 1.00 | 1.00 | 0.92 | 0.11 | 0 |
| position_coupling_s1 | 1.00 | 0.88 | 0 | 0 | 0 |
| position_coupling_s2 | 1.00 | 0 | 0 | 0 | 0 |
| aligned_blankspace_s0, s1, s2 | 1.00 | 0 | 0 | 0 | 0 |
See the GitHub README for what the formats mean and a discussion of the results.