GPT-2 fine-tuned on WikiText: the bergson leaderboard model

The model every row of the bergson leaderboard attributes: gpt2 fine-tuned for 4 epochs on the 4,608 training chunks of EleutherAI/bergson-wikitext-512-chunks with AdamW (betas 0.9/0.999, eps_root 1e-17, lr 4e-4 polynomial with 25% warmup, weight decay 0.01, batch 256, seed 42). Held-out loss 3.545 -> 3.111. Produced by examples/compare_wikitext/2_interval.yaml in bergson.

path	what it is
`exported/checkpoint-72/`	the final model, scored by every leaderboard method
`exported/checkpoint-{12,24,36,48,60}/`	interval checkpoints with `optimizer.pt`, used by SOURCE and the checkpoint-averaged variants
`optimizer.pt`, `config.yaml`	final AdamW second moments (TrackStar-Adam) and the training config
`trak_ensemble/s{0..3}_subset_{0,1}/`	the eight GPT-2 models trained on independent random 50% subsets for the TRAK row (`trak_ensemble/train_s*.yaml`)

Scores, the retrain banks and the LDS/QLD results are in EleutherAI/bergson-wikitext-gpt2-leaderboard-bank.

Leaderboard

Linear datamodeling score (LDS) is the accuracy of a method for producing global data rankings by influence. The query loss difference (QLD) shows how much model loss for a held-out query can be increased by retraining without the most highly ranked data by influence (here the top 1%), compared to a random removal baseline.

Method	Proponent QLD [95% CI]	LDS [95% CI]
MAGIC	0.100 [0.090, 0.112]	0.931 [0.925, 0.936]
Eigenvalue-corrected Shampoo	0.071 [0.060, 0.082]	0.517 [0.491, 0.539]
EK-FAC	0.070 [0.058, 0.082]	0.454 [0.426, 0.479]
KFAC	0.067 [0.056, 0.080]	0.420 [0.391, 0.446]
BM25	0.062 [0.048, 0.076]	0.220 [0.185, 0.252]
Qwen3-Embedding-8B semantic search	0.049 [0.038, 0.061]	0.132 [0.093, 0.169]
TrackStar (no optimizer correction, projection 64)	0.045 [0.036, 0.055]	0.270 [0.240, 0.295]
TRAK (8-model ensemble)	0.032 [0.024, 0.040]	0.138 [0.111, 0.165]
SOURCE (Adam)	0.024 [0.018, 0.030]	0.154 [0.126, 0.181]
Gradient cosine similarity	0.021 [0.016, 0.027]	0.156 [0.131, 0.181]
Activation similarity	0.000 [-0.000, 0.001]	0.110 [0.070, 0.149]