Ablation checkpoints (~180M parameters, 2.5B-token budget)
Checkpoints from architecture and optimizer experiments: four finished runs (final checkpoints) and four runs interrupted mid-training (intermediate checkpoints). Each run is compared with a common reference recipe (16 layers × d_model 896 × 14 heads, RMSNorm, QK-norm, RoPE θ 500 000, value residual, SwiGLU, no biases, Muon lr 0.02 + AdamW lr 6e-4, warmup 500, constant lr, decay over the last 20 % to lr / 10; batch 128 × 4096 tokens; planned 4768 steps = 2.5B tokens; tokenizer: the 32 768-token tokenizer of Slayer1B). VR0 and MU2 change one setting; K12, K22 and GQA change the shape at a similar parameter count (K12 and K22: depth and width; GQA: KV-head count and depth, 18 rather than 16 layers).
Finished runs (final checkpoint, step 4768)
| run | change from the reference | final checkpoint | val bpb PL | val bpb EN | weighted bpb (0.4 PL + 0.6 EN) |
|---|---|---|---|---|---|
| R1 | none (reference), seed 1 | R1/ckpt_4768.pt |
1.086412 | 0.999300 | 1.034145 |
| R2 | none (reference), seed 2 | R2/ckpt_4768.pt |
1.088138 | 1.000682 | 1.035664 |
| R3 | none (reference), seed 3 | R3/ckpt_4768.pt |
1.086788 | 1.000410 | 1.034961 |
| K22 | 22 layers × d_model 768 × 12 heads (deeper, narrower) | K22/ckpt_4768.pt |
1.091207 | 1.003861 | 1.038799 |
Bits per byte on held-out validation text (Polish and English), measured once after the last step. The three reference seeds give mean 1.034923; K22 is 0.003876 higher (worse), beyond the decision threshold of 0.00215 fixed before the runs.
Interrupted runs
| run | change from the reference | stopped at step | checkpoints here |
|---|---|---|---|
| K12 | 12 layers × d_model 1024 × 16 heads (shallower, wider) | 3060 | 500, 1000, 1500, 2000, 2500, 3000 |
| GQA | 18 layers × d_model 896 × 14 heads, 2 KV heads (grouped-query attention) | 2920 | 500, 1000, 1500, 2000, 2500 |
| VR0 | 16 × 896 × 14 without value residual | 2860 | 500, 1000, 1500, 2000, 2500 |
| MU2 | 16 × 896 × 14, Muon learning rate × 2 (0.04) | 2340 | 500, 1000, 1500, 2000 |
These runs did not finish. They were stopped externally on 2026-10-23 at 10:11 UTC, before the learning-rate decay;
no final evaluation exists for them. The checkpoints are published so that the runs can be resumed or inspected.
Each file holds weights and optimizer state (torch.save dictionary of train_gpt_ref_r23.py: model, opt, step,
config); load with torch.load(path, map_location="cpu", weights_only=True).
test_wznowienia/: checkpoints of a 40-step resume test of the trainer (continuous run vs. resumed run), not a model.
SHA-256
Finished runs — recorded off the machine before the copy (in the result file of each run); files on this repository are downloaded back and checked against them:
c2999a7b90a8fd5a27b23e44e05c793b7359196f667d8118071fe51ae4d43df1 R1/ckpt_4768.pt
f4a06ade988a35bfa2d47f686253907400eff9eacd62c91818457fd863d48394 R2/ckpt_4768.pt
ea6748f1bcd72a38d3707bc33ef319125fe9c5e9d273e2d424d733dbe32d4564 R3/ckpt_4768.pt
0840bf3078d0f84fc20e68337cc24e8a9e4bba145f7ea0d27541792c1dd7f189 K22/ckpt_4768.pt
Interrupted runs — recorded by the checkpoint copier on the training machine (files dated before the interruption). These sums were stored on the same machine; no independent copy of them exists. Every file listed was downloaded back and matches them.
846c7cc9f3c88fcda1394e757dd024eebfefc3b43cba549c4dfaa0c0efeb1594 K12/ckpt_500.pt
84f431e41c1a7dcd3cb3a1eda8a572094350a7dbd998669bca640e036c9fd923 K12/ckpt_1000.pt
d3bd736d8c42dbd52568c0598ad52862dbddbf69829cbf6325faa3bf0a4083cc K12/ckpt_1500.pt
cdc444a304c2f57727937a5ca0f58599d220eb0741a3920ddf73f3965e2117ca K12/ckpt_2000.pt
8f5298ada65ab57bfe1b63dcd49b956cf0f37f26e7ec5e2bab072e2e8da21e84 K12/ckpt_2500.pt
c54e324a6d5e2b9e33bf9731ff4162df18fb774e57e7bab253576f096dd24511 K12/ckpt_3000.pt
8296514387f2f1fbc20f5c61822007daa823f86b541dc64e830b1361358fba6a GQA/ckpt_500.pt
fb4a4a805f6e88204a3a859238567ff28f40c9f0adbf5a8fdf6c07119245dfec GQA/ckpt_1000.pt
13ddaf2ea734dd720aaa3c55807621d3b66f44ce4d1a58d782970491fafc8699 GQA/ckpt_1500.pt
e8e632635f4bca6f8d4885502d46503c8ac16f2e83d5d1aa65129535215f828e GQA/ckpt_2000.pt
3366e88771415a893cb14f3d209b7759bde43b7475e594bf8a81a88c68843f2f GQA/ckpt_2500.pt
fd93ad481e1ce29f3bf7d2fe7406b8e2b8821a7a8310bf1795a168236181353f VR0/ckpt_500.pt
83c73ec8302a95a1560a6c1777d3a03786c4f0ae69071babd3774c5dd129981b VR0/ckpt_1000.pt
c5678f18d42e9e6f62adaaa6b2e309fadf470d2a7dc4e614217c00e155a86840 VR0/ckpt_1500.pt
ee399f392ba9fe44419a840a75060eafa26d42ce0227d2538ec47ecf353a0ec0 VR0/ckpt_2000.pt
09ffda1f09c8b0e3a86b350bbbe59afb2b352a3246047bd3ba5f3a544a5cd918 VR0/ckpt_2500.pt
dafb39311a0abbd298efc00372e8d574ef255c2e254dd964f99f92df7ad5116e MU2/ckpt_500.pt
d1c16074f435fed71a47e0c377bc3332344ad5bc29c4dfef86dee55c66a7f396 MU2/ckpt_1000.pt
edf8bdeae17f21b10e59fe5f56e793a3bfe92a5a28da86d1d8eaf21d8fad6602 MU2/ckpt_1500.pt
72507b5463e7716781fe783b1599cac8dd2fcb9c10823d47aada75c2bf5b7248 MU2/ckpt_2000.pt
Resume-test files (test_wznowienia/) — sums from the copier's lists and the test's result files on the training
machine (dated before the interruption):
1c501300ddaaee8bdc063e42db798a394087f6169a3e84446c5346474cbb6136 test_wznowienia/mini_R1/ckpt_20.pt
9bc8d9445b7c2df14b8a4bf34a90e35095d8e7b31c8b27f18b878b563a87a75e test_wznowienia/A/ckpt_20.pt
f5877f84b95282b73d3b112299f14cd79aa28ca7a18d070ef91ee872eefb7d9c test_wznowienia/A/ckpt_40.pt
e41bdc94415849333814f1ae6a9b6a879c0695f2dadc18130c52830e595fce98 test_wznowienia/B/ckpt_40.pt
License
Apache-2.0.