Slayer1B 1.07 — training checkpoints
Native training checkpoints of the run that is producing Slayer1B 1.07,
for resuming or studying training (full trainer state, not a Transformers/safetensors package): pretraining continued
from the final 1.06 checkpoint ckpt_13848.pt (sha256 1fc7396c…, published in
Slayer1B-1.06-checkpoints). Each file holds the model
weights and the optimizer state.
Status: training in progress. The table below lists only the files that are in this repository now; further checkpoints are added as training saves them (the planned end is step 22327, which will also be released as the final 1.07 weights in the model repository). Every file listed was compared after upload with the checksum list written by the trainer: the LFS object id (sha256) and the size match.
| Start | ckpt_13848.pt (final state of 1.06, constant lr 6e-4) |
| Steps | 13849 → 22327 planned (8 479 steps × 1 048 576 tokens), 256 sequences × 4096 tokens per step |
| Schedule | constant lr 6e-4, no decay |
| Data | SlayerLab/slayer-1.07-pretrain, 8 891 005 707 tokens, one pass |
| Hardware | 4 × NVIDIA RTX 4090, DDP, gradient accumulation 32; global batch identical to the 8-GPU runs |
| Interruptions | stopped on 4 November at step 17130 and resumed on 5 November from ckpt_17000.pt; steps 17001–17130 were trained twice, and the loss at step 17010 was identical in both runs. Stopped again on 5 November at step 19330 and resumed from the rolling checkpoint at step 19250 (not published); steps 19251–19330 were trained twice, and the loss at step 19260 was identical in both runs. Checkpoints up to step 17000 were saved before the first interruption, 18000–19000 between the two, later ones after the second resume |
| Trainer | train_gpt_ref r24.2 (sha256 46f83b51…); model definition train_gpt_ref_r23.py (unchanged) |
| Training curve and evaluations | https://track.fabryka.ai/run/4973c068-8dd5-59d8-b6c4-671ca3beb925 |
SHA-256
| step | sha256 |
|---|---|
| 13998 | f1ec1380a956f7fa002446902464df26314901d9cf2afe98ecbd00dec34d3529 |
| 14000 | 752d9d8ec48a6473be78dbccdebfd61621d3b24d78932b1b5f4616da415d28b8 |
| 15000 | f89d7f3c6c223ec00ac4d85cb85546d08194808e419dcda3821ebd7ba4915ae1 |
| 15730 | dd7170909294dbdec1282617a12d8c215037c27be5075e421a68ae6b1245e70b |
| 16000 | c18be246a5ae83be67a61fddf479fe6f906936ccd31f93dccf52fb4f03ca2b48 |
| 17000 | c4961c9ced633fef2efd8d28cc16217154f82b25f4a178ce31865b4523eed95b |
| 18000 | fd6e2360badc1ed79e931ab36864440235d67d86c835592b6c9d788a9b9f0980 |
| 18590 | 6779c0bbfe84af003e5a708c1e9a02914e492a40c3f24278bbd096635a254467 |
| 19000 | 9ae730323e29b929f2ba0f753ee75be43626b804f1a5c452d5cee16d0dacfaf7 |
| 20000 | 33146051461e6102cb97f72b6d5a0913e228ca55f165fa9dc1db8eaa057aeefe |
| 21000 | 05fc22d03e934f39bf70d57ac8f7cf20186df5aa4458250338f76d650c5a5233 |
| 21450 | 7c1dcd9e3d0d4bc03f296788c196eadf1e3855e76f414bb2ea2b96d5e9aff10d |
Available training-time evaluations are in the Slayer1B 1.07 model card.
File format
The files are PyTorch pickles (torch.save dictionary of the trainer: model, opt (Muon + AdamW), step,
config); they contain only dictionaries, numbers, strings and tensors. Load with
torch.load(path, map_location="cpu", weights_only=True) (checked with torch 2.13 on ckpt_17000.pt; mmap=True
avoids reading the whole file into memory). Model class: train_gpt_ref_r23.py in the model repositories
(1.00–1.06). Every file is 8 860 287 151 bytes.
License
Apache-2.0 for the checkpoints. The training data includes CC BY-SA sources (the share-alike parts of the Polish core and legal texts, repeated from slayer-pl-8x3b); see the dataset cards of slayer-pl-8x3b and slayer-1.07-pretrain for the licenses of the data.