Slayer1B 1.06
Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. Slayer1B 1.06 is Slayer1B 1.05 trained to the end of the corpus: training resumed from the final checkpoint of 1.05 (step 12870, constant learning rate 6e-4) and continued to step 13848, which completes one full pass over the prepared corpus (14 520 680 448 tokens in total, ≈ 13.5 tokens per parameter).
1.06 is a constant-learning-rate state: no learning-rate decay and no context extension were applied. It is the final checkpoint of the first pass over the corpus; further training of the Slayer1B line may continue from it.
- Previous versions: SlayerLab/Slayer1B-1.05, SlayerLab/Slayer1B-1.04, SlayerLab/Slayer1B-1.03, SlayerLab/Slayer1B-1.02, SlayerLab/Slayer1B-1.01, SlayerLab/Slayer1B-1.00
- Training checkpoints (with optimizer state): SlayerLab/Slayer1B-1.06-checkpoints (steps 12900–13848); the same files are also in SlayerLab/Slayer1B-1.04-checkpoints (whole continuation 7200–13848)
- Training curve and training-time evaluation: https://track.fabryka.ai/run/f056bb5c-f1da-56d5-9776-cc33063f10a5
Training
- Same model, tokenizer, corpus order and global batch as 1.05: 256 sequences × 4096 tokens = 1 048 576 tokens per step, constant learning rate 6e-4.
- Steps 12870 → 13848 (978 steps, 1 025 507 328 tokens) on 8× NVIDIA L40S. This last segment is shorter than the 2860-step segments of 1.04 and 1.05 because it ends where the corpus ends; it was not padded with repeated data.
- Data: SlayerLab/slayer-pl-8x3b, one pass, each training token seen once.
Training-time evaluation
Measured during training with the run's automatic evaluator (one instrument, NVIDIA A100, zero-shot). These are not the lm-evaluation-harness benchmark used in the Results table of 1.00–1.03 and are not directly comparable with it.
| metric | 1.04 (10010) | 1.05 (12870) | 1.06 (13848) |
|---|---|---|---|
| BLiMP (EN), accuracy % | 74.17 | 75.53 | 76.12 |
| ARC-Easy (EN), accuracy % | 50.59 | 52.19 | 52.99 |
| WikiText-2, byte perplexity ↓ | 2.3078 | 2.2816 | 2.2849 |
| WikiText-2, token perplexity ↓ | 31.29 | 29.86 | 30.03 |
| MultiBLiMP (PL), accuracy % | 98.87 | 98.47 | 98.41 |
| ARC-Easy (PL, eu20), accuracy % | 44.02 | 45.20 | 45.03 |
The lm-evaluation-harness benchmark (HellaSwag, PIQA, LAMBADA, ARC-Challenge, BoolQ, MMLU) has not been run on 1.06 yet.
The benchmark caveat of the 1.03 card (n-gram overlap of the English shared part with BoolQ, MMLU and ARC test items)
applies to this release as well: steps up to 7192 (1.00–1.03 and the first 42 steps of the continuation) used the
unfiltered data. From step 7193 (pack-04) on, the 5133 documents with n-gram overlap with BoolQ / MMLU / ARC test
items are excluded (list sha256 84caed78733c25a7d7bbe5f0c9c1ef1d2bb249793ef559f1b73f4b390995b445).
Files
model.safetensors (weights), config.json, tokenizer.json, load_slayer.py (loader), train_gpt_ref_r23.py and
SHA256SUMS. The config.json records the sha256 of the source checkpoint ckpt_13848.pt
(1fc7396ca1790805128244b2f827e02b9fe88e352534cd887d0c488f121d1e4d, also published in the checkpoints repository).
train_gpt_ref_r23.py is the model definition (GPT class) imported by load_slayer.py; trainer_sha256 in
config.json refers to this file. The continuation from 1.03 was trained with train_gpt_ref r24.1
(sha256 881cb055c38b73a24154d96e2359072e9c526f68875c41e0c1b4ed1c4b9eb62c, steps 7150–10010) and r24.2
(sha256 46f83b51fcec18ffc3273d80fb9eab29e182c52d8667800dab27b80f7ff18afc, steps 10010–13848); the architecture is unchanged.
- Downloads last month
- 26