SlayerLab/Slayer1B-1.05

Slayer1B 1.05

Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. Slayer1B 1.05 is Slayer1B 1.04 trained further: training resumed from the final checkpoint of 1.04 (step 10010, constant learning rate 6e-4) and continued to step 12870 (13 495 173 120 tokens in total, ≈ 12.6 tokens per parameter).

Like 1.02–1.04, 1.05 is a constant-learning-rate state: no learning-rate decay was applied. Pre-training continued from this checkpoint to the end of the corpus pass (release 1.06, step 13848).

• Previous versions: SlayerLab/Slayer1B-1.04, SlayerLab/Slayer1B-1.03, SlayerLab/Slayer1B-1.02, SlayerLab/Slayer1B-1.01, SlayerLab/Slayer1B-1.00
• Training checkpoints (with optimizer state): SlayerLab/Slayer1B-1.05-checkpoints (steps 10050–12870); the same files are also in SlayerLab/Slayer1B-1.04-checkpoints (whole continuation 7200–13848)
• Training curve and training-time evaluation: https://track.fabryka.ai/run/f056bb5c-f1da-56d5-9776-cc33063f10a5

Training

• Same model, tokenizer, corpus order and global batch as 1.04: 256 sequences × 4096 tokens = 1 048 576 tokens per step, constant learning rate 6e-4.
• Steps 10010 → 12870 (2860 steps, 2 998 927 360 tokens) on 8× NVIDIA L40S. The run moved from 4× to 8× L40S at the 1.04 checkpoint; the first update on the new hardware reproduced the global batch of the 4× run (same 256 × 4096 samples), so 1.05 continues the same training trajectory rather than starting a new run.
• Data: SlayerLab/slayer-pl-8x3b, continuing the single pass over the corpus started in 1.00.

Training-time evaluation

Measured during training with the run's automatic evaluator (one instrument, NVIDIA A100, zero-shot). These are not the lm-evaluation-harness benchmark used in the Results table of 1.00–1.03 and are not directly comparable with it.

metric	1.04 (10010)	1.05 (12870)
BLiMP (EN), accuracy %	74.17	75.53
ARC-Easy (EN), accuracy %	50.59	52.19
WikiText-2, byte perplexity ↓	2.3078	2.2816
WikiText-2, token perplexity ↓	31.29	29.86
MultiBLiMP (PL), accuracy %	98.87	98.47
ARC-Easy (PL, eu20), accuracy %	44.02	45.20

Single checkpoints differ: e.g. step 13000 scored ARC-Easy (EN) 51.56 and MultiBLiMP (PL) 98.81 (all points in the training curve above); no cause is attributed.

The lm-evaluation-harness benchmark (HellaSwag, PIQA, LAMBADA, ARC-Challenge, BoolQ, MMLU) has not been run on 1.05. The benchmark caveat of the 1.03 card (n-gram overlap of the English shared part with BoolQ, MMLU and ARC test items) applies to this release as well: steps up to 7192 (1.00–1.03 and the first 42 steps of the continuation) used the unfiltered data. From step 7193 (pack-04) on, the 5133 documents with n-gram overlap with BoolQ / MMLU / ARC test items are excluded (list sha256 84caed78733c25a7d7bbe5f0c9c1ef1d2bb249793ef559f1b73f4b390995b445).

Files

model.safetensors (weights), config.json, tokenizer.json, load_slayer.py (loader), train_gpt_ref_r23.py and SHA256SUMS. The config.json records the sha256 of the source checkpoint ckpt_12870.pt (435082ff36becfbb96803622139f38d2afa0ba1a9d2fc284bad96f5a8edc1181, also published in the checkpoints repository).

train_gpt_ref_r23.py is the model definition (GPT class) imported by load_slayer.py; trainer_sha256 in config.json refers to this file. The continuation from 1.03 was trained with train_gpt_ref r24.1 (sha256 881cb055c38b73a24154d96e2359072e9c526f68875c41e0c1b4ed1c4b9eb62c, steps 7150–10010) and r24.2 (sha256 46f83b51fcec18ffc3273d80fb9eab29e182c52d8667800dab27b80f7ff18afc, steps 10010–12870); the architecture is unchanged.