SlayerLab/Slayer1B-1.02

Slayer1B 1.02

Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. Slayer1B 1.02 is Slayer1B 1.01 trained further: training resumed from the last checkpoint of 1.01 before its learning-rate decay (step 2250) and continued to 5720 steps, i.e. twice the tokens of 1.01 (5 997 854 720 tokens, ≈ 5.6 tokens per parameter).

From 1.02 on, the learning rate is not decayed at the end of a release. Pre-training continues on the whole prepared corpus of about 24 billion tokens, and the final learning-rate decay is applied once, at the end of the whole Slayer1B line. Each release until then is a constant-learning-rate state; 1.00 and 1.01 ended with a decay. Scores of 1.02 and later releases are therefore not directly comparable to 1.01: on the same batches at step 2800, the training loss of 1.01 (in its decay) was 2.507 and that of 1.02 (constant lr) 2.581.

• Previous versions: SlayerLab/Slayer1B-1.01, SlayerLab/Slayer1B-1.00
• Training checkpoints (with optimizer state) and run record: SlayerLab/Slayer1B-1.02-checkpoints
• Training curve and evaluations during training: https://track.fabryka.ai/run/21ae1234-82c2-42b7-887b-cbcc69901ae1

Results

Leaderboard protocol of https://track.fabryka.ai/models (glint_parity_eval.py, sha256 d7a38ca0…: zero-shot, BLiMP raw summed log-probabilities, ARC-Easy raw accuracy, WikiText-2 non-overlapping 256-token chunks), fp32, final checkpoint of each release.

	1.00	1.01	1.02
training tokens	1.50 B	3.00 B	6.00 B
end of training	lr decay	lr decay	constant lr (no decay)
ARC-Easy, accuracy % (2 376)	42.89	47.98	48.99
BLiMP, accuracy % (67 000)	71.51	73.63	74.71
WikiText-2, byte perplexity ↓	2.5857	2.4230	2.3735
MultiBLiMP Polish, accuracy % (3 272)	97.89	98.29	98.38
ARC-Easy-PL, accuracy % (2 376)	36.91	40.36	41.84

Paired comparison 1.01 (step 2860) → 1.02 (step 5720), same items, paired bootstrap (10 000 resamples, 95 % interval):

task	difference, pp	95 % interval
BLiMP	+1.08	[+0.78; +1.37]	positive on this test
ARC-Easy-PL	+1.47	[−0.13; +3.07]	unresolved
ARC-Easy	+1.01	[−0.67; +2.74]	unresolved
MultiBLiMP Polish	+0.09	[−0.40; +0.58]	unresolved

This comparison combines two changes at once — twice the training tokens and no final learning-rate decay — so no difference can be attributed to either of them separately.

Usage

The weights are a plain PyTorch state dict in safetensors; they are not a transformers AutoModel. The model class is GPT in train_gpt_ref_r23.py (included).

import torch, load_slayer
from tokenizers import Tokenizer
model = load_slayer.load(".")                      # directory with the files of this repository
tok = Tokenizer.from_file("tokenizer.json")
ids = torch.tensor([tok.encode("Stolicą Polski jest").ids])
logits, _ = model(ids)                             # [batch, seq, 32768]

Model

Parameters	1 073 912 851 (input and output embeddings tied, counted once)
Architecture	decoder-only transformer, 20 layers, d_model 2048, 16 attention heads (MHA, head dim 128)
Blocks	pre-norm RMSNorm, QK-norm, RoPE (θ = 500 000), value residual, SwiGLU MLP (hidden 5462), no biases
Context	4096 tokens
Tokenizer	byte-level BPE, 32 768 tokens (tokenizer.json, sha256 0640d3bd…), EOS id 0

Training data

Source: SlayerLab/slayer-pl-8x3b, revision d50df04a6ad46876fa559a87cca5d3e2a96e7098. One stream of 4096-token windows, each window at most once; the same order as 1.01 up to its step 2860:

steps	pack	train.bin sha256	tokens available	batches used
1–1801	pack-00	760ef0cc…	1 889 519 627	1801
1802–3594	pack-01	0494df58…	1 880 520 445	1793
3595–5389	pack-02	294a444d…	1 883 249 429	1795
5390–5720	pack-03	7d675734…	1 890 850 536	331

Each pack has its own disjoint web slice; core, legal and English documents of each pack are selected from a document-hash interval disjoint from those of earlier packs. Mixing weights (cl100k tokens): web 0.63, core 0.19, legal 0.08, English 0.10. Shares of this tokenizer's tokens in the training split: pack-02 web 52.09 %, core 21.63 %, legal 8.66 %, English 17.62 %; pack-03 51.68 / 22.27 / 8.59 / 17.46 (pack-00 and pack-01: see the 1.01 card). Personal data: same rule-based masking as in 1.00 (tags [EMAIL], [TELEFON], [PESEL], [NIP], [REGON], [DOWOD]); masking does not guarantee removal of every personal datum. A 13-gram overlap check of pack-02 and pack-03 against the evaluation sets has not been run (pack-01: see the 1.01 card).

Training

Hardware	8 × NVIDIA RTX 5090 (32 GB), DDP
Batch	256 × 4096 = 1 048 576 tokens per step
Steps	resumed at 2250 (1.01 ckpt_2250.pt, sha256 aeddfb02…), ran 2251 → 5720 (22:17 → 07:23 UTC, ≈ 113 000 tokens/s)
Tokens	5720 × 1 048 576 = 5 997 854 720 in total
Schedule	constant lr 6e-4 throughout. The run was started with a final decay planned from step 4578; at step 3300, still at constant lr, it was resumed with the decay removed (training loss at step 3310 before and after equal to four decimal places)
Optimizer	Muon (lr 0.02) for hidden weight matrices + AdamW (lr 6e-4, weight decay 0.1) for the rest
Final training loss	2.4187 (step 5720); mean of the values logged at steps 5630–5720: 2.4379 (1.01 at the end of its decay, steps 2770–2860: 2.5083)
Resume check	training loss at steps 2260, 2270, 2280 equal to 1.01 at the same steps (same batches, same lr) to four decimals
Checkpoints	every 150 steps from 2400, step 4576 and step 5720; all are constant-lr states
Trainer	train_gpt_ref_r24.py: the 1.00/1.01 trainer plus a chain of data streams and extra checkpoint steps; the model definition is unchanged (train_gpt_ref_r23.py in this repository)

Limitations

• 1 073 912 851 parameters trained on 5 997 854 720 tokens (about 5.6 tokens per parameter).
• Released without a final learning-rate decay (see above).
• Polish-dominant data; English is a minority of the mixture.
• Base model: no instruction tuning, no safety tuning.

License

Weights: Apache-2.0. Training data components carry their own licenses, including CC-BY-SA partitions (see the 1.00 card and the dataset card).