SlayerLab/Slayer1B-1.01

Slayer1B 1.01

Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. Slayer1B 1.01 is Slayer1B 1.00 trained further: training resumed from the last checkpoint of 1.00 before its learning-rate decay (step 1050) and continued to 2860 steps, i.e. twice the tokens per parameter of 1.00 (2 998 927 360 tokens, ≈ 2.8 tokens per parameter).

• Previous version: SlayerLab/Slayer1B-1.00
• Training checkpoints (with optimizer state): SlayerLab/Slayer1B-1.01-checkpoints
• Training curve and evaluations during training: https://track.fabryka.ai/run/4934429a-d393-42e0-8aac-03eae6b71ff7

Results

Leaderboard protocol of https://track.fabryka.ai/models (glint_parity_eval.py, sha256 d7a38ca0…: zero-shot, BLiMP raw summed log-probabilities, ARC-Easy raw accuracy, WikiText-2 non-overlapping 256-token chunks), fp32, final checkpoint (step 2860, trainer checkpoint sha256 b0a28879…). The same instrument reproduces the published 1.00 numbers exactly. The released model.safetensors (sha256 607d1a6c…) was exported twice from the step-2860 checkpoint, on two machines, with identical bytes.

task	metric	n	1.00 (step 1430)	1.01 (step 2860)
BLiMP (67 phenomena)	accuracy	67 000	71.51 %	73.63 % (49 335 correct)
ARC-Easy (test)	accuracy	2 376	42.89 %	47.98 % (1 140 correct)
WikiText-2 (raw, test)	byte perplexity	1 292 008 bytes	2.5857	2.4230
eff (leaderboard score from BLiMP, ARC-Easy and WikiText-2 byte perplexity)	leaderboard formula	—	69.50	72.29
MultiBLiMP, Polish (jumelet/multiblimp pol)	accuracy, summed log-probability, leading-space prefix	3 272	97.89 %	98.29 % (3 216 correct)
ARC-Easy-PL (Eurolingua/arcx EU20, test)	accuracy, LL(q + choice) − LL(q)	2 376	36.91 %	40.36 % (959 correct)

Evaluation history of 1.00 and 1.01

Measured values only (no fits, no trend lines): one checkpoint every 150 steps of both runs, same instrument. Data: analysis/scores-history.csv (step, tokens, checkpoint sha256, scores, and which of three GPUs computed the row; each GPU was checked against a known point before use). 1.00 at 1.26 B and 1.42 B tokens is already in its learning-rate decay; 1.01 at the same token counts is not.

Training loss vs. windows fixed before the run

Before the run, windows for the mean training loss were written down from the training-loss scenarios on the 1.00 card (continuation of the 1.00 curve from step 1050):

steps	window fixed before the run	measured mean of logged values
1700–1800 (constant lr, pack-00)	2.692 – 2.882	2.6977
2770–2860 (end of the decay, pack-01)	2.367 – 2.660	2.5083

Both measurements fall inside their windows; the first one at the lower edge. This is one run, not a test of the scenarios.

Usage

The weights are a plain PyTorch state dict in safetensors; they are not a transformers AutoModel. The model class is GPT in train_gpt_ref_r23.py (included).

import torch, load_slayer
from tokenizers import Tokenizer
model = load_slayer.load(".")                      # directory with the files of this repository
tok = Tokenizer.from_file("tokenizer.json")
ids = torch.tensor([tok.encode("Stolicą Polski jest").ids])
logits, _ = model(ids)                             # [batch, seq, 32768]

Model

Parameters	1 073 912 851 (input and output embeddings tied, counted once)
Architecture	decoder-only transformer, 20 layers, d_model 2048, 16 attention heads (MHA, head dim 128)
Blocks	pre-norm RMSNorm, QK-norm, RoPE (θ = 500 000), value residual, SwiGLU MLP (hidden 5462), no biases
Context	4096 tokens
Tokenizer	byte-level BPE, 32 768 tokens (tokenizer.json, sha256 0640d3bd…), EOS id 0

Training data

Source: SlayerLab/slayer-pl-8x3b, revision d50df04a6ad46876fa559a87cca5d3e2a96e7098.

• Steps 1–1050 (from 1.00) and 1051–1801: the pack-00 stream of 1.00 (train.bin sha256 760ef0cc…), each 4096-token window at most once, the same windows as 1.00 up to its step 1050.
• Steps 1802–2860: a new stream from pack-01 (its own disjoint web slice; core, legal and English documents selected from a hash interval disjoint from the one used for pack-00), train.bin sha256 0494df58…, 1 880 520 445 tokens available, of which 1 059 × 1 048 576 were consumed.

Mixing weights of the pack (cl100k tokens): web 0.63, core 0.19, legal 0.08, English 0.10. In this tokenizer's tokens the training split (train.bin) of the pack-01 stream is web 52.16 %, core 22.14 %, legal 8.20 %, English 17.49 % (pack-00 stream: 51.79 / 22.19 / 8.46 / 17.56). Personal data: same rule-based masking as in 1.00 (tags [EMAIL], [TELEFON], [PESEL], [NIP], [REGON], [DOWOD]); masking does not guarantee removal of every personal datum.

Overlap with evaluation sets (13-gram, pack-01 stream vs. the task files): no evaluation item has 90 % or more of its 13-grams in the stream; MultiBLiMP-pl has 10 of 3 272 items with ≥ 50 % overlap, WikiText-2 has 26 of 62 documents with any shared 13-gram (none ≥ 50 %).

Training

Hardware	8 × NVIDIA RTX 5090 (32 GB), DDP
Batch	256 × 4096 = 1 048 576 tokens per step
Steps	resumed at 1050, ran 1051 → 2860 (13:56 → 18:54 UTC, ≈ 108 000 tokens/s)
Tokens	2860 × 1 048 576 = 2 998 927 360 in total
Schedule	warmup 150, constant lr 6e-4 through step 2289, 1 − sqrt decay over steps 2290–2860 to lr / 10 (6.05e-5 at step 2860)
Optimizer	Muon (lr 0.02) for hidden weight matrices + AdamW (lr 6e-4, weight decay 0.1) for the rest
Final training loss	2.5067 (step 2860); mean of the values logged at steps 2770–2860: 2.5083 (1.00, steps 1340–1430: 2.6990)
Resume check	training loss after the resume vs. the 1.00 run at the same steps (same batches, same lr): mean difference about 1e-4 nats (two independent computations: −0.00008 over steps 1060–1140, −0.00013 over 1060–1130), largest single-step difference 0.0007
Checkpoints	every 150 steps + final; last before the decay: step 2250

Limitations

• 1 073 912 851 parameters trained on 2 998 927 360 tokens (about 2.8 tokens per parameter).
• Polish-dominant data; English is a minority of the mixture.
• Base model: no instruction tuning, no safety tuning.

License

Weights: Apache-2.0. Training data components carry their own licenses, including CC-BY-SA partitions (see the 1.00 card and the dataset card).