SlayerLab/Slayer1B-1.00

Slayer1B 1.00

Decoder-only language model, 1 073 912 851 parameters, trained from scratch on Polish-dominant text (pack-00 of SlayerLab/slayer-pl-8x3b).

Training curve: https://track.fabryka.ai/run/48744ed2-3155-4294-b899-a6fda1f851da.

Results

Leaderboard protocol of https://track.fabryka.ai/models (glint_parity_eval.py, sha256 d7a38ca0…: zero-shot, BLiMP raw summed log-probabilities, ARC-Easy raw accuracy, WikiText-2 non-overlapping 256-token chunks), fp32, final checkpoint (step 1430, trainer checkpoint sha256 7d092d4d…). Computed twice with two independent loaders (these weights via load_slayer.py, and the trainer checkpoint directly); both gave the same numbers.

task	metric	n	score
BLiMP (67 phenomena)	accuracy	67 000	71.51 % (47 913 correct)
ARC-Easy (test)	accuracy	2 376	42.89 % (1 019 correct)
WikiText-2 (raw, test)	token perplexity (this tokenizer)	—	49.9734
WikiText-2 (raw, test)	byte perplexity	1 292 008 bytes	2.5857
eff (leaderboard score from BLiMP, ARC-Easy and WikiText-2 byte perplexity)	leaderboard formula	—	69.50

Polish columns of the same leaderboard (also computed twice with two independent implementations; identical item by item):

task	metric	n	score
MultiBLiMP, Polish (jumelet/multiblimp pol)	accuracy, summed log-probability, leading-space prefix	3 272	97.89 % (3 203 correct)
ARC-Easy-PL (Eurolingua/arcx EU20, test)	accuracy, LL(q + choice) − LL(q)	2 376	36.91 % (877 correct)

Usage

The weights are a plain PyTorch state dict in safetensors; they are not a transformers AutoModel. The model class is GPT in train_gpt_ref_r23.py (included).

import torch, load_slayer
from tokenizers import Tokenizer
model = load_slayer.load(".")                      # directory with the files of this repository
tok = Tokenizer.from_file("tokenizer.json")
ids = torch.tensor([tok.encode("Stolicą Polski jest").ids])
logits, _ = model(ids)                             # [batch, seq, 32768]

Model

Parameters	1 073 912 851 (input and output embeddings tied, counted once)
Architecture	decoder-only transformer, 20 layers, d_model 2048, 16 attention heads (MHA, head dim 128)
Blocks	pre-norm RMSNorm, QK-norm, RoPE (θ = 500 000), value residual, SwiGLU MLP (hidden 5462), no biases
Context	4096 tokens
Tokenizer	byte-level BPE, 32 768 tokens (tokenizer.json, sha256 0640d3bd…), EOS id 0

Training data

Source: SlayerLab/slayer-pl-8x3b, revision d50df04a6ad46876fa559a87cca5d3e2a96e7098, pack-00 (its own disjoint web slice + the shared core, legal and English components), sampled per document with the pack's mixing weights (defined in cl100k tokens): web 0.63, core 0.19, legal 0.08, English 0.10.

component	share of tokens (this tokenizer)	license
web (HPLT v3 PL, disjoint slice)	51.8 %	CC0-1.0
core (register-diverse; includes CC-BY-SA partition)	22.2 %	mixed, incl. CC-BY-SA
legal / public administration (includes CC-BY-SA partition)	8.5 %	CC-BY-4.0 / PD / CC-BY-SA
English	17.6 %	ODC-BY

English is 10 % of the mixture in cl100k tokens and 17.6 % in this tokenizer's tokens: the tokenizer is trained on Polish and splits English into more pieces.

Available training stream: 1 889 519 627 tokens (train.bin, sha256 760ef0cc…), of which 1 499 463 680 were consumed (1430 steps × 1 048 576; each window at most once). Held-out validation from the same pack, disjoint by document id: 1 562 647 PL tokens, 1 643 997 EN tokens.

Personal data: before tokenization, e-mail addresses, phone numbers (Polish and international), PESEL, NIP, REGON and Polish ID-card numbers were replaced with tags [EMAIL], [TELEFON], [PESEL], [NIP], [REGON], [DOWOD] (rule-based; counts in the data manifest: e-mail 19 080, phone 21 606, NIP 1 602, REGON 1 421, ID card 259, PESEL 131). The source data also contains upstream placeholders such as [PII]. Rule-based masking does not guarantee removal of every personal datum.

Training

Hardware	8 × NVIDIA RTX 5090 (32 GB), DDP
Batch	256 × 4096 = 1 048 576 tokens per step
Steps	1430 (02:35:48 → 06:47:13 UTC, about 4 h 11 min; ~100 000 tokens/s)
Tokens consumed	1430 × 1 048 576 = 1 499 463 680 (about 1.4 tokens per parameter)
Final training loss	2.6969 (step 1430, nats per token)
Checkpoints	every 150 steps (10 in total), each copied off the training node with sha256
Optimizer	Muon (lr 0.02) for hidden weight matrices + AdamW (lr 6e-4, weight decay 0.1) for the rest
Schedule	warmup 150 steps, constant, then 1 − sqrt decay over the last 20 % of steps (from step 1144) to lr / 10
Data order	each 4096-token window of the stream used at most once (epoch sampler), data seed 20261023
Precision	bf16 autocast, torch.compile, activation checkpointing

Training-loss scenarios (not a forecast)

analysis/scaling-training-loss.png shows the measured training loss and conditional scenarios for a total budget of 3, 6 and 12 billion tokens. Training continues from checkpoint 1050 (the last one before the learning-rate decay that starts at step 1144) at constant learning rate, and the learning rate is decayed only over the last 20 % of the new budget. Up to step 1050 this is the same schedule as a new warmup-stable-decay run with that budget. The scenarios come from the learning-rate-annealing scaling law (Tissue et al., arXiv:2408.11029) fitted to steps 200–1430, for λ = 0.99 / 0.995 / 0.999. Checkpoint 1050 and the other intermediate checkpoints (with optimizer state) are in SlayerLab/Slayer1B-1.00-checkpoints.

• The backtest of this method failed on this run: fitted on earlier steps, it missed the measured loss by 0.077–0.088 nats per token (threshold 0.03). The bars of ±0.09 show that historical error; they are not a confidence or prediction interval.
• The spread between λ values (up to 0.6 nats at 12 B tokens) is larger than the gain being estimated. One 1.5 B-token run does not determine it.
• Assumptions: training loss (not validation loss), new data from the same distribution beyond the 1.89 B tokens that exist, one seed. Benchmarks are not projected.
• Numbers: analysis/scaling-training-loss.json; input loss log: analysis/training_metrics.jsonl. Two independent implementations agree to 3e-7 nats.

Limitations

• 1 073 912 851 parameters trained on 1 499 463 680 tokens (about 1.4 tokens per parameter). Compare with the token counts of other models when reading the scores.
• Polish-dominant data; English is a minority of the mixture.
• Base model: no instruction tuning, no safety tuning.
• Masking of personal data is rule-based (see above).

License

Weights: Apache-2.0. Training data components carry their own licenses (table above), including CC-BY-SA partitions.