SlayerLab/Slayer1B-1.08

Slayer1B 1.08

Arkadiusz Słota, Anna Sieczka, Piotr Styła / Fabryka AI (SlayerLab)

Slayer1B 1.08 is an experimental base language model continued from Slayer1B 1.07. Training completed on 9 November 2026 after 1,709 updates and 1,792,016,384 processed token positions, using Polish analytical text, English technical text and replay. It is not instruction-tuned and is distinct from the earlier 1.08b ablation.

Public training trace.

Run objective

Train on as many approved token positions as the fixed compute budget permits, recording actual progress and expense at the spot prices observed during the run. Synthetic speed, real-data training and downstream evaluation are separate measurements. The expense report covers the billed session, including preparation and evaluation; steady-state cost per billion tokens is an extrapolation, not that bill.

Model and starting point

The run initializes from SlayerLab/Slayer1B-1.07, revision a090b54e658fb57dc5166c1fcc90cc6bf5c80106. Its published card describes a Polish-dominant decoder-only model with 1,073,912,851 parameters, a 32,768-token v6 vocabulary, and final checkpoint 22,327. These are base-model properties.

The continuation preserves that architecture and tokenizer. Its task is causal text continuation, not instruction tuning. The objective is greater exposure to Polish analytical and English technical text while retaining general-language coverage.

Frozen launch recipe

Setting	Selected configuration
Initialization	Public 1.07 weights; fresh optimizer, rather than optimizer-state resume
Trainer and optimizer	Existing r24.2; Muon with AdamW
Batch and context	256 sequences × 4,096 token positions
Parallel layout	Eight GPU ranks; microbatch 8 × accumulation 4; activation checkpointing disabled
Schedule	300 warmup updates, then WSD with decay over the final 25% to a nonzero floor of 0.00006
Checkpoints	Every 250 updates and at the end; FINAL fixed in advance for evaluation

Training execution and cost

Observation	Value and scope
Hardware	Eight NVIDIA B200 GPUs, Nebius spot; USD 0.99 per GPU-hour
Completed updates	1,709
Processed positions	1,792,016,384 at the stated global batch; approximately 0.64 passes through the prepared input
Training window	58.6 minutes, including compilation and checkpoints; not the full billed session
Steady interval throughput	Approximately 572,000 positions/s; not positions divided by the whole training window
Training loss	Approximately 2.20 at update 10 to 1.99 at completion; not an evaluation score
Training GPU cost	Approximately USD 7.73, or USD 4.32 per billion processed positions, calculated from the observed window and spot rate
Cost through evaluation completion	Approximately USD 21.5; preparation ≈11.2, training 7.73, evaluation/export ≈2.6. Provider-counter attribution, not a separate settled invoice
Complete allocation cost	Approximately USD 24.1; the allocation was preempted shortly afterwards and released, rather than reused for 1.09

Input capacity, processed positions, steady interval speed and the complete billed session remain separate quantities.

Training inputs

Input	Pinned revision
Slayer PL Logika 2B	e24c8b71526ca1dc56a280100f88c2f09a79ecdd
English Technical 1B	3b833d36bd3ba9aeecf46f39605c9ac849bb96a3
Slayer 1.07 replay	6a337bbfba518d88f14eb54f7d0a9b9f297df0d1

Replay uses pl_web, en_fineweb_edu and en_mix: previously seen material, not fresh web. Reference v2 contains 2,800,806,882 native token positions including EOS, after source selection and filtering; this is input capacity, not observed training exposure. The prepared TRAIN digest is b73a69eaf399bb594300e7d1914a967a38b80b5bc35505fbc6459a6a27be496f. GPL/GFDL and mixed GPL/LGPL sources are excluded. English validation/test remain outside TRAIN, and Polish held-out sections are grouped by source book.

Comparison limits

A replay-only continued-pretraining control is not budgeted for the priority run. A difference from 1.07 therefore combines data, fresh optimizer initialization and learning-rate decay: 1.07 was released from a constant-LR checkpoint without decay. It does not isolate the new datasets' effect. Inherited exposure in the base model is an additional limitation.

Evaluation records identify checkpoints, evaluator, split, sample count, prompting, normalization and test-read history.

Standard benchmark measurements

We thank BenchTeam / Fabryka AI for its benchmark implementations, protocol work and evaluation proposals. The team's independent full benchmark report is a separate measurement source. The table below reports SlayerLab evaluations, also published in the 1.07 training record and the 1.08 training record; it is not a copy of that independent run.

All nine tasks use their full evaluated split, zero-shot likelihood scoring and float32. Scores are percentages with pointwise 95% confidence intervals. acc is accuracy from summed continuation likelihood; acc_norm is the evaluator's length-normalized accuracy. The metric is named per task rather than combined into a score. WinoGrande uses partial-sentence likelihood; MMLU-PL is micro-averaged over 14,042 questions. Polish HellaSwag and MMLU use the OpenGPTX task definitions.

For comparison, the continuation columns are 1.09, not additional 1.08 measurements: updates 5,000, 10,000, 15,000 and 20,000 are distinct intermediate working checkpoints of the ongoing 1.09 run, not inference-ready weight releases or its eventual budget-limited endpoint. Evaluator SHA, dataset pins, file digests, task protocols and sample counts match across these rows.

Task	Metric	n	1.07	1.08	1.09, update 5,000	1.09, update 10,000	1.09, update 15,000	1.09, update 20,000
ARC-Challenge	acc_norm	1,172	27.30 [24.74, 29.86]	30.03 [27.39, 32.68]	29.35 [26.79, 32.00]	30.46 [27.82, 33.11]	32.51 [29.86, 35.24]	33.11 [30.46, 35.84]
HellaSwag	acc_norm	10,042	43.38 [42.43, 44.34]	43.42 [42.43, 44.39]	46.53 [45.55, 47.52]	48.93 [47.97, 49.91]	49.77 [48.80, 50.76]	50.38 [49.39, 51.35]
PIQA	acc_norm	1,838	66.97 [64.80, 69.15]	66.21 [64.15, 68.34]	68.28 [66.21, 70.40]	68.99 [66.92, 71.11]	68.99 [66.87, 71.06]	70.78 [68.72, 72.85]
WinoGrande	acc	1,267	53.59 [50.83, 56.35]	53.67 [50.91, 56.43]	54.93 [52.17, 57.70]	54.93 [52.17, 57.62]	56.75 [54.06, 59.43]	55.01 [52.25, 57.77]
BoolQ	acc	3,270	58.29 [56.61, 59.97]	59.97 [58.26, 61.59]	61.80 [60.18, 63.49]	61.13 [59.45, 62.78]	59.88 [58.20, 61.53]	51.41 [49.69, 53.12]
OpenBookQA	acc_norm	500	33.60 [29.40, 37.80]	34.80 [30.60, 39.00]	35.60 [31.40, 40.00]	34.40 [30.20, 38.60]	36.00 [31.80, 40.20]	35.20 [31.00, 39.40]
Belebele, Polish	acc	900	26.00 [23.11, 28.89]	24.00 [21.22, 26.89]	24.11 [21.33, 26.89]	24.11 [21.33, 27.00]	22.56 [19.89, 25.33]	24.67 [21.89, 27.56]
HellaSwag-PL	acc_norm	10,042	43.83 [42.84, 44.79]	43.90 [42.90, 44.87]	44.73 [43.76, 45.71]	45.51 [44.51, 46.48]	44.18 [43.21, 45.16]	44.24 [43.26, 45.22]
MMLU-PL	acc	14,042	24.01 [23.30, 24.71]	23.04 [22.34, 23.74]	24.10 [23.39, 24.82]	23.32 [22.62, 24.03]	23.50 [22.80, 24.21]	23.86 [23.17, 24.58]

Intervals use the percentile bootstrap with 10,000 replicates and seed 20261110. The updated approved aggregate does not supply the random-stream derivation or CI-generator SHA; the previous aggregate documented model-specific streams. Items are treated as independent; subject, template and source clusters are not modeled. These are intervals for individual results, not paired intervals for model differences or estimates of training-seed variance. They do not support a difference claim based on interval overlap.

Machine-readable aggregate record: exact scores, intervals, counts, dataset revisions and file digests, checkpoint and evaluator identities, and metric selection. It contains no prompts, answers or per-item predictions. ARC-Challenge, OpenBookQA and MMLU-PL use test splits; HellaSwag, PIQA, WinoGrande, BoolQ and HellaSwag-PL use validation splits; Belebele uses the full Polish evaluation set.

The approved aggregate SHA-256 is fc6f67a20bdc6ada687349d35de26f6811e27766dfe1aa197082000c03535796. The earlier 1.07 and 1.08 scores, intervals and dataset pins are unchanged. The continuation comparison does not extend or replace the separate GSM8K/SST-2 protocol below.

Descriptive results; contamination scan of the training lineage against these tasks is pending. Belebele and MMLU-PL results are near their four-choice 25% random baseline. No overall benchmark average, ranking or model-release threshold is inferred from this table. Results from other evaluators can differ with prompting, BOS handling, normalization or library version; they should not be silently substituted.

Earlier internal diagnostics and their measurements remain available in the versioned evaluation record. They are historical measurements, not this card's release criteria.

Additional GSM8K and SST-2 measurements

SlayerLab measured this two-task subset on 11 November 2026 using the pinned BenchTeam code, lm-eval 0.4.9.1, the slayer adapter, float32, batch 1 and context limit 4,096. These are full evaluated splits, not a full 50-task run or a copy of the community's independent report. The evaluated 1.07 and 1.08 weight identities match the completed checkpoints above.

Task	Metric / prompting	n	1.07, update 22,327	1.08, update 1,709
GSM8K	strict exact match, 5-shot	1,319	1.52 [0.91, 2.20]	0.91 [0.45, 1.44]
SST-2	accuracy, 0-shot	872	51.03 [47.71, 54.36]	50.46 [47.13, 53.67]

Scores are percentages with reported pointwise 95% bootstrap intervals (10,000 draws, seed 20261028). These intervals describe individual results, not model differences or training-seed variance. This protocol and its intervals are recorded separately under additional_measurements in EVALUATION.json; the preceding nine-task protocol is unchanged.

Descriptive results; training-lineage contamination scan pending. GSM8K strict-match scores are near zero. SST-2 zero-shot accuracy is calibration-sensitive; these scores alone do not establish a general reasoning or sentiment-understanding gain. No paired-difference or internal calibration analysis is published here.

Licensing and limitations

Weights are released under Apache-2.0, as 1.07; source corpora keep their own obligations, with attributions in the data card. FinePDFs compilation terms and its TDM rationale do not settle rights to every PDF. Book, translation and technical-source notices must be preserved where required.

A base language model can generate inaccurate, biased, offensive or personal information. Translated reasoning and technical text can contain errors; their presence does not demonstrate reasoning, technical reliability or instruction following. Privacy filtering is heuristic, and technical or tax material is not professional advice.

Acknowledgements

We thank the entire BenchTeam / Fabryka AI community for benchmark implementations, original-protocol documentation, evaluation runs and independent checks. The team is coordinated by @LoIlita and @Elwenor, as documented in the repository's contribution guide.

The following accounts contributed to benchmarks-fabryka-ai. This is a summary of documented contributions, not a ranking or a complete membership list.

GitHub account	Documented contribution
@LoIlita	Team coordination and onboarding; evaluation workflow, methodology and benchmark reviews (PR #92, PR #185).
@Elwenor	Team coordination; EWoK implementation, evaluation runner, multi-model benchmark run and reports (PR #93, PR #179).
@adgr3	Original-benchmark cards, numerical-verdict validation and dataset-licence review (PR #94, PR #144, PR #175).
@gzawadzki	Slayer integration with lm-eval and WinoGrande evaluation (PR #64, PR #100).
@KateMajzel	MathQA and PubMedQA evaluation work (PR #158, PR #156).
@jakublvndsky	HellaSwag reproduction and Slayer evaluation (PR #149).
@janbanot	ARC-Easy original-benchmark card (PR #70).

We invite BenchTeam, DatasetTeam and other contributors to propose datasets for subsequent Slayer training runs. The collaboration aims to assemble a DatasetTeam corpus of one trillion (1T) tokens. Source proposals should include provenance, version, language and domain coverage, applicable rights and known privacy or evaluation-overlap risks. This is a collection objective, not a claim that the corpus has already been assembled or used in Slayer1B 1.08.