Slayer139 1.02
A small decoder-only English language model (139.3M parameters) by Fabryka AI, built for the Glint Tiny-ML leaderboard (≤150M parameters). It is Slayer139 1.01's pretrained base, given a short continued pretraining and a new ARC fine-tune.
Author: Arkadiusz Słota (Fabryka AI / SlayerLab). Training, evaluation and safety gates were run by the Fabryka AI agent team under his direction.
Identity: Slayer139 1.02 is the training checkpoint with sha256 50f07cab0e1eedb4b3c1e65119a53fb6105f9c6e1fe7984b2b604c77c0c5ec7c (internal run name POL4-CEP s1491). No other checkpoint is published under this name.
Result
lm-evaluation-harness 0.4.13, 0-shot, board protocol (C2) — the same protocol and setup as the Slayer139 1.01 row.
| measure | Slayer139 1.02 | Slayer139 1.01 | Δ |
|---|---|---|---|
| BLiMP (67 tasks) | 79.22 | 79.13 | +0.09 |
| ARC-Easy (test, 2,376, acc) | 65.91 | 63.09 | +2.82 |
| WikiText-2 byte perplexity | 1.8620 | 1.8782 | −0.0162 |
| WikiText-2 score | 99.98 | 99.83 | +0.15 |
| Overall | 81.70 | 80.68 | +1.02 |
Overall is the mean of BLiMP, ARC-Easy and the WikiText-2 score. Board size multiplier 1.003100 (efficiency score 81.96).
Provenance (sha256 prefix of the lm-eval results.json): 8b21f803.
Numbers are our measurements with the board's protocol, not official scores until the leaderboard lists them.
How this checkpoint was selected (read this before comparing)
- 15 fine-tune variants were compared on ARC-Easy validation (one half, 285 questions) plus 500 ARC-Easy train questions held out from all fine-tuning. The other validation half was read twice in the series (once per test candidate), only to forecast the test score.
- The test split was read only after a selection rule written in advance was met. For this model the test set was read once. Across the whole series the test set was read twice: an earlier candidate scored Overall 81.14 and was not published.
- Within the chosen variant the last epoch was taken (fixed in advance, not selected).
- Our Slayer149-ARC model scores Overall 81.50 with the same protocol, so the lead of this model over it is only +0.20 Overall, within about one standard error; treat the two as tied. This checkpoint has not yet been re-benchmarked by the leaderboard maintainers.
Model
- Architecture: 17 layers × 768, 12 heads (12 KV heads), SwiGLU, decoder-only; vocabulary 24,576 (BPE), context 1,024. Identical to Slayer139 1.01.
- Parameters: 139,279,760 unique parameters (input and output embeddings tied: one shared tensor, counted once; counting the shared tensor twice gives 158,154,128).
- Checkpoint lineage: pretrained base (step 90,310, sha256
ba223162…, the base of Slayer139 1.01) → continued pretraining (1,145 steps, sha25605815bf3…) → ARC fine-tune → this model (sha25650f07cab…).
Training
- Pretraining: identical to Slayer139 1.01 (~23.67B tokens, no repeated epoch; see the Slayer139 1.01 card for the optimiser, schedule and hardware).
- Continued pretraining: 1,145 steps, ~303M tokens: 85% the same data mix as pretraining, 15% WikiText-103 train.
- ARC fine-tune (after continued pretraining): listwise cross-entropy over the summed log-probabilities of the answer tokens, prompt
Question: {q}\nAnswer:/{a}(the lm-evaluation-harness format). Data: ARC-Easy train and ARC-Challenge train, after removing questions that overlap the ARC-Easy test set; 500 ARC-Easy train questions held out for selection. ARC-Challenge questions were used in epochs 1–2 only. AdamW, constant LR 2e-5, betas (0.9, 0.95), weight decay 0.01, 8 questions per step, 5 epochs (last epoch taken), seed 1491. Replay: the loss is 0.15 × answer-choice loss + 0.85 × next-token loss on pretraining text windows, to limit forgetting.
Training data
Aggregate corpus; each source keeps its upstream terms.
- ARC-MIX pool (
SlayerLab/gollem-v5-arcmix-9b, 9.39B tokens before retokenization): the 15-source baseSlayerLab/minimal-en-corpus-5b(FineWeb-Edu, DCLM-baseline, open-web-math, FineMath, StarCoderData, StackExchange, LoC public-domain books, Wikipedia, Project Gutenberg, scientific papers, UltraChat, WildChat, CC-News, tiny-textbooks, OpenSubtitles) + a FineWeb-Edu expansion + OpenStax textbooks (×4) + extra copies of FineWeb-Edu documents scored as related to ARC science topics (+2 copies for the highest score, +1 for the next). Removed before training: 4,362 documents by the WikiText-2 / ARC validation+test scan and 1,175 documents marked CC BY-NC-SA. - Top-up (11.4M documents, 57.45 GB of text, build manifest
b73f45f5…): FineWeb-Edu expansion (37.1%, ODC-By 1.0, plus the same dose of ARC-related copies as the pool), FineWeb-Edu (14.9%, ODC-By 1.0), DCLM-baseline (10.9%, CC BY 4.0), StackExchange via RedPajama (6.1%, CC BY-SA content), LoC public-domain books (5.4%, CC0), StarCoderData (5.1%, The Stack terms), Wikipedia 20231101.en (4.6%, CC BY-SA 3.0 / GFDL), Project Gutenberg (2.9%, US public domain), PMC Open Access commercial-use subset (2.8%, CC0 / CC BY / CC BY-SA per article; ND and NC excluded), open-web-math (2.7%, ODC-By), FineMath (2.6%, ODC-By), WildChat-1M (2.0%, ODC-By; English, non-toxic), CC-News (1.4%, licence unknown), UltraChat 200k (0.7%, MIT), tiny-textbooks (0.7%, Apache 2.0). The top-up scan removed 25,808 documents (0.23%), each together with its ARC-related copies. - Licences upstream include ODC-By, CC BY 4.0, CC BY-SA (Wikipedia, StackExchange, some PMC articles), CC0 / public domain, and sources with unknown or restrictive terms. Commercial use: review upstream terms, in particular CC-News, scientific papers, StarCoderData and model-generated chat data (UltraChat, WildChat, tiny-textbooks).
- OpenStax textbooks (55 titles) are used under CC BY 4.0; attribution: OpenStax, Rice University (see OpenStax attribution below).
- Continued-pretraining data: 85% the same mix as the pretraining above, 15% WikiText-103 train (Merity et al., 2016, Salesforce;
Salesforce/wikitext, CC BY-SA 3.0 — text from English Wikipedia). WikiText-103 train shares its source (Wikipedia Good/Featured articles) with the WikiText-2 benchmark but not its test articles; see Limitations. - Fine-tune data: ARC-Easy train and ARC-Challenge train from
allenai/ai2_arc(Clark et al., 2018, Allen Institute for AI), CC BY-SA 4.0. Questions overlapping the ARC-Easy test set were removed from both. ARC-Easy validation and 500 held-out ARC-Easy train questions (never trained on) were used only for selection; the test split was not used for training or selection.
Limitations
- English only; small model; one seed.
- The pretraining mix deliberately upweights web documents related to ARC science topics (decontaminated against the ARC test set); ARC-Easy is therefore an in-domain benchmark for this model, and it was fine-tuned on ARC-Easy and ARC-Challenge train.
- Continued pretraining includes WikiText-103 train, which comes from the same Wikipedia article pool as the WikiText-2 benchmark (different articles); the WikiText-2 score (99.98 vs 99.83) should be read with that in mind.
- The improvement over Slayer139 1.01 was measured with one test read after selection on validation and held-out train questions; the margin over our Slayer149-ARC (+0.20 Overall) is within noise.
OpenStax attribution
The training data of this model includes text extracted from the following OpenStax textbooks, each licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0), © Rice University. Download for free at https://openstax.org. The texts were obtained from the Hugging Face dataset crumb/openstax-text (revision 8f502ca45f9f05cb5673eae445b7a97a4e8c4349). Modified: text extracted, chunked and filtered (chunks matching benchmark test/validation sets were removed).
| Title (from source file name) | © year |
|---|---|
| APBiology | 2018 |
| APCollege Physics | 2017 |
| APMacroeconomics 2e | 2017 |
| APMicroeconomics 2e | 2017 |
| Algebra and Trigonometry 2e | 2021 |
| American Government 3e | 2021 |
| Anatomy and Physiology 2e | 2022 |
| Anatomyand Physiology | 2017 |
| Astronomy 2e | 2022 |
| Astronomy | 2018 |
| Biology 2e | 2020 |
| Business Ethics | 2018 |
| Chemistry 2e | 2019 |
| Chemistry Atoms First 2e | 2019 |
| College Algebra 2e | 2021 |
| College Algebra Corequisite Support 2e | 2021 |
| College Physics | 2020 |
| College Physics 2e | 2022 |
| College Physics for AP Courses 2e | 2022 |
| College Success | 2020 |
| College Success | 2023 |
| Concepts Biology | 2017 |
| Contemporary Mathematics | 2023 |
| Economics 2e | 2018 |
| Economics 3e | 2022 |
| Elementary Algebra 2e | 2020 |
| Entrepreneurship | 2020 |
| Intermediate Algebra 2e | 2020 |
| Introduction to Intellectual Property | n/a |
| Introduction to Philosophy | 2022 |
| Introduction to Political Science | 2022 |
| Introductionto Anthropology | 2022 |
| Introductionto Sociology 3e | 2021 |
| Introductory Business Statistics | 2018 |
| Introductory Statistics | 2018 |
| Macroeconomics 2e | 2018 |
| Macroeconomics 3e | 2022 |
| Microbiology | 2021 |
| Microeconomics 2e | 2018 |
| Microeconomics 3e | 2022 |
| Physics | n/a |
| Prealgebra 2e | 2020 |
| Precalculus 2e | 2021 |
| Preparing for College Success | 2023 |
| Principles Marketing | 2023 |
| Principlesof Finance | 2022 |
| Psychology 2e | 2020 |
| Statistics | n/a |
| USHistory | 2021 |
| University Physics Vol 1 | 2021 |
| University Physics Volume 2 | 2021 |
| University Physics Volume 3 | 2021 |
| World History Volume 1 | 2023 |
| World History Volume 2 | 2022 |
| Writing Guide | 2021 |
Titles licensed CC BY-NC-SA 4.0, non-English titles, and three CC BY 4.0 titles whose text contains elements marked ‘CC BY-NC-SA’ (Introduction to Business, Organizational Behavior, Principles of Management) were not used.
How to load
from modeling_slayer139 import load_slayer139, generate
model, tok = load_slayer139(".") # directory with config.json, model.safetensors, tokenizer.json
print(generate(model, tok, "The capital of France is", max_new_tokens=40))
Requirements: torch, safetensors, tokenizers. The output head is tied to the token embedding (stored once in model.safetensors, re-tied on load).
Reproduce
Evaluation: lm-evaluation-harness 0.4.13, tasks blimp, arc_easy, wikitext (board protocol C2), 0-shot. Method and full run history: paper „Same recipe, 4.4× faster on 8× RTX 5090” (Fabryka AI, in preparation).
- Downloads last month
- 17