Slayer9-DQA-9M

An 8,786,185-parameter decoder-only language model by Fabryka AI (Arkadiusz Słota), trained from scratch, annealed with a question-answering mix and fine-tuned on ARC train. It is a research artifact for the Glint Tiny-ML leaderboard, not a general-purpose or instruction-following model.

Name. Slayer is the SlayerLab model line; 9 is the ~9M-parameter family of this line; DQA stands for decay-phase QA: question-answering data (reading passages with their questions, and short multiple-choice questions) placed in the decay (annealing) phase of training, which is what distinguishes this recipe from our baseline; this release uses the licence-clean QA pool listed below. 9M is the size (8.79M unique parameters, the tied input/output embedding counted once).

Evaluation

Status: test split read once for the selected checkpoint (seed 1373, see below). Weights are not uploaded yet.

Selected checkpoint (test, read once)

metric	Glint-1.3 protocol	lm-evaluation-harness 0.4.13 (second column)
BLiMP (67 tasks, 67,000 pairs)	73.83	not measured
ARC-Easy test (2,376 questions)	44.19	not measured
WikiText-2 test, byte perplexity	2.3971	not measured
Overall / Efficiency (leaderboard formulas)	71.16 / 79.60	—

For comparison, the leaderboard row of Qielbas Tiny 9M M2 (as of 2026-10-12) is BLiMP 75.68, ARC-Easy 47.43, WikiText-2 2.3367, Efficiency 81.42; this model does not beat it.

Validation overestimated this model. Its validation Efficiency was 81.88; on test it is 79.60. Two causes, both visible in the numbers: ARC-Easy 46.84 (validation) vs 44.19 (test), expected from choosing the best of fourteen variants on validation; and BLiMP 77.24 on the validation half (first 34 tasks) vs 73.83 on all 67 tasks. For Qielbas M2 the same two BLiMP numbers are 77.65 on our validation half A and 75.68 on the leaderboard (all 67 tasks), so the half-A validation axis overestimates this recipe more than it overestimates M2.

Glint-1.3: zero-shot raw summed log-likelihood of the answer, first 256 tokens, raw accuracy; the protocol adds no BOS of its own (the tokenizer's template does, see Model). lm-evaluation-harness uses a different prompt (Question: …\nAnswer:) and reports acc; the two columns are not comparable as identical measurements. The leaderboard row uses the Glint-1.3 column.

Runs so far (validation split, descriptive)

Validation axis: ARC-Easy validation (570 questions), BLiMP half A (first 34 of the 67 BLiMP tasks, 34,000 pairs) and WikiText-2 validation byte perplexity; the test axis is ARC-Easy test, full BLiMP and WikiText-2 test. Mean ± SD over seeds. The baseline recipe is identical except that its annealing contains no question-answering pool.

stage	n seeds	ARC-E val	BLiMP-A	WikiText-2 val byte ppl	Efficiency (val)
baseline recipe, pretrained	7	40.37 ± 1.21	77.22 ± 0.60	2.3660 ± 0.0038	79.53 ± 0.56
this recipe (QA annealing), pretrained	7	40.40 ± 2.05	78.58 ± 0.63	2.3662 ± 0.0043	80.05 ± 0.82
baseline recipe + ARC fine-tune	7	42.96 ± 1.68	76.83 ± 0.57	2.3803 ± 0.0041	80.31 ± 0.74
this recipe + ARC fine-tune	7	43.58 ± 1.59	78.10 ± 0.57	2.3811 ± 0.0026	81.01 ± 0.67
Δ (this − baseline) after ARC fine-tune, paired by seed	7	+0.62 ± 2.72 (5/7 > 0)	+1.27 ± 0.99 (7/7)	+0.001 ± 0.004	+0.70 ± 1.15 (5/7)
this recipe + ARC fine-tune + 1 extra ARC-Easy epoch	7	44.61 ± 1.64	77.95 ± 0.53	2.3874 ± 0.0037	81.32 ± 0.61
Δ (extra ARC-Easy epoch − ARC fine-tune), paired by seed	7				+0.31 ± 0.39 (5/7)

Validation Efficiency of the seven QA-annealed models after the ARC fine-tune (seeds 1369–1375): 80.93 / 81.07 / 79.80 / 80.62 / 81.20 / 81.76 / 81.72; after the extra ARC-Easy epoch: 81.69 / 81.56 / 80.26 / 80.69 / 81.88 / 81.75 / 81.44.

How the checkpoint was chosen. Seven training seeds (1369–1375) were run with the same recipe. Rule (1) was fixed before any result was known: take the seed with the highest leaderboard Efficiency on the validation axis (Efficiency formula on ARC-Easy validation, BLiMP half A and WikiText-2 validation byte perplexity) after the ARC fine-tune (ties → lower seed). Rule (2) was added at the owner's request after the pre-fine-tune validation results were known, but before any fine-tuned or extra-epoch result was known: one extra epoch on ARC-Easy train only replaces the fine-tune for all seeds if it improves validation Efficiency by at least +0.20 on average and for at least 5 of 7 seeds, in which case the highest of the seven extended models is taken. Rule (2) was met (+0.31, 5/7), so the selected checkpoint is seed 1373 with the extra epoch (validation Efficiency 81.88). The test-axis evaluation was run once, for that checkpoint only; the test splits were never used for training, fine-tuning, epoch choice or seed choice. The test split was read without a separate confirmation run on fresh seeds (owner's decision). Choosing the best of fourteen variants on validation inflated its validation number; the test numbers above are the result.

Model

• 10 layers, d_model 224, 4 heads, RoPE (θ 100,000, interleaved), SwiGLU (×2.667), RMSNorm, QK-norm, value residual; output head tied to the token embedding (stored once). Context 1,024 tokens.
• Tokenizer: BPE, 12,288 tokens (tokenizer.json, sha256 235abda6ba9591b7…). Its template prepends <|endoftext|> (id 12285) to every encoded text: in training every document was preceded by this separator, so evaluation starts the same way. Training data were tokenized without the template (the separator came from the document stream); all reported scores use the tokenizer as shipped (default encode()).
• Files (after selection): model.safetensors (fp32 weights only), config.json, tokenizer.json, inference code, README.md. Loading reproduces the logits of the training checkpoint exactly.

Training

1. Pretraining + annealing, 2.0B tokens total (8,720 steps × 229,376 tokens). Pretraining mix: a subset of SlayerLab/minimal-en-corpus-5b (by Dawid Majewski) restricted to FineWeb-Edu, DCLM-baseline and Wikipedia documents (about 47 / 36 / 16 percent of tokens; 2.29B tokens available, ~0.87 epoch used); documents overlapping WikiText-2 test/validation or ARC-Easy/Challenge test/validation/train were removed before selection (13-gram scan; BLiMP was not scanned).
2. QA annealing (final 1,744 steps, 400M tokens): 19.0 percent from a question-answering pool (each passage once, followed by all of its questions and answers; 4 passes over 18.99M unique tokens), 15.0 percent WikiText-103 train (documents overlapping the union of evaluation test sets removed by a 13-gram filter), 66.0 percent the pretraining mix. Pool sources (train splits only): QASC, Cosmos QA, Quoref, ROPES (CC BY 4.0), CommonsenseQA (MIT), SQuAD v1.1, DROP (CC BY-SA 4.0), BoolQ (CC BY-SA 3.0). Questions that appear in ARC train were removed; 72 documents overlapping WikiText-2 test/validation or ARC-Easy/Challenge test/validation/train were removed by a 13-gram scan (BLiMP was not scanned).
3. ARC fine-tune: ARC-Easy + ARC-Challenge train (CC BY-SA 4.0; questions overlapping ARC test/validation removed), listwise cross-entropy over raw answer likelihoods, 2 epochs (fixed, not selected), AdamW lr 5e-5 (constant), 15 percent language-model replay from the annealing mix; then one extra epoch on ARC-Easy train only with the same settings (adopted by rule (2) above).

Licence and attribution

Weights: CC BY-SA 4.0 (share-alike sources in training: Wikipedia and WikiText-103 (CC BY-SA 3.0 + GFDL), ARC, SQuAD, DROP (CC BY-SA 4.0), BoolQ (CC BY-SA 3.0)). Other sources: FineWeb-Edu (ODC-By), DCLM-baseline (CC BY 4.0), QASC, Cosmos QA, Quoref, ROPES (CC BY 4.0), CommonsenseQA (MIT). The pretraining corpus keeps each document's upstream source and terms.

Sources (repository @ revision, licence): QASC allenai/qasc@a34ba204 CC BY 4.0 · Cosmos QA allenai/cosmos_qa@ed50fe83 (parquet conversion) CC BY 4.0 · Quoref allenai/quoref@41fed5bc CC BY 4.0 · ROPES allenai/ropes@d59f1e2e CC BY 4.0 · CommonsenseQA tau/commonsense_qa@94630fe3 MIT · SQuAD rajpurkar/squad@7b6d24c4 CC BY-SA 4.0 · DROP ucinlp/drop@95cda593 CC BY-SA 4.0 · BoolQ google/boolq@35b264d0 CC BY-SA 3.0 · ARC allenai/ai2_arc@210d026f CC BY-SA 4.0 · pretraining corpus SlayerLab/minimal-en-corpus-5b@bb3aec23, built from FineWeb-Edu HuggingFaceFW/fineweb-edu@87f09149 ODC-By, DCLM-baseline mlfoundations/dclm-baseline-1.0-parquet@817d6752 CC BY 4.0 and Wikipedia wikimedia/wikipedia@b04c8d1c (20231101.en) CC BY-SA 3.0 + GFDL · WikiText-103 Salesforce/wikitext@b08601e0 CC BY-SA 3.0 + GFDL.

Limitations

Very small model (8.8M parameters); trained on the leaderboard's ARC task (in-domain fine-tune, described above); one selected seed out of seven.