Slayer149-ARC

Open models. Measured output.

A 149M-parameter language model from Fabryka AI — fine-tuned from the Slayer149-balanced base for strong multiple-choice question answering, without losing the general language ability that makes a model actually useful.

---

ARC-Easy	66.58% — best on the board
BLiMP	78.38%
WikiText-2 byte PPL	1.907
Raw overall	81.51 — #1
Efficiency	81.52 — #1

Intelligence, in production.

Why it matters

• Highest raw and efficiency score on the Tiny-ML leaderboard. 81.51 raw overall — every benchmark measured with the board's own pinned harness, on full public test sets, zero shortcuts.
• Best ARC-Easy result on the board: 66.58%. The model answers grade-school science questions better than any other submission, at any size.
• A factory, not a black box. Base model, tokenizer, and evaluation protocol are all public. Numbers you can reproduce.

Measured output

Evaluation protocol: lm-evaluation-harness 0.4.13, document-level WikiText, rolling likelihood at 1024-token context, ARC-Easy zero-shot, BLiMP sentence-pair likelihood. No test or validation data was used in training; the fine-tuning set is the clean ARC-Easy train split only.

Architecture

Property	Value
Parameters	149,333,081
Layers	26
Hidden size	576
Attention	GQA (9 query / 3 KV heads, 64-dim)
Vocabulary	24,576 BPE tokens
Context	2,048 tokens
Embeddings	tied

Compact decoder-only transformer with grouped-query attention, RMSNorm, SwiGLU MLPs and rotary position embeddings.

Training

• Pretraining (base model): ~31B tokens of filtered educational and general web text — FineWeb-Edu, DCLM-Edu, Cosmopedia, FineMath.
• Fine-tuning: the clean ARC-Easy training split (2,235 examples) mixed with a general-text and grammar replay stream, so the model learns the question-answering format while keeping its general language quality.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("SlayerLab/Slayer149-ARC", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("SlayerLab/Slayer149-ARC")

prompt = "Question: Which property of a mineral can be determined just by looking at it?\nAnswer:"
inputs = tokenizer(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=8)
print(tokenizer.decode(out[0]))

A research base model, not an instruction-tuned assistant. Trained for English multiple-choice QA and general text.

---

_{Fabryka AI · Warsaw · fabryka.ai
Open models ✳ Real machines ✳ Measured output ✳ Independent thinking}