LuzCommand/lil-beastie-4b

lil'beastie — a 4B that knows what it doesn't know

"The beginning of wisdom is 'I do not know.'" — Lt. Cmdr. Data, Star Trek: TNG

Most small models fail silently: they guess, or reason until they run out of room. This one says \boxed{IDK} instead, and sampling it a few times tells you how far to trust the answer.

TL;DR

• Qwen3-4B fine-tune that abstains natively with \boxed{IDK} and keeps its reasoning short
• Controlled test, only the weights changed (same prompt, sampler, 10-sample vote, engine): abstains on 21% of rollouts vs 1%, calibration error 0.113 vs 0.162, and under the same vote rule it is confidently wrong half as often (9.9% vs 17.7% of questions) — the difference comes back as explicit, routable IDKs instead of wrong answers — on 40% fewer tokens
• On 7,102 never-trained MMLU-Pro questions the 10-sample vote reaches ECE 0.082 (published single-pass figures: Qwen3-32B 0.218, base 0.328 — a different method; see below)
• On free-form math it is right 98–99% of the time it commits (58 never-trained AIME problems × 10 rollouts per prompt wording, full traces), and says \boxed{IDK} on 7–14% instead of guessing
• No coding cost: HumanEval pass@1 92.9% vs 92.8% for the stock base on the identical harness (164 tasks × 5 samples, paired t = 0.12), with 22% less generation time and zero abstentions on code
• Runs in ~6 GB on a Mac (MLX q8) or 10 GB on a GPU (bf16). Q4 is not supported — use q8. Every measured number on this card rescores from the shipped raw rows; the core 645-question run re-runs in about an hour on one A100 ($2).
• Built as the honest front tier of a model cascade: its IDKs and vote-spreads are the routing signal.

Quickstart

Transformers (bf16, repo root):

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("LuzCommand/lil-beastie-4b", torch_dtype="bfloat16")
tok = AutoTokenizer.from_pretrained("LuzCommand/lil-beastie-4b")

MLX on a Mac:

from mlx_lm import load, generate
model, tok = load("LuzCommand/lil-beastie-4b")

(That loads bf16, ~10 GB. For the 6 GB version, download the mlx-q8/ folder and load("path/to/mlx-q8") — behaviorally equivalent, verified below.)

llama.cpp / Ollama: download gguf/lil-beastie-4b-bf16.gguf and ollama create lil-beastie -f Modelfile with FROM ./lil-beastie-4b-bf16.gguf.

Prompting — the IDK line is load-bearing

The abstention behavior is activated by offering the exit. Append this to your question (verbatim from the eval harness):

Reason briefly, then give the final option letter in \boxed{}. If you genuinely cannot determine it, answer \boxed{IDK} instead of guessing.

(For free-form problems swap "the final option letter" for "your final answer".) Use exactly one instruction per prompt, matching the deck's answer type. Do not stack the letter instruction onto a free-form problem: the model reads an unsatisfiable format as "cannot determine" and abstains on answers it has already found (that mistake sat under an earlier version of this card's AIME line — see the eval bundle's results/aime_ab_brief_q8/FINDING.md).

Omit the IDK offer and calibration silently degrades. Sample at temperature 1.0, top_p 0.95, top_k 20 for vote-spreads; the model's confidence signal = how many of G samples agree. All numbers on this card are at temperature 1.0; the repo's generation_config.json defaults to Qwen's 0.6. "Reason briefly" is the mode benchmarked below. Swapping it for "Reason it through" is a confidence dial, not a capability dial: on the same 58 never-trained AIME problems it produced essentially the same length (+0.6% tokens), 0.66 fewer correct rollouts per 10 (paired t = −3.54), +6.6 points IDK, and 0.3% vs 0.9% wrong — fewer commits, slightly safer ones. Use it as a caution rung, not as a solve-rate rescue.

Headline numbers (measured, never-trained-on questions)

7,102 fresh MMLU-Pro questions with zero training overlap, × 10 rollouts each:

metric	lil'beastie (G=10 vote)	base Qwen3-4B*	Qwen3-32B*
ECE ↓	0.082 [95% CI 0.074–0.092]	0.328	0.218
AUROC ↑	0.780	0.743	0.817
accuracy	75.7%	(74.0 published, forced-choice)	—

*Published single-pass calibration, arXiv 2505.23854. Stated caveat: these results use G=10 self-consistency (multi-sample); part of the gap is attributable to method. The controlled ablation below isolates the fine-tune's own contribution.

The consensus operating curve

One fixed rule ("answer iff ≥k of 10 rollouts agree"), no domain knowledge, no tuning:

agree ≥k	answered	precision	confident-wrong
1 (plurality)	94%	76%	22.8%
5	74%	84%	11.9%
7	62%	88%	7.1%
10 (unanimous)	33%	94%	2.0%

The rule is difficulty-adaptive with no band labels: on easy questions rollouts converge (high coverage); on hard ones they scatter or abstain (automatic escalation). Everything below threshold is an explicit, routable "escalate" outcome — not a silent failure.

The ablation: voting cannot create abstention

Identical deck, prompt (including the IDK instruction), sampler, G=10, engine, and GPU — only the weights differ:

	base	lil'beastie
idk rollouts	1%	21%
runaway/truncated rollouts	22%	6%
total tokens (same workload)	24.8M	14.8M
one pass: right when it answers	75.5%	75.6%
one pass: confidently wrong	19.0%	17.8%
one pass: right overall	58.3%	55.1%
10-sample vote: ECE	0.162	0.113
10-sample vote: right when it commits (gate: ≥5 agree, <2 IDK)	77.0%	81.9%
10-sample vote: confident-wrong (same gate, share of all questions)	17.7%	9.9%

Given ten samples and an explicit IDK option, the base model abstains on 1% of rollouts. The abstention channel exists only because of the fine-tune; self-consistency sharpens calibration but cannot create the behavior.

One answer on its own is no more accurate than the base's. The difference is how it fails: an explicit IDK instead of running out of room. Confidence comes from agreement across samples.

(ECE values in this table are from the A100/vLLM engine pairing on the 645-question bench deck — the controlled comparison. The same model measures 0.093 on this deck on Mac/MLX and 0.082 on the 7,102-question fresh set; engine and deck are stated with every number in the eval bundle.)

Calibration tracks self-verifiability (free-form results)

• AIME (free-form integer answers, no options), 58 never-trained problems × G=10, clean prompt, Mac MLX q8, cap 10k: commit precision 97.8% (220/225) under "Reason briefly", 98.9% (182/184) under "Reason it through". Coverage 39% / 32%, abstention 7.4% / 14.0%, wrong 0.9% / 0.3%; the remaining ~54% of rollouts hit the token cap (budget, not confidence). Earlier versions of this line used 96 problems: 38 of them (all 32 of a curated set, plus 6 others) turned out to be in the training lineage and are excluded here; on the full 96 the precision was 96.9% / 98.3%. Before that, an A100 run stacked the option-letter instruction onto the free-form deck and the model vetoed half of its own correct answers (precision the same, coverage halved). Both findings ship in the eval bundle.
• SimpleQA (free-form obscure trivia): 86.6% abstention — question-level modal verdict over the 1,792 questions run (the first 1,792 of a 4,326-question deck; per-rollout abstention is 76.7%; semantically graded, grading files included). Committed answers are only ~15% precise — near-miss entities and small date errors rather than fabricated strings.
• The pattern: on tasks where the output can be checked by the model's own reasoning (computation), what it commits is right ~98% of the time; on pure-recall tasks it abstains heavily, and its rare commitments should be treated as retrieval leads, not answers.

Coding: no capability cost

HumanEval, 164 tasks × 5 samples at temperature 1.0, the identical harness for both models (Mac, MLX, bf16):

	base Qwen3-4B-Thinking-2507	lil'beastie
pass@1	92.8%	92.9%
pass@5	97.6%	96.3%
generation time	17.2 h	13.4 h
abstained	0 / 820	0 / 820
tool-use probes (called and correct)	49/50	49/50

Paired per task, lil'beastie minus base: t = 0.12 (better on 18, worse on 14, tied on 132). The pass@5 gap is three tasks the base solves at least once and lil'beastie never does (HumanEval/91, 115, 132), against one the other way (HumanEval/32). The abstention fine-tune left coding ability unchanged, and it never abstains on code.

Reliability by domain (fresh MMLU-Pro, gate ≥5)

Strong: math (97% gate precision), physics/chemistry (93–94%), engineering (89%). Abstains where weak: history/philosophy (42–49% of rollouts abstain; ~72% gate precision on what clears). Known blind spots: law (46% gate precision) and health (74%) — confidence miscalibrates in these domains; route them past the model. Every number here is reproduced by the bundle's score_g10.py from the raw rows; full per-category table included.

Formats & hardware

format	memory (single answer)	verified
bf16 (as benched)	~10 GB	all results above
q8	~6 GB	behaviorally equivalent: rates identical within sampling noise, 89% same-answer id-matched (100% on easy), ~1.3× faster batched

Q4 is not supported. Quantizing that deep degrades the calibration behavior in a model this small, and abstention is the first thing to go: a Q4's IDKs and answers don't match this card. Use q8 — it works well (verified above) and is only ~6 GB. Full G=10 vote mode requires ~16 GB (bf16) or ~14 GB (q8). Single-answer mode runs on an 8 GB Mac or any 12 GB GPU.

Third-party GGUF quants exist (mradermacher/lil-beastie-4b-GGUF, mradermacher/lil-beastie-4b-i1-GGUF). Use their Q8_0, the closest match to what was verified here. Q4 and below are not supported; Q5/Q6 are untested — check IDK rate and accuracy against the numbers above on the 645-question deck before trusting one.

Verify this card for ~$2

Everything is in the companion dataset repo: LuzCommand/lil-beastie-4b-eval — reproduce the numbers from the shipped raw rows with no GPU at all (python harness/score_g10.py --run results/ck23_g10_fresh2 --exclude 4849,10565), or re-run the core 645-question evaluation (about 66 minutes on one A100-80GB, ~$2):

pip install vllm==0.27.1
python harness/runpod_g10_eval.py --model LuzCommand/lil-beastie-4b --set decks/_full646_clean.jsonl --out my_run

The eval bundle ships: the harness (verbatim prompts including the load-bearing IDK instruction), decks, trained-id manifests + the contamination audit (zero id or exact-text overlap between training and the MMLU-Pro evaluation decks, two passage-siblings excluded conservatively; for AIME, the 38 training-lineage problems are listed and excluded), raw per-rollout results for every run, full reasoning traces, and the scoring code that reproduces every number above from the raw rows (score_g10.py for multiple choice, score_freeform.py for AIME, score_humaneval.py for coding).

Trace disclosure: full per-rollout reasoning traces are included for the Mac 645-question run, the 7,102-question fresh set, AIME, SimpleQA, and both HumanEval runs. Three early runs (the first fresh sample, the A100 replication, and the base-model ablation) have per-rollout outcome rows only — an early harness version discarded completion text; trace capture has since been made unconditional in the published harness.

Known limitations

• This is a specialist model: calibrated short-form QA and computation. Open-ended generation beyond ~1,500 tokens drifts toward the training distribution (math reasoning). Long-form output works when generated in ≤900-token passes.
• ~2% of questions defeat any consensus gate: all ten rollouts converge on the same wrong answer (shared-misconception errors). Voting cannot catch consistent errors; those require routing or retrieval.
• Free-form trivia commitments are retrieval leads, not answers (see SimpleQA).
• Training and selection decks are disclosed by id; the fresh-set numbers above contain none of them.

Provenance

Base: Qwen/Qwen3-4B-Thinking-2507 (Apache 2.0). Trained in stages with LoRA: a light supervised pass and short GRPO runs on math reasoning, a second light supervised pass to seed the \boxed{IDK} format, then several rounds of GRPO with abstention-aware rewards. Training questions come from MMLU-Pro (339 ids, listed in the eval bundle's contamination manifest) plus grade-school and competition math (GSM8K, MATH, AIME). The reward design, data selection and hyperparameters are not published. Community model — not affiliated with Alibaba/Qwen. Evaluations: one 33-hour Mac Studio run plus two rented A100 days totaling ~$40 of cloud compute. Every number on this card has a raw-data file behind it.

A model that knows what it doesn't know. That is the trick.