qev-0.8b

Qev = Qwen + Jev: a Qwen3.5-0.8B-based Jev-style decision model.

LoRA-tuned and merged. It reads a state and a set of typed questions (choice, noul yes/no, score) and returns calibrated probabilities for every option in one forward pass, without generating text.

Results (held-out test sets)

Test	Accuracy	ECE
banking77	81.2%	0.0419
clinc150	80.8%	0.019
synthetic	74.1%	0.1566
typed_decisions	78.1%	0.1293
holdout (10% of training mix)	81.8%	0.0556

Pooled typed-decisions + BANKING77 + CLINC150 accuracy: 79.1%.

Training

• Data: ~40.8k rows: 36k from Unsloth's public decision mixture (banking77, clinc150, mnli, snli, wanli, boolq, ag_news, sst5, mmlu aux-train, commonsense_qa, arc, prompt_injections; xlam skipped, gated), LocalLLaMA/typed-decisions train (1,030 rows after removing rows whose 13-grams mostly appear in the test split; 4 augmented copies), and a small hand-written synthetic set (131 rows, 5 copies). Public rows decontaminated against all test sets.
• LoRA r=64, lr 0.0001, head lr 0.0003, effective batch 16, 1 epoch (2535 steps, 261.3 min on a Kaggle T4), max length 4096, float32 training. Temperatures fitted on the holdout.
• Caveat: the typed-decisions decontamination is looser than Unsloth's reference script (which drops any 13-gram overlap).

Use

from unsloth import FastDecisionModel
model, tok = FastDecisionModel.from_pretrained("Manitchahar/qev-0.8b", max_seq_length = 4096)
FastDecisionModel.for_inference(model)
FastDecisionModel.predict(model, tok, "I was charged twice, refund me", {
    "team": {"type": "choice", "instructions": "Which team?", "criteria": {"billing": "payments", "tech": "bugs"}},
    "refund": {"type": "noul", "instructions": "Asks for a refund?"}})

Benchmarks (re-measured from the public weights)

JevBench public split (231 decisions; the official board also uses held-out items, so these are not leaderboard numbers):

Tier	Accuracy	Brier	ECE
original (72)	83.3%	0.265	0.094
easy (48)	100.0%	0.014	0.047
hard (111)	43.2%	0.680	0.128
all public	67.5%

CPU latency (Kaggle CPU, 4 threads, fp32, batch 1, JevBench original): p50 2225 ms, p95 2574 ms.

Full comparison against other open sub-1B decision models: https://huggingface.co/datasets/Manitchahar/decision-bakeoff