mev-1b

Mev = MiniCPM + Jev: a MiniCPM5-1B-SFT-based Jev-style decision model.

A Jev/Clef-style decision model: openbmb/MiniCPM5-1B-SFT plus a joint schema head (Unsloth FastDecisionModel, Clef head), LoRA-tuned and merged. It reads a state and a set of typed questions (choice, noul yes/no, score) and returns calibrated probabilities for every option in one forward pass, without generating text.

Results (held-out test sets)

Test	Accuracy	ECE
banking77	77.0%	0.0865
clinc150	69.2%	0.1214
synthetic	56.9%	0.0912
typed_decisions	73.1%	0.0952
holdout (10% of training mix)	76.8%	0.0654

Pooled typed-decisions + BANKING77 + CLINC150 accuracy: 73.1%.

Training

• Data: ~40.8k rows: 36k from Unsloth's public decision mixture (banking77, clinc150, mnli, snli, wanli, boolq, ag_news, sst5, mmlu aux-train, commonsense_qa, arc, prompt_injections; xlam skipped, gated), LocalLLaMA/typed-decisions train (1,030 rows after removing rows whose 13-grams mostly appear in the test split; 4 augmented copies), and a small hand-written synthetic set (131 rows, 5 copies). Public rows decontaminated against all test sets.
• LoRA r=64, lr 0.0001, head lr 0.0003, effective batch 16, 1 epoch (2535 steps, 105.6 min on a Kaggle T4), max length 4096, float16 training. Temperatures fitted on the holdout.
• Caveat: the typed-decisions decontamination is looser than Unsloth's reference script (which drops any 13-gram overlap).

Use

from unsloth import FastDecisionModel
model, tok = FastDecisionModel.from_pretrained("Manitchahar/MiniCPM5-1B-decision", max_seq_length = 4096)
FastDecisionModel.for_inference(model)
FastDecisionModel.predict(model, tok, "I was charged twice, refund me", {
    "team": {"type": "choice", "instructions": "Which team?", "criteria": {"billing": "payments", "tech": "bugs"}},
    "refund": {"type": "noul", "instructions": "Asks for a refund?"}})

Benchmarks (re-measured from the public weights)

JevBench public split (231 decisions; the official board also uses held-out items, so these are not leaderboard numbers):

Tier	Accuracy	Brier	ECE
original (72)	77.8%	0.376	0.130
easy (48)	95.8%	0.081	0.060
hard (111)	36.0%	0.776	0.220
all public	61.5%

CPU latency (Kaggle CPU, 4 threads, fp32, batch 1, JevBench original): p50 2523 ms, p95 3010 ms.

Full comparison against other open sub-1B decision models: https://huggingface.co/datasets/Manitchahar/decision-bakeoff