bdauzats/minicpm5-2b-decision-mlx-4bit

minicpm5-2b-decision (MLX, 4-bit)

A decision model: one text (the state) and typed questions in, a probability per option out, in one forward pass. It never generates text. MiniCPM5-2B with a merged LoRA (r=16) and a pointer head, trained on multi-task decision data, then converted to MLX and quantized to 4 bits (group size 64).

Made for jul — local typed decisions with the interface of TypeSafe's (Jev) Python SDK, on Apple Silicon through MLX or anywhere else through PyTorch.

It follows the architecture and training code of Kev (Jared Palmer, Apache-2.0), which reconstructs a Jev-style model, applied here to a different base model and a different data mix.

Use it with jul

jul models add minicpm5-2b-decision --repo bdauzats/minicpm5-2b-decision-mlx-4bit
jul ask choice "Which team should handle this ticket?" -o billing -o shipping -o access \
    --state "I was charged twice for order 4411" --model minicpm5-2b-decision

from jul import TypeSafeClient, Choice, Noul, Score

client = TypeSafeClient(model="minicpm5-2b-decision")
answer = client.system_one(
    state={"subject": "Charged twice", "body": "I see two charges for order #4411."},
    questions={"team": Choice(instructions="Which team should handle this ticket?",
                              criteria={"billing": "Payments and refunds",
                                        "shipping": "Delivery problems",
                                        "access": "Login and account access"}),
               "angry": Noul(instructions="Is the customer angry?"),
               "priority": Score(instructions="How urgent is this ticket?",
                                 criteria=["low", "normal", "high"])})

jul needs no code specific to this model: the input format, the readout, the pointer head and its temperature are described in decision.json, next to the weights. Kev's own server also serves these weights, from the unquantized checkpoint.

Latency

Measured on an M4 Pro (24 GB) on mains power, through jul, MLX 4-bit. A call encodes the state once; every question then continues from that cached prefix, so several questions about one text cost little more than the first.

Request	p50
3 short options (FinancialPhraseBank)	65 ms
10 options, long texts (Yahoo Topics)	140 ms
32 options (Empathetic)	208 ms
59 long options (Massive)	613 ms
4 questions (choice, choice, noul, score) about one ticket	180 ms

Cost grows with the number and the length of the options, which are read on every request. For comparison, Jev — hosted, over the network — publishes 246 ms on its own benchmark.

Files

File	What it is
model.safetensors, config.json, tokenizer.json	MiniCPM5-2B with the LoRA merged in, quantized to 4 bits
pointer_head.npz	the pointer head (two 2048 → 256 linear layers)
decision.json	delimiter tokens, layout, readout, head file, temperature (1.954), training limits

Results

Development sets only; the Jev benchmark has not been read for this checkpoint.

Development set	Kev-0.8B	this model	Kev-4B	Jev
transfer-v4 (sources never trained on, 656 questions)	0.652	0.721	0.797	0.857
decision-v7 (held-out examples of the training sources, 1264)	0.825	0.846	0.872	—

Measured in PyTorch, fp32, before quantization. transfer-v4 includes Emotion, which is in this model's training mix (it is not for Kev): without it, 0.710 against 0.668 for Kev-0.8B.

On four zero-shot classification sets (BTZSC: Yahoo Topics, Empathetic, Massive, FinancialPhraseBank, 200 examples each), in MLX 4-bit through jul: mean accuracy 0.606, against 0.542 for the same base model read by jul's vector method. See Latency.

On the Jev benchmark

The 300 published rows (AG News, Banking77, Emotion), through jul, zero-shot: the text, the question and the option list, no example.

Model	AG News	Banking77	Emotion	Mean	Mean ECE	p50
this model	0.91	0.79	0.69	0.796	0.133	217 ms
Jev (published)	0.91	0.87	0.48	0.753	0.156	246 ms
GLiNER2.5 (published)	0.70	0.61	0.44	0.583	0.101	128 ms

Jev is a trained decision model too, which is what makes this a like-for-like row. One asymmetry is known: this model trained on the three datasets' training splits (the benchmark rows are test rows), while Jev's training data is not published. Emotion is where it wins — one of the tasks it learned — and Banking77 stays 8 points behind Jev even though it learned that one as well: 72 fine-grained intents are the hard part. For tasks it has never seen, read the transfer-v4 line above.

Calibration

A temperature of 1.954, fitted once on the in-distribution development set, is applied at load time. It never changes an answer. On transfer-v4 (sources never trained on) it takes calibration error from 0.182 to 0.092, and confident errors — wrong with probability ≥ 0.9 — from 12.5% to 3.2%, about Jev's 3.7%. Brier 0.436 → 0.377. Set KEV_TEMPERATURE=1.0, or pass your own, for the raw logits.

On the four zero-shot sets above the calibration error is 0.127 (FinancialPhraseBank), 0.107 (Yahoo Topics), 0.134 (Empathetic) and 0.241 (Massive, 59 options) — mean 0.152, against 0.109 for the same base model read by jul's vector method, whose temperature was fitted on those very sets. Trust the answer before the number: quantizing to 4 bits keeps the answer on almost every example but moves probabilities by up to ~0.3, which can flip a borderline decision. The unquantized weights match the PyTorch reference within 0.03. If the probabilities themselves matter to you, fit a calibration on your own data (jul does this from a Context).

Training data

Kev's decision-v7 suite (12,576 records: 1,000 per public source, plus generated policy examples and rule structures) and 2,800 of our own: dair-ai/emotion (1,000) and six BTZSC sets (300 each: CAP SOTU, Manifesto, app reviews, Bias Frames intent, Wikipedia toxicity, TrueTeacher).

The public sources include AG News, Banking77, DBpedia14, TREC, SST-5, Yelp, Amazon reviews, IMDB, BoolQ and MNLI, each from its training split, with its own license. Results on those tasks are therefore not zero-shot, which matters if you compare this model with one that has never seen them.

Two epochs, LoRA r=16 α=32 on q/k/v/o_proj and gate/up/down_proj, pointer head from scratch, cross-entropy, lr 5e-5 (one-cycle), batch 8, bf16 autocast with fp32 master weights; about 90 minutes on one L40S.

Limitations

• English only, like the data it was trained on.
• A state longer than 384 tokens, or a question longer than 1024, goes beyond what it saw in training.
• Weak on knowledge and arithmetic: on transfer-v4, MMLU 0.45 and date arithmetic 0.35 (Jev: 0.90 and 0.93). It sorts and decides; it does not reason over facts it was never given.
• No evaluation outside the sets above, and none of them is a benchmark of your own data. Measure it.