Kev-0.8B for MLX, 4-bit, group 32, with the Gated DeltaNet projections and the MLP gate/up projections kept at 8-bit

What it is. A decision model: it reads a document (the state) and answers typed questions about it — choice over named options, noul (the probability that a statement is true), score over ordered levels — with calibrated probabilities, in one forward pass, without generating text. Use it to classify, route, triage, score or check documents and threshold the probabilities (send the uncertain cases to a human).

Why this checkpoint. The published jaredpalmer/kev-0.8b (Jared Palmer, Apache-2.0) is a rank-16 LoRA plus a pointer head on a Qwen3.5 base: running it means downloading the base model, folding the adapter in at every load and keeping bf16 weights in memory. This repository is the merged result quantized for mlx-lm at 4-bit, group 32, with the Gated DeltaNet projections and the MLP gate/up projections kept at 8-bit (0.66 GB): one download, loads in a few seconds, same answers within the measured tolerance below. The pointer head (head.safetensors) stays in float32; decision.json records the recipe and the source checkpoint.

How accuracy is protected. A decision model's value is its calibration, and a plain 4-bit quantization damages it: every linear layer at 4 bits (group 64) costs Kev-4B a point of accuracy and moves a tenth of its answers by more than 0.10 probability. The recipes here were chosen by measuring every candidate against the bf16 weights on the model author's public development set (1,328 records, 1,592 questions: argmax agreement, mean and maximum probability shift, accuracy, Brier, ECE), per module group and group size. Quantization sensitivity sits in the Gated DeltaNet projections and the group size; keeping those projections at 8 bits with 32-element groups keeps the accuracy, and the 8-bit checkpoint is indistinguishable from bf16. The companion checkpoints are AbstractFramework/kev-0.8b-mlx-8bit and AbstractFramework/kev-0.8b-mlx-4bit; prefer 8-bit when you threshold probabilities, 4-bit when size matters.

Measured

public-pool-v6 development set (1,328 records): accuracy 0.754 (bf16 0.763), ECE 0.024; argmax agreement with bf16 0.960, mean |Δp| 0.028, 4.2 % of questions moved by more than 0.10; 24 ms per record on an M5 Max. The 4B recipe alone (DeltaNet projections at 8-bit) gave 0.954 agreement on this smaller model; keeping the MLP gate/up projections at 8-bit too recovers most of it for 15 % more size. The 8-bit checkpoint (0.8 GB) is indistinguishable from bf16.

Use with AbstractCore

pip install "abstractcore[apple]"
abstractcore models download mlx AbstractFramework/kev-0.8b-mlx-4bit

from abstractcore import create_llm

llm = create_llm("mlx", model="AbstractFramework/kev-0.8b-mlx-4bit")
result = llm.generate(
    "Shoes arrived two weeks late and in the wrong size. Also I see two charges on my card.",
    output="decision",
    questions={
        "department": {"type": "choice", "instructions": "Which team should handle this?",
                       "criteria": {"returns": "Exchanges, refunds, wrong or damaged items",
                                    "shipping": "Delivery status, delays, lost packages",
                                    "billing": "Charges, invoices, payment problems"}},
        "escalate": "Does this need urgent human attention?",   # shorthand for a noul question
        "frustration": {"type": "score", "instructions": "How frustrated is the customer?",
                        "criteria": ["Calm", "Frustrated", "Very angry"]},
    },
)
result.decisions["department"].choice          # "returns"
result.decisions["department"].probabilities   # {"returns": 0.50, "shipping": 0.37, "billing": 0.13}
result.decisions["escalate"].noul              # 0.73
result.decisions["frustration"].score          # expected level, 0..2

Over HTTP (abstractcore serve), the same request is POST /v1/decisions with {"model": "mlx/AbstractFramework/kev-0.8b-mlx-4bit", "state": ..., "questions": ...}.

The state cache (KV cache) and what to expect

The document is read once per request and reused for every question in it; AbstractCore also keeps the computed state of the last four documents, so a second request about the same document only pays for its questions. Measured on an M5 Max with this family's 8-bit Kev-4B:

Request	First time	Same document again
a 100-token support ticket, three questions	~95 ms	~45 ms
a 6,300-token contract, three questions	~1.4 s	~0.1 s

Reading the document is the cost (about 3,000 tokens per second for Kev-4B, four to five times faster for Kev-0.8B); the questions themselves cost tens of milliseconds. Ask everything you need about a document in one call, or in consecutive calls while it is still cached. To disable the cache (every request re-reads its document): create_llm("mlx", model=..., decision_prefix_cache_size=0); to keep more documents: decision_prefix_cache_size=16. The cache lives in memory for the life of the provider.

Docs

Decision models in AbstractCore: https://github.com/lpalbou/AbstractCore/blob/main/docs/decision-models.md (engines, every supported model, conversion recipes, the full measurements)
AbstractFramework: https://github.com/lpalbou/abstractframework

Attribution

Weights derived from jaredpalmer/kev-0.8b (Jared Palmer) on Qwen/Qwen3.5 base models (Alibaba), both Apache-2.0. Converted and measured with python -m abstractcore.decision.convert.