alibiserikbay/JevK5-GGUF

JevK5 GGUF: typed decisions on almost any GPU, or on a CPU

GGUF builds of JevK5 (4B) and JevK5-2B for llama.cpp, which runs on NVIDIA, AMD, Intel and Apple GPUs and on plain CPUs.

JevK5 is an Apache-2.0 open alternative to TypeSafe's Jev. It reads a document and a yes/no, choice, or score question and returns a probability for every option in one forward pass, with nothing generated. It is not affiliated with TypeSafe AI.

Files

Every file was checked against its unquantized model on JevBench's 231 public decisions, scored the same way: how often it gives the same answer, and its accuracy per tier.

File	Model	Size	Same answer as bf16	Easy	Standard	Hard
jevk5-4b-v0.2-Q8_0.gguf	JevK5 4B	4.48 GB	228 / 231	1.000	0.958	0.721
jevk5-4b-v0.2-Q4_K_M.gguf	JevK5 4B	2.71 GB	219 / 231	1.000	0.931	0.730
jevk5-2b-v0.2-Q8_0.gguf	JevK5-2B	2.01 GB	226 / 231	1.000	0.792	0.622
bf16 reference	JevK5 4B	8.4 GB		1.000	0.958	0.739
bf16 reference	JevK5-2B	3.8 GB		1.000	0.806	0.604

• Q8_0 is effectively lossless for both sizes: the few answers that change are borderline items, and hard accuracy moves by two items (4B down two, 2B up two). A Q8_0 of the 4B built the same way and run on an M1 Pro with Metal matched 230 / 231.
• Q4_K_M (4B) fits 4 GB GPUs and 8 GB Macs. It changes 12 answers, 10 of them on the hard tier.
• No 2B Q4_K_M. It changed 29 answers and dropped hard accuracy from 0.604 to 0.514, so it is not published.

Which one to pick: a GPU with 6 GB or more, jevk5-4b-v0.2-Q8_0; 4 GB, jevk5-4b-v0.2-Q4_K_M or jevk5-2b-v0.2-Q8_0; no GPU, jevk5-2b-v0.2-Q8_0.

Run it

JevK5's answer is a probability over the options, read from the next-token log-probabilities of the answer letters. That needs llama-server, which exposes them. Start it with one of the files:

llama-server --hf-repo alibiserikbay/JevK5-GGUF --hf-file jevk5-2b-v0.2-Q8_0.gguf -c 8192 -ngl 99

Then ask it typed questions with this standalone client (Python standard library only):

import json, math, urllib.request

URL = "http://127.0.0.1:8080"  # llama-server
T = 1.532                       # calibration temperature: 1.532 for JevK5-4B, 1.42 for JevK5-2B
LETTERS = "ABCDEFGHIJKLMNOP"
SYSTEM = ("Apply the supplied criterion to the supplied evidence. Choose exactly one listed option. "
          "Respond with only its uppercase letter, with no explanation or reasoning.")

def post(path, body):
    req = urllib.request.Request(URL + path, json.dumps(body).encode(), {"Content-Type": "application/json"})
    return json.load(urllib.request.urlopen(req))

def decide(evidence, criterion: str, options: dict) -> dict:
    """options maps an id to its description; returns a calibrated probability per id."""
    ids = list(options)
    user = json.dumps({"evidence": evidence, "criterion": criterion,
                       "options": [{"letter": LETTERS[i], "description": f"{k}: {options[k]}"}
                                   for i, k in enumerate(ids)]}, ensure_ascii=False)
    prompt = (f"<|im_start|>system\n{SYSTEM}<|im_end|>\n<|im_start|>user\n{user}<|im_end|>\n"
              "<|im_start|>assistant\n<think>\n\n</think>\n\n")
    tokens = post("/tokenize", {"content": prompt, "add_special": False, "parse_special": True})["tokens"]
    top = post("/completion", {"prompt": tokens, "n_predict": 1, "n_probs": 40, "temperature": 0,
                               "cache_prompt": False})["completion_probabilities"][0]["top_logprobs"]
    seen = {t["token"]: t["logprob"] for t in top}
    z = [seen.get(LETTERS[i], min(seen.values()) - 2.0) for i in range(len(ids))]
    w = [math.exp((v - max(z)) / T) for v in z]
    return {k: x / sum(w) for k, x in zip(ids, w)}

decide("I was billed twice for order #4411. Please refund the duplicate charge today.",
       "Which team should handle this?",
       {"billing": "Payments and refunds", "tech": "Bugs", "sales": "New purchases"})

• Yes/no questions: pass {"true": ..., "false": ...} in that order, for example {"true": "The proposition is true.", "false": "The proposition is false."}.
• Score questions: pass the levels in order as {"0": ..., "1": ..., ...}.
• The prompt is tokenized with parse_special before it is sent, so the chat markers stay single tokens. This client gives exactly the same probabilities as the reference implementation used for the table above (largest difference 0.0 on yes/no, choice and score items).

We have only tested llama-server. Apps built on llama.cpp, such as Ollama or LM Studio, can load the files, but the calibrated probabilities need an API that returns the answer letters' log-probabilities for a tokenized prompt; we have not checked whether theirs do.

Speed

Measured, one decision at a time, over HTTP:

Hardware	JevK5-2B Q8_0	JevK5 4B Q8_0
CPU only, 48 threads (server)	~0.23–0.27 s	~0.57 s
Apple M1 Pro, Metal	not measured	~0.6 s

Times are for short documents (~170 tokens) and rise with document length: about 0.7–1.0 s (2B) and 1.8 s (4B) on the CPU for the benchmark's hard items. Consumer NVIDIA, AMD and Intel GPUs have not been measured yet. For the lowest latency on an NVIDIA GPU, the transformers runtime at github.com/allebee/jevk5 uses CUDA graphs (about 9-14 ms per decision on an H100).

How the files were made

convert_hf_to_gguf.py from llama.cpp commit 9575389 with --no-mtp (Qwen3.5 configs otherwise gain a speculative block); Q4_K_M with llama-quantize from a bf16 GGUF. No patches. The checks above ran on llama.cpp's CPU backend at the same commit. SHA-256 sums are in SHA256SUMS.

Evaluation and training disclosure

The numbers above are our own runs on JevBench's 231 public items, not official JevBench results. JevK5 v0.2 (4B) is ranked #2 of 76 on JevBench v1.4; JevK5-2B has not been submitted. No JevBench item and no output of Jev was used for training, tuning or selection. The models' cards describe the training data, the known weak spots, and two corrections: echoes of JevBench wording in our hand-written calibration set, and MMLU-Pro test items in the training data (both in the repository's CHANGELOG.md).

Credits

Qwen3.5 by the Qwen team (Apache-2.0). The one-pass readout and prompt come from SemIf by TheoLeeCJ (MIT). llama.cpp by ggml-org (MIT). Evaluated with JevBench (MIT).