JevK5-9B v0.3 — open-weight Jev alternative, 9B

JevK5-9B is the larger sibling of JevK5: the same v0.3 training data and recipe on Qwen3.5-9B. It is an independent, Apache-2.0 open-source alternative to TypeSafe's Jev for typed decisions. It reads a state and a yes/no (noul), choice, or score question and returns a probability for every option in one forward pass, with zero generated tokens. It runs on the JevK5 runtime, which also serves a TypeSafe-style /v1/systemone endpoint. This is not Jev's model or architecture and is not affiliated with TypeSafe AI.

Use the JevK5 runtime shown below to read option probabilities. Generic text-generation examples on the Hub call generate() and do not perform JevK5's decision readout.

• Base: Qwen3.5-9B, with a LoRA (rank 16, attention projections) merged into the weights
• Readout: SemIf's protocol (TheoLeeCJ/SemIf, MIT): a softmax over the answer letters' next-token logits, divided by one calibration temperature. Questions with more than 16 options are read in several passes and combined with a second temperature. Both are in jevk5_config.json: temperature 1.049 and knockout_temperature 1.2
• Runtime: github.com/allebee/jevk5, 0.3.0 or later, with CUDA graphs: about 31 ms per short decision on an H100 (the 4B: 13 ms), measured while another job shared the GPU. Needs a CUDA GPU with about 19 GB for bf16
• License: Apache-2.0

When to use the 9B

Choose the 4B by default. The 9B is ahead of it on our held-out checks, but behind it on JevBench's public items, and it is larger and slower.

• Held-out checks, 9B against 4B: index proxy 0.762 against 0.731, teacher questions 0.851 against 0.834, hand-written hard set 0.781 against 0.766 (one item). It also leads on bev-decision's test sample (0.700 against 0.663) and by 8 points on questions with more than 16 options.
• JevBench's public items, 9B against 4B: hard tier 0.730 against 0.784 (4 items fixed, 10 broken), and much worse hard-tier calibration (ECE 0.126 against 0.054). The standard tier is the same (0.944).
• It needs about 19 GB in bf16 and is about 2-3x slower.

Other sizes and formats

• JevK5 (4B, v0.3): the same data on Qwen3.5-4B, ~9 GB in bf16.
• JevK5-GGUF: GGUF builds for llama.cpp (NVIDIA, AMD, Intel and Apple GPUs, or a CPU). The 9B Q8_0 gives the same answer as these weights on 229 of 231 public JevBench items, and the Q5_K_M on 225. A 9B Q4_K_M agreed on 218 and is not published.

Results

Held-out checks (used to choose between models)

The dev set is held-out rows only: teacher questions from three domains that training never saw (residential leases, public-sector permits, manufacturing QC), a hand-written hard set, and a hashed 5% of every public train split. All columns are read the same way, through the runtime.

	JevK5 v0.2 (4B)	JevK5 v0.3 (4B)	JevK5-9B v0.3
Index proxy (16 sources, see below)	0.620	0.731	0.762
Held-out teacher questions (362), accuracy	0.801	0.834	0.851
Hand-written hard set (64), accuracy	0.766	0.766	0.781
ECE on the teacher questions, calibrated	0.034	0.035	0.032

The index proxy is our own estimate, not an index score. It is the chance-corrected skill, averaged over the 16 dev sources that are held-out train-split rows of Jev Decision Index benchmarks. Each source has 40 rows, so each one alone is noisy (about ±0.15). Against the 4B, the 9B gains most on HoVer (0.30 → 0.60), WinoGrande (0.70 → 0.85) and GSM8K (0.61 → 0.70). Its clearest loss is NLI4CT (0.75 → 0.55).

bev-decision-150K (another group's decision mix, evaluation only)

On 2,500 hashed rows of the test split of avbiswas/bev-decision-150K (4,723 questions, every question type): the 9B scores 0.700 (the v0.3 4B 0.663, v0.2 0.665), with ECE 0.038. By type: choice 0.732, yes/no 0.777, score 0.483. Leaving out the 34 questions whose document also appears in our training data changes accuracy by 0.002.

JevBench v1.2 public items (report only)

231 public items through JevBench's own runner (jevk5_direct adapter): 231/231 valid, 0 failures. The untrained row is Qwen3.5-9B with the same prompt, without the LoRA or the temperature.

Split	n	Untrained Qwen3.5-9B	JevK5 v0.3 (4B)	JevK5-9B v0.3	9B ECE (4B)
easy	48	1.000	1.000	1.000	0.009 (0.018)
original (standard)	72	0.958	0.944	0.944	0.041 (0.057)
hard (public half)	111	0.676	0.784	0.730	0.126 (0.054)

• Against the untrained Qwen3.5-9B, JevK5-9B fixes 15 hard items and breaks 9 (McNemar p = 0.31). Its hard-tier ECE is 0.126 (untrained 0.146). Distance to the exact gold distributions on the 10 probability items: 0.284 (untrained 0.282).
• Against the v0.3 4B it is worse on these items: 4 hard items fixed and 10 broken (p = 0.18), and its hard-tier calibration is much worse (ECE 0.126 against 0.054; distance to the gold distributions 0.284 against 0.164).
• By family on the hard tier, 4B → 9B: judging answers 0.65 → 0.71, but probability 1.00 → 0.70, long policies 0.68 → 0.58, ambiguous 0.86 → 0.71 and trade-offs 1.00 → 0.83.
• Latency (H100, in-process, batch 1, CUDA graphs), measured while another job shared the GPU: p50 32 ms, p95 36 ms on easy and standard items; hard items p50 74 ms, p95 373 ms.

More than 16 options

These use the runtime's knockout readout (groups of up to 16, then a final). The runs are 500 train-split items per dataset in the Decision Index's request shape, with every option offered. None of these items is in the training or dev data.

Train split	Options	JevK5 v0.3 (4B) accuracy / ECE	JevK5-9B accuracy / ECE	9B macro-F1
MASSIVE en-US (fitting set)	60	0.738 / 0.045	0.818 / 0.036	0.815
BANKING77	77	0.652 / 0.044	0.734 / 0.044	0.722
CLINC150 with out-of-scope	151	0.700 / 0.056	0.780 / 0.061	0.815

• The 9B is 8 points more accurate than the 4B on each set. On CLINC150, out-of-scope recall is 0.46 (the 4B's 0.27), with precision 0.76.
• Second temperature: the 9B carries its own, 1.2, fitted by NLL on the MASSIVE items only (the two halves give 1.23 and 1.15). BANKING77 and CLINC150 are reported, not fitted. With v0.2's 0.77 the 9B would be overconfident (ECE 0.070, 0.129, 0.088).

Jev Decision Index and JevBench

JevK5-9B has not been run by the Jev Decision Index or submitted to JevBench yet. For reference, the index reran JevK5 0.2.2 (v0.2 4B weights, with the runtime that answers any number of options) on its previously refused rows (discussion #13). It scored 36.31, 15th of 49, and its calibration was 4th best (ECE 0.031). JevBench v1.4 ranked JevK5 v0.2 #2 of 76 systems.

How it was trained

Exactly as JevK5 v0.3 (4B), on the same 47,460 rows; see that card for the full data table with licenses.

• Teacher questions (17,408): 3,270 from Qwen3.6-27B (Apache-2.0, self-hosted, thinking on) and 14,138 from GPT-6 Luna (OpenAI, through OpenAI's API). Each question was answered twice, independently, by its teacher and kept only when both answers matched the intended one. Luna's outputs were generated under OpenAI's terms, which govern their use; review them for your use case.
• Public replay (30,052 items): train splits of 26 public datasets, with no test or validation split of any dataset, and no split of MMLU or MMLU-Pro. The sources are GSM8K, WinoGrande, HellaSwag, When2Call, HoVer, iSarcasmEval, ARC, CommonsenseQA, CLINC150, SGD, Amazon ESCI, RAGTruth, ToolACE, WANLI, AQuA-RAT, OpenBookQA, Cosmos QA, SWAG, BoolQ, MultiNLI, BANKING77, MASSIVE, the New Yorker caption contest, QASC, RuleTaker and Glaive function calling v2. Each dataset's license is in the JevK5 card's table.
• Training: cross-entropy on the option-letter logits, SemIf's prompt format, 1 epoch, learning rate 3e-5, inputs up to 2,048 tokens. One temperature is fitted on the held-out teacher questions: ECE 0.034 → 0.032. The second temperature (1.2) is fitted by NLL on the 500 MASSIVE items above.
• Two development 9Bs are not released because of their data's terms:
  • one that also trained on ANLI (CC BY-NC 4.0) and NLI4CT (no stated license);
  • one that also trained on 2,373 rows of MMLU's auxiliary_train, most of them RACE reading passages (RACE is for non-commercial research only).
  • The second one scored higher on our held-out checks than this model. Index proxy: 0.788 against 0.762. Teacher questions: 0.870 against 0.851. Hand-written set: 0.844 against 0.781 (54 against 50 of 64). We chose the openly licensed data.

Data rules.

• No JevBench item, public or held out, and no output of Jev was used for training, tuning or selection. JevBench's public items were only used to report the numbers above.
• Every public and Luna row was checked against the test and validation text of 35 Decision Index benchmarks, and against JevBench's public items. A row was dropped for an exact match or for any shared 8-word sequence. GPQA and HLE are gated and were not checked; no source is built from them.

Declared overlap with the Jev Decision Index. These are train splits of index benchmarks, deduplicated against their test and validation items: ARC, OpenBookQA, CommonsenseQA, GSM8K, WinoGrande, HellaSwag, BANKING77, CLINC150, SGD, Amazon ESCI, When2Call, iSarcasmEval, RAGTruth, HoVer and the New Yorker caption contest. Separately, 53 of the Qwen-written training questions share at least one 8-word sequence with ContractNLI (41) or SGD (12) test or dev text. They were reported by the scan and kept. JevK5-9B was not trained on any split of MMLU, MMLU-Pro, ANLI or NLI4CT.

Known weak spots

• Worse than the 4B on JevBench's public items: hard tier 0.730 against 0.784, and hard-tier ECE 0.126 against 0.054. It is also below JevK5 v0.2 there (0.739).
• Below its unreleased RACE variant on every held-out check (see "How it was trained"). The hand-written set is the biggest gap: 50 against 54 of 64.
• On the hard tier, probability questions (0.70) and long policies (0.58) are weaker than the 4B. Dates and numbers stay weak (0.47).
• On bev-decision, score questions are the weakest type (0.483).
• About 2-3x slower than the 4B.
• Accuracy drops sharply on JevBench's fresh sealed decisions (measured for JevK5 v0.2). Real-world workflow performance against Jev has not been measured.
• English only. Needs a CUDA GPU with ~19 GB for bf16. Inputs over 16,384 tokens are refused, not cut.

Use

from jevk5 import JevK5

model = JevK5("alibiserikbay/JevK5-9B")
model.decide(
    "I was billed twice for order #4411. Please refund the duplicate charge today.",
    {"type": "choice", "instructions": "Which team should handle this?",
     "criteria": {"billing": "Payments and refunds", "tech": "Bugs", "sales": "New purchases"}},
)
# {'type': 'choice', 'choice': 'billing', 'confidence': 0.997, 'probabilities': {...}, ...}

Or as a server that answers TypeSafe-style /v1/systemone requests: jevk5-serve --model alibiserikbay/JevK5-9B --port 8090. Both read temperature and knockout_temperature from this repo's jevk5_config.json.

Credits

Qwen3.5-9B and Qwen3.6-27B by the Qwen team (Apache-2.0). GPT-6 Luna by OpenAI. The one-pass readout and prompt come from SemIf by TheoLeeCJ (MIT). The public datasets listed above belong to their authors, under their licenses (table on the JevK5 card). Evaluated with JevBench (github.com/fstandhartinger/jevbench, MIT) and bev-decision-150K. Not affiliated with TypeSafe AI or Jev.