autotrust/JEV — an open-weights student of TypeSafe Jev 1.13 (Qwen3.5-9B, calibrated System One decisions)

Two different models, two different organisations — please read the names carefully. TypeSafe Jev 1.13 is the hosted, closed model made by TypeSafe AI; it is the teacher whose public outputs this model was trained on. autotrust/JEV (this repository) is an independent open-weights student built by autotrust from the Apache-2.0 corpus SargeDev/jev-distill-corpus-v3. autotrust/JEV is not affiliated with, endorsed by, or a product of TypeSafe AI. Throughout this card the teacher is always written TypeSafe Jev 1.13 and this model is always written autotrust/JEV.

One forward pass in, a calibrated probability distribution out. autotrust/JEV answers typed questions (noul yes/no · choice over 2–16 options · score on a 0–5 scale) about any text or JSON state. No text generation, no JSON parsing, no prompt engineering — it was distilled from the full output distributions of TypeSafe Jev 1.13 onto a pretrained Qwen3.5-9B backbone.

Headline results

Held-out test set: 29,955 questions, 53 domains, all three question types. Numbers are for autotrust/JEV.

What is measured	autotrust/JEV	How to read it
How closely it reproduces the teacher's probabilities — mean KL divergence, lower is better, 0 = identical	0.021	The student's probability distributions are nearly indistinguishable from TypeSafe Jev 1.13's
Yes/no questions (noul) — AUROC, higher is better	0.996	Ranks true vs. false almost perfectly
Multiple-choice questions (choice) — picks the same option as the teacher	89.8 %	Over all questions, including those where the teacher itself is nearly tied
Multiple-choice questions where the teacher has a clear favourite (its top two options differ by ≥ 0.1)	95.4 %	On the remaining, near-tie questions any faithful copy would agree only about half the time
Rating questions (score, 0–5 scale) — mean error of the expected rating, lower is better	0.10	About one tenth of a rating step
Calibration — expected calibration error, lower is better	0.0007	A stated 80 % is an 80 %; no post-hoc temperature correction was needed (fitted temperatures ≈ 1.00)
Tasks never seen in training (Open-Jev OOD split) — mean KL divergence	0.234	Still tracks the teacher closely on unfamiliar task families
Latency — one request on one B200 GPU, median	≈ 90 ms	2.5 ms per decision when requests are batched

Only 40.2 M parameters (0.5 % of the 8 B backbone) were trained, for ≈ 3 GPU-hours on a single B200 (0.93 epoch ≈ 608 k rows). Full evaluation, including robustness and per-domain breakdowns, is in Evaluation details and in code/reports/eval_s2_9b_epoch1.md.

What it does

kind	question	returns
noul	"Is this statement true?"	[P(false), P(true)]
choice	"Which of these 2–16 options?"	one probability per option, aligned with your options
score	"Where on this ordered 0–5 scale?"	a distribution over the six levels (+ expected score)

[kind] choice
[state] SKU AX-330 stock at 8% of safety level; supplier late twice this quarter.
[question] Supplier response for this scenario.
[options]
A) issue_warning
B) renegotiate
C) dual_source
D) maintain
[decision]:

The template is tokenised as one string; the last token's final-norm hidden state goes through a linear fp32 head H → 24 slots (noul → slots 0–1, score → 2–7, choice → 8–23). Inactive slots are masked, a per-kind temperature is applied, and a softmax yields the distribution aligned with your options. One prefill pass — no decoding.

Usage

Option A — HTTP service with a TypeSafe-Jev-style request/response schema (recommended)

The training and serving code ships in this repository under code/.

hf download autotrust/JEV --local-dir JEV            # ~16 GB
pip install -e "JEV/code[dev]"                        # torch ≥ 2.13, transformers ≥ 5.16, flash-linear-attention
python -m jev_judge.server --export JEV --port 18080

curl -s localhost:18080/v1/decisions -H 'content-type: application/json' -d '{
  "kind": "choice",
  "state": "SKU AX-330 stock at 8% of safety level; supplier late twice this quarter.",
  "question": "Supplier response for this scenario.",
  "options": ["issue_warning", "renegotiate", "dual_source", "maintain"]}'

{"id": "req_…", "kind": "choice",
 "options": ["issue_warning", "renegotiate", "dual_source", "maintain"],
 "distribution": [0.334, 0.142, 0.523, 0.001],
 "decision": {"noul": null, "choice": "dual_source", "score": null, "expected_score": null},
 "confidence": 0.523, "model": {"name": "jev-judge-qwen35-9b", "version": "0.8.0", "calibrated": true},
 "latency_ms": 91.3, "batch_size": 1}

POST /v1/decisions:batch ({"items": [...]}, ≤ 256, order preserved) · GET /healthz · 422 on invalid kind/options · 413 on over-long input unless "truncate": true · header X-Jev-Judge-Version · dynamic batching (≤ 4 ms window; 8 concurrent clients → 32 req/s) · refuses to start without calibration.json.

Option B — plain transformers (no extra code)

import json, torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "autotrust/JEV"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="cuda")  # lm_head is not shipped; the warning is expected
head = load_file(hf_hub_download(repo, "head.safetensors"))
cfg  = json.load(open(hf_hub_download(repo, "judge_config.json")))
temp = json.load(open(hf_hub_download(repo, "calibration.json")))["per_kind"]
W, b = head["proj.weight"].cuda(), head["proj.bias"].cuda()

def decide(kind, state, question, options):
    letters = "ABCDEFGHIJKLMNOP"
    lines = options if kind != "choice" else [f"{letters[i]}) {o}" for i, o in enumerate(options)]
    text = f"[kind] {kind}\n[state] {state}\n[question] {question}\n[options]\n" + "\n".join(lines) + "\n[decision]:"
    ids = tok(text, return_tensors="pt", add_special_tokens=False).to("cuda")
    with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
        h = model.model(**ids).last_hidden_state[0, -1].float()          # backbone only, last token
    z = (W @ h + b) / temp[kind]
    s, _ = cfg["slots"]["ranges"][kind]
    p = torch.softmax(z[s : s + len(options)], 0)
    return dict(zip(options, p.tolist()))

print(decide("noul", "Customer says the parcel arrived damaged and wants their money back.",
             "Is the customer asking for a refund?", ["false", "true"]))
# {'false': 0.009, 'true': 0.991}

options are validated: noul must be ["false","true"], score must be ["0".."5"], choice takes 2–16 free-text options.

Why start from a pretrained LLM

A System One judge is "just a classifier", so one could train a small encoder from scratch or fine-tune a BERT-class model. autotrust/JEV starts from a pretrained 9 B LLM instead, and that choice — not the 3 GPU-hours of distillation — is where most of the result above comes from:

1. Pretraining does most of the work; distillation only sharpens. The 24-slot head is initialised from the backbone's own lm_head rows for the verbalizer tokens (false/true, 0–5, A–P), so at step 0 the head's output is exactly the pretrained model's zero-shot restricted next-token distribution (verified to |Δp| < 1e-5; measured 8.6e-07). Without seeing a single label that starting point already agrees with TypeSafe Jev 1.13 on 53 % of choice questions and reaches noul AUROC 0.82; distillation takes it to 90 % / 0.996. World knowledge, instruction reading and long-context handling are inherited, not learned.

2. Extreme sample- and compute-efficiency. Only 40.2 M parameters are trained — LoRA r=16 on the decoder projections plus the 98 k-parameter head, 0.5 % of the 8.0 B text backbone. KL 0.028 was reached after 0.49 epoch (≈ 320 k rows, ≈ 1.7 GPU-hours on one B200) and KL 0.021 after 0.93 epoch (≈ 3 GPU-hours); after the first 64 k rows validation KL was already below 0.10.

3. Transfer to unseen task families. On the Open-Jev OOD split (game-state and workflow-policy decisions never seen in training) autotrust/JEV reaches KL 0.234 and top-1 0.918 — a pretrained LLM reads the content of a new task instead of matching surface patterns of the 53 training domains.

4. Reads real, long, structured states. States in the corpus are prose, JSON game states, policy documents and retrieved passages up to 856 tokens; the backbone supports far longer. BERT-class encoders stop at 512 tokens and lack the code/JSON pretraining to parse them.

5. Reads options, not positions. With 30 % option-permutation augmentation during distillation, the top-1 flip rate under shuffled choice options is 3.9 %; the same backbone before distillation flips 38 % of the time. The pretrained attention binds the letter to the option text almost immediately.

6. Calibration falls out for free. Distilling full teacher distributions with KL (plus an ordinal RPS term for score) on a pretrained backbone gives fitted temperatures of 1.002 / 0.984 / 1.012: the model is calibrated to the teacher with no post-hoc correction (ECE 0.0007).

7. The recipe scales without code changes. The head-initialisation identity holds for any qwen3_5 checkpoint without logit soft-capping; the equivalence gate also passes on Qwen3.8-27B (4.5e-07). 9 B was kept because it already meets the distributional targets.

Evaluation details

Additional metrics (test_set_30k, temperature applied)

metric	autotrust/JEV
noul Brier score against the teacher's probability (lower is better)	0.0015
score ranked probability score (lower is better)	0.0085
Fitted temperatures noul / choice / score	1.002 / 0.984 / 1.012
Top-1 flip rate when choice options are shuffled (1,000 rows × 4 permutations)	3.9 %
Out-of-distribution split — top-1 agreement · noul AUROC	0.918 · 0.989
Throughput — batch of 128 requests on one B200	2.5 ms per decision (≈ 400 decisions/s)
Throughput — 8 concurrent clients, single requests	32 requests/s, median 250 ms

Per source × primitive (test_set_30k, temperature applied)

source	kind	n	KL	top-1	ECE	noul AUROC	score MAE
yuri_v3 — synthetic operational scenarios, labelled by TypeSafe Jev 1.13	noul	8,537	0.005	0.961	0.001	0.994	—
yuri_v3	choice	8,312	0.028	0.902	0.002	—	—
yuri_v3	score	8,527	0.023	0.883	0.002	—	0.103
openjev_v2 — Open-Jev programmatic tasks	noul	1,432	0.004	0.998	0.003	1.000	—
openjev_v2	choice	887	0.176	0.857	0.020	—	—
yuri_v1 — placeholder [0.5, 0.5] labels (see Limitations)	noul	2,260	0.000	—	0.005	—	—

OOD split (13,058 Open-Jev rows from task families not in training): KL 0.234, top-1 0.918, noul AUROC 0.989; choice KL 0.351 / top-1 0.837 (game-state decisions are the hardest slice).

Choice option-permutation consistency (1,000 rows × 4 random permutations): mean max |Δp| 0.024, p90 0.055, top-1 flip rate 3.9 %.

How fast it got there (most recent first)

Validation KL on a fixed 4 k-row subset; test_set_30k metrics after calibration.

stage	rows seen	val KL	t30k KL	choice top-1	score MAE	noul AUROC	ECE
autotrust/JEV v0.8.0 — released weights (4,750 steps ≈ 0.93 epoch, LR annealed to ≈ 0.07×)	608 k	0.019	0.0210	0.898	0.103	0.996	0.0007
v0.7.0 (step 2000 + 500-step LR cool-down)	320 k	0.026	0.0276	0.884	0.119	0.994	0.0014
step 2000	256 k	0.0325	0.037	0.865	0.143	0.992	0.004
step 1500	192 k	0.038	0.040	0.861	0.151	0.991	0.0025
step 500	64 k	0.094	0.081	0.817	0.224	0.977	0.022
untrained backbone with the initialised head (reference point, not the model)	0	0.485	0.510	0.532	1.130	0.824	0.094

Annealing matters: the v0.7.0 cool-down (500 steps, lr ×0.9 → ×0.02 from step 2000) lowered KL by 25 %; continuing on the unseen remainder of the epoch with the learning rate decayed to ≈ 0.07× (v0.8.0) lowered it by another 24 % and added 1.4 points of choice agreement. Checkpoints taken at high learning rate should be annealed.

Training details

item	value
teacher / data	TypeSafe Jev 1.13 full output distributions (as published in the corpus), SargeDev/jev-distill-corpus-v3 (740,957 rows; train 655,806)
backbone	Qwen/Qwen3.5-9B text tower only (vision tower and MTP head dropped), bf16, frozen
trainable	LoRA r=16, α=32, dropout 0.05 on in_proj_qkv, in_proj_z, out_proj, q/k/v/o_proj, gate/up/down_proj (40.1 M) + 24-slot head (98 k, fp32, initialised from lm_head rows)
loss	KL(teacher ‖ model) over active slots + 0.5 · RPS (ranked probability score) for score
augmentation	30 % random permutation of choice options (targets permuted consistently)
batching	128 rows / step, kind-stratified (≥ 1/6 per primitive), length-bucketed, micro-batches capped at 24 k padded tokens, gradient checkpointing
optimiser	AdamW (fused), β=(0.9, 0.98), lr head 2e-4 / LoRA 1e-4, cosine, warmup 3 %, grad-clip 1.0 for 2,500 steps; then continued on the unseen remainder of the epoch (fresh AdamW state, warmup 2 %, lr ×0.9 → cosine) and stopped after 2,250 more steps at lr ≈ ×0.07 — 4,750 steps ≈ 0.93 epoch in total
label hygiene	yuri_v1 rows carry exact-uniform [0.5, 0.5] placeholder labels (137,203 rows, 100 %); down-weighted ×0.05 in training and excluded from temperature fitting
calibration	per-kind scalar temperature (L-BFGS on the calibration split, 10,954 rows): noul 1.002 · choice 0.984 · score 1.012
compute	1× NVIDIA B200 (183 GB); ≈ 1.4 h (2,500 steps) + ≈ 1.5 h (2,250 steps) ≈ 3 GPU-hours; ≈ 7–9 k tokens/s
software	torch 2.13 + cu130, transformers 5.16, peft 0.21, flash-linear-attention 0.5.2

Limitations

• autotrust/JEV mirrors TypeSafe Jev 1.13, including its mistakes. This is a distillation, not an independent judge: where the teacher was wrong or uncalibrated, so is autotrust/JEV. Published evaluations of the teacher show it is unreliable for multi-hop reasoning, arithmetic, dates, counting, and adversarial inputs; autotrust/JEV inherits all of that. It cannot generate text or explanations.
• Choice agreement is capped by teacher ambiguity. The teacher's choice distributions are soft (median top-1 probability 0.70). On the 14 % of rows where the teacher's top-2 options are within 0.1 of each other, argmax agreement is near chance for any faithful mimic (0.46 where the gap is < 0.05). On teacher-decisive rows agreement is 0.954, and autotrust/JEV's argmax captures 97.7 % of the teacher probability mass a perfect mimic could (0.693 vs 0.709).
• Fixed option sets. noul and score accept only their canonical options; choice accepts 2–16 options. Inputs longer than 1,024 tokens are truncated (state only, head 60 % / tail 40 %) at serving unless you raise the limit.
• English-centric. The corpus is English; multilingual behaviour is inherited from the backbone but was not measured.
• Placeholder labels in the corpus. The yuri_v1 memory-relevance stream is 100 % exact-uniform [0.5, 0.5] — those rows teach nothing about relevance. autotrust/JEV outputs ≈ 0.5 on them by design; do not use it for memory-relevance scoring without further training.
• Not for high-stakes decisions. Use confidence gating: act automatically only above a threshold you validated on your own data, and route the rest to a stronger model or a human.

Files

model-0000{1..4}-of-00004.safetensors   merged text backbone (bf16, Qwen3_5ForCausalLM layout, no lm_head)
model.safetensors.index.json · config.json
head.safetensors                        24-slot decision head (fp32): proj.weight [24, 4096], proj.bias [24]
judge_config.json                       slot layout, verbalizer token ids, template version, provenance
calibration.json                        per-kind temperatures (+ fit diagnostics)
tokenizer.json · tokenizer_config.json · chat_template.jinja
code/                                   jev_judge package, scripts (train / calibrate / evaluate / export / serve),
                                        tests, configs, DESIGN.md and all evaluation reports

License and acknowledgements

Weights: Apache-2.0 (base model Qwen/Qwen3.5-9B is Apache-2.0; training corpus SargeDev/jev-distill-corpus-v3 is Apache-2.0, its openjev_v2 stream additionally CC0). The System One framing and the noul / choice / score primitives originate with TypeSafe AI's Jev; autotrust/JEV is an independent student model trained on public data and shares no weights, code or affiliation with TypeSafe AI.

@misc{autotrust_jev_2026,
  title  = {autotrust/JEV: an open-weights student of TypeSafe Jev 1.13 for calibrated System One decisions (Qwen3.5-9B)},
  author = {autotrust},
  year   = {2026},
  url    = {https://huggingface.co/autotrust/JEV}
}