IdeaNJEV

A calibrated decision model that answers typed questions about a JSON state in one forward pass, generating no tokens. It is a rank-16 LoRA adapter plus a linear score head on Qwen3.5-4B, trained on LocalLLaMA/typed-decisions, and intended as a classification gate in agent workflows (route a ticket, decide whether a run needs human review, pick the next step).

Code, demo UI, training script and full results: https://github.com/bhaweshkrsingh/IdeaNJEV (see docs/RESULTS.md).

Read this first. IdeaNJEV is not trained from scratch, and at 4B parameters it is not faster than small encoder-based decision models: about 300 ms for one question and 1.6 s for five on an NVIDIA GB10. It is a calibration-and-accuracy result, not a speed result.

Results (full 2,000-decision test split)

Metric	Value
Accuracy	0.779 (95% CI 0.762 to 0.796)
ECE of confidence (T=0.4, cross-fit, no leakage)	0.0154
Brier vs soft human distribution, sum over options (probabilities, T=1)	0.0459
Accuracy by type: choice / noul / score	74.7% / 86.0% / 74.3%

Measured end to end through the shipped batched decide() path on all 2,000 decisions: accuracy 0.784, ECE 0.0138 (the small difference is bf16 near-tie noise, not an improvement).

Against the original Qwen3.5-4B writing the same JSON on the same GPU (200 decisions): IdeaNJEV 73.5% accurate in 373 ms; the base model with thinking off 49.5% in 1,118 ms; with thinking on, 16 of 20 decisions did not finish within a 3,000-token budget (median 168 s). The base model is prompted while IdeaNJEV is trained on this benchmark, so this measures what the training bought. Full table and caveats: docs/RESULTS.md section 5b in the GitHub repo.

Published figures for the other open models on the same benchmark are in the table below. We did not re-run them, their Brier and ECE definitions differ, and the accuracy gaps are about one standard error, so treat this as "in the same range, at the top of it on accuracy", not as a win.

Model	Params	Accuracy	Brier (published)	ECE (published)
IdeaNJEV	4B + LoRA	0.779	0.0459 (soft, T=1)	0.0154
openJev-verdict-2.0	149.6M	0.771	0.0636	0.0144 (separate confidence head)
Laya	421M	0.766	0.062	0.213
TypeSafe Jev 1.13.0	undisclosed	0.727	0.148	n/a

The checkpoint of openJev-verdict-2.0 could not be downloaded when we checked (its issue #2), so its numbers are unverified.

How to use

from ideanjev.model import IdeaNJEV   # pip install from the GitHub repo

jev = IdeaNJEV(base_model="Qwen/Qwen3.5-4B", adapter="BhaweshSingh/IdeaNJEV")
out = jev.decide(
    state={"alert": {"service": "checkout-api", "symptom": "p99 9.2s, errors 14%"}, "customers_affected_pct": 30},
    questions={
        "rollback": {"type": "noul", "instructions": "Should the last deployment be rolled back now?"},
        "page": {"type": "choice", "instructions": "Who should be paged?",
                 "criteria": {"nobody": "No page.", "oncall": "Primary on-call.", "commander": "Incident commander."}},
        "severity": {"type": "score", "instructions": "Incident severity?",
                     "criteria": ["SEV4", "SEV3", "SEV2", "SEV1"]},
    },
)

Question types: choice (criteria = {label: description}), noul (boolean; criteria optional) and score (criteria = ordered list of level descriptions). For each question you get answer, confidence, and probabilities.

The two probability views are computed from the same logits:

• probabilities: temperature 1.0, matches the human-consensus distribution the model was trained on.
• confidence: probability of the chosen answer at temperature 0.4, calibrated against correctness. Threshold on this.

ideanjev_config.json in this repo carries both temperatures. score_head.safetensors is the linear head (2,560 to 1); adapter_model.safetensors is the LoRA. The base model is downloaded separately from Qwen/Qwen3.5-4B.

How it works

Each candidate answer of a question is written as its own short prompt (State: ... Question: ... Candidate answer: ... How likely is this candidate answer to be correct?). All candidates are scored together in one padded forward pass; a linear head reads the last token's hidden state, and a per-question softmax turns the scalars into a distribution. Trained with soft cross-entropy against the benchmark's human-consensus probabilities: batch 8, AdamW 5e-5, LoRA r=16 on all attention, MLP and DeltaNet projections, about 0.6 epoch of the 6,000 training decisions (step 450 of 500 planned).

Limitations and risks

• Evaluated on one benchmark (English, four enterprise workflows). No evidence outside it.
• About 1 in 5 answers is wrong. Use confidence to send uncertain cases to a person.
• Batched inference can flip near-tied answers relative to the per-option evaluation (2.5% of a 200-decision sample, all with a top-2 logit margin below 0.07); decide(..., exact=True) reproduces the evaluation exactly.
• The temperature was fit on the same benchmark's test set with a leakage-free 2-fold cross-fit (both folds chose 0.4), not on a held-out slice of training data. Training-time monitoring also read 100 test decisions every 50 steps.
• Not evaluated for adversarial or prompt-injected state content, or for fairness. Do not use as the sole decision-maker for legal, financial or safety-critical outcomes.

License

MIT for the adapter and code. The base model (Qwen3.5-4B) and the training dataset are Apache-2.0 and keep their own terms (see NOTICE in the GitHub repo). Check the license of LocalLLaMA/typed-decisions before redistributing derived data.

This card was drafted with help from Claude (Anthropic) and reviewed by the authors.