MedJev Flash 9B (MLX 8-bit, Apple silicon)

MedJev is [OpenJev Flash 9B] adapted to medicine. This repository adds a zero-shot medical evaluation, Open MedJev: How Far Do Zero-Shot System-One Decisions Go in Medicine?

MedJev is a System-One decision model. It reads a state (text or JSON) and answers typed questions, either a Choice over named options or a Noul (the probability that a statement is true), with a full calibrated probability distribution from one forward pass, without generating text. It is a fine-tune of Qwen3.5-9B, converted to 8-bit MLX (about 9.5 GB, text only).

Results (zero-shot, no medical training)

Accuracy on medical knowledge benchmarks, with expected calibration error (ECE). 95% bootstrap CIs in the paper.

Benchmark	n	MedJev	Qwen3.5-9B (base)	MedGemma-1.5-4B	ECE (MedJev)
MedQA (5 options)	1,000	72.5%	68.6%	47.7%	0.039
USMLE (4 options)	1,273	75.4%	70.8%	54.0%	0.028
NEJM-MedQA	512	80.1%	77.0%	60.4%	0.063
DiagnosisArena	915	55.5%	54.4%	42.3%	0.071

AUROC on MIMIC-IV and MIMIC-IV-ED prediction tasks (1,000 held-out cases each), against a supervised XGBoost model tuned by cross-validation on local training data. MedJev uses no local data.

Task	MedJev (zero-shot)	XGBoost (supervised)
In-Hospital Mortality	0.926	0.902
ED Triage Admission	0.745	0.809
ED Admission at 2 h	0.742	0.824
Length of Stay > 7 Days	0.688	0.719
30-Day Readmission	0.671	0.684

MedJev ranks patients well, but its probabilities do not reflect local base rates. A two-parameter Platt recalibration fitted on a few hundred local cases fixes this (for example, ECE on In-Hospital Mortality drops from 0.286 to 0.007) without changing the ranking. Code to reproduce every number: the Open MedJev repository.

Run it

uv venv mlx --python 3.12
uv pip install --python mlx/bin/python "mlx==0.32.2" "mlx-lm==0.31.3" \
  "transformers==5.17.0" "openai==3.16.2" "httpx==0.28.1" "huggingface-hub==1.32.0"

mlx/bin/hf download YOUR_HF_USERNAME/MedJev-Flash-9B-MLX --local-dir MedJev-Flash-9B-MLX

TOKENIZER=MedJev-Flash-9B-MLX SHIM_MODEL=MedJev-Flash-9B-MLX \
READOUT_T=1.07 READOUT_NOUL_T=1.074766 READOUT_NOUL_BIAS=0 \
READOUT_TARGETED=1 READOUT_INSTR_STYLE=pyrepr SHIM_STAGGER=1 \
  mlx/bin/python MedJev-Flash-9B-MLX/helper/shim_mlx.py \
  --helper MedJev-Flash-9B-MLX/helper/shim.py --model MedJev-Flash-9B-MLX --port 3000

The server listens on 127.0.0.1:3000 and answers POST /v1/systemone with the request format described on the OpenJev card. Pass the five READOUT_* settings above: the helper's built-in defaults belong to a different model. Append --selfcheck for a smoke test. It prints the helper hash, which should begin 81a22f1b.

Example request, a Noul on a clinical state:

curl -s localhost:3000/v1/systemone -H 'content-type: application/json' -d '{
  "state": {"patient": {"age": 81, "sex": "female"}, "labs_latest": ["Lactate 5.2 mmol/L (high)"],
            "infusions": ["norepinephrine"]},
  "questions": {"label": {"type": "noul",
    "instructions": "Will this patient die before being discharged from the current hospital admission?",
    "criteria": {"true": "the patient dies in hospital", "false": "the patient is discharged alive"}}}
}'

Intended use and limitations

MedJev is a research artefact, not a medical device. It must not be used for diagnosis, triage or any decision about a real patient. It has not been prospectively validated, and its zero-shot probabilities are miscalibrated for any specific hospital until recalibrated on local data. Its evaluation covers English text, US benchmarks and one health system (Beth Israel Deaconess, through MIMIC), so performance elsewhere is unknown. Like any language model, it can reflect biases in its training data and in the records it reads.

No MIMIC data, nor anything derived from individual patients, is included in this repository.

Licence and attribution

• Weights: CC BY-NC 4.0 (LICENSE), as released by the OpenJev project. Non-commercial use only. For a commercial licence, contact OpenJev (support@loopai.com).
• helper/: Apache 2.0 (LICENSE-APACHE-2.0)
• Base model: Qwen/Qwen3.5-9B, Apache 2.0. Full attribution in NOTICE.

MedJev is an independent research release. It is not affiliated with or endorsed by the OpenJev project, Qwen or TypeSafe (Jev is TypeSafe's product).

Citation

@misc{openmedjev2026,
  title  = {Open MedJev: How Far Do Zero-Shot System-One Decisions Go in Medicine?},
  author = {Lee, Simon},
  year   = {2026}
}

Please also cite OpenJev Flash 9B and Qwen3.5 (see their model cards).