PathJev-P: decision-native vision–language model for histopathology patches

Research use only. Not a medical device. Not validated for clinical use. Do not use it to diagnose or treat patients or to make decisions that affect a person's health.

PathJev-P answers a question about one histopathology patch with a probability distribution over candidate answers that you supply at run time. You can also add the option "other tissue / none of these". It applies the OmniMed-Jev decision-native recipe (LoRA adapter plus a small decision head, one forward pass for all candidates) to histopathology, on top of MedGemma-1.5-4B.

This repository contains seed 1 of the three trained seeds (checkpoint selected on validation data only):

• adapter_model.safetensors: the LoRA adapter (rank 16, all linear layers of the vision encoder and the language model);
• medjev_head.safetensors: the decision head;
• adapter_config.json.

The base model is not included; you must accept its terms on Hugging Face to download it.

Paper: PathJev: Decision-Native Vision–Language Modelling for Histopathology (arXiv preprint, link to be added on announcement). Demo (precomputed examples): https://huggingface.co/spaces/Cristiandieti/PathJev-P-demo

Code, data-construction scripts and split manifests: https://github.com/CosmoIknosLab/PathJev

How to use

# pip install "transformers>=4.50" peft bitsandbytes accelerate pillow   (4-bit needs a CUDA GPU)
import sys
from huggingface_hub import snapshot_download
local = snapshot_download("Cristiandieti/PathJev-P")          # weights + the pathjev_infer/ inference code
sys.path.insert(0, local)
from pathjev_infer import PathJev

model = PathJev.load(local, token=True)              # needs access to google/medgemma-1.5-4b-it
probs = model.predict("patch.png", [{
    "instructions": "Which Gleason pattern predominates in this prostate tissue?",
    "candidates": [{"id": "benign", "text": "benign (non-cancerous) prostate tissue"},
                   {"id": "gp3", "text": "Gleason pattern 3 carcinoma"},
                   {"id": "gp4", "text": "Gleason pattern 4 carcinoma"},
                   {"id": "gp5", "text": "Gleason pattern 5 carcinoma"}]}], add_other=True)
print(probs[0])   # {'benign': ..., 'gp3': ..., 'gp4': ..., 'gp5': ..., 'other': ...}

Input: one RGB patch, resized to 896×896 like the training images, so the aspect ratio is not preserved. The model was trained on H&E patches of 150–512 px at 10–20×.

Output: one normalised distribution per question. Several questions can be asked in one call, and each is answered over its own candidates.

Training data

PathJev-P was trained on 40,000 expert-labelled patch states, built from SICAPv2 (prostate, CC BY 4.0), BRACS (breast, CC BY-NC 4.0) and the four organ subsets of SPIDER (colorectal, lung, breast, skin; CC BY-NC 4.0, research only). The labels were turned into typed Choice, Noul (yes/no) and Score (ordered) questions. No LLM-generated labels were used.

Evaluation (main Choice question; mean over 3 seeds, before any temperature scaling)

All the numbers below come from the preprint.

Test sets	PathJev-P bAcc / ECE	Generative SFT, same data (summed read-out)	Generative SFT (length-normalised read-out, post hoc)	CONCH zero-shot	OmniMed-Jev (released)
Clean, all	0.541 / 0.204	0.492 / 0.279	0.492 / 0.201	0.345 / 0.403	0.213 / 0.517
Unseen organs (liver, stomach)	0.390 / 0.301	0.277 / 0.446	0.289 / 0.110	0.328 / 0.487	0.253 / 0.540
In domain (BRACS, SICAPv2 test)	0.692 / 0.108	0.707 / 0.113	0.696 / 0.293	0.363 / 0.319	0.173 / 0.493

bAcc is balanced accuracy, higher is better. ECE is expected calibration error (15 bins), lower is better. "Clean" means no documented overlap with the training data of the backbone; undisclosed backbone data cannot be excluded.

How to read these results:

• Accuracy. PathJev-P is more accurate than the generative model trained on the same data. The difference is +0.049 [+0.038, +0.063] balanced accuracy on the clean sets, and comes almost entirely from the two organs absent from training.
• Calibration depends on the read-out. PathJev-P has lower calibration error than the generative model when the latter is read out by summed candidate log-likelihood, the read-out OmniMed-Jev used. With a length-normalised read-out the generative model is about as well calibrated on average, better calibrated on the unseen organs and worse in domain.
• Unseen organs. Accuracy on organs not seen in training is low (0.390). The model is over-confident there: ECE 0.301, and a high probability does not mean a reliable answer.
• Supervised probe. A linear probe on H-optimus-0 features is better calibrated in every setting (clean ECE 0.047). It is more accurate wherever labelled data for the target question exist.

Limitations

• Only single patches were tested. Whole slides are handled only by clustering tiles and labelling cluster representatives, shown qualitatively on four slides.
• Only two unseen organs were tested. Fine-grained classes of new organs (e.g. gastric categories) are mostly not recognised.
• The "other / none of these" candidate was never the correct answer during training. It is not a validated open-set rejector.
• Patient-level independence cannot be verified for every test set; see the paper.
• One backbone (4-bit MedGemma-1.5-4B) and one wording per candidate were used. Outputs depend somewhat on candidate order, and wording was not studied.
• Several training datasets are non-commercial, which restricts use of these weights.

Licence and terms

• MedGemma terms. This model is a derivative of MedGemma and is subject to the Health AI Developer Foundations Terms of Use (https://developers.google.com/health-ai-developer-foundations/terms), including the Prohibited Use Policy; see NOTICE.
• Training-data licences. Some training data are licensed CC BY-NC 4.0 and research-only (SPIDER, BRACS).
• Licence for this repository: see LICENSE: MedGemma (HAI-DEF) Model Derivative terms, non-commercial research use only; the code in pathjev_infer/ is Apache-2.0.

Citation

Cristian Tommasino and Antonio M. Rinaldi. PathJev: Decision-Native Vision–Language Modelling for Histopathology. Preprint, 2026 (arXiv link to be added). Department of Electrical Engineering and Information Technology (DIETI), University of Naples Federico II.

Please also cite OmniMed-Jev, MedGemma and the datasets listed above.