Access this MedGemma derivative

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

This model is a derivative of MedGemma and is governed by the Health AI Developer Foundations terms of use, including the use restrictions in Section 3.2 and the Prohibited Use Policy. It is not a medical device. By requesting access you agree to these terms.

medgemma-decide-4b

A typed decision model on MedGemma 1.5 4B, trained with Unsloth's FastDecisionModel. It reads a medical text and a set of typed questions, and returns a probability for every option in one forward pass. It doesn't generate text.

Not a medical device. Not for clinical decisions, diagnosis, triage or patient care. Research use only, under the HAI-DEF terms (see NOTICE).

Results on the medical-typed-decisions test set

One training seed (3407); seed-to-seed variation is not measured. "Constant answer" is the best any single fixed answer scores on that question; a model has to beat it, and balanced accuracy shows whether it does on the rare classes too. Scored with eval.py from Amaljith/medical-typed-decisions, where the predictions and every baseline are published.

Question	n	Accuracy	95% CI (first seed)	Constant answer	Balanced acc.	ECE
`medline/adverse`	1500	0.928	0.915 to 0.941	0.926	0.625	0.014
`medline/animals`	1500	0.935	0.922 to 0.947	0.757	0.896	0.014
`medline/design`	605	0.944	0.924 to 0.962	0.476	0.796	0.027
`medline/disease_area`	314	0.949	0.924 to 0.971	0.185	0.952	0.043
`medline/drug_therapy`	1500	0.929	0.916 to 0.942	0.885	0.791	0.010
`medline/group_adolescents`	356	0.882	0.848 to 0.913	0.854	0.668	0.036
`medline/group_adults`	313	0.821	0.776 to 0.863	0.550	0.823	0.036
`medline/group_children`	322	0.947	0.922 to 0.969	0.907	0.881	0.024
`medline/group_female`	304	0.826	0.780 to 0.868	0.520	0.828	0.066
`medline/group_infants`	308	0.984	0.968 to 0.997	0.968	0.750	0.013
`medline/group_male`	335	0.869	0.833 to 0.904	0.534	0.869	0.053
`medline/group_older_adults`	316	0.854	0.813 to 0.892	0.807	0.835	0.048
`medline/humans`	1500	0.929	0.916 to 0.943	0.751	0.912	0.025
`medmcqa/answer`	4162	0.510	0.495 to 0.525	0.322	0.508	0.046
`medmcqa/subject`	4160	0.688	0.673 to 0.703	0.315	0.677	0.023
`medqa/answer`	1273	0.603	0.578 to 0.630	0.277	0.602	0.039
`pubmedqa/answer`	500	0.774	0.738 to 0.810	0.538	0.578	0.090
`symptoms/diagnosis`	212	0.854	0.807 to 0.901	0.047	0.855	0.076

Against the baselines (balanced accuracy)

The baselines are deliberately cheap. Constant always picks the most common answer for a question. TF-IDF turns each text into word weights (frequent in this text, rare across texts) and fits a logistic regression on them: it counts words and understands nothing, so a question it solves doesn't need a 4B model. S1-PubMedBERT is a 110M-parameter biomedical encoder trained for MEDLINE indexing.

"n/a" means the baseline doesn't apply (TF-IDF has no fixed labels for exam answer options) or was trained on those test rows (S1-PubMedBERT on PubMedQA). The control is trained with the same data and recipe.

Question	This model	Constant	TF-IDF	S1-PubMedBERT
`medline/adverse`	0.625	0.500	0.721	0.861
`medline/animals`	0.896	0.500	0.830	0.933
`medline/design`	0.796	0.167	0.770	0.878
`medline/disease_area`	0.952	0.157	0.516	0.397
`medline/drug_therapy`	0.791	0.500	0.775	0.884
`medline/group_adolescents`	0.668	0.500	0.592	0.787
`medline/group_adults`	0.823	0.500	0.729	0.816
`medline/group_children`	0.881	0.500	0.695	0.872
`medline/group_female`	0.828	0.500	0.752	0.846
`medline/group_infants`	0.750	0.500	0.513	0.863
`medline/group_male`	0.869	0.500	0.764	0.877
`medline/group_older_adults`	0.835	0.500	0.740	0.829
`medline/humans`	0.912	0.500	0.868	0.934
`medmcqa/answer`	0.508	0.250	n/a	n/a
`medmcqa/subject`	0.677	0.050	0.359	n/a
`medqa/answer`	0.602	0.250	n/a	n/a
`pubmedqa/answer`	0.578	0.333	0.395	n/a
`symptoms/diagnosis`	0.855	0.045	0.895	n/a

Published generative scores from Google's MedGemma 1.5 card, with Google's prompts (not comparable to the numbers above): MedGemma 1.5 4B 69.1 MedQA, 59.8 MedMCQA, 68.2 PubMedQA; Gemma 3 4B 50.7, 45.4, 68.4.

Calibration: the probabilities above use Unsloth's temperature per question type (choice, yes/no), fitted on a separate calibration split, so ECE differs by source. decision_format.json also holds one temperature per training question, fitted on the same split; decide(), systemone() and predict.py apply it when a request names its source.

Release gate: FAILED

Every question with at least 100 test decisions and 30 of each answer must reach balanced accuracy of chance + 0.05 at the lower end of its 95% interval, and must not fall below the best cheap baseline (constant answer, TF-IDF) on that question.

Failed on at least one seed:

medline/adverse (cheap baseline: 0.721)
medline/adverse: About 93% of test abstracts are labelled "no adverse effects reported". The reported scores use plain argmax decisions, fixed before the test set was opened, so the model rarely answers "yes": accuracy is high but balanced accuracy falls below TF-IDF's. The Gemma 3 control fails the same way. Training answer shares ship in decision_format.json; dividing them out (logit adjustment) trades accuracy for recall on "yes". Don't rely on this question to find adverse-effect reports.

MedGemma vs Gemma 3 control (pre-registered)

Accuracy difference, MedGemma minus Gemma 3 4B trained on the same data, averaged over 1 seed pair(s); 95% interval from a bootstrap that resamples the same test rows for every seed. A question is won when the interval is above 0 and every seed's difference is above 0.

Question	n	Difference	95% CI	Per seed	Won
`medqa/answer`	1273	+0.079	+0.052 to +0.106	+0.079	yes
`medmcqa/answer`	4162	+0.054	+0.038 to +0.070	+0.054	yes

Verdict: win.

Against Jev (zero-shot, closed)

TypeSafe's Jev 1.14 answered the same test questions through OpenRouter (jev_eval.py in the benchmark repo). Macro balanced accuracy: Jev 0.761, this model 0.689. Jev leads on exam knowledge (MedQA 0.885 vs 0.609, MedMCQA 0.749 vs 0.519). This model leads on 7 of 18 questions, where the task depends on this data's labels: symptoms +0.142, MEDLINE humans +0.107, population groups (men +0.096, women +0.059), medical subject +0.044, adverse effects +0.028, drug therapy +0.014 (paired 95% intervals exclude 0).

Training


Base	`unsloth/medgemma-1.5-4b-it`
Method	LoRA r=16, alpha 16, Clef decision head, Gemma turn format, Unsloth 2026.10.8
Epochs	1
Hardware	NVIDIA A100-SXM4-40GB, 71.8 minutes
Data	22132 training rows from 5 public datasets

Use

The repository follows the layout of Cloudflare's Clef releases: merged weights and the joint schema head at the root, loaded with transformers alone by joint_schema_model.py, which answers Jev/SystemOne requests. Gemma's own turn format replaces Clef's Qwen chat markers, as the model was trained.

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("Amaljith/medgemma-decide-4b")
sys.path.insert(0, path)
from joint_schema_model import load_release_model, systemone

model, processor = load_release_model(path, device = "cuda")
print(systemone(model, processor, {
    "model": "medgemma-decide-4b",
    "source": "medqa",  # optional: applies the temperatures fitted for that training source
    "state": "A 25-year-old presents with fever, severe headache, neck stiffness and photophobia...",
    "questions": {"answer": {"type": "choice", "instructions": "Which option is the best answer?",
                             "criteria": {"A": "Viral meningitis", "B": "Bacterial meningitis"}}},
}))

decide() in the same file returns the calibrated probabilities directly. Loading downloads the merged weights (about 8.6 GB in bfloat16). The base model's terms (HAI-DEF) apply.

The scores above were computed with the 4-bit adapter in adapter/, as pre-registered. The merged bfloat16 weights at the root score the same test split as follows; they agree with the adapter on 94.2% of dev decisions, the rest being near-ties the two precisions break differently.

Question	n	4-bit adapter (scored)	bf16 release (download)
`medline/adverse`	1500	0.928	0.932
`medline/animals`	1500	0.935	0.931
`medline/design`	605	0.944	0.944
`medline/disease_area`	314	0.949	0.943
`medline/drug_therapy`	1500	0.929	0.927
`medline/group_adolescents`	356	0.882	0.885
`medline/group_adults`	313	0.821	0.818
`medline/group_children`	322	0.947	0.941
`medline/group_female`	304	0.826	0.812
`medline/group_infants`	308	0.984	0.984
`medline/group_male`	335	0.869	0.881
`medline/group_older_adults`	316	0.854	0.861
`medline/humans`	1500	0.929	0.930
`medmcqa/answer`	4162	0.510	0.519
`medmcqa/subject`	4160	0.688	0.685
`medqa/answer`	1273	0.603	0.609
`pubmedqa/answer`	500	0.774	0.786
`symptoms/diagnosis`	212	0.854	0.849

To reproduce the scored setup exactly, load the adapter with Unsloth:

pip install unsloth==2026.10.8 unsloth_zoo==2026.10.8

from huggingface_hub import hf_hub_download
import importlib.util
spec = importlib.util.spec_from_file_location("predict", hf_hub_download("Amaljith/medgemma-decide-4b", "adapter/predict.py"))
predict = importlib.util.module_from_spec(spec); spec.loader.exec_module(predict)
model, tokenizer, fmt = predict.load("Amaljith/medgemma-decide-4b")
predict.predict(model, tokenizer, fmt, state, questions, source = "medqa")

predict.py applies the same prompt format and temperatures (decision_format.json). Loading the adapter with FastDecisionModel alone skips them and gives different numbers and, because this model uses the Gemma turn format, lower accuracy.

Limits

Exam sources (MedQA, MedMCQA, PubMedQA) are old public sets, probably in the base model's training data.
Data comes from US (MedQA) and Indian (MedMCQA) exams, English-language indexed literature (MEDLINE) and LLM-paraphrased symptom descriptions; it doesn't represent real patient messages or other populations.
MEDLINE labels are MeSH indexing decisions, not clinical ground truth.
Text only; the image encoder was not trained or evaluated.
It always answers: probability is split across the options given, even when none of them is right. Include the right answer among the options.
It scores options in one pass and never explains; it is weak on questions that need multi-step reasoning.

Downloads last month: 4

Safetensors

Model size

4B params

Tensor type

BF16

Model tree for Amaljith/medgemma-decide-4b

Base model

google/medgemma-1.5-4b-it

Finetuned

unsloth/medgemma-1.5-4b-it

Finetuned

(12)

this model

Dataset used to train Amaljith/medgemma-decide-4b

Evaluation results

medline/adverse balanced accuracy on medical-typed-decisions
test set self-reported

0.625
medline/animals balanced accuracy on medical-typed-decisions
test set self-reported

0.896
medline/design balanced accuracy on medical-typed-decisions
test set self-reported

0.796
medline/disease_area balanced accuracy on medical-typed-decisions
test set self-reported

0.952
medline/drug_therapy balanced accuracy on medical-typed-decisions
test set self-reported

0.790
medline/group_adolescents balanced accuracy on medical-typed-decisions
test set self-reported

0.668
medline/group_adults balanced accuracy on medical-typed-decisions
test set self-reported

0.823
medline/group_children balanced accuracy on medical-typed-decisions
test set self-reported

0.881