CodingBad02/chhaya-medgemma-lora-v2

Chhaya-MedGemma (LoRA)

A QLoRA fine-tune of google/medgemma-1.5-4b-it for Chhaya, a skin & heat-health companion for outdoor workers. It reads a skin photo and emits a structured findings JSON directly — no chain-of-thought preamble — with a concern level calibrated against real clinical labels.

Usage

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel

base = "google/medgemma-1.5-4b-it"
proc = AutoProcessor.from_pretrained(base)
model = AutoModelForImageTextToText.from_pretrained(base, torch_dtype=torch.bfloat16).to("cuda")
model = PeftModel.from_pretrained(model, "CodingBad02/chhaya-medgemma-lora-v2").merge_and_unload()

messages = [{"role": "user", "content": [
    {"type": "image", "image": img},          # image BEFORE text
    {"type": "text", "text": "skin check"},
]}]
inputs = proc.apply_chat_template(messages, add_generation_prompt=True,
    tokenize=True, return_dict=True, return_tensors="pt").to(model.device, dtype=torch.bfloat16)
out = model.generate(**inputs, max_new_tokens=400, do_sample=False)
print(proc.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Output schema:

{"what_i_see","spot":{"type","color","borders","symmetry","texture"},
 "heat_sun_signals":[],"concern":"low|watch|see_doctor","concern_reason",
 "image_quality":"good|limited","summary"}

Training

• Data: CodingBad02/chhaya-skin-extract (ISIC-2024 biopsy-anchored + SCIN real-photo, 1,406 rows). see_doctor class oversampled 3× to improve recall.
• QLoRA (4-bit nf4, r=16, α=32), frozen vision tower (only the language model adapts), 2 epochs, A100. ~$6 of compute.

Eval (141-image held-out test set, vs base)

metric	base	this model (v2)
Valid JSON	0.993	1.0
Concern accuracy	0.333	0.695
Malignant recall	0.936	0.83
Output tokens/answer	770	156

Base's higher recall is achieved by labelling 60% of cases "watch" (33% accuracy); this model gives a real triage at 5× fewer tokens. Residual under-warning risk is mitigated by a deterministic ABCDE backstop in the app.

Limitations

Not a medical device. Does not diagnose. A research/education demo built for a hackathon. Misses some malignant-type lesions (recall 0.83). Always pair with clinician review. No heat-rash/sunburn class in training (ISIC/SCIN gap).

Inspired by Sunny by Daniel Bourke.