akmaier/qwen3vl-8b-figcaption-lora

Figure-captioning LoRA adapters (Qwen3-VL-8B)

PEFT LoRA adapters (r=16, alpha=32) fine-tuned on medical figure captions on top of Qwen/Qwen3-VL-8B-Instruct. They accompany a SPIE Medical Imaging study on caption fine-tuning for medical figure captioning, and are released for research and evaluation only. Not for clinical use. Manuscript under submission.

Conditions (subfolders; seeds 42/43/44)

• c1_raw_s{seed}: fine-tuned on raw figure captions.
• c2_clean_s{seed}: fine-tuned on cleaned captions.
• c3_aux_s{seed}: cleaned captions plus an auxiliary content loss.

C0 (zero-shot) is the base model with no adapter.

Loading

from peft import PeftModel
from transformers import AutoModelForImageTextToText
base = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3-VL-8B-Instruct", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base, "akmaier/qwen3vl-8b-figcaption-lora", subfolder="c2_clean_s42")

Notes

Trained on medical figure captions from open-access and text-and-data-mining sources (figures only; no captions or images are shipped). Findings: fine-tuning raises content overlap and hallucination together; the auxiliary loss does not help; caption cleaning lowers hallucination for this 8B model. The models do not reliably perceive imaging artifacts. For research and evaluation, not for clinical decisions.