How Do Medical MLLMs Fail?  A Study on Visual Grounding in Medical Images
Authors: Guimeng Liu
Tianze Yu
Somayeh Ebrahimkhani
Lin Zhi Zheng Shawn
Kok Pin Ng
Ngai-Man Cheung
Venue: ICLR 2026 Poster (2026)
Keywords: Medical MLLM
Visual Grounding
Primary area: datasets and benchmarks

Abstract: Generalist multimodal large language models (MLLMs) have achieved impressive performance across a wide range of vision-language tasks. However, their performance on medical tasks—particularly in zero-shot settings where generalization is critical—remains suboptimal. A key research gap is the limited understanding of why medical MLLMs underperform in medical image interpretation.
**In this work**, we present a pioneering systematic investigation into the visual grounding capabilities of state-of-the-art medical MLLMs. To disentangle *visual grounding* from *semantic grounding*, we design VGMED, a novel evaluation dataset developed with expert clinical guidance, explicitly assessing the visual grounding capability of medical MLLMs. 
We introduce new quantitative metrics and conduct detailed qualitative analyses. Our study across **eight** state-of-the-art (SOTA) medical MLLMs validates that they often fail to ground their predictions in clinically relevant image regions. We note that this finding is specific to medical image analysis; in contrast, prior work has shown that MLLMs are capable of grounding their predictions in the correct image regions when applied to natural scene images.
Motivated by these findings, we propose VGRefine, a simple yet effective inference-time method that refines attention distribution to improve visual grounding in medical settings. Our approach achieves SOTA performance across  6 diverse Med-VQA benchmarks (over 110K VQA samples from 8 imaging modalities) 
without requiring additional training or external expert models.  Overall, our work, for the first time, systematically validates inadequate visual grounding as one of the key contributing factors for medical MLLMs' under-performance.
Additional experiments are included in the Supp. Project Page: https://guimeng-leo-liu.github.io/Medical-MLLMs-Fail/

Published as a conference paper at ICLR 2026
How Do MEDICAL MLLMs FAIL? A STUDY ON VISUAL GROUNDING IN MEDICAL IMAGES
Guimeng Liu'
Tianze Yu' Somayeh Ebrahimkhani*
Lin Zhi Zheng Shawn
Kok Pin Ng Ngai-Man Cheung
Singapore University of Technology and Design, Singapore
2 Department of Neurology, National Neuroscience Institute, Singapore 3 Duke-NUS Medical School, Singapore
ABSTRACT
Generalist multimodal large language models (MLLMs) have achieved impressive performance across a wide range lision-language tasks. However, their perfor- mance on medical pasks-particularly in zero-shot settings where generalization is critical-remains suboptimal. A key research gap is the limited understanding of why medical MLLMs underperform in medical image interpretation. In this work, we present a pioneering systematic investigation into the visual grounding capabilities of state-of-the-art medical MILMs.. To disentangle visual ground- ing from semantic grounding, we design VGMED, a novel evaluation dataset de- veloped with expert clinical guidance, explicitly assessing the visual grounding capability of medical MLLMs. We introduce new quantitative metrics and con- duct detailed qualitative analyses. Our study across eight oftte-of-theaar (SOTA) medical MLLMs validates that they often fail to ground their predictions in clin- ically relevant image regions. We note that this finding specific to medical image analysis; in contrast, prior work has shown that MLLMs are capable of grounding their predictions the correct image regions when applied to natural scene images. Motivated by these findings, effective inference- time method that refines attention distribution to improve vi- sual grounding in medical settings. Our approach achieves SOTA performance across 6 diverse Med-VQU benchmarks (over 110K VQA samples from imag- ing modalities) without requiring additional training external expert models. Overall, our work, for the first time, systematically validates inadequate visual grounding as one of the key contributing factors for medical MLLMs' under- performance. Additional experiments are included in the Supp. Project Page: httpps:// hatpss:/ggunnnng--e liu, github. io Medical- MiddiccalMMLMss MULLMs-Fail..
propose VGRefine, simple yet
INTRODUCTION
Generalist multimodal large language models (MLLMs) have demonstrated strong performance across a broad range of vision-language tasks, including visual question answering (VQA) (Wang cet al.] 22024] Dai et al. [2233 Liu al. 2023 et al. 2024b Liu et al., 2024b). image captioning | Wu et al. 2024). science and mathematical reasoning (Liu et al. [022dd] Zhuang al. 2025] Shi et al. 2024). Recent efforts have extended these models to the medical domain,
(Li et al. 2022bb
with the goal of developing medical MLLMs that can leverage their generalization capabilities to support diverse clinical deciisoonmmainng tasks.
"Equal first author. Authors are permitted list their name first in their CVs.
Conresonniigg author.
I Projecct I Lead and Corresponding author.

---

Published as a conference paper at ICLR 2026
Source
HuTv HuatuoGPTV--Bo
LLaVA-Med
LLaVA-Tri
Source
LLaVA-v1.5
Q: Eaity infiltration
skin lesion visible image?
picture?
Q: there
pneumothorax?
you
Q: brain edema picture?
(a)
(b)
Figure I: Visual grounding issues in state-of-the-art medical MLLMs. (a) Column 1 shows in- put medical images with expert-annotated ground-truth regions (red boxes). Columns 2-5 display attention distributions from representative medical MLLMs. (b) Column shows natural scene im- ages with annotated ground-truth bounding boxes, and column shows attention distributions from LLa For the first time, we systematically validate that state-of-the-art medical MLLMs often suffer from inadequate visual grounding- regions that are clinically relevant to the question. We note that, in contrast, when applied to natu- ral images, MLLMs are capable of grounding their predictions in the correct image regions (Zhang al. 2025ab. Attention maps taken from the LLM layers identified most relevant visual grounding (see Sec. 2 for details).
fail to accurately localize and interpret image
Medical MILMs. Recent work has explored extending general-purpose MLLMs to the medical domain, with many approaches focusing on constructing multimodal medical datasets and incorpo- rating external expert models. In Li et al. (003a).. large-scale biomedical figure-caption dataset is built from PubMed Central fine-tune LLaVA, resulting in LLaVA-Med However, its perfor- mance in zero-shot settings remains suboptimal and heavily reliant on dataset-specific fine-tuning. HuamooGTT-Vision (Chen al., 2024a) leverages GPT-4V construct large image-text dataset with refined annotations, but also lacks strong zero-shot generalization. VILA-M3 (Nath et al., 2024) incorporates external medical expert models assist medical image analysis tasks. Recently, Xic et al. (2025), the authors introduce Mcd a dataset comprising 25 million medical images, and propose VA-TT,, model pretrained on this dataset to improve regional focus in medical images. (See Supp. for additional review of related work.)
Despite these advances, most existing medical MLLMs strongly rely on training or fine-tuning with samples from downstream datasets. They continue to underperform zero-shot setting-wher -whee no downstream task samples are seen during training or fine-tuning- falling short of the goal of developing truly generalist medical MLLMs. This raises key ques- tion: Why do medical MLLMs struggle with medical image interpretation, despite their success in general-domain tasks?
medical VQA tasks in the
Research Gap. There remains a lack of deeper analysis into the underlying causes of medical MLLMs' suboptimal performance in the important zero-shot setting. Particularly, there is lack of studies to systematically examine the internal failure modes of these models models-particulalyy particularly in terms of how and where predictions are derived from visual inputs. Without such analysis, it remains
2

---

Published as a conference paper at ICLR 2026
unclear whether performance limitations stem from a lack of clinical task understanding (semantic grounding) or from an inability to accurately localize and interpret relevant image regions (visual grounding). Advancing our understanding of these failure modes is essential for building robust generalist medical MILMs for real-world clinical deployment.
Our work underscores the importance of explicitly distinguishing between semantic grounding LLu eet al. 2024 Lyre 2024) and visual grounding (Xiao et al.) (224) in medical tasks. This distinction particularly critical for Med-VQA, which-unlike general-domain VQA-often requires deep domain-specific reasoning. For example, answering a question like "What diseases are included in the image?" requires the model reason about the anatomical structures and visual features that are relevant to specific pathologies, A model may experience failure in semantic grounding that is, it lacks the medical knowledge to determine what to look for. Alternatively, it may experience failure in visual grounding- it cannot accurately localize and interpret the relevant regions in the medical image, even when knows what to look for. As medical MLLMs increasingly incorporate large-scale biomedical knowledge to enhance semantic grounding, we argue that visual grounding may emerge as the primary bottleneck limiting further progress.
Our Contribution. In this work, we present a pioneering systematic investigation aimed at advanc- ing the understanding of failure modes and the visual grounding capabilities of medical MLLMs (Fig. D.. To disentangle visual grounding from semantic grounding. we co-create novel evalua- tion dataset with 3 clinicians, named VGMED, a dataset for Visual Grounding analysis of MEDical MILMs. VGMED ensures focused evaluation of whether MLLMs can accurately localize and in- terpret the relevant regions in medical images. We introduce new quantitative metries and qualitative analyses assess visual grounding performance-that is, the extent to which model predictions are grounded in clinically relevant visual evidence. Critically, by using VGMED to evaluate eight SOTA medical MLLMs, we reveal for the first time that even the most advanced models frequently rely on spurious or irrelevant regions, highlighting inadequate visual grounding as pervasive and funda- mental failure mode. We note that this finding is specific to medical image analysis; in contrast, prior work has shown that MILMs are capable of grounding their predictions in the correct image regions when applied to natural images (Zhang et al. 2025a).
To address this, propose VGRefine, simple yet effective inference-time method that improves visual grounding by refining internal attention distributions. VGRefine requires no additional train- ing. Across 6 diverse Med-VQA benchmarks, comprising over 110K VQA samples from 8 imaging modalities (CT, MRI, X-ray, OCT, dermoscopy, microscopy, fundus, ultrasound), VGRefine consis- tently achieves improved and SOTA performance. Overall, our work offers new insights into the failure modes of medical MI.LMss and establishes visual grounding analysis as a necessary diagnos- tic tool for advancing medical MILMs in clinical applications.
2 INVESTIGATION OF VISUAL GROUNDING IN MEDICAL MLLMS
Despite recent advances, medical MILMS continue to underperform on complex medical image reasoning tasks, particularly in medical VQA (Hu et al. 2024 Jeong et al.) 2024). In this work, we conduct a systematic study to validate that a key limitation lies in inaccurate visual grounding. As starting point, we analyze attention maps from the model layers most relevant to visual ground- ing (details on layer selection are provided in Sec. 2.4). As shown in Fig. I for medical images, MLLMs' attentions often fail to align with clinically relevant regions.
2.1 A NEW DATASET FOR VISUAL GROUNDING ANALYSTS
Existing medical VQA datasets are ill-suited for visual grounding analysis. To rigorously eval- uate medical MILMS visual grounding, we aim to systematically assess the extent which their outputs are supported by clinically relevant regions of the image (e.g., organs, tissues, or lesions essential for answering a given question). However, existing medical VQA datasets are ill-suited for this purpose, as illustrated in Fig. 2 (a). Many questions, such as "What diseases included in the picture?", can be answered without referencing specific image regions. In contrast, questions like "What diseases what to look for, since different diseases, including their stages or subtypes, can manifest with var- ied and often subtle visual patterns. These patterns are not always declenece in text and may
included in the picture?" require substantial medical knowledge to determine
3

---

Published as a conference paper at ICLR 2026
(A) existing dataset
(b)
VGMED clinicians
bat popppppee
Medaaty-speclicc Pronept
Qumvina Htllennent
Blad dh
gius
pepeto diploy's
besting peneigraeeeee cslar
Bullee thr CIT perepe sodiie
parano:
busaeind und unch
Revken grndons siceeni srprod lica voaal pranding
You dese on tofolit
all cer in nicecca
bedlluck firm Sielical biokee
and oodiol bas
Quality Contrel
Modally Dicrftadian
Bocy Part Dutriattke
Figure 2: Co-creation of VGMED with clinicians for visual grounding assessment. Existing Med- VQA datasets often include questions about image modality plane, which can be answered without referencing specific image regions. They also contain many abnormality- or knowledge- based questions that require substantial medical expertise to determine what to look for. As a result, existing datasets are not well-suited for analyzing visual grounding. In contrast, our dataset lever- ages LLM prompting and clinical expert guidance generate clinically meaningful localization and attribute questions that are explicitly grounded in annotated image regions, enabling rigorous as- sessment of the visual grounding capabilities of medical MLLMs. Best viewed in color and with zoom.
depend on clinical interpretation. making it difficult to determine whether model failures stem from inadequate semantic grounding or from visual grounding alone.
VGMED: A new dataset for Visual Grounding analysis of MEDical MLLMs co-created with clinicians. To address this gap, we build an evaluation dataset VGMED, focusing on visual ground- ing analysis, as illustrated in Fig. (b). VGMED was co-created with three certified medical doctors (general practice, neurology, radiology) to ensure annotation accuracy and clinical relevance, includ- ing two senior clinicians with over ten years of experience. One expert also serves as Director (AI and Data Science) at national medical center. Their contributions included: (1) co-designing GPT prompts to elicit clinically meaningful and visually grounded questions, (2) reviewing and refining all samples for clinical relevance and grounding focus, and (3) verifying that all samples require reference to the annotated region.
Our VGMED dataset is constructed from over 40 publicly available medical image segmentation datasets, with detailed information summarized in Table C.2 The original segmentation masks are converted into bounding boxes to support visual grounding analysis. To ensure diversity across imaging modalities and anatomical regions, we filter 13,962 samples, each consisting of medical image paired with ground-truth bounding box. The distributions of modalities and body parts are illustrated in Fig. (b).
For each image-bbox pair, we construct clinically meaningful questions that target specific anatom- ical or pathological regions, guided by input from clinical experts. This allows us to conduct fine- grained visual grounding analysis. The questions are first generated using GPT-4 and subsequently reviewed and validated by medical professionals. They fall into two categories: localization and attribute questions. Localization questions inquire about the presence or identification of a specific organ or lesion, whereas attribute questions focus visual properties such as size, shape, or abnor- mality (see Fig. Fig for details). GPT-4 is prompted to ensure that questions are both clinically relevant and visually grounded, requiring attention to the entire annotated region. In total, our dataset tains approximately 28K image babox-question triplets.
As the reference point, randomly draw the same number of samples from MS COCO Lin al. 2014], using the same question generation pipeline for the evaluation of natural scene images. We include all prompts used in localization and attribute questions generation in Supp
4

---

Published as a conference paper at ICLR 2026
Remark: Co-created with 3 clinicians, VGMED is a dataset for evaluation and analysis of visual grounding in medical domain. The size of VGMED (28K samples) is comparable to datasets typi- cally used in general-domain visual grounding evaluation and studies (see Supp
2.2 QUANTIFYING MLLMS' VISUAL GROUNDING WITH ATTENTION MAPS
Measuring MLLMs' visual grounding. To evaluate how multimodal large language models (MILMM) ground their predictions in visual evidence, we analyze internal attention maps that in- dicate which image regions the model attends to. Attention maps are widely used in recent studies to evaluate visual grounding in general- -ommrl MLLMs (Zhang et al. 2025a; Kang et al. 2025 Kaduri et al.] 2024). Importantly, Zhang et al. (2025a) demonstrated that attention distributions can reliably capture visual grounding in natural scene images. This enables us directly compare the visual grounding in medical images and natural scene images.
Alternative grounding indicators, such gradient-based saliency and causal perturbation, are in principle applicable but are considerably more expensive at scale. Gradient-based saliency methods (Selvaraju et al., 2017: Ismail et al. (02T) require backpropagation for each input, making them substantially more computation-intensiee than directly using attention maps from the forward pass. Causal perturbation techniques (Fong & Vedaldi 2017 Hooker et al. 2019) demand a new forward pass for each perturbed input (e.g., region masking or token removal), which may quickly become prohibitive for large-scale medical grounding analysis.
Attention maps in MLLMs. We extract cross-attention weights from the last input text token to each of the N image tokens across all I, layers and H attention heads of the L.LM (Zhang et al. 2024 Kang et al., 2025). For each layer E and head h, we denote the attention vector as a Fh E N
and compute the average across heads to obtain a per-layer attention map A = H E a f.h, Then we reshape A into spatial attention map of size N X N.
Attention Ratio (AR). We aim to measure the alignment from the model's attention map to the ground truth bounding box. For this purpose, we apply attention ratio (AR), defined as the sum of attention inside the ground truth bounding box divided by the average attention inside the bounding box of the same size Zhang et al. 2025ap. Let A E N denote the attention map over image patch tokens, and let M E E xI represent the binary ground-trun truth mask indicating the anno- tated region (e.g., bounding box), where Mij = 1 if patch (i, is inside the region, and otherwise.
Formally, AR is defined as AR Σ/ EN Ag. Maj
where A| D. Σj" Aij and similarly M
TAIL
N'
for || 1-
New metrics to quantify model's attention map alignment. We note that AR only considers the amount of attention within the bounding box, ignoring how the attention is distributed. Particularly, a uniform distribution of attention within the bounding box region would be preferable, as questions in VGMED are specifically designed to require attention to entire bounding box regions. To take at- tention distribution into account, we propose to use the Kullback-Leibler (KL) and JSnsen-Shannon (JS) divergence, which measure the difference between the attention map and bounding box by viewing them as two probability distributions.
Kullback-Leibler (KL) divergence. the normalized g ground-truth mask (M || A) = En E/= My log
We compute the KL divergence between and the normalized attention map Á as where Å. = At// ||] and My, = M2g M|]]
M My Ass
Jensen-Shannon (JS) divergence.
To obtain compute
symmetric and as
bounded divergence (is(M Á) = DRt. (M || it) + + (AI It), R.j = (M,, + A.) divergences allow us to quantify not only whether the model attends to the correct region, but also how its attention is distributed within that region. A lower divergence indicates better alignment
metric,
we
the JS.
divergence The KL and JS
and more consistent attention over clinically relevant areas, offering a complementary perspective to AR.
5

---

Published as a conference paper at ICLR 2026
Localization
Attribute
Attention Ratio (AR)
Divergnnce
Divergence
Attention Ratio (AR)
Divergence
J5 Diveggnnes
(Bii)
LLaki Med
04
20
16
20
10
LLAW
VILAA
1.98
28
28
Hamaun PTV
546
MedFooo
348
Layer
Layor
Layer
Layor
Medical
Natural Scone
Figure 3: Medical MLLMs demonstrate suboptimal visual grounding when applied to med- ical images. Analysis using our proposed VGMED cettaasttddsiiinn specifically to assess vi- sual grounding in medical MLLMs-Shool that all evaluated medical MILMS exhibit substantial weaker alignment between their attention distributions and ground-truth annotations on medical images compared natural scene images (from MS COCO). Additional comparison with general domain MILM LLaVA-vI. 1.5 on natural images (below the dashed line) further confirms that med- ical MLLMs consistently exhibit reduced alignment with annotated regions. Best viewed in color and with zoom.
2.3 EXPERIMENTAL SETUPS
We conduct our analysis 8 SOTA medical MLLMs, including LLaVA-Med (Li al. 2023a), LLaVA-Tri (Xie et al., 2025). HuatuoGPT- Vision- HuatuoGPT-V), BLLA-M--BB/133 (Nath Vision-7B/34B to as HuatuoGPTT--Bii
(Chen et al. 2024a) (abbreviated as 2024). MedRegA (Wang et 2025), and vari- the original CLIP vision encoder
ant of HuatuoGPT-V--efeee replaced with BiomedCLIIP domain-specifie encoder trained on biomedical data (see Supp H for details). To analyze attention behavior, compute the each LLM layer. Inspired by Zhang et al. (2025a), we normalize the attention map using a reference attention map obtained from the generic prompt: "Write a general description of the image.". This normalization helps highlight regions relevant to the specific question.
attention map across all heads in
We also include -LaVA-v1.5-7B (Liu et al. 2024a) results both medical and natural scene im- ages. As general-domain MLLM. LLa VA demonstrates strong performance and exhibits good visual grounding, with attention distributions that align closely with ground-truth regions (Zhang

---

Published as a conference paper at ICLR 2026
al. 2025a Kang et al., 2025). This serves useful reference point for interpreting attention ratios, KL and JS divergence associated with effective visual grounding.
2.4 EMPIRICAL ANALYSIS
Medical MILMs exhibit inadequate visual grounding medical images. We plot the attention ratio, KL divergence and JS divergence across all LLM layers for all models in Fig. 3 As shown in the figure, all evaluated medical MLLMs demonstrate weaker alignment between their attention distributions and ground-truth annotations when applied to medical images, compared to natural images. This is quantitatively and consistently supported by lower AR and higher values in our pro- posed KI. and JS divergence metrics for measuring attention alignment. These trends persist across most network layers and are consistent for both attribute and localization tasks. Further comparison with LLa 1.5 natural images reinforces this observation: medical MLLMs show significantly lower aligmment with annotated regions, as measured by AR, KL, and JS highlightgt deficiencies in visual grounding for medical image analysis. Note that general-domad domain model (LLaVA-vil vA.11. also fails to ground its predictions clinically relevant regions on medical images and medical VQA tasks, whereas medical-domaii models can ground their predictions when applied ral images and general VQA tasks. This indicates that the grounding failure is not due to model weakness, but fundamentally specific the medical domain, consistent with our central findings.
natu-
For qualitative analysis, we visualize the attention map from the layer with the lowest KL divergence [i Lower KL divergence reflects closer alignment between the model's s attention distribution and the annotated regions, indicating that these layers are most relevant for visual grounding analy- sis. Comprehensive qualitative analysis and visualization are included in Supp
VISUAL GROUNDING REFINEMENT
STEP
Hoart
LunDE
Which organ abnoema), hears lung?
hoars
Ditter fooa on Magastade
Muskin
Soices Type
Imagee
Figure 4: Illustration of the proposed VGRefine method: two-step inference-time method to improve visual grounding in medical MLLMs. In Step I (Attention Triage), we aggregate attention from the model's most visually sensitive heads and suppress low-confident attention, obtaining binary mask. In Step II (Attention Knockout), we use this mask to refine the model's 's attention dis- tribution, improving its focus on relevant regions during inference. In the lower triangular attention matrix, each row represents the attention score of query token to all key tokens.
Our analysis in Sec. suggests that current medical MLLMs attend clinically and irrel- evant regions. In this section, we propose Visual Grounding Refinement (VGRefine), an inference- time method that enhances visual grounding in medical MLLMs by suppressing attention to clin- ically irrelevant regions. Specifically, as shown in Fig. our method consists of two steps: 1) Attention Triage and 2) Attention Knockout.
Step I: Attention Triage More Focusing on Clinically Relevant Regions. As illustrated in Fig. [ medical MLLM's attention maps are often noisy-while they do attend to relevant areas, they also include a substantial focus irrelevant regions, which diminishes interpretability and precision. To better focus on clinically meaningful regions, we move beyond layer wise average attention and instead examine visual sensitivity at the head level across all layers. Following the same evaluation in Sec. 2.4] we identify the top K attention heads that most consistently align with visually relevant regions, using our proposed evaluation dataset (Sec. 2.1 and metric (Sec. 2.2
7

---

Published as a conference paper at ICLR 2026
We then aggregate the attention maps from these selected heads with their average. We suppress low-activation regions based on magnitude of attention, as these are likely to represent irrelevant or noisy attention (see Supp. for further details and motivation). This results in sparse attention map with high-confidence. We convert this filtered attention map into a binary mask M E {0. {0,1) by simply setting all non-zero entries to I and keeping the zeros unchanged.
Step I:: Attention Knockout - Suppressing Irrelevant Visual Input. To enhance the visual grounding ability of medical MLLMs. we aim to guide the model's attention toward clinically rele- vant regions. Intuitively, improving focus on these regions can suppress distractions from irrelevant areas and yield more interpretable predictions. Similarly, recent advances in attention manipulation (Zhang et al.) 2024] Geva et al., 2023 Zhang et al. 2025c) have shown that attending to redun- dant information potentially detriment prediction as they distract the model's focus, they improve model behavior by preventing attention to uninformative tokens.
Inspired by this, propose knock out attention connections between question tokens and clin- ically irrelevant visual tokens. Specifically, we apply the binary mask M obtained in Step to the attention weights a.th,, where ag denotes the cross-attention from the qth question token to all visual tokens layer é and head h. We compute the masked attention as a a. @ M, and use a for the subsequent attention computation in model's forward pass. @ denotes element-wise multiplication. The masking operation explicitly restricts question tokens from receiving informa- tion from irrelevant visual regions at the selected layer. This modification encourages the model to attend selectively to meaningful regions, reducing distraction from irrelevant areas and therefore enhancing models' visual grounding capability.
EXPERIMENTS
4.1 EVALUATION SETTINGS
Baselines. We compared two types of open-source models: (1) Medical MLLMs. We evaluated
with the latest medical MLLMs, including Med-Flamingo (Moor al. 2023), RadFM (Wu al.
2023
LLaVA-Med-78 7B (Li et al. 2023a), VLaVA-Tri (Xie et al. 2025) MedPLIB (Huang et al.
2025 VILt- et al. 2024). HuatuoGPT-V (Chen et al.) 2024a). (2) General MLLMs. We
compared with two latest models pretrained on natural scene domain, LL (Liu et al.)
2024a) and Qwen-VL--Chat (Bai et al. 2023). We include the comparison of larger models in Supp-
Benchmarks. We follow the exact evaluation protocol of Chen et al. (2024a). Specifically, we adopt six benchmarks that are designed for biomedical MLLM evaluation, including VQA-RAD (Lau et al. 2018), SLAKE (Liu et al. 2021a). PathVQA (He et al. (220)), PMC-VQA (Zhang al.) 2023b), OmniMedVQA (Hu et al. 2024) (open-access split), and MMMU (Health & Medicine track) Yue et al. (2024). All evaluations were conducted in zero-shot setting using question tem- plates provided by LLaVA (details in Supp.).
VGRefine. We applied our inference- time method HuatuoGPT-V (Chen al. 2024a) All experiments are conducted using the same hyperparameters across benchmarks. Specifically, for Step I, we aggregate the attention maps from the top K heads with the highest alignment visual relevant regions, as measured by KL divergence on our curated evaluation set built using COCO images. This setup prevents data leakage from medical evaluation benchmarks and demonstrates that our method generalizes from natural images to biomedical domains. Low-activation regions are suppressed based on percentile threshold p over attention magnitude. We discuss the choice of K and p in Sec. (4.4. For Step II we apply the attention knockout only at layer & = 16 layer, which, according to our analysis in Fig. demonstrates the most relevancy to visual grounding among all the layers.
4.2 EXPERIMENTAL RESULTS
We follow exactly the evaluation setup of HuatuoGPT-V (Chen et al. 2024a to ensure consistency across all benchmarks. Since the original papers of HuatuoGPT- V-BB (Chen et al.] 2024a) and VILA-M3-8B (Nath et al. 2024) do not report results on certain benchmarks, we re-evaluate both models under the same zero-shot setting. For models with complete benchmark results available
8

---

Published as a conference paper at ICLR 2026
in their original publications suchh such MedPLIB (Huang al. 2025 and LLaVA- (Xie al. 2023) we directly report the official numbers. For all other baselines, we use the results provided the HuatuoGPT-V paper, as it adopts the same evaluation protocol.
is important to note that some models include benchmark training sets during pretraining, making zero-shot evaluation unfair. Specifically, VILA-M3 (Nath al. (024) and MedPLIB (Huang al., 2025) incorporate training data from VQA-RAD, SLAKE, Path VQA, and PMC-VQA, and thus are excluded from our zero-shot comparison on those datasets.
Medical VQA Benchmarks. Table shows results on four standard medical VQA datasets. Here, report the closed-ended question accuracy and weighted average (Avg.) that scales by the number of samples in each benchmark (Additional results are in Supp.). Our inference-time method VGRefine applied HuatuoGPT-V consistently improves its performance. We observe notable gains of +5.6% on VQA-RAD and + +1.3% on PathVQA, with the overall average increasing from 65.3% 68.4%, outperforming all baselines. These results underscore that enhanced visual ground- ing contributes better performance medical VQA tasks. On the MMMU benchmark (Table VGRefine achieves the highest accuracy across all five sub-domains, increasing the overall average from 45.8% to 47.2%. This demonstrates that enhancing visual grounding at inference time also improves complex multimodal medical reasoning. As shown Table 3 VGRefine improves per- formance across all eight imaging modalities, with significant boosts CT (+7.5%). MRI (+6,4%), and X- X-Ray (+8.1 1..1% 1%) on the OmniMedVQA benchmark. These results confirm the generalizability of visual grounding refinement across diverse medical imaging tasks. Overall, method raises average accuracy from 71.3% to 74.4%, demonstrating its robustness and generalizability across wide range of modalities.
Table Accuracy on medical VQA datasets. To align with the evaluation protocol with HuatuoGPT- V (Chen et al. 2024a), we specifically used the closed-ende subset for evaluation. Evaluation on other subsets Supp.
Model
VQA-RAD
SLAKE
PathVQA
PMC-VQA
Avg.
Qwen-VL-Chat
47.0
56.0
55.1
36.6
48.9
LLaVA-v1.6-7B
52.6
57.9
47.9
35.5
48.5
Mcd-Flamingo
45.4
43.5
54.7
23.3
41.7
RadFM
50.6
34.6
38.7
25.9
37.5
LLaVA-Med-7B
48.6
56.8
24.7
45.4
LLaVA-Tr
59.8
43.4
59.0
HuatuoGPT-V-7D
67.4
76.5
60.7
53,9
65.3
VGRefine (Ours)
71.2
76.9
67.6
56.2
68.4
Table 2: Accuracy on MMMU Health & Medicine benchmark. BMS, CM, DLM, P. PH denote Basic Medical Science, Clinical Medicine, Diagnostics & Laboratory Medicine, Pharmacy, Public Health respectively.
Model
BMS
CM
DLM
P
PH
Avg.
Qwen-VL--Cht
36.5
31.7
32.7
28.4
34.6
32.7
LLaVA-v1.6-7B
40.5
36.9
32.1
32.3
26.9
33.1
Med-Flamingo
29.6
28.1
24.8
25.3
31.2
28.3
RadFM
27.5
26.8
25.8
24.7
29.1
27.0
LLaVA-Med-77
39.9
39.1
34.6
37.4
34.0
36.9
LLaVA-Tri
37.1
27.8
VILA-M3-8B
39.3
39.7
34.0
32.1
28.7
34.0
HuatuoGPT-V-7B
58.9
57.2
43.8
37.2
38.3
45.8
VGRefine (Ours)
59.5
59.1
45.7
38.6
39.3
47.2
4.3 HUMAN EVALUATION: VGREFINE IMPROVES TRUSTWORTHINESS
We conducted blinded study with five experienced clinicians using 20 medical VQA cases from VGMED, Each case presented attention maps: from the baseline model and one from the

---

Published as a conference paper at ICLR 2026
Table 3: The accuracy of OmniMedVQA within different modalities. Specifically, FP denotes
Fundus Photography, MRI denotes Magnetic Resonance Imaging, OCT denotes Optical Coherence Tomography. Der denotes Dermoscopy, Mic denotes Microscopy Images, US denotes Ultaasound
Model
CT
FP
MRI
OCT
Der
Mic
X-R
X-Ray
US
Avg.
Awen-VL--Ciee (Bai et al. 2023
51.5
43.9
54.0
55.4
49.5
49.5
LLaVA-v1.6-2 LLu et al. 2024a
40.1
39.5
54.8
58.4
54.0
48.8
53.3
47.9
49.6
Med-Flamingo Moor ct al. 202.3
34.6
33.3
27.5
26.0
28.3
28.1
30.1
33.2
30.2
RadFM (Wu et 2023
33.3
35.0
22.0
31.3
36.3
28.0
31.5
26,1
30.5
LLaVA-Med-7B (Li al.
2023a
25.3
48.4
35.9
42.1
45.2
44.0
31.7
34.4
35.8
VII.A-M3-8B (Nath
2024
60.2
35.7
51.5
56.9
51.5
51.7
65.4
46.1
53.0
McdPLIB (Huang al. 2025
62.7
65.0
67.0
75.1
51.5
64.4
60.3
38.8
60.6
HuatuoGP1-v-/1 (Chen al.
2024a
62.6
80.3
67.7
86.2
71.7
74.2
74.2
79.7
71.3
VGRefine (Ours)
67.3
82.4
72.0
74.9
80.2
74.4
Table Ablation study on the choice of top K heads and p precentile of magnitude- baaeed filtering.
VQA-RAD
SLAKE
PathVQA
PMC-VQA
Avg.
(%)
VQA-RAD
SLAKE
PathVQA
PMC-VQA
Avg.
68.62
75.81
64.85
53.65
68.28
30
70.78
76.84
67.55
55.70
68 22
67.28
70.09
76.56
66.52
$4.30
68.08
71.24
76.88
67.61
56.20
68.42
70.70
76.52
66.64
55.60
68.
70.70
76.66
67.28
55.65
68.05
10
70.86
76.81
67.67
56.05
68.34
70
70.47
76.66
67.61
55.80
68.17
70.78
76.84
67.43
55.40
68.
70.78
76.73
67.55
55,80
68.21
71.24
76.88
67.61
56.20
68.42
70.55
76.34
68.11
55.50
68.20
same model after applying VGRefine. The source of each attention map was not disclosed, and their order randomized. Clinicians were asked which map appeared more clinically reasonable and trustworthy. VGRefine was preferred in 76% of cases, with feedback noting improved focus and reduced noise. These results suggest that VGRefine enhances clinician trust by producing more interpretable visual. See human evaluation details in S.Ip
4.4 ABLATION STUDIES
Table presents ablations on the number of top attention heads K and the percentile threshold used for magnitude-based filtering. Performance improves consistently as K increases, with the best average accuracy (68.42%) achieved at = 20, indicating that aggregating more heads helps capture richer grounding signals. For the percentile p, the model remains stable across values, with optimal performance also at 50%, confirming the effectiveness of moderate filtering in removing noisy regions without discarding relevant information.
CONCLUSION
In this work, presented the first systematic analysis of visual grounding in medical MILMss. Us- ing our clinically guided VGMED dataset and newly introduced metrics, we showed across 8 SOTA medical MLLMs frequent failures grounding predictions in clinically relevant regions. This fail- ure mode persisted even in recent medical MLLMs and contributed to their underperformance in zero-shot medical image understanding. To address this, we proposed VGRefine, an inference-time attention refinement method to improve medical MLLMs' visual grounding. Across 6 diverse Med- VQA benchmarks, comprising over 10K VQA samples from 8 imaging modalities, VGRefine con- sistently achieves SOTA performance. We remark that improvements using VGRefine are achieved without retraining or introducing any new medical knowledge. If visual grounding were not a limit- ing factor, such consistent gains would not occur. Therefore, VGRefine results further support that visual grounding deficiency is general, widespread issue. Overall, our proposed VGMED helps uncover and confirm inadequate visual grounding, while VGRefine experiments demonstrate its broad prevalence and generalization across different modalities and clinical scenarios. Our findings underscored the importance of grounding-awar analysis achieve more reliable and generalizable medical MLLMs. Additional experiments, limitation and ethical consideration are included in Supp.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This research is supported by the National Research Foundation, Singapore under its AI Singapore Programmes (AISG Award No.: AISG2-TC-2002-0007; The Agency for Science, Technology and Research (A STAR)) under its MTC Programmatic Funds (Grant No. M23L7b0021). This research supported by the National Research Foundation, Singapore and Infocomm Media Development Authority under its Trust Tech Funding Initiative. Any opinions, findings and conclusions or rec- ommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore and Infocomm Media Development Authority. The work is sponsored by the SUTD Decentralised Gap Funding Grant.
REFERENCES
W. Al-Dhabyani, M. Gomaa, H. Khaled, and A. Fahmy, Dataset of breast ultrasound images. Data in Brief, 28:104863, February 2020. doi: 10. (.110//..dbb.11..1444 Retrieved: September 25, 2025.
Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald M Summers, et al. The medical segmentation decathlon. Nature communications, (3(1):4125 128. 2022.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond. arXiv preprint arXiv:2308. 12966, 2023.
Ujjwal Baid, Satyam Ghodasara, Suyash Mohan, Michel Bilello, Evan Calabrese, Errol Colak, Keyvan Farahani, Jayashree Kalpathy-Crameet Felipe C. Kitamura, Sarthak Pati, Luciano M. Prevedello, Jeffrey D. Rudie, Chiharu Sako, Russell T. Shinohara, Timothy Bergquist, Rong Chai, James Eddy, Julia Elliott, Walter Reade, Thomas Schaffter, Thomas Yu, Jiaxin Zheng, Ahmed W. Moawad, Luiz Otavio Coelho, Olivia McDonnell, Elka Miller, Fanny E. Moron, Mark C. Oswood, Robert Y. Shih, Loizos Siakallis, Yulia Bronstein, James R. Mason, An- thony F. Miller, Gagandeep Choudhary, Aanchal Agarwal, Cristina H. Besada, Jamal J. Der- akhshan, Mariana C. Diogo, Daniel D. Do-Dai, Luciano Farage, John L. Go, Mohiuddin Hadi, Virginia B. Hill, Michael Iv, David Joyner, Christie Lincoln, Eyal Lotan, Asako Miyakoshi, Mariana Sanchez-Montane tiz Jimenez, Kerem Ozturk, Bojan D. Petrovic, Chintan Shah, Lubdha M. Shah, Manas Sharma, Onur Simsek, Achint K. Singh, Salil Soman, Volodymyr Statsevych, Brent D. Weinberg, Robert J. Young. Ichiro Ikuta, Amit K. Agarwal, Sword C. Cambron, Richard Silbergleit, Alexandru Du- soi, Alida A. Postma, Laurent Letoumeau-Guiilon Gloria J. Guzman Perez-Carrillo Atin Saha, Neetu Soni, Greg Zaharchuk, Vahe M. Zohrabian, Yingming Chen, Milos M. Cekic, Akm Rah- man, Juan R. Small, Varun Sethi, Christos Davatzikos, John Mongan, Christopher Hess, Soonmee Cha, Javier Villanueva-Meyer, John B. Freymann, Justin S. Kirby, Benedikt Wiestler, Priscila Crivellaro, Rivka R. Colen, Aikaterini Kotrotsou, Daniel Marcus, Mikhail Milchenko, Arash Nazeri, Hassan Fathallah-Shayke Roland Wiest, Andras Jakab, Marc-Andre Weber, Abhishek Mahajan, Bjoern Menze, Adam E. Flanders, and Spyridon Bakas. The maaa-ssn-micces brats 2021 benchmark on brain tumor segmentation and radiogenomic classification, 2021. URL https:://axxis http///axxvv.voaaa//a/aaaa
Jaya Nath, Xuan V. Nguyen, Manal Nicolas-Jilwan, Johanna Or-
abs/ wttpg//ab//21...00 02314
Spyridon Bakas, Hamed Akbari, Aristeidis Sotiras, Michel Bilello, Martin Rozycki, Justin Kirby, John Freymann, Keyvan Farahani, and Christos Davatzikos. Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features. Scientific data, 4 (1):1-13, 2017.
Jorge Bemnal F Javier Sánchez, Gloria Fernández-EEpparrchh, Debora Gil, Cristina Rodríguez, and Fernando Vilariño. Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized medical imaging and graphics, 43:99-111, 2015.
Aditya Bharatha, Masanori Hirose, Nobuhiko Hata, Simon K. Warfield. Matthieu Ferrant, Kelly H. Zou, Eduardo Suarez-Santana, Juan Ruiz-Alzola, Anthony D' A'Aico, Robert A. Cormack, Ron

---

Published as a conference paper at ICLR 2026
Kikinis, Ferene A. Jolesz, and Clare M. C. Tempany. Evaluation of three-dimensional fi- nite element- based deformable registration of pre- and intraoperative prostate imaging. Medi- cal Physics, 28(12):2511-2500 / //aapm. on 1 i ne wwrrrrrrr........ wi ley. www.cm/cccc...oommmm.o..........
2001. doi: https://bb..rg//0001111111111100
URL https:
http:/do//bbb//000 11t.111181111 wwtpcoo/wo./ab////////e
1414009
Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xi- dong Wang, Zhenyang Cai, Ke Ji, Xiang Wan, and Benyou Wang. Towards injecting medical visual knowledge into multimodal LLMS at scale. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods Natu- ral Language Processing, pp. 7346-7370, Miami, Florida, USA, November 2024a. Associa- tion for Computational Linguistics. doi: 10. 1/2554///2222.mmmlp.moo //aclanthology. /ttp:////2000.. wwwwp-main....... 418/
URL https:
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. Aleš Leonardis, Elisa Ricei, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (eds.). Computer Vision - ECCV 2024, pp. 19-35, Cham, 2025a. Springer Nature Switzerland. ISBN 978-3-031-73004-7
Shiqi Chen, Tongyao Zhu, Ruochen Zhou, Jinghan Zhang, Siyang Gao, Juan Carlos Niebles, Mor Geva, Junxian He, Jiajun Wu, and Manling Li. Why is spatial reasoning hard for vims? an attention mechanism perspective on focus areas. In Forty-second International Conference on Machine Learning, 2025b.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer 2024b.
Vision and Pattern Recognition, pp.
Noel Codella, Veronica Rotemberg, Philipp Tschandl, M. Emre Celebi, Stephen Dusza, David Gut- man, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, Harald Kittler, and Allan Halpern. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic), 2019, URL https: //arxiv..ivvvv//aa//aa org/ abs/ 1902. 03368
Noel C. F. Codella, David Gutman, M. Emre Celebi, Brian Helba, Michael A. Marchetti, Stephen W. Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, and Allan Halpem. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international sympo- sium biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 168-172, 2018a. doi: 10.1 10.110///BB..01 10.1109911112001833333777 83665547.
Noel C. F. Codella, David Gutman, M. Emre Celebi, Brian Helba, Michael A. Marchetti, Stephen W. Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra. Harald Kittler, and Allan Halpem. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international sympo- sium biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 168-172, 2018b. doi: 10.1 10.1109/1S...2018.888 83665477.
Hejie Cui, Lingjun Mao, Xin Liang, Jieyu Zhang, Hui Ren, Quanzheng Li, Xiang Li, and Carl Yang. Biomedical visual instruction tuning with clinician preference alignment. Advances in neural information processing systems, 37:96449-96467, 2024.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In seiethh Conference on Neural Information Processing Systems, 2023. URL
OpenAl et al. Gpt-4 technical report, 2024. URL.
Huihui Fang, Fei Li, Huazhu Fu, Xu Sun, Xingxing Cao. Fengbin Lin, Jaemin Son, Sunho Kim, Gwenole Quellec, Sarah Matta, Sharath M. Shankaranarayana, Yi-Ting Chen, Chuen-Heng Wang.
12

---

Published as a conference paper at ICLR 2026
Nisarg A. Shah, Chia- Yen Lee. Chih-Chung Hsu, Hai Xie, Baiying Lei, Ujjwal Baid, Shubham Innani, Kang Dang. Wenxiu Shi, Ravi Kamble, Nitin Singhal, Ching Wei Wang, Shih-Chang Lo, José Ignacio Orlando, Hrvoje Bogunović, Xiulan Zhang, and Yanwu Xu. Adam challenge: Detecting age-related macular degeneration from fundus images. IEEE Transactions on Medical Imaging, (1(0))28828 2847, 2022. doi: 10. 00.1199/MM.2222.317777
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful pertur- bation. In Proceedings of the IEEE international conference on computer vision, pp. 3429-3437, 2017.
Huazhu Fu, Fei Li, José Ignacio Orlando, Hrvoje Bogunovic, Xu Sun, Jingan Liao, Yanwu Xu, Shaochong Zhang, and Xiulan Zhang. ichallenge-palm: Pathologic myopia challenge, 07 2019. URL https: www..iii........................ ac. /ttp://////8002222221212221
Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Interpreting clip's image representation via text-based decomposition. In The Twelfth International Conference on Learning Representa- tions, 2024.
Sergios Gatidis, Tobias Hepp, Marcel Früh, Christian La Fougère, Konstantin Nikolaou, Christina Pfannenberg, Berhhard Schölkopf, Thomas Küstner, Clemens Cyran, and Daniel Rubin. A whole- body fdg-pet/ct dataset with manually annotated tumor lesions. Scientific Data, 9(1 (1))601, 2022.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual asso- ciations in auto-regressive language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12216-12235, 2023.
Michal Golovanevsky, William Rudman, Vedant Palit, Ritambhara Singh, and Carsten Eickhoff. What do vims notice? mechanistic interpretability pipeline for noise-free text-image corr tion and evaluation. CoRR, abs/2406.166220 2024. URL https: http:///10o..rr/////// 4855C/
arxiv 2406. 16320
Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ ques- tions for medical visual question answering. 2020. URL
10286
Nicholas Heller, Niranjan Sathianathen, Arveen Kalapara, Edward Walezak, Keenan Moore, Heather Kaluzniak, Joel Rosenberg, Paul Blake, Zachary Rengel, Makinna Oestreich. Joshua Dean, Michael Tradewell, Aneri Shah, Resha Tejpaul, Zachary Edgerton, Matthew Peterson, Shaneabbas Raza, Subodh Regmi, Nikolaos Papanikolopoulos, and Christopher Weight. The kits19 challenge data: 300 kidney tumor cases with clinical context, ct semantic segmentations, and surgical outcomes, 2020. URL bttps:/: //rxiv. org/ abs /aap//bbs//////// 1904. 00445
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretabil- ity methods in deep neural networks. Advances neural information processing systems, 32, 2019.
Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical Ivim. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 22170-22183, June 2024.
Rui Huang, Zijie Chen, Yuanyuan Chen, Hongsheng Li, et al. StructSeg2019 Grand Challenge 2019. Re-
Dataset.
trieved: September 25, 2025.
Xiaoshuang Huang, Lingdong Shen, Jia Liu, Fangxin Shang, Hongxiang Li, Haifeng Huang. and Yehui Yang. Towards a multimodal large language model with pixel-level insight for biomedicine. Proceedings of the AAAI Conference on Artificial Intelligence, 3974)3377--3787 Apr. 2025, doi: 10. 10.1669/aai..v. 33944.3234 URL htLp::////// wwwpaaiiirgginndedeeeeeee php/ http//aticcl//eee
32394
J Igelsias, M Styner, T Langerak, B Landman, Z Xu, and A Klein. Miccai multi-atlas labeling beyond the cranial worksowo and challenge. In Proc. MICCAI Atlai-Atlas Labeling Beyond Cranial Vault Workshop Challenge, 2015.
13

---

Published as a conference paper at ICLR 2026
Aya Abdelsalam Ismail, Hector Corrada Bravo, and Soheil Feizi. Improving deep learning inter- pretability by saliency guided training. Advances in Neural Information Processing Systems, 34: 26726-26787 26739, 2021.
Daniel P Jeong, Saurabh Garg, Zachary Chase Lipton, and Michael Oberst. Medical adaptation of large language and vision-language models: Are we making progress? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference Empirical Methods in Natural Language Prucessing, pp. 22143-12170, Miami, Florida, USA, November URL
2024. Association for Computational Linguisties. doi: 10. http:///...222..2maaaaaaaa hat///acaaiiii http:s////alaathooooo..ooooo wwmp:///..222222......mmm ewnnl-nnneea. ma manoomaii... 6777
Debesh Jha, Pia H Smedsrud, Michael A Riegler, Päl Halvorsen, Thomas De Lange. Dag Johansen, and Hävard D Johansen. Kvasir-seg; A segmented polyp dataset. In International conference on multimedia modeling, pp. 451 4621.462.220 Springer, 2019.
Debesh Jha, Nikhil Kumar Tomar, Vanshali Sharma, Quoc-Huy Trinh, Koushik Biswas, Hongyi Pan, Ritika K. Jha, Gorkem Durak, Alexander Hann, Jonas Varkey, Hang Viet Dao, Long Van Dao, Binh Phuc Nguyen, Nikolaos Papachrysos, Brandon Rieders, Peter Thelin Schmidt, Enrik Geissler, Tyler Berzin, Pål Halvorsen, Michael A. Riegler, Thomas de Lange, and Ulas Bagci. Polypdb: A curated multi-center dataset for development of ai algorithms in colonoscopy, 2025.
URL https: / org/ abs/ /tlbs/2444 2409. 00045
Yuanfeng Ji, Haotian Bai. Chongjian GE, Jie Yang, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhanng, Wanling Ma, Xiang Wan, and Ping Luo. Amos: A large-scale abdominal multi- organ benchmark for versatile medical image segmentation. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Informa- tion Processing Systems, volume 35, pp. 36722-36732. Curran Associates, Inc., 2022. www.ffflwweeee/aeeeae 2aaper//022/fllee ee604e (rt.aaaaaabbbbeee222200003-----0
URL httppp///ppreeecdinneeeeerr...pp
and Benchmarks.
pdf
Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. Devils in middle layers of large vision-language models: Interpreting. detecting and mitigating object hallucina- tions via attention lens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 25004-25014, June 2025.
Omri Kaduri, Shai Bagon, and Tali Dekel. What's in the image? deep-dive into the vision of vision language models, 2024. URL https: http://aavvvvvvo org www.rrggbb........ abs/ 2411 17491
Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. Your large vision-language model only needs few attention heads for visual grounding, 2025. URL htttss://axii...rr
abs/2503. 06287
A. Emre Kavur, N. Sinem Gezer, Mustafa Barış, Sinem Aslan, Pierre-Henri Conze, Vladimir Groza, Due Duy Pham, Soumick Chatterjee, Philipp Ernst, Savaş Özkan, Bora Baydar, Dmitry Lachi- nov, Shuo Han, Josef Pauli, Fabian Isensee, Matthias Perkonigg, Rachana Sathish, Ronnie Ra- jan, Debdoot Sheet, Gurbandurdy Dovletov, Oliver Speck, Andreas Nürnberger, Klaus H. Maier- Hein, Gözde Bozdağı Akar, Gözde Ünal, Oguz Dicle, and M. Alper Selver. Chaos challenge- combined (ct-mr) healthy abdominal organ segmentation. Medical Image Analysis, 69:101950, 2021. ISSN 1361-8415.
doi: htt.a/awww..cc..cc.c.eeeeeeeeaea 7/ 7/www. www..scieneen. www.ssseeecccicccc rect. com
1019500 URL https:
Jason J. Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman A dataset of clinically generated visual questions and answers about radiology images. Scientific Data, 5:1-10, 11 2018.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tris- tan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large language- and-vision assistant for biomedicine in one day. In A. Oh, Naumann, A. Glober- son, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Pro- cessing Systems, volume 36, pp. 28541-28564 Curran Associates, Inc., 2023a. URL twwes/papee/eeeeeeeppeeaaa httpppppppas222 file/ http:://ppoceedinns..eerrrssss.s cc cc/paper folfffille files / baboffBeddaccaa22266272855557773
-ataeett and Benchmarks
pdf
14

---

Published as a conference paper at ICLR 2026
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre- pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Pro- ceedings of Machine Learning Research, pp. 19730-19742 PMLR, 23-29 Jul 2023b. URL
https: http:///ooeeeeennnnee.e mlr pre.s/ppsss... wwwppppwsssse...ee DDISss/220//11 2tress//202/1123.hh00 html
Tsung-Yl Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects context. In David Fleet, Tomas Pajdla, Bernt Schicle, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, pp- 740-755, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10602-11
Geert Litjens, Robert Toth, Wendy van de Ven, Caroline Hoeks, Sjoerd Kerkstra, Bram van Gin- neken, Graham Vincent, Gwenael Guillard, Neil Birbeck, Jindang Zhang, Robin Strand. Filip Malmberg, Yangming Ou, Christos Davatzikos, Matthias Kirschner, Florian Jung, Jing Yuan, Wu Qiu, Qinquan Gao, Philip "Eddie" Edwards, Bianca Maan, Ferdinand van der Heijden, Soumya Ghose, Jhimli Mitra, Jason Dowling, Dean Barratt, Henkjan Huisman, and Anant Mad- abhushi. Evaluation of prostate segmentation algorithms for mri: The promise 12 challenge. Medical Image Analysis, ((22):559373. 2014. ISSN 1361-8415. doi: https://doi.org//0..u 1011// 1mmedia.013.122.00 URL https::/ www.sscciieeecccoo.omoo com/ wwa.///ceene//artiicoo 1111101133773
Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma. Yan Yang, and Xiao-Ming Wu. Slake: A semantically- labeled knowledge-enhane dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp. 1650-1654, 2021a. doi: 10.1109/
ISB148211.22 1.221...44444000
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In shirth-eeventh Conference on Neural Information Processing Systems, 2023. URL https:
//openrevi wwt./forumm d=w0H2xGH11 kw
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 26296-26306, June 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang. Sheng Shen, and Yong Jae Lee. Leava-next: Improved reasoning, ocr, and world knowledge, January 2024b. URL https:/7
llava-vis vl. www.wvl...giiiiiiiio.oo wtl.uu..oo/bbbe wttp://w...../ooo/////o///// -012224-11330 01-30- llava lavaa-extt//a///eaa
Pengbo Liu, Hu Han, Yuanqi Du, Heqin Zhu, Yinhao Li, Feng Gu, Honghu Xiao, Jun Li, Chunpeng Zhao, Li Xiao, et al. Deep learning segment pelvic bones: large-scale et datasets and baseline models. International Journal of Computer Assisted Radiology and Surgery, 16(5):749-756, 2021b.
Shi Liu, Kecheng Zheng, and Wei Chen. Paying more attention to image: training-fre free method for alleviating hallucination in Ivims. In European Conference on Computer Vision, pp. 125-140. Springer, 2024c.
Wentao Liu, Qianjun Pan, Yi Zhang, Zhuo Liu, Ji Wu, Jie Zhou, Aimin Zhou, Qin Chen, Bo Jiang, and Liang He. Cmm-math: A chinese multimodal math dataset to evaluate and enhance the mathematics reasoning of large multimodal models. CoRR, abs/2409.0283. 2024d. URL attpp:://et //doi. wwr.org/io /0/////.000 48550/ arxiv. 2409. 02834
Jaying Lu, Jinmeng Rao, Kezhen Chen, Xiaoyuan Guo, Yawen Zhang, Baochen Sun, Carl Yang, and Jie Yang. Evaluation and enhancement of semantic grounding in large vision-language mod- els. In AAAI-24 Workshop on Responsible Language Models, 2024. URL
erg/abs 2309. 04041
Xiangde Luo, Wenjun Liao, Jianghong Xiao, Jieneng Chen, Tao Song, Xiaofan Zhang, Kang Li, Dimitris N. Metaxas, Guotai Wang, and Shaoting Zhang. Word: A large scale dataset, bench- mark and clinical applicable study for abdominal organ segmentation from et image. Medi- cal Image Analysis, 82: 10242, 2022. ISSN 1361-8415. doi: htps://doi.org/10. 1016//.media.
15

---

Published as a conference paper at ICLR 2026
2022.102642. $1361841522002705
http:///wwwa.aaaaaeaaaaccc/////a //axsiv.org/abs/220..2
Holger Lyre. Understanding ai: Semantic grounding in large language models, 2024. URL https:
Jun Ma, Yao Zhang, Song Gu, Xingle An, Zhihe Wang. Cheng Ge, Congcong Wang, Fan Zhang, Yu Wang. Yinan Xu, Shuiping Gou, Franz Thaler, Christian Payer, Darko Stern, Ed- ward G.A. Henderson, Dónal M. McSweeney, Andrew Green, Price Jackson, Lachlan McIn- tosh, Quoc-Cuong Nguyen, Abdul Qayyum, Pierre-Henri Conze, Ziyan Huang, Ziqi Zhou, Deng-Ping Fan, Huan Xiong, Guoqiang Dong, Qiongjie Zhu, Jian He, and Xiaoping Yang. Fast and low-gpu-memory abdomen et organ segmentation: The flare challenge. Medical Image Analyvis, 82: 102666, 2022a. ISSN 1361-8415 doi: htpps//bbb.rrg/1000106/.mmee 2022. 1022110 URL / 6136184152200224
1016/ /www www.scccinceirecceoo. com/ http://eeeetttcccooocceeeeeeeere
Jun Ma, Yao Zhang, Song Gu, Cheng Zhu, Cheng Ge, Yichi Zhang, Xingle An, Congcong Wang, Qiyuan Wang, Xin Liu, Shucheng Cao, Qi Zhang, Shangqing Liu, Yumpeng Wang, Yuhui Li, Jian He, and Xiaoping Yang. Abdomenct-Iss lk: Is abdominal organ segmentation a solved problem? IEEE Transactions on Pattern Analysis and Machine Intelligence, (4(10):6695-677) 2022b. doi: 10.1 1..1099TTAMMI222..1000000
Jun Ma, Yao Zhang, Song Gu, Cheng Ge, Shihao Mae, Adamo Young, Cheng Zhu, Xin Yang, Kangkang Meng, Ziyan Huang, et al. Unleashing the strengths of unlabelled data in deep learning- assisted pan-cances abdominal organ quantification: the flare22 challenge. The Lancet Digital Health, 6(11):e815-e88 2024.
Oskar Maier, Bjoern H. Menze, Janina von der Gablentz, Levin Häni, Mattias P. Heinrich, Matthias Liebrand, Stefan Winzeck, Abdul Basit, Paul Bentley, Liang Chen, Daan Christiaens, Fran- cis Dutil, Karl Egger, Chaolu Feng, Ben Glocker, Michael Götz, Tom Haeck, Hanna-Leena Halme, Mohammad Havaei, Khan Iftekharuddin, Pierre-Marc Jodoin, Konstantinos Kam- nitsas, Elias Kellner, Antti Korvenoja, Hugo Larochelle, Christian Ledig, Jia-Hong Lee, Fred- erik Maes, Qaiser Mahmood, Klaus H. Maier-Hein, Richard McKinley, John Muschelli, Chris Pal, Linmin Pei, Janaki Raman Rangarajan, Syed M.S. Reza, David Robben, Daniel Rueck- ert, Eero Salli, Paul Suetens, Ching- Wei Wang, Matthias Wilms, Jan S. Kirschke, Ulrike M. Krämer, Thomas F. Münte, Peter Schramm, Roland Wiest, Heinz Handels, and Mauricio Reyes. Isles 2015 public evaluation benchmark for ischemic stroke lesion segmentation from multispectral mri. Medical Image Analysis, 35:250-269, 2017. ISSN 1 1361-8415. doi: http:///oi..or//10. http://wwww.attt....../aaaaaaaaa
009. URL https:: www. sci enced rect. com/ bccaane/arrrceee///1//3333333332
Bjoern H. Menze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, Levente Lanezi, Elizabeth Gerstner, Marc-André Weber, Tal Arbel, Brian B. Avants, Nicholas Ayache, Patri- cia Buendia, D. Louis Collins, Nicolas Cordier, Jason J. Corso, Antonio Criminisi, Tilak Das, Hervé Delingette, Çağatay Demiralp. Christopher R. Durst, Michel Dojat, Senan Doyle, Joana Festa, Florence Forbes, Ezequiel Geremia, Ben Glocker, Polina Golland, Xiaotao Guo, An- dac Hamamci, Khan M. Iftekharuddin. Raj Jena, Nigel M. John, Ender Konukoglu, Danial Lashkari, José António Mariz, Raphael Meier, Sérgio Pereira, Doina Precup. Stephen J. Price, Tammy Riklin Raviv, Syed M. S. Reza, Michael Ryan, Duygu Sarikaya, Lawrence Schwartz, Hoo-Chang Shin, Jamie Shotton, Carlos A. Silva, Nuno Sousa, Nagesh K. Subbanna, Gabor Szekely, Thomas J. Taylor, Owen M. Thomas, Nicholas J. Tustison, Gozde Unal, Flor Vasseur, Max Wintermark, Dong Hye Ye, Liang Zhao, Binsheng Zhao, Darko Zikic, Marcel Prastawa, Mauricio Reyes, and Koen Van Leemput. The multimodal brain tumor image segmentation benchmark (brats). IEEE Transactions on Medical Imaging, 34(10):1993-2024, 2015. doi: 10. D0.1109/TM1.2004223776
Anna Montoya, Hasnin, kaggle446, shirzad, Will Cukierski, and yffud. Ul- trasound Nerve Segmentation.
bttpss:/kaagee com/ competitions/ ultrasound--erre seggeetttteeneetttttiiiiiei 2016. Retrieved: September 25, 2025.
16

---

Published as a conference paper at ICLR 2026
Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. Med-flamingg a multimodal medical few-shot learner. In Stefan Hegselmann, Antonio Parziale, Divya Shanmugam, Shengpu Tang, Mercy Nyamewaa Asiedu, Serina Chang, Tom Hartvigsen, and Harvineet Singh (eds.), Proceed- ings of the 3rd Machine Learning for Health Symposium, volume 225 of Proceedings of Machine Learning Research, pp. 353-367. PMLR, 10 Dec 2023. htwptttsss/pppeeeeeecc Fiees/v225/mont
mmlr.
html
Vishwesh Nath, Wenqi Li, Dong Yang, Andriy Myronenko, Mingxin Zheng, Yao Lu, Zhijian Liu, Hongxu Yin, Yee Man Law, Yucheng Tang, et al. Vila-m3: Enhancing vision-language models with medical expert knowledge. arXiv preprint arXiv:2411.12915, 2024.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint 1XXi:22209.18999 2022.
Danielle F Pace, Adrian V Dalca, Tal Geva, Andrew J Powell, Mchdi H Moghari, and Polina Gol- land. Interactive whole-heart segmentation in congenital heart disease. In International Con- ference Medical Image Computing and Computer-Assisted Intervention, pp. 80-88. Springer, 2015.
Vedant Palit, Rohan Pandey, Aryaman Arora, and Paul Pu Liang. Towards vision-language mech- anistic interpretability: causal tracing tool for blip. Proceedings of the IEEE/CVV Interna- tional Conference on Computer Vision (ICCV) Workshops, pp. 2856-2861, October 2023.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruy Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Arnaud Arindra Adiyoso Setio, Alberto Traverso, Thomas de Bel, Moira S.N. Berens, Cas van den Bogaard, Piergiorgio Cerello, Hao Chen, Qi Dou, Maria Evelina Fantacei, Bram Geurts, Rob- bert van der Gugten, Pheng Ann Heng, Bart Jansen, Michael M.J. de Kaste, Valentin Kotov, Jack Yu-Hung Lin, Jeroen T.M.C. Manders, Alexander Sónorra-Mengana, Juan Carlos Gareía- Naranjo, Evgenia Papavasileiou. Mathias Prokop. Marco Saletta, Cornelia M Schaefer-Prokop. Ernst T. Scholten, Luuk Scholten, Miranda M. Snoeren, Ernesto Lopez Torres, Jef Vandemeule- broucke, Nicole Walasek, Guido C.A. Zuidhof, Bram van Ginneken, and Colin Jacobs. Vali- dation, comparison, and combination of algorithms for automatic detection of pulmonary nod- ules in computed tomography images: The lunal6 challenge. Medical Image Analysis, 42:1- 13, 2017. ISSN 1361-8415. doi: https://ooi.or// 10. mew.::/.........111111.......... 7.00.015. URL https: 7/www. www.ssciencediret..c
www. wtt.eee...oscsiiieeeeee http://ww.......eaa..a..aaaaaaaa
Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See- Ng, Lidong Bing, and Roy Ka-Wei Lec. Math-Ilava: Bootstrapping mathematical reasoning for multimodal large language models. CoRR, abs/2406.17294 2024. URL httpp::/ /ttt::://dci.oog/1000 http://.................0/ 48540/85550..
2406. 17294
Aliyun Tianchi. Chest image dataset for pneumothorax segmentation. htttp:://tianeh 2020. Accessed: September 25, 2025.
aliyun. com/
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint 17X1v:1905.00411. 2019.
Lehan Wang, Haonan Wang, Honglong Yang, Jiaji Mao, Zehong Yang, Jun Shen, and Xiaomeng
Li. Interpretable bilingual multimodal large language model for diverse biomedical tasks. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
7/ //openreviee ew. net. net./form? d
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vasion-language model's
17

---

Published as a conference paper at ICLR 2026
perception of the world at any resolution, 2024. URL https:///rx...org//aaa// 12191
Jakob Wasserthal, Haans-CChisste Breit, Manfred T. Meyer, Maurice Pradella, Daniel Hinck, Alexander W. Sauter, Tobias Heye, Daniel T. Boll, Joshy Cyriac, Shan Yang. Michael Bach, and Martin Segeroth. Totalsegmentatoo:: Robust segmentation of 104 anatomic structures in et images. Radiology: Artificial Intelligence, 5(3)2233022, 2023. doi: 10. //.114//4ai.230024. URL
httpss//doi. www.logg.1... htte://..111///111
11w.1118//11 1ya4///yyi.... 230024
Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards generalist foun- dation model for radiology. CoRR, abs/2308.02463, 2023. URL https::/dbi.czg///
48550/ 2 2308 02463
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. NEXT-GPT: Any-to-any multi- modal L.M. In Proceedings of the International Conference on Machine Learning, pp. 53366- 53397, 2024.
Linhui Xiao, Xiaoshan Yang. Xiangyuan Lan, Yaowei Wang, and Changsheng Xu. Towards visual grounding: A survey, 2024. URL peattts://aaxiie wrg./b/////........//// 20206
Yunfei Xie, Ce Zhou, Lang Gao, Juncheng Wu, Xianhang Li, Hong-Yu Yu Zhou, Sheng Liu, Lei Xing, James Zou, Cihang Xie, and Yuyin Zhou. Medtrinity-25i1 A large-scale multimodal dataset with multigranular annotations for medicine. In The Thirteenth International Conference on Learning Representations, 2025.
http:::::pppreeieeeeeee// net httee//eooummnnnwnnnoiee
Tianyun Yang, Ziniu Li, Juan Cao, and Chang Xu. Mitigating hallucination in large vision-language models via modular attribution and intervention. In Adaptive Foundation Models: Evolving Al for Personalized and Efficient Learning, 2025.
Qinan Yu, Jack Merullo, and Ellie Pavlick. Characterizing mechanisms for factual recall in language models. In The Conference on Empirical Methods Natural Lunguage Processing, 2023.
Zeping Yu and Sophia Ananiadou. Understanding multimodal 1lms: the mechanistic interpretabil- ity of llava visual question answering, 2025. URL httpp://aar.iv.org/aabaa 10950
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9556-9567, June 2024.
Anna Zawacki, Carol Wu, George Shih, Julia Elliott, Mikhail Fomitchev, Mohannad Hussain, Paras Lakhani, Phil Culliton, and Shunxing Bao. Simm-acr pncumothorax segmentation. https:
//k 1/kaggla 1 com/ compet tiiittissssse si im acr- www.orrrpeetttoooooo www.ehowwww-eeetttti...oo 2019. Kaggle.
Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip Ilievski. Milms know where to look:
Training-fre free perception of small visual details with multimodal 1lms. In The Thirteenth Interna- tional Conference on Learning Representations, 2025a. URL https:://ppereeeeeeee Berum?id=Dgee hrr.un/e...gggaammmmmmmtttttttte http::////....htbbccooeeceeeeeee com/ htt.://........cccccccccoeaeeeea
.nt/
Minghui Zhang, Yangqian Wu, Hanxiao Zhang, Yulei Qin, Hao Zheng, Wen Tang, Corey Amoll, Chenhao Pei, Pengxin Yu, Yang Nan, Guang Yang, Simon Walsh, Dominic C. Marshall, Matthieu Komorowski, Puyang Wang, Dazhou Guo, Dakai Jin, Ya nan Wu, Shuiqing Zhao, Runsheng Chang, Boyu Zhang, Xing Lu, Abdul Qayyum, Moona Mazher, Qi Su, Yonghuang Wu, Ying'n ao Liu, Yufei Zhu, Jiancheng Yang, Ashkan Pakzad, Bojidar Rangelov, Raul San Jose Estepar, Car- los Cano Espinosa, Jiayuan Sun, Guang- Zhong Yang, and Yun Gu. Multi-site, multi-domain airway tree modeling. Medical Image Analysis, 90:102957, 2023a. ISSN 1361-8415. doi:
102957. ht www. www.sciencediret.. com/ https///oi.or//10.0ii///.j httpa///eeeee1111111111520000000
sci cccienne/artiillelli///S///
18

---

Published as a conference paper at ICLR 2026
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Pre- ston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, Andrea Tupini, Yu Wang, Matt Mazzola, Swadheen Shukla, Lars Liden, Jianfeng Gao, Angela Crabtree, Brian Piening, Carlo Bifulco, Matthew P. Lungren, Tristan Naumann, Sheng Wang. and Hoifung Poon. A multimodal biomedi- cal foundation model trained from fifteen million image text pairs. NEJM AI, 2(1): 2():Aloa2440064H, 10556
2025b, doi: AIoa2400640
Utthttps: / ai ne jm.
Xiaofeng Zhang. Yihao Quan, Chen Shen, Xiaosong Yuan, Shaotian Yan, Liang Xie, Wenxiao Wang, Chaochen Gu, Hao Tang, and Jieping Ye. From redundancy to relevance: Enhancing explainability in multimodal large language models. Anmual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics, 2025c.
Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-vqa: Visual instruction tuning for medical visual question answering. CoRR, abs/2305.10415 2023b. /tttp::::///oii wtg/000........... 48550 2tt.xxv.230..1000
Zhi Zhang, Srishti Yadav, Fengze Han, and Ekaterina Shutova. Cross-modal information flow in multimodal large language models. arXiv preprint 2. 2024.
Zhongchen Zhao, Huai Chen, and Lisheng Wang. A coarse-to-fine framework for the 2021 kidney and kidney tumor segmentation challenge. In Kidney and Kidney Tumor Segmentation: MIC- CAI 2021 Challenge, KiTS 2021, Held in Conjunction with MICCAI 2021, Strasbourg. France, September 27, 2021, Proceedings, pp. 53-58, Berlin, Heidelherg, 2021. Springer- Verlag. ISBN 1111197888-----98-8
978-33-030-98384-0.. doi:
8. URL https: /tw/////.////// oor/110.
Wenwen Zhuang, Xin Huang, Xiantao Zhang, and Jin Zeng. Math-puma: Progressive upward multimodal alignment enhance mathematical reasoning. Proceedings of the AAAI Conference on Artificial Intelligence, 39(24):26183 26111111111111,,,,1111111 Apr. 2025. doi: 10. wttp://.....i.22....4.3. URI.
https://ojs.. aaai. org wwwgg//idxxx.eee wwtppppppppaa/aeta / htt./////rriceeeeee//////55
19

---

Published as a conference paper at ICLR 2026
Avg.
U5
PH
BMS
X-Ray
MR
HosuunPPT
OmniMedVQA
MMMU
Figure A.1: Our proposed inference- time method VGRefine achieve state-of-the- art performance on OnniMedVQA (Hu et al. 2024) and MMMU (Health & Medicine track) Yuue al. 2024). Many existing medical MLL.Ms remain to underperform on medical VQA tasks in the zero-shot setting as shown in this figure, but there is lack of systematic study understand the reasons. Compared to existing medical MLLMs, our proposed VGRefine demonstrates consistently stronger zero-shot performance across all modalities and sub-domains, highlighting its effectiveness in mitigating the issue of inadequate visual grounding as revealed in our study.
APPENDIX OVERVIEW
In this supplementary material, provide additional experiments, ablation studies, and repro- ducibility details to support our findings. These sections are not included the main paper due space constraints.
Please find the following link for code and other resources: github.ic wwtp:iicooeeicoooo -ML Fail/
https:///giieee leo liu.
CONTENTS
More Discussion Related Work
22
B VGMED Scale Comparison with Related Attention Analysis Works
22
Detailed Information of Datasets Used in VGMED
23
More details of VGRefine
24
Experiments Open-ended Medical VQA
25
Comparison with Other Attention-hased Methods
25
More Experiments on Larger Models
25
HuatuoGP1- with BiomedCLIn Vision Encoder
27
20

---

Published as a conference paper at ICLR 2026
I Prompts for VGMED and QA evaluation
27
I.1 Prompts for constructing VGMED
27
1.2 Prompis for Zero-shot Evaluation
27
J
Additional Qualitative Evaluation
37
Human Evaluation
37
0.2
Additional Qualitative Analysis on Medical MLLM's Attention Maps
38
K
Additional Evaluation of Latest General MLLMs
48
Attention Maps Derived from Different Text Tokens
49
M Representative Failure Cases
50
Clinical Validation During VGMED Curation
51
Limitations
53
Experimental Setting/Details and Computing Resources
53
Broader Impacts and Ethical Considerations
53
R Safeguards
53
S Licenses
53
Use of Large Language Models (LLMs)
54
21

---

Published as a conference paper at ICLR 2026
A MORE DISCUSSION ON RELATED WORK
Medical Multimodal Large Language Models (MLLMs). Recent advances in medical multi-
modal large language models (MLLMs) have focused on leveraging image-text pairs from sources
like PubMed central (Zhang et al. 2023b Moor et al. 2023 Chen et al. 2024a Li et al.]
2023a)
and medical textbooks (Moor al. 2023) to enable generative VQA and medical reasoning. Mod-
els such as LLaVA-Med (Li et al. 2023a MedVInT (Zhagg et al. (023b), Mlaminaaingo (Moor
al., 2023), GuuatooGP--Vision (Chen et al. 2024a), and BioMed- VITAL (Cui et al.
2024) intro-
duce GPT-4 eet al. (024) generated instruction followiig datasets and expert-validated responses to
improve medical VQA performance. More recent studies have begun to explore different ideas to
improve region awareness in biomedical MILMs: explicit fine-tuning with additional supervision, such as annotated bounding boxes (Wang et al. [ie et al. (025) or segmentation masks (Jeong et al. 2024), along with architectural modifications to support spatial reasoning. For instance, mod- els like MedRegA (Wang et al.) 2025) and VL- (Xie et al. 2025) rely on additional datasets. Other recent models focus scale domain expertise. VILA-M3 (Nath et al. (024). for instance, incorporates domain-specific expert models during training, arguing that generic Vision-Language Models (VLMs) lack the fine-grained expertise needed for healthcare. Given their dependence on task-specific fine-tuning (Nath et al. 2024 Xie et al.] (225)) and sub-optimal generalization in zero- shot settings (Li et al., 2023a), it remains unclear whether current medical MLLMs ground their predictions in meaningful visual evidence within medical images. To our knowledge. no prior work has conducted comprehensive analysis of visual grounding of medical MILMs.
Visual Grounding Analysis in General Domain MLLMs. Some recent studies have investigated the internal attention mechanisms of general-domate MLLMs, revealing their potential for implicit visual grounding. Zhang et al. (2025a) demonstrated that MILMs can identify the correct spa- tial regions relevant to a given query, even without explicit grounding supervision. They introduce training-free free intervention method (e.g., cropping guided by attention gradient maps) that en- hances performance on general-domate VQA tasks. Broader research into MLLM interpretability has studied how visual information is fused into language representations. Techniques such causal intervention and cross-modal attention visualization have been employed to offer insights into how vision and language tokens interact through attention mechanisms (Golovanevsky et al. (2024] [Zhng eet al. 2024. Yu & Ananiadou| 2025 Palit et al. 2023). These studies suggest that middle layers are especially crucial for integrating object-level visual cues with textual context, and that cross-modal attention patterns can encode meaningful spatial alignment signals. However, all of these insights have been drawn from general-domain visual data, such as natural scene images and standard VQA benchmarks. In contrast, to our knowledge, no prior work has performed visual grounding analysis of medical MILMs.
B VGMED SCALE COMPARISON WITH RELATED ATTENTION ANALYSIS WORKS
VGMED comprises approximately 28K image-bbox-quesiun tripicts, including 14K samples for localization questions and another 14K for attribute questions. The scale of VGMED larger than or comparable to the number of samples used in the closely related RGB-domain visual grounding (Zhang et al., 2025a Kang et al. [2055] Kaduri et al., 2024) / attention analysis work (Yang et al., 2025 Jiang et al. 2025 Chen et al. (025aa) (see Table B. Unlike RGB datasets that can be constructed by non-experts, our medical datasets require clinical expertise.
Table B.I: Number of samples used related works.
Related works
No. of Samples
Data Source
MLLMs Know Zhang et al. 2025a
4,370
Text-VQA
Your LVLM (Kang et al. 2025)
1,000
RefCOCO
What's in the Image (Kddur ct al.
2024)
81
COCO
Hallucination Attribution Yang et al. 2025
1,500
COCO
Devils in LVLM (Jiang et al. 2025
2,000
COCO
FastV CCen et al." 2025a)
1,000
4 Tasks
22

---

Published as a conference paper at ICLR 2026
C DETAILED INFORMATION OF DATASETS USED IN VGMED
Table C.2: Detailed information about the 44 datasets incorporated into VGMED, In the "Dataset" column, names such "StructSeg2019 (Task 1)" represent specific task-based subsets. In the "Anatomical Structures" column, "Others" signifies datasets lacking detailed anatomical data from their original sources.
Dataset
Modality
Anatomical Structures
AMOS2022 Ji 2022
CT, MR
Abdomen, Thorax, Pelvic Thorax
ATM2022 Zhang et al.
2023a
CT
AldomenomenCC1-1 (M et
2022b
CT
Abdomen
BTCV [Igelsias et al.
2015
CT
Thorax, Abdomen, Pelvic
BraTS2013
Menze et al. 2015
MR
Head & neck
BraTS2015
Menze et al. 2015
MR
Head & neck
BraTS2018
Menze et al. 2015
MR
Head & neck
BraTS2019
Menze et al. 2015
MR
Head & neck
BraTS2020 Menze et al. 2015
MR
Head & neck
BraTS2021 Bakas et al.
2017:
Baid et al.
2021
MR
Head & neck
CHAOS 4T 4) (Kavur et al.
2021)
MR
Abdomen
CTPelvicIk (Liu et al. 2021b
CT
Pelvic
CVC-ClinieDB (Bernal et al.
2015
Endoscopy
Others
Chest Laags_mage__Pll Pneumm
Tianchi
2020
X-ray
Thorax
FLARE21 1 (Ma et al.
2022a)
CT
Abdomen
FLARE22 (Ma et al.
2024)
CT
Abdomen, Thorax
HVSMR20T6 (Pace et al.
2015
MR
Thorax
ADAM (Task 2) (Fang al. 2022)
Fundus
Head & neck
PALMI9 (Fu al. 2019)
Fundus
Head & neck
ISLES Maier et al. 2017
MR
Head & neck
KiTS2019 (Heller al.
20200)
CT
Abdomen
KiTS2021 (Zhao el al.
2021)
CT
Abdomen
LUNA16 (Setio et 2017)
CT
Thorax
MSD-Brain MSD-BrainnTumor l'mmor Antonellii et al.
2022)
MR
Head & neck
MSD-Liver (Antonelli et al. 2022)
CT
Abdomen
MSD-Pancreas (Antonelli et al. 2022)
CT
Abdomen
MSD-Spleen (Antonella et 2022
CT
Abdomen
CT-ORG (Antonelli et al. 2022
CT
Head & neck, Thorax,
Abdomen
PROMISE09 Bharatha et al 2001
MR
Pelvic
PROMISE12 Liljens et al.
2014)
MR
Pelvic
SIIM-ACR Pneumothorax (Zawacki et al.
2019
X-ray
Thorax
StructSeg2019 (Task 1) (Huang et al.
2019
CT
Head & neck
StructSeg2019 (Task 2) (Huang ct
2019
CT
Thorax, Abdomen
TotalSegmentator (Waseerhall et al.
20233
CT
Head & neck, Thorax, Abdomen, Pelvic
Ultrasound Nerve Segmentation
Ultrasound
Others
(Montoya et al., 2016)
WORD [uo et al. 2022
CT
Thorax, Abdomen Pelvie
autoPET Gatidis et al. 2022
PET
BUSI (Al-Daayyani et al. 2020
Ultrasound
Thorax
Kvasir-SBG (Jha et al. 2019
Endoscopy
Others
ISIC18 (Task 1) Codella la et al.
2019
Dermoscopy
Skin
ISIC17 (Task 1) (Cdeela et al
2018b
Dermoscopy
Skin
ISIC16 (Task 1) (Codella et al.
2018a
Dermoscopy
Skin
SLAKE (Liu et al.
2021a)
CT, MR, X-ray
Head neck, Abdomen,
Thorax
PolypDB (Jha et al. 2025
Endoscopy
Others
23

---

Published as a conference paper at ICLR 2026
D MORE DETAILS OF VGREFINE
Natural Scene
Medical
24
11:.mm-4o..mmm maar maass s 10 me to
Ritaam mam otan pa meree man muma
mome sepammen
22
4-11 t mo Risseent peeeaaaaee
24
113ds
11-
10-10-00 con mesessen 1og tu sERuE ses o
www.
Heads
10 Sorceapon ciser pacorcaan 2200222
14
13-
24-
15
to w
17-
14
18-
28
19
12
220
22 00
22
23 m000 200
25
28 1e
26
Layers
Layers
Figure D.I: We conduct an experiment to analyze the alignment between attention distributions from different attention heads and layers and the ground truth annotations in the images. This fol- lows the evaluation setup deseribed in Section 2.3 of the main paper. The medical MLLM evaluated is HuatuoGPT-V-7B. Each cell in the above figures reflects the degree of alignment, measured us- ing our proposed KL Divergence metric (lower is better). This analysis helps identify the specific heads and layers that are most relevant visual grounding. COCO is used for natural scene im- age analysis, and our dataset VGMED is used for medical image analysis. Interestingly, we find that the attention heads most relevant visual grounding in natural scene images are often also the most relevant for medical images. However, despite this overlap, the overall visual grounding performance on medical images remains lower than on natural scenes, consistent with the findings presented in Figure 3 (main paper). Based on this analysis, we identify the top K attention heads with the strongest alignment (i.e., lowest KL divergence) and aggregate their attention distributions to compute refined attention map. Notably, natural scene images from COCO dataset. to avoid data leakage from medical evaluation bench- marks. This setup also demonstrates that our method generalizes effectively from natural images to the biomedical domain.
select the top K heads using randomly sampled
In this section, we provide more details of our proposed inference-tima method VGRefine (intro- duced in Sec. 3 of the main paper). Particularly, we discuss how we identify top K attention heads most relevant to visual grounding and leverage their attention distributions in Step of VGRefine. Fig. D.I depict the analysis.
We explore attention distributions from different attention heads across all layers, as prior work suggests that individual attention heads in transformers specialize in capturing distinct types of in- formation Voita et al. 2019)) Olsson et al. (2022); Gandelsman et al. (2024); Yu et al.] (2023); Yang al. 2025). This motivates us to examine attention at finer granularity to obtain the attention that focusing more on clinically relevant regions.
See details in Fig. D.I Following the same evaluation setup of Sec. 2.3 of main paper, we assess relevancy to visual grounding of each attention head in HuatuoGPT-V by measuring the alignment between their attention distributions and ground-truth annotations. We perform this analysis using both natural scene images (from MS COCO) and medical images (from our VGMED). The align- ment is measured by our proposed KL Divergence (L) as metric.
As shown in Fig. D.I the visual relevancy patterns are consistent across domains: heads that are relevant to visual grounding in natural scenes also show relative relevancy in medical images, despite exhibiting inadequate visual grounding on medical images compared to natural images (as discussed in Sec. 2.4). Based on this analysis, we select the top K heads with the highest visual grounding relevancy (lowest KI.) on natural images and average their attention maps obtain
24

---

Published as a conference paper at ICLR 2026
refined attention map. This map is used in Step II to guide the model's improved focus on clinically meaningful areas.
E EXPERIMENTS ON OPEN-ENDEL MEDICAL VQA
We present additional experimental results on the open-ended questions from the Medical VQA benchmarks. Specifically, evaluate VQA-RAD (Lau et al. (2018), SLAKE (Liu et al. (0211), and Path VQA (He et al. 2020). which include open-ended formats. As shown in Table E.3 our inference-time method consistently achieve better performance across all datasets, demonstrating its effectiveness in enhancing open-ended medical VQA.
Table E.3: Performance comparison on full medical VQA datasets for open-ended medical VQA. We evaluate all models under the zero-shot setting. These results underscore that enhanced visual grounding with our inference-time method VGRefine contributes better performance on medical VQA tasks. It is important to note that VILA-M3 (Nath et al.) 2024), MedPLII (Huang et al. 2025) and LLaVA-TH at. 2025) incorporate training data from VQA-RAD, SLAKE, and Path VQA, and thus making zero-shot evaluation unfair, and are excluded from zero-shot comparison.
Model
VQA-RAD
SLAKE
PathVQA
Metric
BLB11-1
BERT
OpenRecall
BLEU--1
BERT
OpenRezall
Aug.
RI--1
BERT
OpenRecall
Avg.
Owen-VL-Chut
28.6
63.4
27.0
39.7
28.9
52.0
33.6
38.2
18.7
15.1
9.9
24.6
LLaVA-v1.6-78
22.1
58.0
21.9
34.0
30.8
52.7
36.4
40.0
22.8
47.7
11.2
27.2
Med-Flamingo
27.4
61.9
12.7
34.0
11.3
40.2
21.1
24.4
24.3
50.4
2.4
25.7
LLdV7-Meed7B
21.6
40.5
28.2
30.1
37.0
58.4
39.2
44.9
28.5
60.1
12.3
33.6
HuatuoGPTTVV77
49.7
75.0
50.7
58.5
53.0
78.9
55.6
63.2
34,2
65.8
36.5
45.5
VGRefine-BB (Ours)
76.3
52,3
80.0
56.7
68.1
36.5
F COMPARISON WITH OTHER ATTENTION-BASER METHODS
We conducted an additional experiment comparing VGRefine with three very recent attention- based methods for medical MLLMs. Specifically, PAI (Liu et al. (024c) and AdaptVis (Chen et al. (2025b) aim to refine/manipulate attention maps over visual tokens, while ViCrop (Zhang al. 2025a uses attention maps to enhance visual perception.
For fair comparison, we implemented all methods on HuatuoGPT--V7B, following their official code and hyperparameter settings. The experimental results, shown Tab, F44 indicate that VGRe- fine consistently outperforms all other methods.
Table F.4: Accuracy on closed-ende ended medical VQA datasets.
Model
VQA-RAD
SLAKE
PathVQA
PMC-VQA
Avg.
HuatuoGPT-V--78 (Baseline)
67.4
76.5
60.7
53.9
65.3
PAI[Liu et al. 2024c)
43.7
24.48
20.8
52.8
33.3
Adapt Chen et al.
2025b
68,6
75.1
67.6
52.9
66.7
ViCrop (Zhang et al. 2025a
68.9
70.9
66.7
54.6
65.5
VGRefine (Ours)
71.2
76.9
67.6
56.2
68.4
G MORE EXPERIMENTS ON LARGER MODELS
In this section, we provide more experimental results on larger models (with parameters > 10B). We show comparison on all six benchmarks that are designed for biomedical MLLM evaluation. including VQA-RAD (Lau et al. 2018). SLAKE (Liu et al. 2021a). Vat (He et al. 2020 PMC-VQA (Zhagg et al. 2023b). OmniMed
(Hu et al. (024) (open-access split), and MMMU
(Health & Medicine track) (Yue et al., 2024). All evaluations were conducted zero-shot setting using question templates provided by LLaVA (see Sec. [.
25

---

Published as a conference paper at ICLR 2026
All experiments conducted using the same hyperparametters across benchmarks. Specifically, for Step I, we aggregate the attention maps from the top 20 heads with the highest alignment to visual relevant regions, measured by KL divergence COCO images. This setup prevents data leakage from medical evaluation benchmarks and demon- strates that method generalizes from natural images biomedical domains. Low-activation regions are suppressed based on a percentile threshold
curated evaluation set built using
50% over attention magnitude. For Step 34, 35, 36 layer, which, according our analysis in
apply the attention knockout only Fig. demonstrates the most relevancy to visual grounding among all the layers. We applied our inference- time method VGRefine-34B on HuatuoGPT- V-34B (Chen et al., 2024a). The hyperpa- rameters K and are kept consistent with the VGRefine-7B setting. In Step II, we apply attention knockout more layers, as the 34B model has twice as many layers as 7B variant and requires deeper intervention to achieve significant improvements.
Results in Tab. G.5 demonstrate that our proposed method consistently achieves good performance across all 6 benchmarks, demonstrating its effectiveness in enhancing all types of medical VQA.
Table G.5: Experiment results of larger models (more than 10B parameters). We evaluate all models under the zero-shot setting. Our inference-time method VGRefine outperforms other state-of-the-an medical MI.LMs in most cases. These results underscore that enhanced visual grounding contributes to better performance 2024), MedRegA (Wang et al. 2025) incorporate training data from VQA-RAD (Lau et al. 2018) SLAKE (Liu et al. 2021aa, PathVQA (He et al. 2020), and PMC-VQA (Zhang et al. 2023b
medical VQA tasks. It is important note that VILA-M3 (Nath et al.
thus making zero-shot evaluation unfair, and are excluded from our zero-shot comparison these benchmarks.
Benchmarks
Subset
Metric
LLaVA-116-3B
VILA-M3-138
MedRegA-34B
HustuoGPT.V.34
VGRefine-34B (Ouns
CloseAce
58.6
68.1
72.9
VOA-RAD
BLEU-1
44.5
S0.5
51.6
OpenRecall
43.6
CloseAce
67.3
76.9
79.1
SLAKE
BLRBI-1
BERT
518
48.6
56.3
79.5
$7.2
OpenRecall
CloseAce
59.1
69.7
PahVQA
BERT
57.7
28.1
36.6
37.6
OpenRecall
PatVVA
CloseAce
44.4
58.2
58.7
Avg. Med-VQQs
ClaseAce
574
67.0
70.7
BMS
56.4
36.8
54.3
64.3
66.0
CM
52.8
38.8
53.5
56.5
58.2
MMMU
DLM
ClaseAce
42.6
29.0
37.7
45.4
PH
38.1
32.2
10.7
43.8
44.8
Avg.
45.6
33.3
44.7
50.1
51.3
CT
30.6
56.9
62.5
69.7
71.7
FP
63.1
50.1
80.1
$4.6
$1.1
MRI
60.9
52.9
72.7
69.7
73.9
OCT
OmniMadVQA
Mic
62.8
50.6
71.3
71.1
71.4
X-Ray
74.7
62.5
76.7
83.8
84.7
US
44.5
47.1
Avg.
26

---

Published as a conference paper at ICLR 2026
H HUATUOGPT- WITH BIOMEDCLIP VISION ENCODER
Model Setup. To evaluate the effect of domain-specifie visual encoders, we modified the orig- inal Piisson architecture by replacing its CLIP-based vision encoder with BioMed- CLIP Zhang et al. (2055). a biomedical foundation model pretrained on 15 million scientific im- age-text pairs. All other components of the model (including the Qwen2 language model, the cross- modal connector module, and the training protocol) remain identical to the original configuration. This substitution allows us to isolate the impact of specialized medical image representations on visual grounding performance.
Training Details. Since the original training code for Huatuo was publicly avail- able, we replicated the training pipeline using the LLaVA-Ne.N Liu et al. (2024b) codebase. We follow a two-stage training protocol on the same pretraining and iustruction-tuning datasets used in HuatuoGPT-Vision including LLaVA and PubMedVision. In Stage I, we freeze both the BioMed- CLIP vision encoder and the Qwen2 language model, training only the connector to align visual and textual representations. In Stage II, we fine-tune both the connector and the language model while keeping the vision encoder frozen. The model trained for epoch. BioMedCL.IP processes images fixed resolution of 224 x 224 with patch size of 16, which differs from the resolution and tokenization settings used in the original CLIP-based HuatuoGPT-Vision
Analysis Results. As shown in main paper Fig. 1 and Fig. 3, the issue of suboptimal visual grounding on medical images cannot be solved by using BiomedCLIP vision encoder.
I PROMPTS FOR VGMED AND QA EVALUATION
1.1 PROMPTS FOR CONSTRUCTING VGMED
Localization Question Set
Is there {label} in the image?
Can you see a {label} in the image?
Does the image contain {label]?
Is {label} present in this image?
Do you see {label} in the picture?
Is the [label} visible in the image?
Is there any sign of a [label} in the image?
Can a (label) found in this image?
Does this image show [label}?
Is {label} shown in the picture?
The question from predefined question set is sampled for generating localization ques- tions in both the COCO and VGMED datasets. {label} represents the object (in COCO) or or- gan/lesion (in VGMED) identified by a bounding box the corresponding image.
Figure I.2:
1.2 PROMPTS FOR ZERO-SHOT EVALUATION
We used the LLaVA prompt template during the evaluation for open, closed-ended, and multiple- choice questions.
27

---

Published as a conference paper at ICLR 2026
Prompt for VGMED Attribute Questions (MRI)
Your task to generate clinically meaningful questions for an evaluation dataset to assess visual grounding capability in medical reasoning, without requiring deep semantic grounding.
Semantic Grounding: Anchors linguistic representations
to domain-specific medical knowledge to inform what to look for, ensuring accurate reasoning about diseases anatomical concepts.
Visual Grounding:
Localizes and interprets specific regions in medical images based on relevant features, enabling spatial alignment of language queries to visual elements for accurate analysis.
We will provide the label
medical structure (organ, lesion, or tissue) in (modality) image, derived from bounding box annotation. Your need to generate three clinically relevant questions about visual attributes this stucture.
Guidelines for the question:
Focus on visual grounding, without requiring deep medical semantic grounding.
Ensure clinical relevance.
Require attention to the entire annotated bounding box.
Address only observable visual characteristics (e.g., size, shape, density, erhancement, homogeneity).
Avoid referencing other body parts surrounding structures.
Do not include position, modality, plane.
Exclude diagnoses or treatments requiring deep semantic grounding.
Avoid compound or multi multi-condition questions.
Ensure variety across the three questions.
Example questions:
"Is the lesion hyper or hypointense?"
"Is the lesion erhancing?"
"What does the area of necrosis look like?"
"What pattern of enhancement does the lesion show?"
Now, generate three questions based on the label. Return exactly three questions without any additional text or formatting.
Label:
{label}
Figure L.3: For attribute questions in VGMED, we use a specific prompt for each modality. In the prompt, (modality) denotes the modality of the image. (label} denotes the organ lesion labeled by a bounding box the image.
28

---

Published as a conference paper at ICLR 2026
Prompt for VGMED Attribute Questions (CT)
Your task is to generate clinically meaningful questions for evaluation dataset to assess visual grounding capability in medical reasoning, without requiring deep semantic grounding.
Semantic Grounding: Anchors linguistic representations
to domain-specific medical knowledge to inform what to look for, ensuring accurate reasoning about diseases or anatomical concepts.
Visual Grounding: Localizes and interprets specific regions in medical images based on relevant features, enabling spatial alignment of language queries to visual elements for accurate analysis.
We will provide the label of a medical structure (organ, lesion, tissue) in (modality) image, derived from a bounding box annotation. Your need to generate three clinically relevant questions about visual attributes of this structure.
Guidelines for the question:
Focus visual grounding, without requiring deep medical semantic grounding.
Ensure clinical relevance.
Require attention to the entire annotated bounding box.
Address only observable visual characteristics (e.g., size, shape, density, enhancement, homogeneity).
Avoid referencing other body parts or surrounding structures.
Do not include position, modality, or plane.
Exclude diagnoses treatments requiring deep semantic grounding.
Avoid compound multi-condition questions.
Ensure variety across the three questions.
Example questions:
"Are there ground glass opacities within the lung?"
"Is the kidney enlarged?"
"What is the size of the necrosis?"
Now, generate three questions based the label. Return exactly three questions without any additional text or formatting.
Label: (label}
Figure 1.4: For attribute questions in VGMED, specific prompt for each modality. In the prompt, {modality) denotes the modality of the image. {label} denotes the organ or lesion labeled by a bounding box the image.
29

---

Published as a conference paper at ICLR 2026
Prompt for VGMED Attribute Questions (Ultrasound)
Your task is to generate clinically meaningful questions for an evaluation dataset to assess visual grounding capability in medical reasoning, without requiring deep semantic grounding.
Semantie Grounding: Anchors linguistic representations to domain-specific medical knowledge to inform what to look for, ensuring accurate reasoning about diseases or anatomical concepts.
Visual Grounding: Localizes and interprets specific regions in medical images based on relevant features, enabling spatial alignment of language queries to visual elements for accurate analysis.
We will provide the label of medical structure (organ, lesion, or tissue) in (modality) image, derived from a bounding box annotation. Your need to generate three clinically relevant questions about visual attributes of this structur.
Guidelines for the question:
Focus on visual grounding, without requiring deep medical semantic grounding.
Ensure clinical relevance.
Require attention to the entire annotated bounding box.
Address only observable visual characteristicy (e.g., size, shape, density, enhancement, homogeneity).
Avoid referencing other body parts or surrounding structures.
Do not include position, modality, or plane.
Exclude diagnoses or treatments requiring deep semantic grounding.
Avoid compound or multi-condition questions.
Ensure variety across the three questions.
Example questions:
"Does the thyroid nodule have irregular or microlobulated margins?"
"Does the thyroid nodule have marked hypoechogenicity."
"Does the thyroid nodule have multiple microcalcifications.
"Is the breast lesion homogeneous heterogeneous?"
"Does the breast lesion appear solid or cystic on ultrasound?"
Now, generate three questions based on the label. Return exactly three questions without any additional text formatting.
Label:
{label}
Figure I.5: For attribute questions in VGMED, we use specific prompt for each modality. In the prompt, {modality) denotes the modality of the image. [label} denotes the organ or lesion labeled by bounding box in the image.
30

---

Published as a conference paper at ICLR 2026
Prompt for VGMED Attribute Questions (X-ray)
Your task is to generate clinically meaningful questions for an evaluation dataset to assess visual grounding capability in medical reasoning, without requiring deep semantic grounding.
Semantic Grounding: Anchors linguistic representations to domain-specific medical knowledge to inform what to look for, ensuring accurate reasoning about diseases or anatomical concepts.
Visual Grounding: Localizes and interprets specific regions in medical images based on relevant features, enabling spatial alignment of language queries to visual elements for accurate analysis.
We will provide the label of medical structure (organ, lesion, or tissue) in (modality) image, derived from a bounding box annotation. Your need to generate three clinically relevant questions about visual attributes of this ructure.
Guidelines for the question:
Focus on visual grounding, without requiring deep medical semantic grounding.
Ensure clinical relevance.
Require attention to the entire annotated bounding box.
Address only observable visual characteristics (e.g., size, shape, density, enhancement, homogeneity).
Avoid referencing other body parts or surrounding structures.
Do not include position, modality, or plane.
Exclude diagnoses or treatments requiring deep semantic grounding.
Avoid compound or multi-condition questions.
Ensure variety across the three questions.
Example questions:
"What is the size of the preumothorax?"
Where is the pneumothorax!"
"Does the lung field appear more opaque or translucent in the annotated region?"
Now, generate three questions based on the label. Return exactly three questions without any additional text formatting.
Label: {label}
Figure L.6: For attribute questions in VGMED, we use specific prompt for each modality. In the prompt, {modality) denotes the modality of the image. [label} denotes the organ or lesion labeled by bounding box in the image.
31

---

Published as a conference paper at ICLR 2026
Prompt for VGMED Attribute Questions (Fundus Photography)
Your task is to generate clinically meaningful questions for an evaluation dataset to assess visual grounding capability in medical reasoning, without requiring deep semantic grounding.
Semantic Grounding: Anchors linguistic representations to domain-specific medical knowledge to inform what to look for, ensuring accurate reasoning about diseases or anatomical concepts.
Visual Grounding: Localizes and interprets specific regions in medical images based on relevant features, enabling spatial alignment of language queries to visual elements for accurate analysis.
We will provide the label of medical structure (organ, lesion, or tissue) in (modality) image, derived from a bounding box annotation. Your need to generate three clinically relevant questions about visual attributes of this ructure.
Guidelines for the question:
Focus on visual grounding, without requiring deep medical semantic grounding.
Ensure clinical relevance.
Require attention to the entire annotated bounding box.
Address only observable visual characteristics (e.g., size, shape, density, enhancement, homogeneity).
Avoid referencing other body parts or surrounding structures.
Do not include position, modality, or plane.
Exclude diagnoses or treatments requiring deep semantic grounding.
Avoid compound or multi-condition questions.
Ensure variety across the three questions.
Example questions:
"Is there any pallor observed in the optic disc?"
"Does the optic disc appear swollen elevated?"
"Is there any evidence of swelling or pallor in the optic disc?"
Now, generate three questions based on the label. Return exactly three questions without any additional text formatting.
Label:
{label}
Figure 1.7: For attribute questions in VGMED, we use specific prompt for each modality. In the prompt, {modality) denotes the modality of the image. [label} denotes the organ or lesion labeled by bounding box in the image.
32

---

Published as a conference paper at ICLR 2026
Prompt for VGMED Attribute Questions (Endoscopy)
Your task is to generate clinically meaningful questions for an evaluation dataset to assess visual grounding capability in medical reasoning, without requiring deep semantic grounding.
Semantic Grounding: Anchors linguistic representations to domain-specific medical knowledge to inform what to look for, ensuring accurate reasoning about diseases or anatomical concepts.
Visual Grounding: Localizes and interprets specific regions in medical images based on relevant features, enabling spatial alignment of language queries to visual elements for accurate analysis.
We will provide the label of medical structure (organ, lesion, or tissue) in (modality) image, derived from a bounding box annotation. Your need to generate three clinically relevant questions about visual attributes of this ructure.
Guidelines for the question:
Focus on visual grounding, without requiring deep medical semantic grounding.
Ensure clinical relevance.
Require attention to the entire annotated bounding box.
Address only observable visual characteristics (e.g., size, shape, density, enhancement, homogeneity).
Avoid referencing other body parts or surrounding structures.
Do not include position, modality, or plane.
Exclude diagnoses or treatments requiring deep semantic grounding.
Avoid compound or multi-condition questions.
Ensure variety across the three questions.
Example questions:
"What is the size of the polyp?"
"Does the colorectal polyp have smooth lobulated
surface appearance?"
"What is the mobility of the poly??"
Now, generate three questions based on the label. Return exactly three questions without any additional text formatting.
Label:
{label}
Figure 1.8: For attribute questions in VGMED, we use specific prompt for each modality. In the prompt, {modality) denotes the modality of the image. [label} denotes the organ or lesion labeled by bounding box in the image.
33

---

Published as a conference paper at ICLR 2026
Prompt for VGMED Attribute Questions (PET)
Your task to generate clinically meaningful questions for an evaluation dataset to assess visual grounding capability in medical reasoning, without requiring deep semantic grounding.
Semantic Grounding: Anchors linguistic representations to domain-specise medical knowledge to inform what to look for, ensuring accurate reasoning about diseases anatomical concepts.
Visual Grounding: Localizes and interprets specific regions in medical images based on relevant features, enabling spatial alignment of language queries to visual elements for accurate analysis.
We will provide the label
medical structure (organ, lesion, or tissue) in a (modality) image, derived from bounding box annotation. Your need to generate three clinically relevant questions about visual attributes this strutuur.
Guidelines for the question:
Focus on visual grounding, without requiring deep medical semantic grounding.
Ensure clinical relevance.
Require attention to the entire annotated bounding box.
Address only observable visual characteristics (e.g., size, shape, density, erhancement, homogeneity).
Avoid referencing other body parts surrounding structures.
Do not include position, modality, plane.
Exclude diagnoses or treatments requiring deep semantic grounding.
Avoid compound or multi-condition questions.
Ensure variety across the three questions.
Example questions:
"Does the lesion show increased radiotracer uptake on the
PET scan?"
"Is the lesion hypo- or myper-metabolic?"
"What is the standardized Uptake Value (SUV) of the lesion?"
Now, generate three questions based on the label. Return exactly three questions without any additional text or formatting.
Label:
{label}
Figure 1.9: For attribute questions in VGMED, we use a specific prompt for each modality. In the prompt, (modality) denotes the modality of the image. {label} denotes the organ lesion labeled by a bounding box in the image.
34

---

Published as a conference paper at ICLR 2026
Prompt for VGMED Attribute Questions (Dermoscopy)
Your task to generate clinically meaningful questions for an evaluation dataset to assess visual grounding capability in medical reasoning, without requiring deep semantic grounding.
Semantic Grounding: Anchors linguistic representations
to domain-specific medical knowledge to inform what to look for, ensuring accurate reasoning about diseases anatomical concepts.
Localizes and interprets specific regions in medical images based on relevant features, enabling spatial alignment of language queries to visual elements for accurate aralysis.
Visual Grounding:
We will provide the label
medical structure (organ, lesion, or tissue) in a (modality) image, derived from bounding box annotation. Your need to generate three clinically relevant questions about visual attributes this structure.
Guidelines for the question:
Focus on visual grounding, without requiring deep medical semantic grounding.
Ensure clinical relevance.
Require attention to the entire annotated bounding box.
Address only observable visual characteristics (e.g., size, shape, density, erhancement, homogeneity).
Avoid referencing other body parts surrounding structures.
Do not include position, modality, plane.
Exclude diagnoses or treatments requiring deep semantic grounding.
Avoid compound or multi-conditionn questions.
Ensure variety across the three questions.
Example questions:
"What is the size of the lesion?"
"Is the lesion hypo- or hyper pigmented?"
"Does the lesion have peripheral black dots clods?"
"Does the skin lesion have thick lines (reticular or branched) "
"Does the lesion have Polymorphous vessels?"
Now, generate three questions based on the label. Return exactly three questions without any additional text or formatting.
Label: {label}
Figure 1.10: For attribute questions in VGMED, we use a specific prompt for each modality, In the prompt, (modality) denotes the modality of the image. {label} denotes the organ lesion labeled by a bounding box the image.
35

---

Published as a conference paper at ICLR 2026
Prompt for COCO Attribute Questions
Your task is to generate one simple and meaningful question about a visual attribute an object identified in an image.
We will provide the label the object, which comes from bounding box annotation in an image from the coco dataset.
Guidelines for the question:
Ensure variety in the questions generated.
Focus only on the visual characteristics (e.g., color, size, material, etc.) of the given object.
Do not reference other parts of the image.
Do not ask questions about the position of the object or the surrounding structure.
Avoid compound multi-condition questions.
Now, generate one question based on the following label: Label: {label}
Figure 111: In order compare the results between VGMED datasets and natural scene images, we have also generated the attribute questions for COCO examples. {label} refers to the object label from the image's bounding boxes.
Short Answer (e.g., VQA-RAD, SLAKE, PathVQA)
<question>
Answer the question using single word phrase.
Prompt for evaluating the open and closed-ended questions in VQA-RAD, SLAKE, and VQA benchmarks.
Figure 1.12:
Option-only for multiple-choice (e.g., PMC-VQA, OmniMed and MMMU)
<question>
A. <option_1>
B. <option_2>
C. <option 3>
D. <option 4>
Answer with the option's letter from the given choices directly.
Figure I.13: Prompt for evaluating the multiple-choiee VQA benchmarks.
36

---

Published as a conference paper at ICLR 2026
J ADDITIONAL QUALITATIVE EVALUATION
J.1 HUMAN EVALUATION
We conducted a blinded human evaluation involving five experienced clinicians of them have over 10 years of clinical practice). The study was based on 20-case questionnaire. For each case, clinicians were shown a medical image with a VQA question and two corresponding attention maps: (1) from the baseline model and (2) from the same model after applying VGRefine. The source of each attention map was not disclosed, and their order was randomized. Clinicians were asked: "Which model's s attention visualization (shown heatmap) appears more clinically reasonable and trustworthy?""
Source image
Model A
Model
Does the picture contain lung?
Which model's highlighted region appears more clinically ressonable and trustworthy?
Model
Please share any comments on what influenced your choice (optional):
appeers
Figure J.14: Example of blinded human evaluation case, showing medical image with VQA question, baseline attention map, and VGRefin attention map, assessed by an experienced clinician for clinical reasonableness. Clinician feedback highlighted that VGRefine attention maps were less noisy, better localized, and more aligned with expected clinical focus points.
37

---

Published as a conference paper at ICLR 2026
J.2 ADDITIONAL QUALITATIVE ANALYSIS ON MEDICAL MLLM'S ATTENTION MAPS
HuatuoGPT-V
Source
Layer
Layer
Layer
Layer
Layer
Layer
you right kidmey in the picture?
of fully inGitmation liver?
Source
Layer
Layer 6
Layer
Layer
Layer 25
Layer 31
LLaVA-Tri
enhancing tumor be image!!
Do you atelectacis in the picture?
Layer
Layer 6
Layer
Layer
Layer
Layer
LLLVA-v1.5
What shape clock?
Is there parking meter image?
Figure J.15: Qualitative evaluation. We visualize attention maps across different layers, including those with the lowest KI. divergence (highlighted with an orange boundary), which are indicative of layers most relevant to visual grounding in MLLMs. For medical images, the attention maps of medical MILMs show limited alignment with the ground-truth annotated regions. In contrast, general-domain MLLM VLa applied to natural images exhibits strong alignment with relevant regions, consistent with other study of general-domain MLLMs (Zhang et al., (2025)). This highlights gap in MLLM's visual grounding performance between the medical and natural image domains. Best viewed in color and with zoom. Additional results in Supp J.2
38

---

Published as a conference paper at ICLR 2026
HuatuoGPi-y
Source
Layer
Layer
Layer 10
Layer 16
Layer
Layer
toilet shown the picture?
What color keyboard?
Is there any sign of fork the image?
LLaVA-Tri
Source
Layer
Layer
Layer 12
Layer 17
Layer 25
Can you see boat in image?
What color is the handbag?
Does this show a stop sign?
MedReg
Source
Layer 0
Layer 15
Layer 25
Layer 35
Layer 45
Layer 55
Does the image contain cell phone?
crust of the pizza?
Do you sea domut the picture?
Layer
Layer
Layer
Layer
Layer
Tayer
LLaVA-v1.5
Can you see a lesion in image?
there any evidence
infiltation the liver?
you pneumomia in the image?
Figure J.16: Qualitative evaluation of (i) medical MILMSs HuatuoGPT-V, LLaVA-Tri and MedRegA COCO, and (ii) LLaVA-v1.5 on VGMED. We visualize attention maps across dif- ferent layers, including those with the lowest KL divergence (highlighted with an orange bound- ary), which are indicative of layers most relevant to visual grounding in MILMs. We observe that LLaVA-v1.5 fails to ground predictions in clinically relevant regions when operating on medical images and medical VQA tasks. Furthermore, medical-domain models ground their predictions when applied to natural images. This is consistent with our quantitative analysis in Fig. 3 of the main paper. Together, they show that medical MILMs possess good visual grounding capabilities in general-domain settings. Overall, this confirms that the grounding failure is not due to model weakness, but is fundamentally specific the medical domain, consistent with our central findings. Inadequate visual grounding is medical- failure mode.
39

---

Published as a conference paper at ICLR 2026
HuatuoGPT-V-7B
Source
Layer
Layer 5
Layer 10
Layer 16
Layer 20
Layer 27
Is the bladder wall thickened?
Can liver found this image!
Is there small bowel the image?
Is spleen calarged!
Is there liver the imsaee?
Does the brain enhancing tumer show helcrogemoous enhancement?
Souree
Layer
Laver 6
Laver 14
Laver 18
Laver 25
Laver 31
LLaVA-v1.5
Do you see motorcycle picture?
Does this image show a umbrella?
Is there any sign parking meter in
image?
Figure 1.17: Qualitative evaluation. We visualize attention maps across different layers, including those with the lowest KL divergence (highlighted with an orange boundary), which are indicative of layers most relevant to visual grounding in MLLMs. For medical images, the attention maps of medical MILMs show limited alignment with the ground-truth annotated regions. In contrast, general-domain MLLM LLaVA-vi.5 applied to natural images exhibits strong alignment with relevant regions, consistent with other study general-domain MILMs Zhang et al. (2025a). This highlights gap in MLLM's visual grounding performance between the medical and natural image domains.
40

---

Published as a conference paper at ICLR 2026
HuatuoGPT-V-344
Layer
Layer
Layer
Layer
Layer
Layer
Does the
Is brain enhaneing tumer visible the image?
Is the brain edema visible image?
Is the cardiac silhouette
significantly enlarged?
Does show any signs of internal necrosis heterogeneity?
Does the lung cancer any evidence cavitation?
Source
Layer 0
Layer 6
Laver 14
Laver
Layer
Layer 31
LLLV
Does this show
present
in this image?
Can frishee be found image?
Figure J.18: Qualitative evaluation. We visualize attention maps across different layers, including those with the lowest KL divergence (highlighted with orange boundary), which are indicative of layers most relevant to visual grounding in MIL.Ms. For medical images, the attention maps of medical MILMs show limited alignment with the ground-truth annotated regions. In contrast, a general-domain MLLM LLaVA-v1.5 applied to natural images exhibits strong alignment with relevant regions, consistent with other study of general-domain MLLMs Zhang et al. (2025a). This highlights a gap in MLLM's visual grounding performance between the medical and natural image domains.
41

---

Published as a conference paper at ICLR 2026
HuatuoGPT---Bio
Source
Layer
Layer 5
Layer
Layer 15
Layer 20
Layer
Can a brain edema
found this
Is there a liver in image?
you cancer the image?
Does the image contain lung cancer?
image atelectusis?
Can you effusion the image?
Source
Layer O
Layer
Layer 14
Layer 18
Layer 25
Layer 31
LLaVA-v
What color is dog?
Do you see clock in the picture?
What
02
Figure J.19: Qualitative evaluation. We visualize attention maps across different layers, including those with the lowest KL. divergence (highlighted with orange boundary), which are indicative of layers most relevant to visual grounding in MILMs. For medical images, the attention maps of HuatuoGPT-V-Bio with a specialized vision encoder (BiomedCLIP) show limited alignment with the ground-truth annotated regions. In contrast, a general-domain MLLM LLaVA-v 1.5 applied natural images exhibits strong alignment with relevant regions, consistent with other study of general-domate doman MLLMs Zhang et al. (2025a).
42

---

Published as a conference paper at ICLR 2026
AedRegA
Source
Layer O
Layer i5
Layer 25
Layer 35
Layer 45
Layer 55
any evidence abnornal enhancement
kidney?
the liver any areas of abnormal density signal?
right mandible in the image?
Is there sign of left lang image?
Is there evidenos of consolidation the lung?
Is right mandible present image?
Source
Layer 0
Layer
Layer 14
Layer 18
1 Layer 25
Tayer 31
LLa-1-
buseball bat??
00
What color leaves of the potted plant?
you handbagg the picture?
Figure J.20: Qualitative evaluation. We visualize attention maps across different layers, including those with the lowest KL divergence (highlighted with orange boundary), which are indicative of layers most relevant to visual grounding in MLLMs. For medical images, the attention maps of medical MLLMs show limited alignment with the ground-truth annotated regions. In contrast, general-domain MLLM LLaVA-vi.5 applied to natural images exhibits strong alignment with relevant regions, consistent with other study general-domain MILMs Zhang et al. (2025a). This highlights gap in MLLM's visual grounding performance between the medical and natural image domains.
43

---

Published as a conference paper at ICLR 2026
VILA-MB-SB
Source
I 0
Layer
13
Layer 18
Layer 25
Does image show small howel?
Drees this image show brain edemal?
there evidence abnormal masses Jesions the left kidney?
any sign left temporal lobe in the image?
Does the liver show any abnormal density or attenuation?
the cardiac silhouette significantly en largd??
Source
Laver D
Laver
Laver 14
Layer
Layer
Layer 31
LLaVA-v1.5
Can see sports in image?
What color person's clothing?
What color the skis?
Figure J.21: Qualitative evaluation. We visualize attention maps across different layers, including those with the lowest KL divergence (highlighted with orange boundary), which are indicative of layers most relevant to visual grounding in MLLMs. For medical images, the attention maps of medical MLLMs show limited alignment with the ground-truth annotated regions. In contrast, a
MLLM LLaVA-v1.5 applied to natural images exhibits strong alignment with relevant regions, consistent with other study of domain MLLMs Zhang et al. (2025a). This highlights a gap in MLLM's visual grounding performance between the medical and natural image domains.
44

---

Published as a conference paper at ICLR 2026
VILA-MB-13B
Source
Layer 0
Tayer
Layer 16
Layer 23
Layer 32
Layer 39
Is the pacumothorax visible the image?
Do you liver in picture?
right kidney shown in the picture?
Does spleen show evidence enlargement?
Is there evidence of incressed swelling the brain edema region?
Does the enhancing tumer the brain show beterageneouu uniforn cohancement?
Spurce
(1
Layer
Layer 14
Layer 25
Layer 31
traffic light?
Is cup visible in image?
Can you see person image?
Figure J.22: Qualitative evaluation. We visualize attention maps across different layers, including those with the lowest KL divergence (highlighted with orange boundary), which are indicative of layers most relevant to visual grounding in MLLMs. For medical images, the attention maps of medical MLLMs show limited alignment with the ground-truth annotated regions. In contrast, general-domen domainn MLLM LLaVA-v1.5 applied to natural images exhibits strong alignment with relevant regions, consistent with other study of ceneral-domain MLLMs/ Zhang al. (202a)) This highlights gap in MLLM's visual grounding performance between the medical and natural image domains.
45

---

Published as a conference paper at ICLR 2026
LLaVA-Med-v1.5
Source
Layer
Layer 7
Layer 15
Layer 20
Layer 25
Laye 31
there evidenec abnornal cnlargement swelling left kidney?
enhancing tumor have any areas necrosis cystic change?
evidence of consolidation the left luag?
Is pneumonia present in this image?
you liver the image?
Can mass be found in image?
Source
Lnyer
Layer
Layer
Layer 18
Layer 25
Layer
What color the bottle?
any sign af vase in the image?
24
Can you see clock the image?
Figure J.23: Qualitative evaluation. We visualize attention maps across different layers, including those with the lowest KL divergence (highlighted with orange boundary), which are indicative of layers most relevant to visual grounding in MLLMs. For medical images, the attention maps of medical MILMs show limited alignment with the ground-truth annotated regions. In contrast, general-domain MLLM LLaVA-v1.5 applied to natural images exhibits strong alignment with relevant regions, consistent with other study general-domain MILMs Zhang et al. (2025a). This highlights gap in MLLM's visual grounding performance between the medical and natural image domains.
46

---

Published as a conference paper at ICLR 2026
TraVA-Tri
Source
Layer 0
Layer
Layer 12
Layer 18
Layer 25
Laye 31
there evidence consolidation the right hung?
there ovidence fatty infiltration in the liver?
Does the hruin non- enhancing tumor show evidence of surrounding edema?
there of left kidney in the image?
Is there sign of pncumenta
image?
right keidney in the image??
Source
Layer 0
Layer
Laver
Layer 18
Layer 25
Layer
LLaVA-v1.5
3.0
Do you see person in the picture?
the TV visible in the image?
What color truck?
Figure J.24: Qualitative evaluation. We visualize attention maps across different layers, including those with the lowest KL divergence (highlighted with an orange boundary), which are indicative of layers most relevant to visual grounding in MLLMs. For medical images, the attention maps of medical MILMS show limited alignment with the ground-truth annotated regions. In contrast, general-domain MILM I.L. applied to natural images exhibits strong alignment with relevant regions, consistent with other study of general-domain MLLMs Zhang et al. (2025a). This highlights gap in MLLM's visual grounding performance between the medical and natural image domains.
47

---

Published as a conference paper at ICLR 2026
K
ADDITIONAL EVALUATION OF LATEST GENERAL MLLMS
(a)
Localization Divergence
15 Divergence
InternVL
140
Owerz 5.
Attribute
D.vergencee
15 Divergence
IntOITALS 60
345
MW
VGMED
(b)
Qwen2.6-VT.
Qwen2.5-VI.
HISS
this image spleen?
this image tennis racket?
show any calcification?
chair?
Figure K.25: (a) Quantitative and (b) qualitative evaluation of Intern VL3-8B and Qwen2.5-VL--7 on VGMED and COCO. We observe that the visual grounding deficiency in medical domain persists even these latest general-purpose models.
48

---

Published as a conference paper at ICLR 2026
L
ATTENTION MAPS DERIVED FROM DIFFERENT TEXT TOKENS
Localization
Attribute
(AR)
Divergence
J5 Divergence
Attention
Divergence
Divergence
10
HuatuoGPT-V 78
Last
26
10
10
LLaVA-v1.5
366
Layer
Layer
Layer (VGHED
Layer
Layer
Scene (CDCO)
Figure L.26: Comparison of visual grounding when using all input tokens, question-only tokens, or the last token to derive attention maps. Using two representative MLLMs (HuauuGGPT-V-7B and LLaVA-v1.5 1,5), we evaluate how different token-selection strategies affect attention alignment on VGMED and COCO. Across all metrics and layers, attention maps computed from the last token achieves equal or better alignment with ground-truth truth regions compared to the alternative options.
49

---

Published as a conference paper at ICLR 2026
M
REPRESENTATIVE FAILURE CASES
(a)
Source
HuatuoGPT-Vissln
VGRefine (ours)
(c)
Q: Does the picture contain colon? HustuoOPT-Vision:
A: Yes
VGRefine (ours):
What color the bench?
Does the picture contain heart?
A: Yes
HuatuoGPT-Vision No
VGRafine (ours): Yes
picture?
(b)
Source
HustuoGPT-Visior
VGRefine (ours)
there pulmonary findings? HustuoGPT-Vision: Yes
Do toothbrush the picture?
VGRefine (ours): Yes
Q: is the liver parenchyma norma??
A: Yes
MustuoGPT-Visionn
VGRefine (ours):
Q: What color the bottle?
Figure M.27: Representative failure cases of HuatuoGPT-Visson on medical benchmarks. (a) The model correctly interprets the question but attends to the wrong anatomical region, leading an incorrect answer. After applying VGRefine, the model's attention shifts toward more clinically relevant region, resulting in the correct prediction. (b) The model misunderstand the question, sulting in both semantic and visual grounding failure. (c) Additionally, we include examples from LLLV on natural images as reference of accurate visual grounding. While multiple factors contribute to poor generalization, weak visual grounding consistently emerges major and mea- surable issue, though not the sole cause.
50

---

Published as a conference paper at ICLR 2026
N CLINICAL VALIDATION DURING VGMED CURATION
As part of the VGMED curation process, clinicians reviewed each sample to verify that (i) the question properly focused on visual grounding, (ii) it does not require deep diagnostic-level semantic medical reasoning, and (iii) it remains clinically appropriate and meaningful. An example of the rating interface used during the curation process shown in Fig. N.28
colon
Attribute Question:
there evidence of abnormal density masses the colon?
Clinccal Relevance:
3 4 5
Visual Grounding:
Minimum Semantic Grounding
Localization Question:
Does this image show a colon?
Clinical Relevance:
Visual Grounding:
2
Minimum Semantic Grounding:
Figure N.28: Example of the clinician rating interface used during VGMED curation.
Clinical Relevance
1: Inrelevant misleading: the question is clinically inappropriate nonsensical in this con-
text.
2: Marginally relevant; the question has limited medical value or loosely pertains the case.
3: Acceptable; the question is reasonable in clinical significance.
4: Clinically useful; the question is clearly relevant and meaningful to medical interpretation.
5: Highly relevant and valid; the question is well-phrased, accurate, and directly supports clinical reasoning.
Visual Grounding
1: It refers to other anatomy ignores the boxed area entirely; ignores the region.
2: The question has only weak or incidental connection to the boxed region; the area is largely irrelevant to the text.
3: It reasonably overlaps or implies the boxed region.
51

---

Published as a conference paper at ICLR 2026
4: Clear reference to the boxed region.
5: Perfectly aligned, the question precisely refers to the boxed region.
Minimum Semantic Grounding
Very deep semantic grounding; requires advanced, multi-step clinical reasoning, such as staging, prognosis, mechanisms, or treatment decisions. Examples:
1:
"What is the appropriate treatment for this condition?"
"How does this imaging pattern affect the patient's prognosis?"
2:
High semantic grounding; requires reasoning about specific diseases well-defined diag- nostic entities. Substantial medical knowledge is needed. Example:
"What diseases included in the image?"
3: Moderate semantic grounding: requires linking features to broad categories of pathology, such as distinguishing between growth, inflammation, or degeneration. Example:
"Do the changes suggest a Long-standing damage?"
4: Low-moderate semantic grounding: requires recognition of more specific medical descrip-
tors, but does not involve broad pathology categories or diagnostic reasoning. Examples:
"Does the structure appear to be pushing against or displacing nearby tissues?"
"Is there region that appears more diffuse rather than well-demarcated?""
5: Low semantic grounding requires only basic clinical or anatomical recognition (c.g., body parts, organs, simple structures, fractures, nodules). Examples:
"Does the bone show a visible fracture line?" "Is there nodule in this region?"
Therefore, rating of represents acceptable threshold across all three dimensions: the sample clinically relevant, visually grounded, and does not require deep semantic knowledge.
During the benehmark curation process, all samples receiving any score below 3 were discarded. Consequently, every VGMED sample satisfies 3 or above on all criteria. This ensured that retained samples genuinely test visual grounding rather than medical reasoning.
Furthermore, as summarized in Tab. N.6 the vast majority of clinician ratings are in the upper categories (4-5), with only minor proportion samples receiving a rating of 3 across any axis.
Table N.6: Percentage distribution of clinician ratings (3-5) across all axes for Attribute and Local- ization questions.
Type
Category
Rating (%)
Rating 4 (%)
Rating 5 (%)
Clinical Relevance
3.31
4.11
92.58
Attribute
Min. Semantic Grounding
0.37
10.38
89.25
Visual Grounding
4.04
12.18
83.77
Clinical Relevance
0.02
0.52
99.46
Localization
Min. Semantic Grounding
0.05
5.76
94.19
Visual Grounding
3.96
11.79
84.25
52

---

Published as a conference paper at ICLR 2026
O LIMITATIONS
While our work provides a systematic and detailed investigation into visual grounding as a key failure mode in medical MLLMs, focuses exclusively on this aspect. We do not examine other potential sources of failure, such as deficiencies in semantic grounding or reasoning capabilities. In practice, failures may also arise from an inability recognize what clinical concepts are relevant or to integrate multimodal information effectively. Additionally, our proposed method, VGRefine, designed improve visual grounding at inference time but does not address other underlying limitations, such as dataset biases insufficient domain-specific knowledge. Future work will ex- plore complementary methods to assess and improve semantie grounding and extend our analysis framework uncover other failure modes.
P EXPERIMENTAL SETTING/DETALE AND COMPUTING RESOURCES
For both VGRefine-7B and VGRefine-34B, select the top 20 attention raaked by align- ment with visually relevant regions-and average their outputs to obtain the filtered attention map. A percentile threshold of 50% is used to suppress low-activation regions during attention knockout. For VGRefine-75 7B, the attention knockout is applied at layer 16, while for EGRefine-34B, is applied at layers 34, 35, 36, identified through our quantitative analysis most relevant vi- sual grounding. We follow a zero-shot evaluation protocol across six biomedical VQA benchmarks: VQA-RAD, SLAKE, Vath PMC-VQA, OmniMedVQA. and MMMU (Health & Medicine track). The full set of prompts used for zero-shot evaluation is provided in Section F22 All experi- ments are conducted on server with SxNVIDIA A100 80GB GPUs.
Q BROADER IMPACTS AND ETHICAL CONSIDERATIONS
This work involves the analysis of medical MILMs using publicly available datasets that are de- identified. No private or sensitive patient data is used. We acknowledge that the deployment of medical MLLMs carries potential risks, including misinterpretation of clinical images, over-reliance on automated outputs by clinicians, and disparities in performance across patient populations. Our work aims mitigate such risks by improving the reliability of model predictions through better visual grounding. To promote transpareney and reproducibility. provide open access to code, evaluation metrics, and the VGMED dataset. This enables the broader research community to scru- tinize and build upon work responsibly.
R SAFEGUARDS
Our study does not involve training or releasing new foundation model, but rather evaluates and analyzes existing medical MLLMs in terms of their visual grounding behavior. While our proposed inference-tine refinement method improves grounding performance, it is designed for research use only and does not replace expert validation. We do not claim clinical applicability, and no com- ponents of work should be used for medical diagnosis decision-making without extensive clinical evaluation.
If any models or code are released, access will be gated under research-use license, and accompa- nied by usage guidelines clearly stating that they are intended solely for non-commercial, academic use. The evaluation dataset we construct contains only de-identified medical images drawn from publicly available datasets, and all visual content has been reviewed to ensure it does not pose safety, privacy, or dual-use risks.
LICENSES
All datasets and models used in this work are publicly available and cited appropriately in the main paper. We do not scrape any new data from the web or repackage any existing datasets; all visual assets have been used in accordance with their licenses.
53

---

Published as a conference paper at ICLR 2026
T USE OF LARGE LANGUAGE MODELS (LLMs)
LLMs were used solely as a writing aid to improve clarity, grammar, and style. They were not involved in generating research ideas, designing methodology, analyzing data, or drawing conclu- sions.
54


Reviews:
[{"review_id": "KS4kWnTrO8", "reviewer": "Reviewer_Qozi", "summary": "The paper investigates why medical multimodal large language models (MLLMs) underperform in medical image interpretation tasks, particularly in zero-shot settings where generalization is critical.\nThrough an empirical analysis of eight state-of-the-art medical MLLMs, the authors demonstrate that these models often fail to ground their predictions in clinically relevant image regions—a deficiency that appears specific to medical imaging. In contrast, prior studies show that general-domain MLLMs exhibit appropriate visual grounding when applied to natural image understanding tasks.\nMotivated by this finding, the paper introduces VGRefine, a simple yet effective inference-time method that optimizes the model’s attention distribution to improve visual grounding and reasoning in medical contexts. The proposed method achieves state-of-the-art results on six Med-VQA benchmarks covering eight imaging modalities and more than 110,000 samples.", "questions": "1. What do the authors believe are the reasons why this phenomenon occurs in medical scenarios?\n\n2. In Figure 1 of the main text, the paper compares several domain-specific medical models on medical VQA tasks with LLaVA-1.5 on general VQA tasks. What if this comparison were reversed? Is the problem caused by these medical models themselves? If the proposed method were directly applied to stronger and more recent general MLLMs, such as Qwen2.5-VL or Qwen3-VL, would the same conclusions be obtained?\n\nIf the authors can address the above questions, the reviewer promises to raise the score.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The study is interesting. By examining whether MLLMs can reuse their reasoning attention for localization, the authors observe that models’ attention during question answering is not fully concentrated on relevant image regions, and finally propose VGRefine to improve model performance.\n\n2. The paper is the first to systematically verify that insufficient visual grounding is one of the key factors leading to poor performance of medical MLLMs. The proposed method achieves state-of-the-art results on six different Med-VQA benchmarks covering eight imaging modalities and more than 110,000 VQA samples.", "weaknesses": "1. The paper could include more mathematical analysis to explain the causes behind the observed phenomenon.\n\n2. The paper claims that this phenomenon has not been observed in other domains. Do “other domains” refer only to general visual question answering on natural scenes? Or does the paper merely show results on some general scenes? If so, can this truly prove that the same phenomenon does not appear in any other specialized domains?\n\n3. The models used in the experiments—such as Qwen-VL, LLaVA-Med, and LLaVA-1.5—are relatively outdated. On the latest visual models, whether domain-specific or general-purpose, can the observed phenomenon and the effectiveness of the proposed method still hold? Could the authors also provide experiments showing how medical models perform in general VQA scenarios?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BXHe3hiWto", "reviewer": "Reviewer_2uoh", "summary": "This work presents VGMED, a clinician-curated benchmark for evaluating visual grounding in medical MLLMs, and proposes VGRefine, an inference-time attention refinement strategy that significantly improves visual grounding and zero-shot Med-VQA performance. The authors systematically analyze failure modes in existing medical MLLMs and demonstrate that inadequate visual grounding is a major bottleneck. The study is well-motivated, carefully designed, and supported by strong empirical results across multiple tasks and modalities.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper isolates visual grounding as a distinct bottleneck in medical multimodal models, a perspective that is under-explored yet highly relevant for trustworthy deployment.\n2. Layer- and head-specific analysis of cross-attention provides an interpretable pathway to understanding failure behavior, in line with mechanistic interpretability trends in multimodal research.\n3. The inference-time attention refinement is conceptually clean, model-agnostic, and introduces no additional data or parameters. The performance gains across diverse modalities and benchmarks are substantial.", "weaknesses": "1. The reliance on attention as the primary grounding signal may be discussed more rigorously; a short reflection on alternative grounding indicators (e.g., gradient-based saliency, causal perturbation) would improve completeness.\n\n2. The benchmark curation process is strong, but clearer quantitative annotation statistics (e.g. agreement checks, diversity distribution) would further support dataset rigor.\n\n3. Method hyperparameters (selected heads/layers) are fixed; small discussion of adaptive or data-driven selection would enhance generality claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FUV2lKUnYW", "reviewer": "Reviewer_Bv76", "summary": "This paper investigates the failure modes of medical MLLMs, focusing on their visual grounding ability in Med-VQA. The authors introduce a diagnostic framework that evaluates attention alignment between model-generated attention maps and annotated regions of interest using three metrics—Attention Ratio (AR), Kullback–Leibler (KL) divergence, and Jensen–Shannon (JS) divergence—on the proposed VGMED benchmark. They further present VGRefine, a two-step training-free method that refines attention through (1) filtering low-confidence visual regions to create a high-confidence binary mask and (2) applying this mask to suppress irrelevant visual attention. Experiments on multiple Med-VQA benchmarks show improved grounding alignment and accuracy, with qualitative visualizations and expert evaluations supporting the method’s interpretability and effectiveness.", "questions": "* Provide an explanation for using only the last input text token to compute attention maps. It would be helpful to test or discuss whether averaging or weighting attention across all input tokens offers a more reliable alignment estimation.\n\n* Explain the rationale for applying Attention Knockout at a single layer (e.g., layer 16) and whether extending it to multiple or higher layers could further improve grounding consistency.\n\n* Consider including newer or stronger vision-language models (such as Qwen2.5-VL or InternVL) in the comparison to confirm whether grounding difficulty is indeed domain-specific.\n\n* Discuss possible training-related causes of grounding failure, such as dataset bias or limited visual diversity, and whether VGRefine could mitigate these underlying issues.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper identifies a clear and meaningful problem—the weak visual grounding ability of medical MLLMs—and addresses it through a structured, quantitative framework. It turns a qualitative interpretability challenge into measurable alignment metrics and integrates them within a unified evaluation setting.\n\n* The proposed inference-time method, VGRefine, is concise, computationally efficient, and easy to deploy. Despite its simplicity, it consistently enhances grounding quality and answer accuracy across diverse Med-VQA benchmarks, demonstrating both technical soundness and practical relevance.\n\n* The experimental section is extensive, including evaluations on multiple datasets, ablation studies, and expert assessments. The combination of quantitative metrics and human evaluation adds credibility and depth to the analysis.", "weaknesses": "* The paper attributes model failure primarily to weak visual grounding but does not clearly disentangle it from semantic grounding or reasoning errors. It remains unclear which factor—semantic or visual—dominates the poor generalization of current medical MLLMs. Providing representative failure cases could clarify whether errors stem from misunderstanding the question or mislocalizing visual evidence.\n\n* The choice to compute attention maps only from the last input text token lacks justification. Averaging or weighting attention across all input tokens might yield a more stable and comprehensive alignment estimation; offering an explanation or comparison for this design choice would strengthen the analysis.\n\n* Attention Knockout is applied at a single layer, yet the broader influence on higher layers or cross-layer propagation of redundant visual information is not analyzed, which may limit the consistency of the refinement effect.\n\n* Comparisons with general-domain vision-language models are limited. Including stronger or more recent baselines, such as Qwen2.5-VL or InternVL3, could better support the argument that grounding difficulty is domain-inherent rather than model-specific.\n\n* While the study provides a thorough diagnostic view, it remains primarily descriptive. Further exploration into why grounding failures arise—such as biases in medical datasets or insufficient visual diversity—would deepen the explanatory value of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)