MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models
Authors: Mohammad Shahab Sepehri
Zalan Fabian
Maryam Soltanolkotabi
Mahdi Soltanolkotabi
Venue: ICLR 2025 Poster (2025)
Keywords: Medical foundation models
Benchmarking
Vision encoding
Radiology
Primary area: datasets and benchmarks
TL;DR: A challenging medical Visual Question Answering (VQA) benchmark dataset, that probes the failure modes of medical MLLMs from a vision perspective.

Abstract: Multimodal Large Language Models (MLLMs) have tremendous potential to improve the accuracy, availability, and cost-effectiveness of healthcare by providing automated solutions or serving as aids to medical professionals. Despite promising first steps in developing medical MLLMs in the past few years, their capabilities and limitations are not well understood. Recently, many benchmark datasets have been proposed that test the general medical knowledge of such models across a variety of medical areas. However, the systematic failure modes and vulnerabilities of such models are severely underexplored with most medical benchmarks failing to expose the shortcomings of existing models in this safety-critical domain. In this paper, we introduce MediConfusion, a challenging medical Visual Question Answering (VQA) benchmark dataset, that probes the failure modes of medical MLLMs from a vision perspective. We reveal that state-of-the-art models are easily confused by image pairs that are otherwise visually dissimilar and clearly distinct for medical experts. Strikingly, all available models (open-source or proprietary) achieve performance below random guessing on MediConfusion, raising serious concerns about the reliability of existing medical MLLMs for healthcare deployment. We also extract common patterns of model failure that may help the design of a new generation of more trustworthy and reliable MLLMs in healthcare.

Published as a conference paper at ICLR 2025
MEDICONFUSION: CAN YOU TRUST YOUR AI RADIOLOGIST? PROBING THE RELIABILITY OF MULTI- MODAL MEDICAL FOUNDATION MODELS
Mohammad Shahab Sepehri,' 1 Zalan Fabian, Maryam Soltanolkotabi, Mahdi Soltanolkotabi DDet. of Electrical and Computer Engineering, University of Southern California Dept. of Radiology and Imaging Sciences, University of Utah sepehri @usc.
ABSTRACT
Multimodal Large Language Models (MLLMs) have tremendous potential improve the accuracy, availability, and cost-effectiveness of healthcare by providing automated solutions or serving as aids to medical professionals. Despite promising first steps in developing medical MLLMs in the past few years, their capabiliites and limitations are not well understood. Recently, many benchmark datasets have been proposed that test the general medical knowledge of such models across variety of medical areas. However, the systematic failure modes and vulnerabilities of such models are severely underexplored with most medical benchmarks failing to expose the shortcomings of existing models in this safety-criticll domain. In this paper, we introduce MediConfusion, challenging medical Visual Question Answering (VQA) benchmark dataset, that probes the failure modes of medical MLLMs from a vision perspective. We reveal that faate-of-the-ss models are easily confused by image pairs that are otherwise visually dissimilar and clearly distinct for medical experts. Strikingly, all available models (open-source or proprietary) achieve performance below random guessing MediConfusion, raising serious concerns about the reliability of existing medical MLLMs for healthcare deployment. We also extract common patterns of model failure that may help the design of new generation of more trustworthy and reliable MILMS in healthcare. The dataset and evaluation code available at / httpb//ggtttt.tt///// com/AIF4S/ Medi Con fusofuusio
INTRODUCTION
Multimodal Large Language Models (MLLMs) have demonstrated unprecedented capabilities in variety of multimodal tasks, including image understanding and visual reasoning, autonomous driving (Cui et al., 20244,, robotics (Wang et al. 2024a) and embodied AI (Driess et al. 2023). Motivated by
this success, a growing body of work (Moor et al. 2023 Li et al. 2024 Lin et al.) 2023) explores the potential of MILMMs in medical applications with the hope of paving the way more accurate, personalized and cost-effective healthcare solutions through modern generative AI.
Even though MLLMs show enormous potential in a wide range of tasks, a swath of challenges have stymied their deployment, including object hallucinations Li et al., relationship hallucinations (Wul al. 2024a), inaccurate object counting (Jain al. 2024) and lack of spatial reasoning capabilities (Kamath et al. 2023). These shortcomings are especially worrisome in safety-critich applications, such as healthcare, where reliability an essential requirement. In fact, recent research efforts on medical MILMs have revealed weak anatomic knowledge (Nan et al. 2024). concerns on toxicity and patient privacy Xia al. (2024), highly unreliable disease diagnosis Wu et al. (2023a), and the fact that even junior doctor far outperforms the most proficient medical MLLMs across wide spectrum of tasks Wang et al. (2024b). As model failure in the medical domain can lead to serious adverse health effects, it is of utmost importance to understand the performance and limitations of generative AI in the medical context.
A flurry of activity has emerged around probing the performance of medical MLLMs in multitude of tasks, including visual question answering (VQA) (Ben Abacha et al. 202T), disease classification and report generation (Royer et al. 2024), and modality recognition (Wu et al. 2023b). Even though

---

Published as a conference paper at ICLR 2025
Discovering confusing pairs
VQA generation from pairs
Radiologist feedback
Eradient-echhe bate--aaedd E8L from pstient with recurrent B-GCV..
+
BiomedCLLIP
Sagillal MRI scan showing complete rupture of the
similarity
correctorss
SB LLM
relevanice quality
BOCO
DINOv2 fentures
What is the primory pachology in the MRI acen?
Recurcent 1-600
similarity
B: Coxpleta ruptace of
Figure 1: Overview of MediConfusion curation pipeline. First, we extract image pairs from the ROCO radiology dataset that clearly distinet in the image domain. but may be challenging to differentiate between for multimodal models (left). Next, we use an automated pipeline leveraging LLM prompting to generate VQA from the confusing pairs and their corresponding captions (center). Finally, we incorporate radiologist feedback to filter questions for correctness, relevance and quality, and to revise the questions and answer options for improved medical language and precision (right).
proposed medical benchmarks offer valuable insights on model performance across variety of anatomic regions and imaging modalities, they focused evaluating the medical knowledge of MLLMs across large evaluation sets, heavily biased towards common or typical scenarios. Therefore, it is unclear how well the measured performance correlates with the actual multimodal medical reasoning capabilities of these models, especially in the face of systematic but perhaps more intricate model failures underrepresented in the dataset. Therefore, developing new evaluation benchmarks that carefully test and probe the capabilities of these systems, expose their vulnerabilities, and facilitate the development of better understanding of failure modes is vital in healthcare applications.
In this work, we introduce MediConfusion, a challenging benchmark for evaluating the failure modes medical MLLMs from vision perspective. We combine novel insights on the image representations of medical MLLMs with the expertise of clinical radiologists to craft benchmark dataset designed to evaluate the ability of state-of-the-an models recognize subtle, yet clinically meaningful differences between medical images. Our work reveals that medical MLLMs often confuse image pairs that otherwise appear very different the image domain. Leveraging this observation, we introduce automated pipeline discover such pairs in the ROCO (Pelka et al.) 2018) multimodal radiology dataset. Then, collaboration with radiologists, we curate a VQA benchmark of clinically relevant multiple-choice problems designed to probe the model's ability to distinguish between such images. By design, relying solely on unimodal (language) priors cannot achieve better than random guessing accuracy MediConfusion directly correlates with multimodal medical reasoning and image understanding capabilities. Remarkably, discover that both state-of-the-an medical MLLMs, well the most advanced proprictary models, are easily confused by the image pairs, resulting in performance below random guessing for most models the time of writing this paper. What striking about this poor performance is that for some of the models (i.e. all medical MLLMs we studied) the images and corresponding captions are part of the training data []] Finally, we leverage our pipeline to categorize failure cases in order to guide future research toward more reliable medical Al solutions.
benchmark, and therefore performance
2 THE MEDICONFUSION BENCHMARK
The majority of existing multimodal foundation models leverage CL.IP (Radford et al. 2021) to encode the input image (Li et al. 2024 Liu et al., 2024: Moor et al. 2023 Li et al. (002). CLIP has been pretrained on internet-scale general domain data and, therefore. may not be suitable for the nuanced representation of medical images due to the considerable distribution shift. Thus, variants
'iien the public nature of the original dataset such images and captions are also likely part of the pre- training of proprietary models such OpenAl's GPT-4o, Google Deep Mind's ' Gemini 1.5 Pro, and Anthropic's Claude 3 Opus.
2

---

Published as a conference paper at ICLR 2025
Spinal
Head and Neck
Gastrointestinal
Pulmonary
Vascular
Nuclear Medicine
Musculoskeletal
Cardiac
Cerebral
Figure Sample confusing image pairs we have extracted from the ROCO dataset across categories.
trained on large-scale medical image-text datasets have been introduced as image encoders for medical agents, including BiomedCLIIP (Zhang al. 2023b) and PMC-CLIP (Lin al., 2023). Due to the specialized training data, these models are able to better capture the structure and semantics of medical images. However, surprisingly, we observe that the feature space of even specialized medical encoders is often not rich enough to clearly differentiate between images that are otherwise highly dissimilar. A growing body of recent work Thrush et (2022); Yuksekgonul al. (2023) has shown that the contrastive pretraining objective, shared by common image encoders for MLLMs, can be optimized via shortcuts that lead to fundamental flaws multimodal understanding. In particular, training consists of aligning image features with their corresponding text features within batch of data. Thus, if the images are clearly distinct within the batch, the task becomes easy and the model is not encouraged to learn embeddings nuanced enough for more intricate downstream tasks, such as medical reasoning. As a result, MLLMs that leverage such pretrained encoders suffer from impaired image understanding and visual reasoning (Tong et al., 2024a b), casting serious doubt on the reliability of such models in critical medical diagnosis. Therefore, designing challenging benchmarks that stress-test the visual capabilities of medical MLLM of utmost importance for gaining a better understanding of the limitations of existing models.
In this work, introduce the MediConfusion Benchmark, challenging multiple choice medical visual question answering benchmark designed to probe the reasoning capabilities of medical MLLMs. The overview of our curation pipeline is summarized in Figure First, extract image pairs that are visually clearly different, but MLLMs will likely confuse them due to their similar features in embedding space. Next, based on the captions corresponding to each of the images in the confusing pairs, we generate large pool of multiple choice problems via LLM prompting. Finally, each question the L.L--ggeerttee pool is scrutinized and revised by an expert radiologist before being added to MediConfusion. We evaluate a range of state-of-the-ee medical and general domain MI..Ms and demonstrate that even flagship proprictary models have performance worse than random guessing.
2. DISCOVERING CONFUSING PAIRS
We find confusing image pairs in ROCO (Pelka et al. 2018), multimodal dataset of xz 80k radiology images and their corresponding captions extracted from PMC-OA (Lin et al. 2023) (Figure left). Inspired by Tong et al. (2024b), we seek out pairs with clear visual differences, but high similarity in the feature space of medical vision-language models. This implies that at least one of the images in the pair is compressed ambiguously, and thus, it is likely that relevant visual information is lost in the encoding. In particular, we base our selection criteria on BiomedCLIP's embedding space, as this model has been specifically trained medical image-tes data, and thus it has more refined feature space for medical images than general-domain encoder, such as CLIP. Simply put, radiology image pair that confuses BiomedCLIP will likely confuse CLIP as well. Moreover, BiomedCI.I has been
3

---

Published as a conference paper at ICLR 2025
that peimary abnamalltty observed tr ragtttal veighted
Vascaller
Head and Necis
Cerebral
15.2%
bernistio the level of c2 vish cilacurant Di..
199
hamission Level of CJ with actererent oe.
3.2%
Nuclese Mediciee
11.0%
4.5%
aftect lexion oe the foranan Kanco.
effect Loolon the focares of Korzo.
Spinal
Palmonsay
9.6%
11.8%
Castraineetinal
Individual score:
Mascnioskoketll
Set score: 0
Candicc
Figure 3: A VQA pair from MediConfusion. A confusing pair shares the same question and an- swer options, but the correct answer is different for the two (A for the image on the left and B for the image on the right). The model receives a ser score only it correctly answers both questions in the confusing pair. Individual score is evaluated separately for each image.
Figure 4: Distribution of question categories in MediConfusion. We assign category to the ques- tion based the category of the corresponding image the VQA. A single image can belong to multiple categories at the same time.
pretrained on the largest dataset of medical image-caption data among publicly available CLIP-style biomedical vision-language models. We measure visual differences between images in the feature space of DINOv2, vision-only foundation model with robust image representations that capture visual details. We randomly sample pairs of images and evaluate their similarity BiomedCLIP (simmed) and DINOv2 (simgee) feature spaces. We consider them confusing pair if sim-med 2 0.9 and gem & 0.75 hold at the same time. The gap simmed simgen can be increased further in order to obtain more difficult pairs, however find that our setting already challenging enough for most contemporary models. We depict sample pairs uncovered by our technique in Figure 2
2.2 VQA GENERATION
Given a pool of candidate confusing pairs, we generate multiple choice medical VQA problems that probe the MLLM's ability to effectively diffcrentiate between the images in the pair (Figure center). First, we filter the candidate pool by removing images with short captions (< 100 characters) that likely contain insufficient detail about the image. Next, we pass the pair of captions to an LLM (GPT-4) and prompt the model to generate question which the answer is different for the two images and to provide the two answer options. Thus, we create two VQA problems for each pair that share the same question and answer options, however the correct answer is different for the two images (Figure 3). Therefore, if the medical MILM is unable to differentiate between the input images, it would only be able answer at most one of the pair of VQA problems correctly, but not both. As a result, our benchmark by construction cannot be solved to higher than 50% accuracy by relying solely on language prior. In particular, we only credit a set score the model on a particular question pair, if the question has been answered correctly for both images. On the other hand, as less strict metric an individual score awarded to the model for each correct answer, irrespective of correctness on the other image in the pair. Furthermore, in order to categorize questions in the VQA, we prompt the LLM to assign the most relevant medical area to each of the questions based on the corresponding image's 's PMC caption. We leverage these categories to break down the performance of existing medical MLLMs across the various categories. We include all prompts used in VQA generation in Appendix A
4

---

Published as a conference paper at ICLR 2025
2.3 DATA FILTERING AND REVISION VIA RADIOLOGIST FEEDBACK
As we generate the questions/answer choices using an LLM, various issues may arise such as factual errors and inconsistencies in quality, format, or language. To ensure the curation of reliable benchmark dataset, oversight and feedback from a radiology expert are crucial. A radiologist has evaluated each of the automatically generated VQA problems focusing on three aspects.
the question has to be valid with respect to both of the images, the problems need to be solvable by looking at the individual images alone, and the corresponding answers have be correct.
Correctness:
Relevance: the question has to be relevant to clinical practice or medical research.
Language: the problem has use proper medical terminology and precise language.
Based on these guidelines, the radiologist has assigned quality score to each question, on a scale 1 - 10. Higher scores correspond better problems, and score of is assigned if correctness violated in any form (e.g., irrelevant question, incorrect answer). We add a VQA pair to MediConfusion only the quality score is at least 5 for both individual problems in the pair. Moreover, the expert has verified the medical categories assigned by the LLM to each of the images and revised the question and answer options to improve language quality and precision. This step is crucial in eliminating model artifacts that originate in L.LM-generaeed text inputs.
The resulting benchmark is well-rounded, with questions touching upon areas (see Figure +
cerebral, spinal, cardiac, gastrointestinal, musculoskeletal, vascular, pulmonary, head and neck, and nuclear medicine with 352 questions in total. The distribution of question categories is depicted in Figure
EXPERIMENTS
3.1 EVALUATION
We evaluate models on MediConfusion based on two notions of accuracy. Set accuracy is the portion of correct confusing pairs, where we only consider pair correct if the model has answered the question correctly for both images in the pair. Individual accuracy is the standard notion of accuracy, that is, the portion of correct answers over all questions. An example is depicted in Figure 3 Furthermore, we report confusion score, which indicates the portion of pairs where the model has chosen the same answer for both images in the pair, out of all pairs (we exclude pairs where the model generated invalid answers or failed to answer). A high confusion score signifies that the model prediction is overwhelmingly invariant to the specific input image within pair, and thus, it is confused by the images.
Extracting the knowledge from MLLMs for VQA benchmarking is often challenging due to sensitivity the specific prompt format and phrasing, strong reliance on language bias and other factors. For instance, instruction tuned models can be directly prompted to answer a multiple choice question with the correct letter option, whereas models without instruction tuning often fail to do so. Therefore, in order to provide a fair comparison, we use range of evaluation techniques to assess performance.
based score (PS) Following Xu et al. (023). we compute the normalized likelihood of lrage--uustioon-anse answer triplets for each answer option, and pick the option with the highest likelihood as the answer. In other words, we select the answer option that the model assigns the highest probability to, given the image and question. To compute the prefix-based score, we concatenate the medical question and the answer sentence directly, stripping the multiple choice question style (e.g., removing "Choose the letter of the correct answer.") and option indicators (e.g.. removing "Option A: ...") in order to ensure that models that have not been specifically trained on multiple choice question answering can also consistently provide valid answers.
Multiple choice prompting (MC) - We directly prompt the model to answer the multiple choice question with the letter of the correct option. As the output formats may vary (e.g., "A" vs. "The answer is A."), we parse the outputs and attempt to match it to one of the answer options.
Free-form evaluation (FF) We prompt the model to answer the question without providing the answer options or requiring any specific output format. Then, we attempt to match the model output to one of the options using L.I.M. In particular, prompt GPT-4 to score how well the generated
5

---

Published as a conference paper at ICLR 2025
Table Experimental results on MediConfusion. Evaluation techniques: PS prefix-based scoring. MC multiple choice prompting, FF free-form evaluation, GD greedy decoding evaluation. We underscore the best accuracy for each method across evaluation techniques and report the overall best in bold.
Set acc. (%)
Indiv, acc.(%)
Confusion
(%)
Best
Method
MC
GD
FF
MC
FF
PS
MC
GD
FF
PS
Set acc.
Indiv, acc.
LLaVA
8.52
9.09
1.70
50.57
51.70
15.06
49.72
85.47
85.80
76.00
97.16
9.09
51.70
BLIP-2
0.57
6.82
1.70
3.98
22.16
50.28
11.65
51.42
92.19
86,93
86.67
94.89
6.82
51.42
InstructBI.Ip
12.50
7.95
2.84
3.41
51.99
53.12
19.60
50.57
80.35
90.34
87.23 94.32
12,50
53.12
DeepSeek-V1.2
15.91
16.48
4.55
6.25
54.26
54.26
16.19
49.43
77.19
75.57
50.0 86.36
16.48
54.26
Molmo
9.66
0,57
0.57
5.11
52.84
49.72
14.77
51.42
86.21
98.3
83.33 92.61
9.66
52.84
LLaVA-Med
0.00
0.00
114
23.58
49.72
18.75
49.72
100.00
99.43
95.92 97.16
1.14
49.72
RadFMM
0.57
1.14
0.57
5.68
35,90
50.28
16.19
48.58
97.54
98.30
95.12 85,80
5.68
50.28
Med-Flamingo
1.14
2.27
0.57
4.55
47.73
50.00
17.05
51.99
98.75
95.45
94.89
98.30
4.55
51.99
GPT-4o
18.75
56.25
75.00
18.75
56.25
ol
21.59
$7.95
72.99
21.59
57.95
Claude 3 Opus
8.52
50.85
84.09
8.52
50.85
Gemini 1,5 Pro
19.89
51.14
58.52
19.89
51.14
Gemini 2.0 Flash
29.55
61.93
67.05
29.55
61.93
Random guessing
25.00
50.00
output matches each of the options, and we pick the answer option with the highest score. We include the specific evaluation prompt in Appendix
Greedy decoding (GD)- Similar to multiple choice prompting, we directly prompt the model to answer the problem with the letter of the correct option, then we pick the option with the highest assigned next-token probability. Greedy decoding evaluation is special case of prefix-based scoring, where the answer options consist of a single letter.
PS and FF evaluations are suitable for models that are not instruction tuned or have not been trained to understand the multiple choice QA format. On the other hand, MC and GD are simpler to evaluate, however these techniques may fail to correctly measure the knowledge of MLLMs unable to understand and follow the multiple choice format. Overall, we represent the performance of each model by their best performance across all evaluation techniques. As observe, proprietary models can consistently pick an answer option for multiple choice questions; for these models, we only provide MC results. Moreover, output logits necessary for PS and GD evaluation are not available for proprictary models.
We evaluate a representative of 13 models, of which are medical MLLMS (LLaVA-Med (Li 2024). Med-Flamingo (Moor et al. 2023), RadFM (Wu et al., )023b), 5 are flagship proprictary models (GPT-40, ol, Claude 3 Opus, Gemini 1.5 Pro, Gemini 2.0 Flash) and open-source general- domain MLLMS VLB77 v1.6 (Liu et al. 2024). BLIP-2 (Li et al. 2023). InstructBLIP (Dai et al. 2024b), Molmo-7B (Deitke et al. 2024 We set generation
et al. 2023), DeepSeek-
(L.2(Wu VetDSSeee-VV2((WW
parameters according to the corresponding code release and recommended settings and use few-shot prompting for Med-Fit ood-Flamingo (more details in Appendix B
3.2 RESULTS
We summarize the performance of MLLMs on MediConfusion in Table Alarmingly, almost all MLLMs perform below random guessing in terms of set accuracy, corroborating our hypothesis that models struggle to differentiate in fine enough detail between the extracted image pairs necessary for accurate medical reasoning. This observation is further supported by the markedly high (often above 90%) confusion scores, indicating that models tend to select the same answer for both images within a confusing pair. Even RadFM, model that does not leverage CLIP- image encoder, is confused on our benchmark (82.39% confusion score) with performance well below random guessing. As most likely proprietary models leverage visual encoders other than CLIP as well, the overall poor performance and extremely high confusion scores suggest that the exposed vulnerability more general and not solely rooted in the specific ambiguities of CLIP-style contrastive pretraining.

---

Published as a conference paper at ICLR 2025
Table 2: Results by category. We report the best set and individual accuracies (%) for each model across all evaluation techniques.
Cerebral
Vascular
Head & Neck
Spinal
Muscutoskel.
Cardiac
Gastroiat.
Palnsonary
Nuclear Med.
Model
Ser
Indiy.
Set
Indiv.
Set
Indiv.
Set
Indiv.
Set
Indiv.
Ser
Indiv.
Ser
Indiv.
Set
Indix.
Set
Indiv
7.50
13.70
54.79
4.48
52.
5.88
52.94
9.52
52.38
27.91
10.00
55.00
14.29
57.14
BLIP3
5.6 54.43
8.23 53,42
4.48
46.27
3.92
49.02
4.76
50,00
7.69
50.00
13.95
51,16
10.00
55.00
28.57
64.29
InstrccBLIP
16.46 59.49
10.96
56.16
7.46
52.24
17.65
52,91
23.81
59.52
7.69
50.00
9.30
51.16
10.00
60.00
14.29
57.14
21.52 56.96
26.03
60.27
23,88
58.21
15.09
52.94
9.52
54.76
11:54
53.85
13.95
48.84
20.00
55.00
28.57
57.14
Modmo
13.92 55.70
13:70
54.79
8.96
50.24
5.88
56.36
19:05
54.76
7.69
51.92
13.05
53.49
10.00
55.00
14.29
50.00
LLaVA-MMdd
1.27 55.16
49.32
0.00
50.25
4.76
50.00
50.00
4.05
53.49
45.00
50,00
RadFM
1.27
49.32
5,97
40.25
3,92
2.38
11:54
50.00
11.63
50.00
0.00
Med Flumingo
7.59 58.23
10.95
56.15
8,96
52.24
0.00
52.94
4.76
52.38
3.85 51.92
2.33
48.84
10,00
50,00
0.00
50.00
GPT-40
15.19 59,49
34.25
67.12
8.96
58.21
15.69
52.94
14:29
52.38
19:23
55.77
16.28
55.81
35,00
65.00
14.29
42.86
ol
30.36 64.56
28.77
63.01
20.90
59.70
19.61
58.82
14:29
57.14
15.38
53.85
16.28
53.49
15.00
50,00
42,86
71.43
Chaude 3 Opus
7.59 55.70
20.55
58.90
0.00
44.78
0.00
52.91
9.52
45.24
11.54
53.85
11.63
51.16
10.00 50,00
14.29
50.00
Gemini 1.5 Pro
25:32
27.40 60.27
16.42
52.24
17.65
26.19
47.62
48.08
23:26
5,00
50.00
28.57
57.14
Gemini 2.0 Flash
29.11
68.49
58.21
23.53
25.58
50.00
71.43
Gemini models are interesting outliers: even though they are the least confused on the dataset, they struggle tackling MediConfusion as well, as Gemini 2.0 barely surpasses random guessing by about 3%. This may suggest that the model's 's visual representations are rich enough to meaningfully distinguish between images; however, the medical knowledge or necessary reasoning skills are lacking to correctly answer the questions.
Furthermore, perhaps surprisingly, medical MLLMs did not outperform other methods, indicating that the shortcomings cannot be addressed exclusively by domain-specific training. These results are especially surprising, as the image-caption pairs used to generate MediConfusion are part of PMC-OA, which is included in the pre-training set of all medical MLLMs in our experiments. We also note that given the public nature of PMC-OA these image-caption pairs are likely included in the training set of proprietary models as well. Finally, we see some performance gap between open- and proprietary models, with Gemini 2.0 achieving the highest individual accuracy of 61.93%.
We further break down the results based on the category of the question in order to identify if specific areas have been more/less challenging to the models. We summarize our findings in Table Even though the overall results across all categorics are close to random guessing performance, proprictary models demonstrate slightly better accuracies questions related to cerebral, vascular, and nuclear medicine images. In particular, Gemini 2.0 achieves 42.86% and 71.43% set and individual accuracy correspondingly on nuclear medicine, overall best across all models and categories
DISCUSSION
4.1 IDENTIFYING PATTERNS IN CONFUSING PAIRS
Our experiments have demonstrated that state-of-the-art MLLMs are easily confused by radiology image pairs that exhibit major differences obvious human experts. The first step towards improving the reliability of such models is to identify and categorize common cases where medical MLLMs tend to break down. We leverage an expert-in-the-logy pipeline to extract failure modes from MediConfusion via combination of LLM prompting and radiologist supervision. In particular, pass the VQA problems from MediConfusion to GPT-4, where we replace the images with their corresponding captions from ROCO. We prompt the model to summarize the key differences between images in pair that the questions are designed to test (details in Appendix (A). The L.LM identifies patterns in the extracted differences and distills them into a set of categories that the radiologist corrects and refines based on the dataset. As a result, we identify the following common patterns that have confused the models:
Pattern 1: Normal/variant anatomy vs. pathology- Models often struggle to differentiate between normal/variant anatomy and pathological structures. For instance, the model often confuses malalignment with normal alignment (e.g.. atlantoaxial dislocation vs. normal atlantoaxial interval) or differentiating pituitary region masses (suprasellar vs. parasellar vs. intrasellar) or various anatomical regions of the spine (cervical vs. thoracic vs. lumbar).
7

---

Published as a conference paper at ICLR 2025
Pattern 2: Lesion signal characteristics- Models fail to correctly identify regions of high signal intensity and their significance, particularly on T2-weighted sequences. This failure is especially of clinical significance in differentiatingg solid vs. cystic entities.
Pattern 3: Vascular conditions- Identifying aneurysms and differentiating them from normal vascular structures or other abnormalities like vascular malformations seems to be challenging for MLLMs. Furthermore, there is often confusion between total occlusions and partial stenosis in coronary arteries.
Pattern 4: Medical devices- Models often fail to detect the presence of stents and have difficulties distinguishing between various types of stents. Identifying the presence or absence guidewires in images of interventional procedures tends to also be challenging for MLLMs.
Most of the above shortcomings can be, to some degree, traced back to known, common failure modes (Tong et al. 2024b) of visual reasoning in MLLMs in the general domain.
Detecting presence (or absence) of specific features: Correct reasoning over medical VQA problems strongly relies on detecting the presence (or absence) of particular features objects relevant to the question. MILMs are known to suffer from object hallucinations (Li et al.) rooted in parts in flawed image encoding, statistical biases and strong reliance language priors (Leng et al. 2024). We can see this specific weakness reflected in Patterns 3 and 4 directly.
Understanding state and condition: In medical VQA, it is crucially important for the model to understand the difference between "normal" and "abnormal" structures. MILMS have difficulties identifying the state and condition of objects in the general domain, such as whether the ground is wet or if a flag is blowing in the air (Tong et al. (024b). These challenges may be amplified in the more nuanced medical setting, which we observe Patterns I and especially.
Positional and relational context: Answering medical VQA problems often necessitate a careful understanding of the spatial relationships of various anatomical features and their specific location. Recent research has uncovered serious limitations in the spatial reasoning capabilities of MLLMs (Kamath et al. 2023), some even failing to distinguish left from right. This pervasive weakness in spatial reasoning may translate to failures in medical VQA seen in Pattern 1.
Color and appearance: Recent work has shown that MILMs can confuse colors and their intensity (bright/dark) (Tong et al. 2024b), which may cause challenges in identifying signal characteristics in radiology images (high/low intensity) reflected in Pattern 2.
4.2 VISUAL PROMPTS IN MEDICONFUSION
Free-form visual prompts are intuitive annotations in the input image, such as a red bounding box or an arrow, aimed at highlighting a specific point or area within the image. It is natural to ask whether well-placed visual prompts in medical images, annotated by a doctor, can potentially guide the attention of MLLMs to important areas in the image and thus help provide accurate answers. Such a capability would greatly facilitate human-machine collaboration in healthcare and provide more reliable Al-assisted diagnosis. In the general domain, research has shown that MILMS typically are unable efficiently interpret visual prompts without incorporating such task specifically into the training procedure (Cai et al., 2024).
We find that some images in MediConfusion include such visual prompts. typically in the form of arrows pointing at the abnormality, and a specific case the correct answer written in the image along with the prompt (Figure 5). We observe that only proprictary models, well as LLaVA v1.6 and BLIP-2, have been able to provide consistently correct answers for this particular image, and none of the medical MILMS. We hypothesize that the success of proprietary models and LLaVA v1.6 can be attributed to their OCR (optical character recognition) capabilities, which is missing from medical MLLMs. In examples where only the visual prompt (e.g., an arrow pointing at the abnormality/reiion of interest) is included we don't I observe a similar trend. We believe that understanding and improving the visual prompting capabilities of medical MLLMs a promising direction for future research.
8

---

Published as a conference paper at ICLR 2025
Q: what: is the condition of the left anterior descanding axtery?
Critical narrowing flow cessation and posaible residual thrombus
Contained dissection in the
mid-segment
Figure 5: Sample VQA from MediConfusion where the solution is directly provided in the image in the form of text and visual prompts (arrows). Medical MLLMs not trained for OCR have been unable to leverage the hint.
5 RELATED WORK
Multimodal Large Language Models Large Language Models (LLMs) such as InstructGPT (Ouyang al., 2022) and LLaMA (Touvron al. 2023) have emerged as powerful models capable of performing complex tasks rooted in natural language. including text summarization, coding, and LLMs are pretrained on massive text corpora and can efficiently adapted to downstream tasks. Beyond textual inputs, the LLM pretraining framework has been extended to further modalities, such as images, resulting in multimodal large language models that demonstrate strong visual understanding and reasoning capabilities. LLaVA (Liu al. 2024) interleaves image representations with the input text pretrained LLaMA model and fine tunes on visual instruction- following data. Flamingo (Alayrac et al. 2022) injects visual information into frozen LLM via cross-attention. BLIP (Li et al. 2022] 2023) proposes the Q-Former architecture for connecting pretrained vision features multimodal inputs directly.
LLM. More recently, GPT-4o has been trained from scratch mixed
Beyond the general domain, MILMs especially promising in automating costly medical tasks, such as analyzing radiology images, generating medical reports or acting as medical conversational agents provide healthcare advice. There has been substantial research recently develop medical MLLMs, most often by adapting popular general domain architectures to medical data. Med- Flamingo (Moor al., 2023) pretrains Flamingo on interleaved image-text medical data sourced from publications and textbooks, unlocking few-shot medical VQA capabilities. Authors of VLLV (Li et al.) 2024) focus on rapid adaptation the medical domain by fine-tuning LL on filtered image-text pairs from PMC-15M (Zhang et al. (023a). Authors in /Zhang et al.) (202cc) generate a large-scale medical VQA dataset from PMC-OA (Lin et al. 2023) which is subsequently used to train a medical MILM. Moreover, authors in Wu et al. (2023b) propose multimodal foundation model, RadFM, for radiology, aligning natural language with 2D and 3D radiology images.
Encoding visual information in MLLMs The prevailing approach to incorporate visual infor- mation MLLM training leverages contrastive language-iman -mmge pretrained models as frozen image encoders. CLIP (Radford al. 2021), and its variants (Cherti al.) 2023), are trained internet- scale paired image-text data, and thus its representations are readily aligned with natural language, then adapted to the feature space of the language model using MLP heads (Liu et al.) 2024). Q-Former (Li et al. 2022). cross-attention (Alayrac al., 2022) other mechanisms. The image encoder acts as the "eye" of the MLLM as it directly determines what visual information enter the model. In fact, imperfect compression of relevant visual information is dominant issue with contemporary MLLMs. resulting in object hallucinations (Li et al.) Gunjal al. 2024), fundamental errors in spatial reasoning (Kamath al. 2023). and inability to understand inter-object relationships (Wu et al., 2024a).
and can be effectively combined with language models. The frozen representations
As the distribution of general 'internet data' and medical image-text data is markedly different, CLIP may be unable to capture the intricate structure medical images with fidelity sufficient for reliable performance. Researchers have proposed CLIP-like models pretrained on large-scale medical data better suited as image encoders for medical MLLMs. LLaVA-Med leverages BiomedCLIP (Zhang et al.) 2023b). a foundation model designed for biomedical image-text processing that has been pretrained on PMC-15M. MedVInT uses PMC-CLIP (Lin et al. 2023). a CLIP-style model pretrained PMC-OA with 1.6M medical image-caption pairs. The limitations of image encoders in medical

---

Published as a conference paper at ICLR 2025
MLLMs have attracted less attention than in the general domain, which is especially troubling due to the safety-critical nature of healthcare upplications. Thus, the lack of in-depth understanding of the shortcomings and possible failure modes of the image encoder in the medical MLLM pipeline is an exceedingly pressing concern.
With the recent rapid advances in developing medical MLLMs, there has been substantial effort in quantifying their performance in a wide range of tasks and areas within the medical domain. -QA-Rad (Lau al. 2018), SLAKE (Liu et al., 2021). Path-VQA (He et al. 2020) and VQA-Med (Ben Abacha et al. 2021 are widely-used to benchmark the performance of MLLMss in medical VQA. Due to their small size and limited scope, there has been a push for more comprehensive and diverse evaluation datasets. OmniMedVVA (Hu et al. (224) introduces the largest medical VQA dataset to date, encompassing 12 data modalities and 20 anatomical regions with a total of more than 100k images. Authors of Asclepius (Wang et al. 2024b) focus on eliminating data leakage present in other benchmarks and providing human evaluations. GMAI-MMBBnch (Chen et al.) 2024) incorporates problems probing the performance of MLLMs at various perceptual granularities, and targets a well-categorized data structure for ease of preparing customized evaluations. Other benchmarks extend the evaluation task beyond VQA in order provide more comprehensive view of model performance. MultiMedEvall (Royer et al. 2024) builds a uniform and fair benchmarking framework for multiple tasks including report generation and classification. Micro-Bench Lozano ett al. (2025) evaluates the visual understanding capabilities of MILMs across diverse microscopy modalities. RadBench (Wu et al. 2023b) focuses on radiology with associated tasks such as modality recognition and disease diagnosis. Authors of CARES (Xia et al., (024)) aim to provide a more holistic view of model performance by focusing on aspects such as fairness, privacy and safety of
Medical VQA benchmarks-
MLLMs as well as factual correctness.
All of these datasets are aimed at probing the medical knowledge of MLLMs and quantifying their average performance on a wide variety of tasks. modalities and anatomic regions. However, none of these benchmarks are specifically designed to probe the reliability, fundamental limitations and failure modes in the medical domain, all critical aspects in healthcare applications. Perhaps the closest work in spirit to ours is RadVUQA (Nan et al. 2024), where authors call attention to the critical deficiencies of existing medical MLLMs, revealing a large gap between ofate-of-the-arr MLLMs and clinicians. Their dataset focuses on more fundamental visual question answering and understanding, such as spatial reasoning, anatomic understanding and quantitative reasoning on medical images. However, we go a step further and design a benchmark that stress-tests the visual capabilities of MLLMs by curating questions expected to be challenging for their image processing pipeline.
Related to our work, Tong et al. 2024b) has investigated the failure modes of MI.LMs originating in ambiguous vision encoding in the general domain. Their study is based on finding CLIP--lind pairs, images that have high similarity in CLIP embedding space, but otherwise have dissimilar low-level image features. However, their methodology not directly applicable in the medical domain for two reasons. First, CLIP has been pretrained on general domain data and thus it is unable to capture the intricate structure of medical images. Second, their methodology relies on human annotators to describe the difference between a large number of image pairs, which is prohibitively costly in our scenario, as only radiologists are qualified to provide such annotations in the medical setting.
6 CONCLUSION
In this paper, we introduce MediConfussin, challenging medical VQA benchmark designed to probe the limitations of multimodal reasoning in medical MLLMs. In particular, we discover radiology image pairs that, due to ambiguities originating in their multimodal embedding spaces, confuse contemporary models despite being dissimilar in the image domain. We leverage an automated pipeline along with the expertise of radiologists to create dataset of VQA problems that tests the ability of MLLMs to effectively distinguish and answer clinically relevant questions about such confusing pairs. Our benchmark, by construction, cannot be solved by leveraging unimodal priors, and thus, it directly probes multimodal capabilities. We find that most existing models achieve performance no better than random guessing on MediConfusion, as models tend to select the same answer option for both images in the pair, raising serious concerns about the reliability of existing MLLMs in a medical setting. In order to guide future research in addressing the limitations of current MLLMs, we identify common failure patterns where models often break and relate them to known limitations in the general domain. We hope that our work sparks further research efforts to improve the reliability of AI for healthcare applications.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We would like to thank Microsoft for Accelerating Foundation Models Research grant that provided the OpenAl credits enabling this work. This research also part supported by AWS credits through an Amazon Faculty research award and NAIRR Pilot award. M. Soltanolkotabi is also supported by the Packard Fellowship in Science and Engineering, a Sloan Research Fellowship in Mathematics, an NSF-CAREER under award #1 #1846369, DARPA FastNICS program, and NSF-CIF awards #1813877 and #2008443. and NIH 0P11.004464-00..
REFERENCES
Jean-Ban nan-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances neural information processing systems, 35:23716-23736, 2022.
Asma Ben Abacha, Mourad Sarrouti, Dina Demner-Fushman Sadid A Hasan, and Henning Müller. Overview of the vqa-med task imageclef 2021: Visual question answering and generation in the medical domain. In Proceedings of the CLEF 2021 Conference and Labs of the Evaluation Forum- Forum-working notes. 21-24 September 2021, 2021.
Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. Vip-llava: Making large multimodal models understand arbitrary visual prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 12914-12923, 2024.
Pengcheng Chen, Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, et al. Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai. arXiv preprint arXiv: 003611 2024.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818 2818-2829, 2023.
Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang. Kuei-Da Liao, et al. A survey on multimodal large language models for autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 958-979, 2024.
Wenliang Dai, Junnan Li. DONGXU LI. Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. InstructBLIP: Towards General- purpose Vision-Language Models with Instruction Tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural In- formation Processing Systems, volume 36, pp- 49250 Curran Associates, Inc., htwpeeeeeeddinggserereeeeeee neur httpp/appppplllssppppppee//e/ Papeee Sappspaaeee--ppper---oeeeeeeeee
2023. URL ht httpppss//prrcceccnn.....rrppppp
e
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang. Jae Sung Park, Moham- madreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.171a. 17146, 2024.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03374 2023.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference Machine Learning, pp. 00835-10866 PMLR, 2023.

---

Published as a conference paper at ICLR 2025
Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18135-18143, 2024.
Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286, 2020.
Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Omnimedvqa: A new large-scale comprchensive evaluation benchmark for medical Ivlm. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22170-22183, 2024.
Jitesh Jain, Jianwei Yang, and Humphrey Shi. Vcoder: Versatile vision encoders for multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27992-28002, 2024.
Amita Kamath, Jack Hessel, and Kai-Wei Chang. What's "up" with vision-language models? investigating their struggle with spatial reasoning. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 9161 91611-917,,2000 2023.
Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1-10, 2018.
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decod- ing. In Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition, pp. 13872-13882, 2024.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2024.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. International conference on machine learning, pp. 12888-17 12888-12900. PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In The 2023 Conference on Empirical Methods in Natural Language Processing.
Weixiong Lin, Ziheng Zhao, Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-clip: Contrastive language-iman pre-training using biomedical documents. In International Conference on Medical Image Computing and Cosiuter-Assisted Intervention, pp. 525-536.
Springer, 2023.
Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically- labeled mowledge-enhancee dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp. 1650-1654. IEEE, 2021.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Alejandro Lozano, Jeffrey Nirschl, James Burgess, Sanket Rajan Gupte, Yuhui Zhang, Alyssa Unell, and Serena Yeung. Miero-bench: microscopy benchmark for vision- language understanding. Advances in Neural Information Processing Systems, 37:30670-30683 2025.
12

---

Published as a conference paper at ICLR 2025
Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (ML4H), pp. 353-367. PMLR, 2023.
Yang Nan, Huichi Zhou, Xiaodan Xing, and Guang Yang. Beyond the hype: A dispassionate look at vision-language models in medical scenario. arXiv preprint arXiv:2408.0088 08704, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Obioma Pelka, Sven Koitka, Johannes Rückert, Felix Nensa, and Christoph M Friedrich. Radiology objects in context (roco): multimodal image dataset. In Intravascular Imaging and Computer Assisted Stenting and Large-Scall Annotation of Biomedical Data and Expert Label Synthesis: 7th Joint International Workshop, CVII-STENT 2018 and Third International Workshop. LABELS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings 3. pp. 180-189. Springer, 2018
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 9748-8766 PMLR, 2021.
Corentin Royer, Bjoern Menze, and Anjany Sekuboyina. Multimedeval: A benchmark and toolkit for evaluating medical vision-language models. arXiv preprint arXiv:2402.09262, 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Can- dace Ross. Winoground: Probing vision and language models for lisouliin compositionality. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 2022.
Shengbang Tong, Erik Jones, and Jacob Steinhardt. Mass-producing failures of multimodal systems with language models. Advances in Nearal Information Processing Systems, 36, 2024a.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9568 -978,,97,,778 2024b.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302 2023.
Jiaqi Wang, Zihao Wu, Yiwei Li, Hanqi Jiang, Peng Shu, Enze Shi, Huawen Hu, Chong Ma, Yiheng Liu, Xuhui Wang, al. Large language models for robotics: Opportunities, challenges, and perspectives. arXiv preprint arXiv:2401.04334 2024a.
Wenxuan Wang, Yihang Su, Jingyuan Huan, Jie Liu, Wenting Chen, Yudi Zhang, Cheng-Yi Li, Kao-Jung Chang, Xiaohan Xin, Linlin Shen, et al. Asclepius: A spectrum evaluation benchmark for medical multi-modal large language models. arXiv preprint arXiv:2402. / /217, 2024b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Seff-consistency improves chain of thought reasoning in language models. arXiv preprint ar0iv:2233.1 1111, 2022.
Chaoyi Wu, Jiayu Lei, Qiaoyu Zheng, Weike Zhao, Weixiong Lin, Xiaoman Zhang, Xiao Zhou, Ziheng Zhao, Ya Zhang, Yanfeng Wang, et al. Can gpt-4v (ision) serve medical applications? case studies on gpt-4v for multimodal medical diagnosis. arXiv preprint arXiv:23 10. 10.09909, 2023a.
13

---

Published as a conference paper at ICLR 2025
Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang. and Weidi Xie. Towards generalist foundation model for radiology. arXiv preprint arXiv:2308. 02463, 2023b.
Mingrui Wu, Jiayi Ji, Oucheng Huang, Jiale Li, Yuhang Wu, Xiaoshuai Sun, and Rongrong Ji. Evaluating and analyzing relationship hallucinations in Ivlms. arXiv preprint arXiv:2406. 240.2446..444, 2024a.
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang. Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang, Liang Zhao, Yisong Wang. and Chong Ruan. Deepseek-v12: Mixture-of-experts vision-language models for advanced multimodal understanding, 2024b. URLpsps:://axxi
axg/abs/2412..2 10302
Peng Xia, Ze Chen, Juanxi Tian, Yangrui Gong, Ruibo Hou, Yue Xu, Zhenbang Wu, Zhiyuan Fan, Yiyang Zhou, Kangyu Zhu, et al. Cares: A comprehensive benchmark of trustworthiness in medical vision language models. arXiv preprint arXiv:2406.066007 2024.
Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lvim-ehub: comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306. 00060 2023.
Joy Qiping Yang, Salman Salamatian, Ziteng Sun, Ananda Theertha Suresh, and Ahmad Beirami. Asymptotics of language model aligmment. arXiv preprint arXiv:2404.01730, 2024.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? The Eleventh International Conference on Learning Representations, 2023.
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, et al. Large-scale -opaifi pretraining for biomedical vision-language processing, arXiv preprint arXiv:2303.00915. 2(3):6, 2023a.
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs, arXiv preprint 05X1v:2303.00915. 2023b.
Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pme-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint 2055 2023c.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A PROMPTS FOR DATASET CURATION
In this section, provide the prompts used to interact with GPT-4o for dataset generation and MLLM evaluation. Wherever we use **TEXT** we mean that TEXT is description or variable that is image/pair specific.
A.1 QUESTION GENERATION
We use the following prompt to generate question for single confusing pair. We describe the output format as detailed as possible to be able to process the answers with little human interaction.
System message:
You are a helpful assistant expert the medical
Here is the description of two medical images that can domain.
Prompt:
see:
Imagel: **CAPTION OF IMAGE 1**
Image2: **CAPTION OF IMAGE 2+*
Your task is to create miltiple-choice questions. Follow the rules below.
1. The question should be about a property that is clearly visible in the images.
2. It should be possible to answer the question by only looking at the images.
3. Pretend that you can only see one image. You are not allowed
to refer to 'Imaee'' 'Image2' or 'images'. You should also not create questions that require comparing the two images.
4. There should be exactly two answer options. You have to come
up with question for which the answer is different for the two images.
5. The answer options should be clearly different.
Please provide the question and the two answer options in the following format:
Question: <YOUR QUESTION>
Optionl: <ANSWER 1>
Option2: <ANSWER 2>
Also, please provide the correct answers the question for the
images corresponding to our captions in the following format:
Imagel: <ANSWER>
Image2: <ANSWER>
Aim for simple, efficient and concise questions to best test someone's knowledge and understanding of the underlying concepts.
A.2 CATEGORIZING IMAGES
To categorize the images, first show GPT-4o captions of several (here 100) images and ask it to separate them into different categories. Afterward, for each image, we ask GPT-4o to pick one of the categories for that image based its caption.
We used the following prompt to extract categories:
System message:
You are a helpful assistant expert in the medical
domain.
I have several medical images related to radiology that each one has corresponding ng caption. have listed the captions below. Can you go through the captions and categorize them?
Prompt:
15

---

Published as a conference paper at ICLR 2025
Please focus the general categories.
Caption 0: ** IMAGE 0 CAPTION*
Caption 1: IMAGE 1 CAPTION**
Caption 99: TMAGE 99 CAPTTON**
Using this prompt, we find 9 categories: Cerebral, Spinal, Cardiac, Gastrointestinal, Musculoskeletal, Vascular, Pulmonary, Head and Neck, Breast, and Other. We used the following prompt assign categories:
System message:
You are a helpful assistant expert in the medi cal
domain.
Prompt: Caption: IMAGE CAPTION**
The above caption describes radiology image. To which of the following categories does this image belong? You should only name the category, and you do not need to specify your reasoning. Categories: Cerebral, Spinal, Cardiac, Gastrointestinal, Musculoskeletal, Vascular, Pulmonary, Head and Neck, Breast, Other"
The final set of categories in our dataset are somewhat different, we incorporated feedback from the radiologist to revise the automatically generated categories.
A.3 FINDING FAILURE MODES
To find common failure modes that our dataset probes, we use the questions and captions of 100 pairs in the following prompt send to GPT-4o:
System message:
You are helpful assistant expert in the medical
domain.
image embedding model. have several image pairs that each one has a corresponding two choice question. know that the embedding model confuses the images about the corresponding question. Can you go through the questions, options, and image descriptions, trying to figure out some general patterns that the embedding model struggles with? Please focus on the visual features and generalize patterns that are important to vision models.
Prompt:
I am analyzing
Pair 0
First image description: *+IMAGE 1 CAPTION*
Second image description: IMAGE CAPTION*
Confusing multiple choice question: **QUESTTON**
Pair
First image description: IMAGE 1 CAPTION*
Second image description:
** IMAGE 2 CAPTION
Confusing multiple choice question: **QUESTION
Pair 99
First image description: Second image description: IMAGE 2 CAPTION* Confusing multiple choice question: **QUESTION**
**AME 1 CAPTION*
16

---

Published as a conference paper at ICLR 2025
A.4 FREE FORM EVALUATION
For the free-form (FF) GPT-4o evaluation, we pass the MLLM's answers with the following prompt GPT-4o to obtain two scores, for each answer option.
System message: You are helpful and precise assistant for checking the quality of the answer.
Prompt: [Question] **QUESTION**
[Answer A]
**OPTION A**
[Answer B]
**OPTION B**
[Assistant]
**RESPONSE*x
[End of Assistant]
[System]
We would like to request your feedback on the performance of an AI assistant in response to the user question displayed above. The observing an image. We have provided
user asks the question two possible answers, [Answer A] and [Answer B] to the question. Your job is to evaluate how close the AI assistant's answer is to each of the answers. You don't have to decide whether the correct or not. Each answer should receive overall score on a scale of 1 to 10, where a higher score indicates the AI assistant's answer is closer to the specific answer. After providing the scores, concisely provide your explanation for the given scores. Remember, you don't need comment on the correctness of the answers. Please provide your answer in the following format:
answers
A: <SCORE>
B: <SCORE>
Your explanation: <EXPLANATION>
These scores are the similarities of the MLLM's answer the different answer options. If the gap between the higher and lower score is at least & = 3, which we call the score gap threshold, we assign the option with the higher score the MLLM's output. Otherwise, mark the answer invalid.
B MODEL DETAILS
In this section, provide details on the versions and hyperparameters of MILMs that should be noted that for the multiple choice (MC) evaluation mode, we set temperature to 0, as we only expect a single letter option be generated.
B. MEDFLAMINGO FEW-SHOT PROMPTING
In order for MedFlamingo to produce valid responses, we need to use few-shot prompting. Here, we show three questions and answers from PMC-VQA benchmarks Zhang et al. (202cc). The following is the prompt used for MC evaluation:
17

---

Published as a conference paper at ICLR 2025
MLLM
Version/LLM
Temperature
Beams
Top p
LLaVA
v1.6/Mistral 7B
0.2
BLIP-2
Opt 2.7B
0.9
InstructBLIP
Vicuna 7B
5
0.9
DeepSeek
VL2
0.7
Molmo
7B
0.7
LLaVA-Med
V1.//Misttal 7B
0.2
RadFM
Med-Flamingo
1
5
0.9
GPT
4o (release 20240513)
0.7
(release 20241217)
0.7
Claude
3 Opus
0.2
Gemini
1.5 Pro
0.2
Gemini
2.0 Flash
0.2
Table 3: MLLM details
You are helpful medical assistant. You are being provided with images, a two choice question about each image and an answer. Follow the examples and answer the last question. <image>Question: What radiological technique was used to confirm the diagnosis?
Prompt:
A: CT Scan
B: Mammography
Answer: B:
What did
the CT scan show?
A: Cerebral edema
B: Intracranial hemorrhage
Answer: A: Cerebral edema endofchunk >ximage>Question: What is the purpose the asterisk shown in the figure? A: To indi cate the formation of lobes around the contracting nucleus.
B: To indicate the normal lentoid shape of hypocotyl nuclei.
Answer: B: To indicate the normal lentoid shape of hypocotyl >><image>
Question: **QUESTION**
A: **OPTION A**
B: **OPTION B**
Answer:
The following the prompt we used for FF evaluation:
You are helpful medical assistant. You are being provided with images, question about each image and an answer. Follow the examples and answer the last question. <image>Question: technique was used to confirm the diagnosis? Answer: htaptogrrppp</eenooooooo peapofffhnnkkxiiaaeeeeeeeeete |xinnage>question: What did the CT scan show? Answer: Cerebral edemalendor endoffhuukk >ximage>Question: What is the purpose of the asterisk shown in the figure?
Prompt:
What radiological
18

---

What did Published as a conference paper at ICLR 2025
Answer: To indicate the normal lentoid shape of hypocotyl
nuclei. endofchunk Suaauuueetiiimmmaeemuettttttti
** **UESSTION** Answer:
The following is the prompt we used for GD evaluation:
helpful medical assistant. You are being provided with images, a two choice question about each image and an answer. Follow the examples and answer the last question. <image>Question: What radiological technique was used to confirm the diagnosis?
Prompt:
You
A: CT Scan
Mammography
Answer: B: (ammoorappyyy/eooocccc//>>>>>>>/ A
the CT scan show?
A: Cerebral edema
B: Intracranial hemorrhage
Answer: A: Cerebral edema endofchunk >>image>Question: What is the purpose of the asterisk shown in the figure? A: To indicate the formation of lobes around the contracting nucleus.
B: To indicate the normal lentoid shape of hypocotyl nuclei. Answer: B: To indicate the normal lentoid shape of hypocotyl muclei.</endoffll Question: **QUESTION**
>><image>
A: **OPTION A**
B: K*OPTION B**
Answer:
The following the prompt we used for PS evaluation:
You are helpful medi cal assistant. You are being provided with images, question about each image and an answer. Follow the examples and answer the last question. <image>Question, What radiological technique used to confirm the diagnosis? Answer: What did the CT scan
Prompt:
show? Answer: Cerebral edema< ennoffhhnk ><image>Question: What is the purpose of the asterisk shown in the figure? Answer: To indicate the normal lentoid shape of hypocotyl nuclei.< pooolei.//endofchiiie /tcpsfchhhnn/iiiiieeeeeeee />cimage-Question. **QUESTION** Answer:
**ANSWR**
C ABLATION ON PROMPTING
In this section, perform an ablation study the effect of prompt variations on model performance. We test GPT-4o, InstructBIIIP and -LLaVA-Med with MC evaluation. We manually create 10 prompt variations different from the original, shown in Table 4 We do not rely LLMs for rephrasing the original prompt to avoid biases and artifacts originating in -LM-generated text. To capture variability in accuracy both due to prompting and stochasticity, sample 10 answers for each model, and for each prompt. Table shows our results.
First, observe that the performance of GPT-4o is robust the prompt format, but the performance still below random guessing. InstructBLIP more sensitive to the particular input prompt, as demonstrated by significantly reduced performance some prompts. Lastly, L.LVA-Medd struggles
19

---

Published as a conference paper at ICLR 2025
with the MC evaluation, as it has not been specifically trained to follow the multiple-choice format, and prompt engineering does not fix this issue.
Table 4: List of prompts we use to evaluate sensitivity of model performance to prompt variations. The second column shows the high-level rationale for the variation, and the specific prompt is included in the last column.
Change
Prompt
Original
image, choose correct option following question.
Question:
A:
B:
Answer with the option's 's letter from the choices directly. Your answer should be just one
Answer:
Shorten
Answer the following question about the image.
Question:
A:
B:
with option's 's letter directly.
Answer:
Move answer options to the end
Answer the following question about the image with the correct option' is letter directly.
Question:
A:
B:
Answer:
Instruct the for extra attention
Carefully and skillfully reviow the image answer following question.
Question:
A:
B:
Answer with the option's 's letter directly.
Answer:
Al expert role
an expert radiologist deep knowledge radiology image analysis. Based on the image, choose the correct option for the following question.
Question:
A:
B:
Answer with option's 's letter directly.
Answer:
Rephrase the instruction to answer with a single letter
Based on the image, choosee the correct option for the following question.
Question:
A:
B:
Simply respond with 'A' based your
Answer:
Ask the model nicely
Based on the image, please choose the correct option for following question.
Question:
A:
B:
Please with the option's letter from given choices directly.
Answer:
Remove now lines from formatting
Based on the image, choose the correct option for following question. Question:
Option A: Option B: Answer with the option's 's letter from the given choices directly.
Your should just one
explanation
are given a radiology image, based which you medical question related to the image. You will he given two answer options: A and The correct
carefully answer
from which you have to choose the right
can always be
determined just by looking at the radiology image itself. Take a close look at the image, think carefillly, and answer the following question. Question:
A:
B:
Answer with the option's 's letter from the given choices dircetly. Your answer should be just one letter.
Answer:
No instractions
Question:
A:
B:
Correct answer option:
Emphusize importance
are given midioloyy image, and your critical the conrect option for the following question. It is extremely important to answer the following question correctly.
look choose
Question:
A:
Answer with the option's is letter from the given choices directly. Your answer should be one letter.
Answer:
20

---

Published as a conference paper at ICLR 2025
Table 5: Results of prompt variations. The table shows the mean and standard deviation of accuracy (in %) across 10 random samples for each prompt. The last column aggregates the results across all prompts.
Prosp13
19.33
19:0:110:0
1501 4.00
17.30
18:00
10.0
10.03 1.82
70.00-1:10
19.03-1330 1:30
19.00
19.19
13.55
5.7
575 5.5.1.0277
13k 1.44
147 1:135
1957
0.00
1.86
1240 3:15
1.0.3
ELNNMM
600
500+0000
600+0000
0:0000000
0:000:000
6.000000
D FINE-TUNING ON MEDICONFUSION
In this section, we investigate the effect of fine-tuning medical MLLMs on MediConfusion. Fine- tuning the specific benchmark typically helps the model better understand the format of the VQA and can greatly boost the performance of the model on the particular benchmark.
We create train and test splits from MediConfusion by randomly sampling 50% of confusing pairs from each category to train on, and hold out the other half for evaluation. We ensure that the two images from a confusing pair always belong to the same split order avoid leakage. As each pair may belong to multiple categories, the above process results in two splits of slightly different size (84 pairs in train and pairs test), but with category distribution close to the full dataset. We following common procedure to fine-tune LLaVA-Med on the train split, involving fine-tuning the language model and the multimodal adapter, keeping the image encoder frozen. We perform a search over learning rates within [10-6 5 10 10-6 10-5 10-1,2 2 10-1 and select 5 10-6, We keep other fine-tuning hyperparameters at their recommended values. We perform full fine-tuning, as opposed to parameter-efffciennt fine-tuning approaches, and train for up to 1000 epochs. Train and test (set) accuracy across epochs are depicted in Figure 6
Interestingly, we observe that the model struggles to achieve 100% set accuracy the training set, even after a very high number of training epochs (typical fine-tuning is within 1 - 50 epochs), hypothesis that the model is unable to differentiate between image pairs due to fundamental ambiguity in their embeddings. In particular, the vision features appear so similar to the model that even fine-tuning the whole language model is insufficient. This experiment suggests that MediConfusion can only be solved by improving the vision encoder directly.
further supporting
Furthermore, we find that fine-tuning helps LLa learn the multiple-choice format (set accuracy on the test split increased from 0% to around 20%), however its performance is still well below random guessing as the knowledge learned on the training set does not generalize to the test set.
LLaVA-Med finetuning
Tuta
Set accuracy
Epochs
Figure 6: Evolution of train and test set accuracies during LLaVM- fine-tuning over epochs.
21

---

Published as a conference paper at ICLR 2025
E ABLATION STUDY ON DATASET DIFFICULTY
We investigate the effect of the difficulty of confusing pairs in our benchmark construction. Specifi- cally, we create "easy" version of MediConfusion from image pairs that look dissimilar medical vision-language models, which we refer to as MediConfusion-easy We extract pairs from ROCO that have low BiomedCLIP similarity (sittmeed S 0.4). As radiologist feedback is prohibitively costly and time-consuming, we build MediConfuuiin--ee as we use in curating MediConfusion, but skip radiologist feedback. Thus, MediConfusion easy may include questions that are clinically irrelevant or have incorrect answers. The resulting dataset consists of 988 pairs (1976 VQA problems).
by following the same automatic pipeline
To make sure that the performance gap between MediConfusion and MediConfusion-caa is not due to the lack of radiologist feedback, we also create MediConfusion MeddiConfusion-uufitteee using identical hyperparameters and procedure to the original benchmark curation, but without radiologist feedback. We match the number of samples to MediConfussonn-aaayy
We evaluate GPT-4o (MC evaluation) and Laa (PS evaluation) on MediConfusion-casy Table O summarizes our results. First, we highlight that removing radiologist feedback does not necessarily improve performance (GPT-4o has slightly lower set accuracy, LLa performed somewhat better on unfiltered data). Furthermore, our experiment demonstrates that models can indeed differentiate between image pairs in our confusing pair-based VQA format, if the images are different enough, as evidenced by the nearly 100% accuracy of GPT-4o on MediConfusion easy.. Thus, this experiment further supports the need for a striet selection criteria for confusing pairs, as easier variants of our VQA may be unable to identify the shortcomings of state-of-the-ard models.
Table 6: Set accuracy, individual accuracy, and confusion (all in %) of GPT-4o (MC evaluation) and I.LaVA-Med (PS evaluation) on the easy variant of our benchmark. MediConfusion-easy is not verified by radiologist, and thus for fair comparison we include an unfiltered version of MediConfusion as well.
McdiConfusion-EE
McdiConffuioonnnniitee
MediConfusion
Model
Set acc.
Confusion
Indiv, acc.
Set acc.
Confusion
Indiv, acc.
Set acc.
Confusion
Indiv; acc.
GPT-40
98.38
1.32
99.04
18.01 4.86
77.83
55.97
18.75
75.00
56.25
LLaVA-Med
34.21
65.59
66.90
95.75
51.06
0.00
97.16
49.72
We believe that selecting challenging pairs with high BiomedCLIP similarity, done in work, contributes to the difficulty in 2 ways. First, due to the ambiguity of input vision features, the model unable resolve enough details in the medical images that is sufficient to answer the questions correctly. Second, the questions themselves are also more challenging, as the difference between hard confusing image pairs is more nuanced. Thus, the model needs more medical knowledge to correctly answer the question, even if the vision embeddings are sufficiently dissimilar. We hypothesize that this medical knowledge lacking in the case of the Gemini model our experiments, where we observe poor performance despite low confusion. To highlight how questions are more general and potentially require less medical knowledge, we show an example from MediConfusion-eayy in 7 where identifying the anatomical region is sufficient for solving the VQA pair.
F INVESTIGATING FEATURE SIMILARITY OF VISION ENCODERS
We find confusing pairs based on BiomedCLIP similarities to construct MediConfusion. The resulting poor performance across all models implies that they struggle differentiating between the confusing pairs, however it is unclear how similar the images are in the respective models' own image feature space. this section, we analyze the similarity of MediConfusion pairs in the feature space of vision encoders used by open-source models in our experiments: CLIP and the visual encoder of RadFM. RadFM has its own vision encoder trained on medical images, including 3D radiology images. Figure 8 shows the histogram of similarities between confusing pairs based on the cosine similarity of BioMedCLIP, CLIP, and RadFM visual embeddings. We average pool across the token dimension in the output of vision encoders to produce the visual embedding.
As these plots show, the images have high similarity in not only BiomedCLIP's s, but other vision encoders" embedding spaces, both trained on general-domain data and medical data.
22

---

Published as a conference paper at ICLR 2025
Question: Is the image showing condition related to the respiratory system or the reproductive system?
A: Respiratory system
B: Reproductive system
Answer:
Answer: B
Figure 7: A sample VQA pair from MediConfusion-eaay. Identifying the anatomical region is sufficient to answer pair correctly.
BiomedCLIP
CLIP
RadFM
0.30
Density
Density
Density
cosine similarity
cosinc similarity
cosine similarity
Figure 8: Histograms of the cosine similarity of confusing pairs in different visual representations.
G REVERSING THE ROLE OF DINO AND BIOMEDCLIP
As an ablation study, we explore samples that have high DINO similarity but low BiomedCLIP similarity. In particular, searched for image pairs for which sim S 0.5 and simgen 0.5. shows
Figure shows some of these samples.
Figure Sample with high DINO similarity but low BiomedCLIP similarity
H EXPLORING TECHNIQUES TO IMPROVE PERFORMANCE ON MEDICONFUSION
In this section, we examine various methods we employed to improve model performance on MediConfusion and analyze their effectiveness.
23

---

Published as a conference paper at ICLR 2025
Table Performance of LLaVA-Med model with added DINOv2 embeddings
Set acc. (%)
Indiv, acc.(%)
Confusion (%)
Best
Method
MC
GD
PS
MC
GD
PS
MC
GD
PS
Set acc.
Indiv, acc.
LLaVA-Med
0.00 0.00 1.14
23.58 49.72
49.72
100.00
99.43
97.16
1.14
49.72
LLaVA-Med DINOv2 embed.
2.84
2.84 2.27
45.45
5U.57
50.00
95.54
95.45
95.45
2.84
50.57
Table 8: GPT-4o Best of 100 performance with an oracle as the scoring metric
Frequency threshold (%) Set acc. (%)
Indiv, acc. (%)
47.16
68.18
2
37.50
62,50
32.95
59.38
4
27.84
55.97
5
24.43
53.41
H.1 MIXTURE OF FEATURES
Following (Tong et al. 2024b), we investigate the effect of incorporating DINOv2 embeddings into the image encoding of LLa VA-Med. Specifically, we concatenate the DINOv2 embeddings with the existing CLIP embeddings to create an enriched image representation. We then apply the feature alignment and instruction tuning procedures outlined (Liu et al. (202)) and train the model from scratch. Additionally, we follow the medical-specife feature alignment and instruction tuning methodologies from (Li et al. 2024).
Table 7 compares the performance of this modified model with the original LLaVA-Med. While observe a slight improvement in accuracy, overall performance remains poor, indicating that richer visual features alone insufficient solve MediConfusion.
H.2 BEST-OF-N
The best-of-N method is widely used approach for enhancing MLLM responses( (Stiennon et al. 2020). (Gao et al., 2023), (Yang al.) (2024)). In this technique, we generate and select the best one based predefined scoring metric. For this study, we use GPT-4o and explore different scoring strategies.
responses(Stieenon N different responses
H.2.1 ORACLE-BASED SCORING
To establish an upper bound on model performance, we employ an oracle-based scoring method, where the oracle has access to the ground truth solutions. We hypothesize that if the model is capable of correctly answering the questions with some reasonable, albeit low, probability, then improved prompting strategies can potentially elicit its medical knowledge. Specifically, we follow the FF evaluation framework, where we first present only the question the model and generate response. We then provide GPT-4o with the question, answer choices, and the generated response, asking to determine which option closest to the generated answer. for each question. Furthermore, we lower the score gap threshold to k 2. We then measure the frequency with which the correct response appears among these generations. We accept an answer the frequency of the correct answer exceeds a certain threshold. Table @able&&pree presents the results of this experiment. As we can see, even under these lenient conditions, the model performs poorly, with its performance declining significantly as the acceptance threshold increases.
sample N 100 responses
H.2.2 RANKING
Now, we consider more realistic setup in which we do not have access to the ground truth. Similar to the previous method, we generate different samples, then we show the question, options, and all of the generated answers GPT-4o and ask it to select the best response. We then pass the selected answer as the final answer of the model. Table provides the results of this section for different choices of N. Note that since using the FF evaluation, in which the model cannot the
24

---

Published as a conference paper at ICLR 2025
Table 9:
GPT-4o best-of-N performance with a judge the scoring metric
Set acc. (%)
Indiv, acc. (%)
Confusion (%)
Invalid (%)
10
5.11
33.24
81,58
41,48
4.55
30.97
81.33
43.75
7.39
36.08
79.76
38.64
Table 10:
GPT-4o Majority voting with N answers
N
Set acc. (%)
Indiv. acc. (%)
Confusion (%)
1
18.75
56.25
75.00
10
23.30
57.39
68.00
100
22.73
57.39
69.32
answer options prior to generating the response, the performance is much lower compared to the MC evaluation results in Table These results show that even when we are aggregating 50 responses, the model's 's performance remains poor, suggesting the presence of fundamental limitations that prevent it from effectively solving these problems.
H.2.3 MAJORITY VOTING
Self-consistency Wang et al. 2022) sampling can improve reasoning performance by marginalizing over multiple reasoning paths to aggregate the final answer. We generate N different answers using the MC evaluation and then choose the most frequent one as the model's final answer. As shown in Table 10, with N 10 we observe 5% improvement, but increasing N further yields negligible gains.
25


Reviews:
[{"review_id": "wJQ0Zl6PTN", "reviewer": "Reviewer_QM8Q", "summary": "The paper introduces MediConfusion, a benchmark specifically designed to assess and expose failure modes in multimodal large language models (MLLMs) used in medical visual question answering (VQA) tasks. The benchmark highlights that current state-of-the-art MLLMs, including both open-source and proprietary models, struggle significantly with distinguishing visually dissimilar yet clinically distinct medical images, often performing below random guessing on the dataset. MediConfusion is created by identifying “confusing image pairs” from the ROCO radiology dataset, where image pairs appear distinct to medical professionals but prove challenging for MLLMs due to limitations in visual feature space alignment.\n\nThe authors reveal key failure patterns among MLLMs, such as difficulties in differentiating between normal and pathological anatomy, recognizing lesion characteristics, and identifying medical devices and vascular conditions. The benchmark leverages radiologist oversight to ensure clinical relevance and accuracy in the question-answer pairs, aiming to guide future MLLM improvements in medical contexts. By exposing these systematic vulnerabilities, the paper underscores the critical need for more reliable and trustworthy AI systems in healthcare applications.", "questions": "1.\tDataset Diversity and Expansion: Could you elaborate on the decision to limit the benchmark to the ROCO dataset? Do you have plans to expand MediConfusion to include other datasets, such as ultrasound or nuclear imaging, to capture a broader range of diagnostic tasks? Additional dataset diversity might reveal new model weaknesses or challenges in different medical imaging modalities.\n\t2.\tImpact of Prompt Engineering: Given the importance of prompt phrasing in VQA tasks, did you experiment with alternative prompt formats or styles (e.g., framing questions in different ways or including detailed instructions)? If so, what impact did this have on model performance, and could further prompt optimization improve reliability? Insights here could guide future research on prompt refinement in medical VQA.\n\t3.\tFuture Directions for Medical MLLM Improvements: Given the identified failure patterns (e.g., difficulties with spatial reasoning or lesion characteristics), do you have specific recommendations for architectural or training adjustments that could address these weaknesses? For example, do you see value in integrating spatial attention mechanisms or training models on more anatomically detailed data?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The paper addresses a critical gap in the assessment of medical MLLMs by introducing MediConfusion, a novel benchmark explicitly designed to test the reliability of these models in healthcare. While prior benchmarks focus on overall performance in typical medical scenarios, this work innovatively emphasizes systematic failure modes by identifying “confusing image pairs” that challenge models with subtle, clinically important distinctions. The benchmark not only probes weaknesses but also highlights specific visual reasoning failures, which is a unique contribution to medical AI evaluation.\n\nQuality: The dataset creation is rigorous, combining automated techniques with expert radiologist validation to ensure that each question is clinically relevant and accurately represents medical knowledge. The experimental design is robust, systematically evaluating a range of state-of-the-art models (both general and medical-domain MLLMs) and comparing their performance against baseline expectations. The detailed breakdown of common model failure patterns enhances the paper’s depth, providing actionable insights for improving MLLM robustness in medical settings.\n\nClarity: The paper is well-structured and presents its findings in a clear, logical manner. Descriptions of the dataset creation process, evaluation methods, and results are easy to follow, with visual aids (e.g., figures illustrating confusing image pairs) that help readers understand the complexity of the visual tasks posed to models. The breakdown of error types further clarifies the distinct challenges MLLMs face, enhancing the reader’s comprehension of the benchmark’s significance.\n\nSignificance: The benchmark’s focus on failure modes has broad implications for the deployment of MLLMs in healthcare. By surfacing limitations that could compromise patient safety, the work serves as a foundational tool for future research aimed at increasing AI reliability in clinical practice. Additionally, MediConfusion highlights the limitations of current visual encoders and training paradigms in handling medical data, offering a clear research direction for building more robust and trustworthy models. The benchmark’s significance is further bolstered by its potential as a standard for evaluating and guiding improvements in medical AI systems.", "weaknesses": "Limited Dataset Size: While the MediConfusion benchmark is rigorous, the dataset size (352 questions across 9 categories) may be relatively small to capture the full spectrum of medical image complexity. Expanding the dataset with a broader range of confusing image pairs, potentially covering additional anatomic regions and diagnostic subtleties, could improve its generalizability and make it more comprehensive for model evaluation across diverse medical contexts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "81tdAhEeHR", "reviewer": "Reviewer_bwv6", "summary": "This paper evaluates reliability of medical VLMs. The authors introduce MediConfusion, a benchmark comprising 352 pairs of “confusing” medical images. Each pair consists of images that are visually distinct yet similar in feature space. The authors leverage GPT-4 to generate questions for which the answers differ between the two images. The benchmark data is further evaluated by a radiologist. Experimental results indicate that MediConfusion poses significant challenges to several existing VLMs, highlighting areas for improvement and setting the stage for future research.", "questions": "1. Are there confusing pairs that appear visually similar but differ in feature space?\n2. How would the Mixture-of-Features method from \"Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs\" apply to medical VLMs?\n3. What are the limitations of the confusing pairs identified here? Could there be potential biases?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper addresses a well-motivated and underexplored problem in the medical domain.\n2. The benchmark creation process is robust, with manual evaluation by a radiologist enhancing its credibility.\n3. The experiments are comprehensive, demonstrating that MediConfusion presents substantial challenges to current VLMs, with promising potential for inspiring future research.", "weaknesses": "While this work provides valuable insights, it appears to closely follow the Multimodal Visual Patterns (MMVP) benchmark framework, potentially limiting its novelty. Many findings align with known challenges in general VLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uokAd6anl0", "reviewer": "Reviewer_cT2V", "summary": "This paper evaluates the reliability of medical Multimodal Large Language Models in healthcare. The authors present MediConfusion, a new VQA benchmark that highlights distinguishing similar medical xray images. They find that all tested MLLMs, open-source and proprietary, perform below random chance on MediConfusion, indicating serious reliability issues.", "questions": "- Does the proposed benchmark dataset only contain 352 image pairs?\n\n- The authors only evaluate the models on the similar pair images. Can the models work well on different image pairs? For example, to use different sim_med and sim_gen thresholds and compare the model performance.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea of using the same question with similar images could offer a interesting way to probe model weaknesses.\n\n- With the help of an expert-in-the-loop pipeline, the paper thoroughly examines why models struggle to differentiate between visually similar medical images.\n\n- The evaluation metrics are comprehensive, particularly accommodating models unable to directly answer multiple-choice questions.", "weaknesses": "- The motivation is not clear. The authors claimed that \"existing benchmark datasets are focused on evaluating the medical knowledge of MLLMs across large evaluation sets, heavily biased towards common or typical scenarios.\" What are the common and typical scenarios? Could you provide a more detailed discussion about this limitation of existing benchmarks and how the proposed benchmark in this paper can overcome the limitations?\n- The results show that all available models (open-source or proprietary) achieve performance below random guessing on MediConfusion. It would be better if the authors conducted more experiments to find the reasons. For example, by using different sim_med and sim_gen thresholds and compare the model performance, we can identify whether the models can differentiate easy image pairs or they even cannot differentiate any image pairs. \n- The writing style needs refinement. For instance, the phrase \"stress-tests the visual capabilities of state-of-the-art models\" is unclear and may confuse readers. The motivation and innovative aspects of the benchmark remain vague until \"image pair\" is mentioned at the end of Section 2, and even then, it is not clarified that the task is framed as multiple-choice questions. It would be easier for the readers to better understand this paper if the writer could clarify these designs in the introduction section.\n- The rationale for focusing on the medical domain is unclear. The methods do not appear specific to this field, resembling a rephrasing of existing VQA datasets where answers are scrambled, and models are tasked with similar image comparisons. Could you explain what is the specific design of this benchmark considering its medical context? What is the difference between two similar medical images and two similar general images?\n- The choice to format the task as multiple-choice is questionable. Have you considered open-ended QA or report generation of CXR images? Additionally, given the relatively small dataset (352 questions), this benchmark may lack the scale needed to effectively assess MLLM capabilities. Could you justify that 352 questions are sufficient for the evaluation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VJnqogir9y", "reviewer": "Reviewer_772B", "summary": "The paper introduces MediConfusion, a challenging medical Visual Question Answering (VQA) benchmark that exposes the vulnerabilities of current medical multimodal large language models (MLLMs) by highlighting their failure modes. Despite promising progress, existing models—open-source and proprietary—struggle significantly, even performing below random guessing in certain tests. This suggests major reliability issues in deploying these models for healthcare. The benchmark was guided by clinician experts, using adversarial examples to stress-test models, offering a valuable resource for the research community to design more trustworthy MLLMs. Personally I like this paper because it hits on a crucial gap in medical AI that’s often overlooked.", "questions": "See the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper brings a novel perspective by focusing on the systematic failure modes of medical multimodal models—something that hasn’t been extensively explored before. The fact that state-of-the-art models performed at or below random guessing levels is a striking finding. I can see this dataset becoming a benchmark everyone wants to use as soon as the paper is out, as it offers a strong challenge to prove model reliability.\n\n2. The presentation and writing are clear and well-structured.", "weaknesses": "1. One significant limitation is that it feels almost obvious to train a model using both public data and this new dataset to demonstrate improved performance over current models. It’s surprising this wasn’t attempted—even for a dataset paper. Adding such experiments could have strengthened the impact by showing how incorporating MediConfusion directly improves model reliability.\n\n2. The authors might also want to discuss the importance of the image encoder in these models. For example, both LLaVA and LLaVA-med use encoders trained on everyday images, which could be a reason why LLaVA-med performs even worse than LLaVA. If the encoder isn’t well-suited for medical images, it limits the model's ability to learn effectively. Highlighting the role of specialized image encoders could have added valuable insights into improving model reliability in the medical domain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)