MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models

## Abstract

Abstract Multimodal Large Language Models (MLLMs) have tremendous potential to improve the accuracy, availability, and cost-effectiveness of healthcare by providing automated solutions or serving as aids to medical professionals. Despite promising first steps in developing medical MLLMs in the past few years, their capabilities and limitations are not well-understood. Recently, many benchmark datasets have been proposed that test the general medical knowledge of such models across a variety of medical areas. However, the systematic failure modes and vulnerabilities of such models are severely underexplored with most medical benchmarks failing to expose the shortcomings of existing models in this safety-critical domain. In this paper, we introduce MediConfusion, a challenging medical Visual Question Answering (VQA) benchmark dataset, that probes the failure modes of medical MLLMs from a vision perspective. We reveal that state-of-the-art models are easily confused by image pairs that are otherwise visually dissimilar and clearly distinct for medical experts. Strikingly, all available models (open-source or proprietary) achieve performance below random guessing on MediConfusion, raising serious concerns about the reliability of existing medical MLLMs for healthcare deployment. We also extract common patterns of model failure that may help the design of a new generation of more trustworthy and reliable MLLMs in healthcare.
The evaluation code and the dataset are available at our github .

## 1 Introduction

Multimodal Large Language Models (MLLMs) have demonstrated unprecedented capabilities in a variety of multimodal tasks, including image understanding and visual reasoning, autonomous driving (Cui et al., ), robotics (Wang et al., ) and embodied AI (Driess et al., ). Motivated by this success, a growing body of work (Moor et al., ; Li et al., ; Lin et al., ) explores the potential of MLLMs in medical applications with the hope of paving the way to more accurate, personalized and cost-effective healthcare solutions through modern generative AI.

Even though MLLMs show enormous potential in a wide range of tasks, a swath of challenges have stymied their deployment, including object hallucinations , relationship hallucinations (Wu et al., ), inaccurate object counting (Jain et al., ) and lack of spatial reasoning capabilities (Kamath et al., ). These shortcomings are especially worrisome in safety-critical applications, such as healthcare, where reliability is an essential requirement. In fact, recent research efforts on medical MLLMs have revealed weak anatomic knowledge (Nan et al., ), concerns on toxicity and patient privacy Xia et al. (), highly unreliable disease diagnosis Wu et al. (), and the fact that even a junior doctor far outperforms the most proficient medical MLLMs across a wide spectrum of tasks Wang et al. (). As model failure in the medical domain can lead to serious adverse health effects, it is of utmost importance to understand the performance and limitations of generative AI in the medical context.

A flurry of activity has emerged around probing the performance of medical MLLMs in a multitude of tasks, including visual question answering (VQA) (Ben Abacha et al., ), disease classification and report generation (Royer et al., ), and modality recognition (Wu et al., ). Even though the proposed medical benchmarks offer valuable insights on model performance across a variety of anatomic regions and imaging modalities, they are focused on evaluating the medical knowledge of MLLMs across large evaluation sets, heavily biased towards common or typical scenarios. Therefore, it is unclear how well the measured performance correlates with the actual multimodal medical reasoning capabilities of these models, especially in the face of systematic but perhaps more intricate model failures underrepresented in the dataset. Therefore, developing new evaluation benchmarks that carefully test and probe the capabilities of these systems, expose their vulnerabilities, and facilitate the development of a better understanding of failure modes is vital in healthcare applications.

In this work, we introduce MediConfusion, a challenging benchmark for evaluating the failure modes of medical MLLMs from a vision perspective. We combine novel insights on the image representations of medical MLLMs with the expertise of clinical radiologists to craft a benchmark dataset that stress-tests the visual capabilities of state-of-the-art models. In particular, our work reveals that medical MLLMs often confuse image pairs that otherwise appear very different in the image domain. Leveraging this observation, we introduce an automated pipeline to discover such pairs in the ROCO (Pelka et al., ) multimodal radiology dataset. Then, in collaboration with radiologists, we curate a VQA benchmark of clinically relevant multiple-choice problems designed to probe the model’s ability to distinguish between such images. By design, relying solely on unimodal (language) priors cannot achieve better than random guessing accuracy on our benchmark, and therefore performance on MediConfusion directly correlates with multimodal medical reasoning and image understanding capabilities. Remarkably, we discover that both state-of-the-art medical MLLMs, as well as the most advanced proprietary models, are easily confused by the image pairs, resulting in performance below random guessing for all models at the time of writing this paper. What is striking about this poor performance is that for some of the models (i.e. all medical MLLMs we studied) the images and corresponding captions are part of the training data!(^1^11Given the public nature of the original dataset such images and captions are also likely part of the pre-training of proprietary models such as OpenAI’s GPT-4o, Google Deep Mind’s Gemini 1.5 Pro, and Anthropic’s Claude 3 Opus.) Finally, we leverage our pipeline to categorize failure cases in order to guide future research toward more reliable medical AI solutions.

Figure: Figure 1: Overview of MediConfusion curation pipeline. First, we extract image pairs from the ROCO radiology dataset that are clearly distinct in the image domain, but may be challenging to differentiate between for multimodal models (left). Next, we use an automated pipeline leveraging LLM prompting to generate VQA from the confusing pairs and their corresponding captions (center). Finally, we incorporate radiologist feedback to filter questions for correctness, relevance and quality, and to revise the questions and answer options for improved medical language and precision (right).
Refer to caption: extracted/5869430/plots/overview_v4.png

## 2 The MediConfusion Benchmark

The majority of existing multimodal foundation models leverage CLIP (Radford et al., ) to encode the input image (Li et al., ; Liu et al., ; Moor et al., ; Li et al., ). CLIP has been pretrained on internet-scale general domain data and, therefore, may not be suitable for the nuanced representation of medical images due to the considerable distribution shift. Thus, variants trained on large-scale medical image-text datasets have been introduced as image encoders for medical agents, including BiomedCLIP (Zhang et al., ) and PMC-CLIP (Lin et al., ). Due to the specialized training data, these models are able to better capture the structure and semantics of medical images. However, surprisingly, we observe that the feature space of even specialized medical encoders is often not rich enough to clearly differentiate between images that are otherwise highly dissimilar. As a result, medical MLLM that leverage such input features suffer from impaired image understanding and visual reasoning, casting serious doubt on the reliability of such models in critical medical diagnosis. Therefore, designing challenging benchmarks that stress-test the visual capabilities of medical MLLM is of utmost importance for gaining a better understanding of the limitations of existing models.

In this work, we introduce the MediConfusion Benchmark, a challenging multiple choice medical visual question answering benchmark designed to probe the reasoning capabilities of medical MLLMs. The overview of our curation pipeline is summarized in Figure 1. First, we extract image pairs that are visually clearly different, but MLLMs will likely confuse them due to their similar features in embedding space. Next, based on the captions corresponding to each of the images in the confusing pairs, we generate a large pool of multiple choice problems via LLM prompting. Finally, each question in the LLM-generated pool is scrutinized and revised by an expert radiologist before being added to MediConfusion. We evaluate a range of state-of-the-art medical and general domain MLLMs and demonstrate that even flagship proprietary models have performance worse than random guessing.

### 2.1 Discovering confusing pairs

We find confusing image pairs in ROCO (Pelka et al., ), a multimodal dataset of $\approx 80k$ radiology images and their corresponding captions extracted from PMC-OA (Lin et al., ) (Figure 1, left). Inspired by Tong et al. (), we seek out pairs with clear visual differences, but high similarity in the feature space of the medical MLLMs. This implies that at least one of the images in the pair is compressed ambiguously, and thus, it is likely that relevant visual information is lost in the encoding. In particular, we base our selection criteria on BiomedCLIP’s embedding space, as this model has been specifically trained on medical image-text data, and thus it has a more refined feature space for medical images than a general-domain encoder, such as CLIP. Simply put, a radiology image pair that confuses BiomedCLIP, will likely confuse CLIP as well. Moreover, BiomedCLIP has been pretrained on the largest dataset of medical image-caption data among publicly available CLIP-style biomedical vision-language models. We measure visual differences between images in the feature space of DINOv2, a state-of-the-art vision-only foundation model with robust image representations that capture visual details. We randomly sample pairs of images and evaluate their similarity in BiomedCLIP ($sim_{med}$) and DINOv2 ($sim_{gen}$) feature spaces. We consider them a confusing pair if $sim_{med}\geq 0.9$ and $sim_{gen}\leq 0.75$ hold at the same time. The gap $|sim_{med}-sim_{gen}|$ can be increased further in order to obtain more difficult pairs, however we find that our setting is already challenging enough for most contemporary models. We depict sample pairs uncovered by our technique in Figure 2.

Figure: Figure 2: Sample confusing image pairs we have extracted from the ROCO dataset across $9$ categories.
Refer to caption: extracted/5869430/plots/samples.png

### 2.2 VQA generation

Given a pool of candidate confusing pairs, we generate multiple choice medical VQA problems that probe the MLLM’s ability to effectively differentiate between the images in the pair (Figure 1, center). First, we filter the candidate pool by removing images with short captions ($<100$ characters) that likely contain insufficient detail about the image. Next, we pass the pair of captions to an LLM (GPT-4) and prompt the model to generate a question to which the answer is different for the two images and to provide the two answer options. Thus, we create two VQA problems for each pair that share the same question and answer options, however the correct answer is different for the two images (Figure 3). Therefore, if the medical MLLM is unable to differentiate between the input images, it would only be able to answer at most one of the pair of VQA problems correctly, but not both. As a result, our benchmark by construction cannot be solved to higher than $50\%$ accuracy by relying solely on language prior. In particular, we only credit a set score to the model on a particular question pair, if the question has been answered correctly for both images. On the other hand, as a less strict metric an individual score is awarded to the model for each correct answer, irrespective of correctness on the other image in the pair. Furthermore, in order to categorize questions in the VQA, we prompt the LLM to assign the most relevant medical area to each of the questions based on the corresponding image’s PMC caption. We leverage these categories to break down the performance of existing medical MLLMs across the various categories. We include all prompts used in VQA generation in Appendix A.

Figure: Figure 3: A sample question pair from MediConfusion. A confusing pair shares the same question and corresponding answer options, but the correct answer is different for the two (A for the image on the left and B for the image on the right). The model receives a set score of 1 only if it correctly answers both questions in the confusing pair. Individual score is evaluated separately for each image ($1$ out of $2$ in the example).
Refer to caption: extracted/5869430/plots/confusing_pair_v3.png

### 2.3 Data filtering and revision via radiologist feedback

As we generate the questions/answer choices using an LLM, various issues may arise such as factual errors and inconsistencies in quality, format, or language. To ensure the curation of a reliable benchmark dataset, oversight and feedback from a radiology expert are crucial. A radiologist evaluated each of the automatically generated VQA problems focusing on three aspects.

Correctness: the question has to be valid with respect to both of the images, the problems need to be solvable by looking at the individual images alone, and the corresponding answers have to be correct.

Relevance: the question has to be relevant to clinical practice or medical research.

Language: the problem has to use proper medical terminology and precise language.

Based on these guidelines, the radiologist assigned a quality score to each question, on a scale $1-10$. Higher scores correspond to better problems, and a score of $1$ is assigned if correctness is violated in any form (e.g., irrelevant question, incorrect answer). We add a VQA pair to MediConfusion only if the quality score is at least $5$ for both individual problems in the pair. Moreover, the expert verified the medical categories assigned by the LLM to each of the images and revised the question and answer options to improve language quality and precision. This step is crucial in eliminating model artifacts that originate in LLM-generated text inputs.

The resulting benchmark is well-rounded, with questions touching upon $9$ areas (see Figure 4): cerebral, spinal, cardiac, gastrointestinal, musculoskeletal, vascular, pulmonary, head and neck, and nuclear medicine with $352$ questions in total. The distribution of question categories is depicted in Figure 4.

## 3 Experiments

### 3.1 Evaluation

We evaluate models on MediConfusion based on two notions of accuracy. Set accuracy is the portion of correct confusing pairs, where we only consider a pair correct if the model has answered the question correctly for both images in the pair. Individual accuracy is the standard notion of accuracy, that is, the portion of correct answers over all questions. An example is depicted in Figure 3. Furthermore, we report confusion score, which indicates the portion of pairs where the model has chosen the same answer for both images in the pair, out of all pairs (we exclude pairs where the model generated invalid answers or failed to answer). A high confusion score signifies that the model prediction is overwhelmingly invariant to the specific input image within a pair, and thus, it is confused by the images.

Extracting the knowledge from MLLMs for VQA benchmarking is often challenging due to sensitivity to the specific prompt format and phrasing, strong reliance on language bias and other factors. For instance, instruction tuned models can be directly prompted to answer a multiple choice question with the correct letter option, whereas models without instruction tuning often fail to do so. Therefore, in order to provide a fair comparison, we use a range of evaluation techniques to assess performance.

Prefix-based score (PS) – Following Xu et al. (), we compute the normalized likelihood of image-question-answer triplets for each answer option, and pick the option with the highest likelihood as the answer. In other words, we select the answer option that the model assigns the highest probability to, given the image and question. To compute the prefix-based score, we concatenate the medical question and the answer sentence directly, stripping the multiple choice question style (e.g., removing "Choose the letter of the correct answer.") and option indicators (e.g., removing "Option A: …") in order to ensure that models that have not been specifically trained on multiple choice question answering can also consistently provide valid answers.

Multiple choice prompting (MC) – We directly prompt the model to answer the multiple choice question with the letter of the correct option. As the output formats may vary (e.g., "A" vs. "The answer is A."), we parse the outputs and attempt to match it to one of the answer options.

Free-form evaluation (FF) – We prompt the model to answer the question without providing the answer options or requiring any specific output format. Then, we attempt to match the model output to one of the options using an LLM. In particular, we prompt GPT-4 to score how well the generated output matches each of the options, and we pick the answer option with the highest score. We include the specific evaluation prompt in Appendix A.

Greedy decoding (GD)– Similar to multiple choice prompting, we directly prompt the model to answer the problem with the letter of the correct option, then we pick the option with the highest assigned next-token probability. Greedy decoding evaluation is a special case of prefix-based scoring, where the answer options consist of a single letter.

PS and FF evaluations are suitable for models that are not instruction tuned or have not been trained to understand the multiple choice QA format. On the other hand, MC and GD are simpler to evaluate, however these techniques may fail to correctly measure the knowledge of MLLMs unable to understand and follow the multiple choice format. Overall, we represent the performance of each model by their best performance across all evaluation techniques. As we observe, proprietary models can consistently pick an answer option for multiple choice questions; for these models, we only provide MC results. Moreover, output logits necessary for PS and GD evaluation are not available for proprietary models.

We evaluate a representative set of $9$ models, $3$ of which are medical MLLMs (LLaVA-Med (Li et al., ), Med-Flamingo (Moor et al., ), RadFM (Wu et al., )), $3$ are flagship proprietary models (GPT-4o, Claude 3 Opus, Gemini 1.5 Pro) and $3$ open-source general-domain MLLMs (LLaVA-7B v1.6 (Liu et al., ), BLIP-2 (Li et al., ), InstructBLIP (Dai et al., )). We set generation parameters according to the corresponding code release and recommended settings and use few-shot prompting for Med-Flamingo (more details in Appendix B).

### 3.2 Results

We summarize the performance of MLLMs on MediConfusion in Table 1. Alarmingly, all MLLMs perform below random guessing in terms of set accuracy, corroborating our hypothesis that models struggle to differentiate in fine enough detail between the extracted image pairs necessary for accurate medical reasoning. This observation is further supported by the markedly high (often above $90\%$) confusion scores indicating that models tend to select the same answer for both images within a confusing pair. Even RadFM, a model that does not leverage a CLIP-style image encoder, is confused on our benchmark ($82.39\%$ confusion score) with performance well below random guessing. As most likely proprietary models leverage visual encoders other than CLIP as well, the overall poor performance and extremely high confusion scores suggest that the exposed vulnerability is more general and not solely rooted in the specific ambiguities of CLIP encoding.

An interesting outlier is Gemini 1.5, which has been the least confused (approx. $60\%$) on the dataset; however, its accuracy is still close to random guessing. This may suggest that the model’s visual representations are rich enough to meaningfully distinguish between images; however, the medical knowledge or necessary reasoning skills are lacking to correctly answer the questions.

Furthermore, perhaps surprisingly, medical MLLMs did not outperform other methods, which indicates that the shortcomings cannot be addressed exclusively by domain-specific training. These results are especially surprising, as the image-caption pairs used to generate MediConfusion are part of PMC-OA, which is included in the pre-training set of all $3$ medical MLLMs in our experiments. We also note that given the public nature of PMC-OA these image-caption pairs are likely included in the training set of proprietary models as well. Finally, we do see some performance gap between open-source and proprietary models, with GPT-4o achieving the highest individual accuracy, however, barely surpassing that of random guessing ($56.25\%$ vs. $50\%$) with set accuracy still well below random guessing.

**Table 1: Experimental results on MediConfusion. Evaluation techniques: PS - prefix-based scoring, MC - multiple choice prompting, FF - free-form evaluation, GD - greedy decoding evaluation. We underscore the best accuracy for each method across evaluation techniques and report the overall best in bold.**
|  |  | Set acc. ($\%$) |  | Indiv. acc.($\%$) |  | Confusion ($\%$) |  | Best |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method |  | MC | GD | FF | PS |  | MC | GD | FF | PS |  | MC | GD | FF | PS |  | Set acc. | Indiv. acc. |
| LLaVA |  | 8.52 | 9.09 | 1.70 | 1.14 |  | 50.57 | 51.70 | 15.06 | 49.72 |  | 85.47 | 85.80 | 76.00 | 97.16 |  | 9.09 | 51.70 |
| BLIP-2 |  | 0.57 | 6.82 | 1.70 | 3.98 |  | 22.16 | 50.28 | 11.65 | 51.42 |  | 92.19 | 86.93 | 86.67 | 94.89 |  | 6.82 | 51.42 |
| InstructBLIP |  | 12.50 | 7.95 | 2.84 | 3.41 |  | 51.99 | 53.12 | 19.60 | 50.57 |  | 80.35 | 90.34 | 87.23 | 94.32 |  | 12.50 | 53.12 |
| LLaVA-Med |  | 0.00 | 0.00 | 1.14 | 1.14 |  | 23.58 | 49.72 | 18.75 | 49.72 |  | 100.00 | 99.43 | 95.92 | 97.16 |  | 1.14 | 49.72 |
| RadFM |  | 0.57 | 1.14 | 0.57 | 5.68 |  | 35.90 | 50.28 | 16.19 | 48.58 |  | 97.54 | 98.30 | 95.12 | 85.80 |  | 5.68 | 50.28 |
| Med-Flamingo |  | 1.14 | 2.27 | 0.57 | 4.55 |  | 47.73 | 50.00 | 17.05 | 51.99 |  | 98.75 | 95.45 | 94.89 | 98.30 |  | 4.55 | 51.99 |
| GPT-4o |  | 18.75 | - | - | - |  | 56.25 | - | - | - |  | 75.00 | - | - | - |  | 18.75 | 56.25 |
| Claude 3 Opus |  | 8.52 | - | - | - |  | 50.85 | - | - | - |  | 84.09 | - | - | - |  | 8.52 | 50.85 |
| Gemini 1.5 Pro |  | 19.89 | - | - | - |  | 51.14 | - | - | - |  | 58.52 | - | - | - | - | 19.89 | 51.14 |
| Random guessing |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 25.00 | 50.00 |

We further break down the results based on the category of the question in order to identify if specific areas have been more/less challenging to the models. We summarize our findings in Table 2. Even though the overall results across all categories are close to random guessing performance, proprietary models demonstrate slightly better accuracies on questions related to cerebral and vascular images. In particular, GPT-4o achieves $34.25\%$ and $67.12\%$ set and individual accuracy correspondingly on vascular images, an overall best across all models and categories

**Table 2: Results by category. We report the best set and individual accuracies ($\%$) for each model across all evaluation techniques.**
|  |  | Cerebral |  | Vascular |  | Head & Neck |  | Spinal |  | Musculoskel. |  | Cardiac |  | Gastroint. |  | Pulmonary |  | Nuclear Med. |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model |  | Set | Indiv. |  | Set | Indiv. |  | Set | Indiv. |  | Set | Indiv. |  | Set | Indiv. |  | Set | Indiv. |  | Set | Indiv. |  | Set | Indiv. |  | Set | Indiv. |
| LLaVA |  | 7.59 | 49.37 |  | 13.70 | 54.79 |  | 4.48 | 52.24 |  | 5.88 | 52.94 |  | 9.52 | 52.38 |  | 7.69 | 51.92 |  | 27.91 | 60.47 |  | 10.00 | 55.00 |  | 14.29 | 57.14 |
| BLIP2 |  | 5.06 | 54.43 |  | 8.22 | 53.42 |  | 4.48 | 46.27 |  | 3.92 | 49.02 |  | 4.76 | 50.00 |  | 7.69 | 50.00 |  | 13.95 | 51.16 |  | 10.00 | 55.00 |  | 28.57 | 64.29 |
| InstructBLIP |  | 16.46 | 59.49 |  | 10.96 | 56.16 |  | 7.46 | 52.24 |  | 17.65 | 52.94 |  | 23.81 | 59.52 |  | 7.69 | 50.00 |  | 9.30 | 51.16 |  | 10.00 | 60.00 |  | 14.29 | 57.14 |
| LLaVA-Med |  | 1.27 | 53.16 |  | 2.74 | 49.32 |  | 0.00 | 50.75 |  | 0.00 | 50.98 |  | 4.76 | 50.00 |  | 0.00 | 50.00 |  | 4.65 | 53.49 |  | 0.00 | 45.00 |  | 0.00 | 50.00 |
| RadFM |  | 1.27 | 49.37 |  | 4.11 | 49.32 |  | 5.97 | 49.25 |  | 3.92 | 50.98 |  | 2.38 | 50.00 |  | 11.54 | 50.00 |  | 11.63 | 53.49 |  | 10.00 | 50.00 |  | 0.00 | 50.00 |
| Med-Flamingo |  | 7.59 | 58.23 |  | 10.95 | 56.16 |  | 8.96 | 52.24 |  | 0.00 | 52.94 |  | 4.76 | 52.38 |  | 3.85 | 51.92 |  | 2.33 | 48.84 |  | 10.00 | 50.00 |  | 0.00 | 50.00 |
| GPT-4o |  | 15.19 | 59.49 |  | 34.25 | 67.12 |  | 8.96 | 58.21 |  | 15.69 | 52.94 |  | 14.29 | 52.38 |  | 19.23 | 55.77 |  | 16.28 | 55.81 |  | 35.00 | 65.00 |  | 14.29 | 42.86 |
| Claude 3 Opus |  | 7.59 | 55.70 |  | 20.55 | 58.90 |  | 0.00 | 44.78 |  | 0.00 | 52.94 |  | 9.52 | 45.24 |  | 11.54 | 53.85 |  | 11.63 | 51.16 |  | 10.00 | 50.00 |  | 14.29 | 50.00 |
| Gemini 1.5 Pro |  | 25.32 | 58.23 |  | 27.40 | 60.27 |  | 16.42 | 52.24 |  | 17.65 | 43.14 |  | 26.19 | 47.62 |  | 7.69 | 48.08 |  | 23.26 | 44.19 |  | 5.00 | 50.00 |  | 28.57 | 57.14 |

## 4 Discussion

### 4.1 Identifying Patterns in Confusing Pairs

Our experiments have demonstrated that state-of-the-art MLLMs are easily confused by radiology image pairs that exhibit major differences obvious to human experts. The first step towards improving the reliability of such models is to identify and categorize common cases where medical MLLMs tend to break down. We leverage an expert-in-the-loop pipeline to extract failure modes from MediConfusion via a combination of LLM prompting and radiologist supervision. In particular, we pass the VQA problems from MediConfusion to GPT-4, where we replace the images with their corresponding captions from ROCO. We prompt the model to summarize the key differences between images in a pair that the questions are designed to test (details in Appendix A). The LLM identifies patterns in the extracted differences and distills them into a set of categories that the radiologist corrects and refines based on the dataset. As a result, we identify the following common patterns that have confused the models:

- •
Pattern 1: Normal/variant anatomy vs. pathology– Models often struggle with differentiating between normal/variant anatomy and pathological structures. For instance, the model often confuses malalignment with normal alignment (e.g., atlantoaxial dislocation vs. normal atlantoaxial interval) or differentiating pituitary region masses (suprasellar vs. parasellar vs. intrasellar) or various anatomical regions of the spine (cervical vs. thoracic vs. lumbar).
- •
Pattern 2: Lesion signal characteristics– Models fail to correctly identify regions of high signal intensity and their significance, particularly on T2-weighted sequences. This failure is especially of clinical significance in differentiating solid vs. cystic entities.
- •
Pattern 3: Vascular conditions– Identifying aneurysms and differentiating them from normal vascular structures or other abnormalities like vascular malformations seems to be challenging for MLLMs. Furthermore, there is often confusion between total occlusions and partial stenosis in coronary arteries.
- •
Pattern 4: Medical devices– Models often fail to detect the presence of stents and have difficulties distinguishing between various types of stents. Identifying the presence or absence of guidewires in images of interventional procedures tends to also be challenging for MLLMs.

Most of the above shortcomings can be, to some degree, traced back to known, common failure modes (Tong et al., ) of visual reasoning in MLLMs in the general domain.

Detecting presence (or absence) of specific features: Correct reasoning over medical VQA problems strongly relies on detecting the presence (or absence) of particular features or objects relevant to the question. MLLMs are known to suffer from object hallucinations () rooted in parts in flawed image encoding, statistical biases and strong reliance on language priors (Leng et al., ). We can see this specific weakness reflected in Patterns 3 and 4 directly.

Understanding state and condition: In medical VQA, it is crucially important for the model to understand the difference between "normal" and "abnormal" structures. MLLMs have difficulties identifying the state and condition of objects in the general domain, such as whether the ground is wet or if a flag is blowing in the air (Tong et al., ). These challenges may be amplified in the more nuanced medical setting, which we observe in Patterns 1 and 3 especially.

Positional and relational context: Answering medical VQA problems often necessitate a careful understanding of the spatial relationships of various anatomical features and their specific location. Recent research has uncovered serious limitations in the spatial reasoning capabilities of MLLMs (Kamath et al., ), some even failing to distinguish left from right. This pervasive weakness in spatial reasoning may translate to failures in medical VQA seen in Pattern 1.

Color and appearance: Recent work has shown that MLLMs can confuse colors and their intensity (bright/dark) (Tong et al., ), which may cause challenges in identifying signal characteristics in radiology images (high/low intensity) reflected in Pattern 2.

### 4.2 Visual prompts in MediConfusion

Free-form visual prompts are intuitive annotations in the input image, such as a red bounding box or an arrow, aimed at highlighting a specific point or area within the image. It is natural to ask whether well-placed visual prompts in medical images, annotated by a doctor, can potentially guide the attention of MLLMs to important areas in the image and thus help provide accurate answers. Such a capability would greatly facilitate human-machine collaboration in healthcare and provide more reliable AI-assisted diagnosis. In the general domain, research has shown that MLLMs typically are unable to efficiently interpret visual prompts without incorporating such task specifically into the training procedure (Cai et al., ).

We find that some images in MediConfusion include such visual prompts, typically in the form of arrows pointing at the abnormality, and in a specific case the correct answer is written in the image along with the prompt (Figure 5). We observe that only proprietary models, as well as LLaVA v1.6 and BLIP-2, have been able to provide consistently correct answers for this particular image, and none of the medical MLLMs. We hypothesize that the success of proprietary models and LLaVA v1.6 can be attributed to their OCR (optical character recognition) capabilities, which is missing from medical MLLMs. In examples where only the visual prompt (e.g., an arrow pointing at the abnormality/region of interest) is included we don’t observe a similar trend. We believe that understanding and improving the visual prompting capabilities of medical MLLMs is a promising direction for future research.

Figure: Figure 5: Sample VQA from MediConfusion where the solution is directly provided in the image in the form of text and visual prompts (arrows). Medical MLLMs not trained for OCR have been unable to leverage the hint.
Refer to caption: extracted/5869430/plots/solution_in_image.png

## 5 Related Work

### 5.1 Multimodal Large Language Models

Large Language Models (LLMs) such as InstructGPT (Ouyang et al., ) and LLaMA (Touvron et al., ) have emerged as powerful models capable of performing complex tasks rooted in natural language, including text summarization, coding, and question-answering. LLMs are pretrained on massive text corpora and can be efficiently adapted to downstream tasks. Beyond textual inputs, the LLM pretraining framework has been extended to further modalities, such as images, resulting in multimodal large language models that demonstrate strong visual understanding and reasoning capabilities. LLaVA (Liu et al., ) interleaves image representations with the input text of a pretrained LLaMA model and fine-tunes on visual instruction-following data. Flamingo (Alayrac et al., ) injects visual information into a frozen LLM via cross-attention. BLIP (Li et al., , ) proposes the Q-Former architecture for connecting pretrained vision features to an LLM. More recently, GPT-4o has been trained from scratch on mixed multimodal inputs directly. The resulting MLLMs have achieved great success in visual question answering in the general domain and as multimodal chat agents able to interpret images in the input prompt.

Beyond the general domain, MLLMs are especially promising in automating costly medical tasks, such as analyzing radiology images, generating medical reports or acting as medical conversational agents to provide healthcare advice. There has been substantial research recently to develop medical MLLMs, most often by adapting popular general domain architectures to medical data. Med-Flamingo (Moor et al., ) pretrains Flamingo on interleaved image-text medical data sourced from publications and textbooks, unlocking few-shot medical VQA capabilities. Authors of LLaVA-Med (Li et al., ) focus on rapid adaptation to the medical domain by fine-tuning LLaVA on filtered image-text pairs from PMC-15M (Zhang et al., ). Authors in Zhang et al. () generate a large-scale medical VQA dataset from PMC-OA (Lin et al., ) which is subsequently used to train MedVInT, a state-of-the-art medical MLLM. Moreover, authors in Wu et al. () propose a multimodal foundation model for radiology, aligning natural language with 2D and 3D radiology images.

### 5.2 Encoding visual information in MLLMs

The prevailing approach to incorporate visual information in MLLM training is to leverage contrastive language-image pretrained models as frozen image encoders. CLIP (Radford et al., ), and its variants (Cherti et al., ), are trained on internet-scale paired image-text data, and thus its representations are readily aligned with natural language, and thus can be effectively combined with language models. The frozen representations are then typically adapted to the feature space of the language model using MLP heads (Liu et al., ), Q-Former (Li et al., ), cross-attention (Alayrac et al., ) or other mechanisms. The image encoder acts as the "eye" of the MLLM as it directly determines what visual information will enter the model. In fact, imperfect compression of relevant visual information is a dominant issue with contemporary MLLMs, resulting in object hallucinations (; Gunjal et al., ), fundamental errors in spatial reasoning (Kamath et al., ), and inability to understand inter-object relationships (Wu et al., ).

As the distribution of general ’internet data’ and medical image-text data is markedly different, CLIP may be unable to capture the intricate structure of medical images with fidelity sufficient for reliable performance. Researchers have proposed CLIP-like models pretrained on large-scale medical data better suited as image encoders for medical MLLMs. LLaVA-Med leverages BiomedCLIP (Zhang et al., ), a foundation model designed for biomedical image-text processing that has been pretrained on PMC-15M. MedVInT uses PMC-CLIP (Lin et al., ), a CLIP-style model pretrained on PMC-OA with $1.6M$ medical image-caption pairs. The limitations of image encoders in medical MLLMs have attracted less attention than in the general domain, which is especially troubling due to the safety-critical nature of healthcare applications. Thus, the lack of in-depth understanding of the shortcomings and possible failure modes of the image encoder in the medical MLLM pipeline is an exceedingly pressing concern.

### 5.3 Medical VQA benchmarks

With the recent rapid advances in developing medical MLLMs, there has been substantial effort in quantifying their performance in a wide range of tasks and areas within the medical domain. VQA-Rad (Lau et al., ), SLAKE (Liu et al., ), Path-VQA (He et al., ) and VQA-Med (Ben Abacha et al., ) are widely-used to benchmark the performance of MLLMs in medical VQA. Due to their small size and limited scope, there has been a push for more comprehensive and diverse evaluation datasets. OmniMedVQA (Hu et al., ) introduces the largest medical VQA dataset to date, encompassing 12 data modalities and 20 anatomical regions with a total of more than $100k$ images. Authors of Asclepius (Wang et al., ) focus on eliminating data leakage present in other benchmarks and providing human evaluations. GMAI-MMBench (Chen et al., ) incorporates problems probing the performance of MLLMs at various perceptual granularities, and targets a well-categorized data structure for ease of preparing customized evaluations. Other benchmarks extend the evaluation task beyond VQA in order to provide a more comprehensive view of model performance. MultiMedEval (Royer et al., ) builds a uniform and fair benchmarking framework for multiple tasks including report generation and classification. RadBench (Wu et al., ) focuses on radiology with associated tasks such as modality recognition and disease diagnosis. Authors of CARES (Xia et al., ) aim to provide a more holistic view of model performance by focusing on aspects such as fairness, privacy and safety (toxicity) of MLLMs as well as factual correctness (trustfulness).

All of these datasets are aimed at probing the medical knowledge of MLLMs and quantifying their average performance on a wide variety of tasks, modalities and anatomic regions. However, none of these benchmarks are specifically designed to probe the reliability, fundamental limitations and failure modes in the medical domain, all critical aspects in healthcare applications.

Perhaps the closest work in spirit to ours is RadVUQA (Nan et al., ), where authors call attention to the critical deficiencies of existing medical MLLMs, revealing a large gap between state-of-the-art MLLMs and clinicians. Their dataset focuses on more fundamental visual question answering and understanding, such as spatial reasoning, anatomic understanding and quantitative reasoning on medical images. However, we go a step further and design a benchmark that stress-tests the visual capabilities of MLLMs by curating questions expected to be challenging for the image processing pipeline of state-of-the-art models.

Related to our work, Tong et al. () has investigated the failure modes of MLLMs originating in ambiguous vision encoding in the general domain. Their study is based on finding CLIP-blind pairs, images that have high similarity in CLIP embedding space, but otherwise have dissimilar low-level image features. However, their methodology is not directly applicable in the medical domain for two reasons. First, CLIP has been pretrained on general domain data and thus it is unable to capture the intricate structure of medical images. Second, their methodology relies on human annotators to describe the difference between a large number of image pairs, which is prohibitively costly in our scenario, as only radiologists are qualified to provide such annotations in the medical setting.

## 6 Conclusion

In this paper, we introduce MediConfusion, a challenging medical VQA benchmark designed to probe the limitations of multimodal reasoning in medical MLLMs. In particular, we discover radiology image pairs that, due to ambiguities originating in image encoding, confuse contemporary models despite being dissimilar in the image domain. We leverage an automated pipeline along with the expertise of radiologists to create a dataset of VQA problems that tests the ability of MLLMs to effectively distinguish and answer clinically relevant questions about such confusing pairs. Our benchmark, by construction, cannot be solved by leveraging unimodal priors, and thus, it directly probes multimodal capabilities. We find that existing models achieve performance no better than random guessing on MediConfusion, as models tend to select the same answer option for both images in the pair, raising serious concerns about the reliability of existing MLLMs in a medical setting. In order to guide future research in addressing the limitations of current MLLMs, we identify common failure patterns where models often break and relate them to known limitations in the general domain. We hope that our work sparks further research efforts to improve the reliability of AI for healthcare applications.

## Acknowledgements

We would like to thank Microsoft for an Accelerating Foundation Models Research grant that provided the OpenAI credits enabling this work. This research is also in part supported by AWS credits through an Amazon
Faculty research award and a NAIRR Pilot award. M. Soltanolkotabi is also supported by the Packard Fellowship in Science and Engineering, a Sloan Research Fellowship in
Mathematics, an NSF-CAREER under award #1846369, DARPA FastNICS program, and NSF-CIF awards #1813877
and #2008443. and NIH DP2LM014564-01.

## Appendix

## Appendix A Prompts for dataset curation

In this section, we provide the prompts used to interact with GPT-4o for dataset generation and MLLM evaluation. Wherever we use **TEXT**, we mean that TEXT is a description or variable that is image/pair specific.

### A.1 Question generation

We use the following prompt to generate a question for a single confusing pair. We describe the output format as detailed as possible to be able to process the answers with little human interaction.

System message: You are a helpful assistant expert in the medical domain.
Prompt: Here is the description of two medical images that I can see:
Image1: **CAPTION OF IMAGE 1**
Image2: **CAPTION OF IMAGE 2**
Your task is to create multiple-choice questions. Follow the rules below.
1. The question should be about a property that is clearly visible in the images.
2. It should be possible to answer the question by only looking at the images.
3. Pretend that you can only see one image. You are not allowed to refer to ’Image1’, ’Image2’ or ’images’. You should also not create questions that require comparing the two images.
4. There should be exactly two answer options. You have to come up with a question for which the answer is different for the two images.
5. The answer options should be clearly different.
Please provide the question and the two answer options in the following format:
Question: <YOUR QUESTION>
Option1: <ANSWER 1>
Option2: <ANSWER 2>
Also, please provide the correct answers to the question for the images corresponding to our captions in the following format:
Image1: <ANSWER>
Image2: <ANSWER>
Aim for simple, efficient and concise questions to best test someone’s knowledge and understanding of the underlying concepts.

### A.2 Categorizing images

To categorize the images, we first show GPT-4o captions of several (here we use 100) images and ask it to separate them into different categories. Afterward, for each image, we ask GPT-4o to pick one of the categories for that image based on its caption.
We used the following prompt to extract categories:

System message: You are a helpful assistant expert in the medical domain.
Prompt: I have several medical images related to radiology that each one has a corresponding caption. I have listed the captions below. Can you go through the captions and categorize them? Please focus on the general categories.
Caption 0: **IMAGE 0 CAPTION**
Caption 1: **IMAGE 1 CAPTION**
…
Caption 99: **IMAGE 99 CAPTION**

Using this prompt, we find 9 categories: Cerebral, Spinal, Cardiac, Gastrointestinal, Musculoskeletal, Vascular, Pulmonary, Head and Neck, Breast, and Other.
We used the following prompt to assign categories:

System message: You are a helpful assistant expert in the medical domain.
Prompt: Caption: **IMAGE CAPTION**
The above caption describes a radiology image. To which of the following categories does this image belong? You should only name the category, and you do not need to specify your reasoning.
Categories: Cerebral, Spinal, Cardiac, Gastrointestinal, Musculoskeletal, Vascular, Pulmonary, Head and Neck, Breast, Other"

The final set of categories in our dataset are somewhat different, as we incorporated feedback from the radiologist to revise the automatically generated categories.

### A.3 Finding failure modes

To find common failure modes that our dataset probes, we use the questions and captions of $100$ pairs in the following prompt to send to GPT-4o:

System message: You are a helpful assistant expert in the medical domain.
Prompt: I am analyzing an image embedding model. I have several image pairs that each one has a corresponding two choice question. I know that the embedding model confuses the images about the corresponding question. Can you go through the questions, options, and image descriptions, trying to figure out some general patterns that the embedding model struggles with? Please focus on the visual features and generalize patterns that are important to vision models.
Pair 0
First image description: **IMAGE 1 CAPTION**
Second image description: **IMAGE 2 CAPTION**
Confusing multiple choice question: **QUESTION**
Pair 1
First image description: **IMAGE 1 CAPTION**
Second image description: **IMAGE 2 CAPTION**
Confusing multiple choice question: **QUESTION**
…
Pair 99
First image description: **IMAGE 1 CAPTION**
Second image description: **IMAGE 2 CAPTION**
Confusing multiple choice question: **QUESTION**

### A.4 Free Form Evaluation

For the free-form (FF) GPT-4o evaluation, we pass the MLLM’s answers with the following prompt to GPT-4o to obtain two scores, one for each answer option.

System message: You are a helpful and precise assistant for checking the quality of the answer.
Prompt: [Question]
**QUESTION**
[Answer A]
**OPTION A**
[Answer B]
**OPTION B**
[Assistant]
**RESPONSE**
[End of Assistant]
[System]
We would like to request your feedback on the performance of an AI assistant in response to the user question displayed above. The user asks the question on observing an image. We have provided two possible answers, [Answer A] and [Answer B] to the question. Your job is to evaluate how close the AI assistant’s answer is to each of the answers. You don’t have to decide whether the answers are correct or not. Each answer should receive an overall score on a scale of 1 to 10, where a higher score indicates the AI assistant’s answer is closer to the specific answer. After providing the scores, concisely provide your explanation for the given scores. Remember, you don’t need to comment on the correctness of the answers. Please provide your answer in the following format:
A: <SCORE>
B: <SCORE>
Your explanation: <EXPLANATION>

These scores are the similarities of the MLLM’s answer to the different answer options. If the gap between the higher and lower score is at least $3$, we assign the option with the higher score as the MLLM’s output. Otherwise, we mark the answer as invalid.

## Appendix B Model details

In this section, we provide details on the versions and hyperparameters of MLLMs that we use. It should be noted that for the multiple choice (MC) evaluation mode, we set temperature to 0 0, as we only expect a single letter option to be generated.

**Table 3: MLLM details**
| MLLM | Version/LLM | Temperature | Beams | Top p |
| --- | --- | --- | --- | --- |
| LLaVA | v1.6/Mistral 7B | 0.2 | 1 | - |
| BLIP-2 | Opt 2.7B | 1 | 5 | 0.9 |
| InstructBLIP | Vicuna 7B | 1 | 5 | 0.9 |
| LLaVA-Med | v1.5/Mistral 7B | 0.2 | 1 | - |
| RadFM | - | - | - | - |
| Med-Flamingo | - | 1 | 5 | 0.9 |
| GPT | 4o (release 20240513) | 0.7 | - | - |
| Claude | 3 Opus | 0.2 | - | - |
| Gemini | 1.5 Pro | 0.2 | - | - |

### B.1 MedFlamingo few-shot prompting

In order for MedFlamingo to produce valid responses, we need to use few-shot prompting. Here, we show three questions and answers from PMC-VQA benchmarks Zhang et al. ().
The following is the prompt we used for MC evaluation:

Prompt: You are a helpful medical assistant. You are being provided with images, a two choice question about each image and an answer. Follow the examples and answer the last question. <image>Question: What radiological technique was used to confirm the diagnosis?
A: CT Scan
B: Mammography
Answer: B: Mammography
<image>Question: What did the CT scan show?
A: Cerebral edema
B: Intracranial hemorrhage
Answer: A: Cerebral edema|endofchunk|><image>Question: What is the purpose of the asterisk shown in the figure?
A: To indicate the formation of lobes around the contracting nucleus.
B: To indicate the normal lentoid shape of hypocotyl nuclei.
Answer: B: To indicate the normal lentoid shape of hypocotyl nuclei.
<image>
Question: **QUESTION**
A: **OPTION A**
B: **OPTION B**
Answer:

The following is the prompt we used for FF evaluation:

Prompt: You are a helpful medical assistant. You are being provided with images, a question about each image and an answer. Follow the examples and answer the last question. <image>Question: What radiological technique was used to confirm the diagnosis? Answer: Mammography
<image>Question: What did the CT scan show? Answer: Cerebral edema|endofchunk|><image>Question: What is the purpose of the asterisk shown in the figure? Answer: To indicate the normal lentoid shape of hypocotyl nuclei.
<image>Question: **QUESTION** Answer:

The following is the prompt we used for GD evaluation:

Prompt: You are a helpful medical assistant. You are being provided with images, a two choice question about each image and an answer. Follow the examples and answer the last question. <image>Question: What radiological technique was used to confirm the diagnosis?
A: CT Scan
B: Mammography
Answer: B: Mammography
<image>Question: What did the CT scan show?
A: Cerebral edema
B: Intracranial hemorrhage
Answer: A: Cerebral edema|endofchunk|><image>Question: What is the purpose of the asterisk shown in the figure?
A: To indicate the formation of lobes around the contracting nucleus.
B: To indicate the normal lentoid shape of hypocotyl nuclei.
Answer: B: To indicate the normal lentoid shape of hypocotyl nuclei.
<image>
Question: **QUESTION**
A: **OPTION A**
B: **OPTION B**
Answer:

The following is the prompt we used for PS evaluation:

Prompt: You are a helpful medical assistant. You are being provided with images, a question about each image and an answer. Follow the examples and answer the last question. <image>Question: What radiological technique was used to confirm the diagnosis? Answer: Mammography
<image>Question: What did the CT scan show? Answer: Cerebral edema
<image>Question: What is the purpose of the asterisk shown in the figure? Answer: To indicate the normal lentoid shape of hypocotyl nuclei.
<image>Question: **QUESTION** Answer: **ANSWER**

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.	Dataset Diversity and Expansion: Could you elaborate on the decision to limit the benchmark to the ROCO dataset? Do you have plans to expand MediConfusion to include other datasets, such as ultrasound or nuclear imaging, to capture a broader range of diagnostic tasks? Additional dataset diversity might reveal new model weaknesses or challenges in different medical imaging modalities.
	2.	Impact of Prompt Engineering: Given the importance of prompt phrasing in VQA tasks, did you experiment with alternative prompt formats or styles (e.g., framing questions in different ways or including detailed instructions)? If so, what impact did this have on model performance, and could further prompt optimization improve reliability? Insights here could guide future research on prompt refinement in medical VQA.
	3.	Future Directions for Medical MLLM Improvements: Given the identified failure patterns (e.g., difficulties with spatial reasoning or lesion characteristics), do you have specific recommendations for architectural or training adjustments that could address these weaknesses? For example, do you see value in integrating spatial attention mechanisms or training models on more anatomically detailed data?
soundness: 3
strengths: Originality: The paper addresses a critical gap in the assessment of medical MLLMs by introducing MediConfusion, a novel benchmark explicitly designed to test the reliability of these models in healthcare. While prior benchmarks focus on overall performance in typical medical scenarios, this work innovatively emphasizes systematic failure modes by identifying “confusing image pairs” that challenge models with subtle, clinically important distinctions. The benchmark not only probes weaknesses but also highlights specific visual reasoning failures, which is a unique contribution to medical AI evaluation.

Quality: The dataset creation is rigorous, combining automated techniques with expert radiologist validation to ensure that each question is clinically relevant and accurately represents medical knowledge. The experimental design is robust, systematically evaluating a range of state-of-the-art models (both general and medical-domain MLLMs) and comparing their performance against baseline expectations. The detailed breakdown of common model failure patterns enhances the paper’s depth, providing actionable insights for improving MLLM robustness in medical settings.

Clarity: The paper is well-structured and presents its findings in a clear, logical manner. Descriptions of the dataset creation process, evaluation methods, and results are easy to follow, with visual aids (e.g., figures illustrating confusing image pairs) that help readers understand the complexity of the visual tasks posed to models. The breakdown of error types further clarifies the distinct challenges MLLMs face, enhancing the reader’s comprehension of the benchmark’s significance.

Significance: The benchmark’s focus on failure modes has broad implications for the deployment of MLLMs in healthcare. By surfacing limitations that could compromise patient safety, the work serves as a foundational tool for future research aimed at increasing AI reliability in clinical practice. Additionally, MediConfusion highlights the limitations of current visual encoders and training paradigms in handling medical data, offering a clear research direction for building more robust and trustworthy models. The benchmark’s significance is further bolstered by its potential as a standard for evaluating and guiding improvements in medical AI systems.
summary: The paper introduces MediConfusion, a benchmark specifically designed to assess and expose failure modes in multimodal large language models (MLLMs) used in medical visual question answering (VQA) tasks. The benchmark highlights that current state-of-the-art MLLMs, including both open-source and proprietary models, struggle significantly with distinguishing visually dissimilar yet clinically distinct medical images, often performing below random guessing on the dataset. MediConfusion is created by identifying “confusing image pairs” from the ROCO radiology dataset, where image pairs appear distinct to medical professionals but prove challenging for MLLMs due to limitations in visual feature space alignment.

The authors reveal key failure patterns among MLLMs, such as difficulties in differentiating between normal and pathological anatomy, recognizing lesion characteristics, and identifying medical devices and vascular conditions. The benchmark leverages radiologist oversight to ensure clinical relevance and accuracy in the question-answer pairs, aiming to guide future MLLM improvements in medical contexts. By exposing these systematic vulnerabilities, the paper underscores the critical need for more reliable and trustworthy AI systems in healthcare applications.
weaknesses: Limited Dataset Size: While the MediConfusion benchmark is rigorous, the dataset size (352 questions across 9 categories) may be relatively small to capture the full spectrum of medical image complexity. Expanding the dataset with a broader range of confusing image pairs, potentially covering additional anatomic regions and diagnostic subtleties, could improve its generalizability and make it more comprehensive for model evaluation across diverse medical contexts.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 4
questions: 1. Are there confusing pairs that appear visually similar but differ in feature space?
2. How would the Mixture-of-Features method from "Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs" apply to medical VLMs?
3. What are the limitations of the confusing pairs identified here? Could there be potential biases?
soundness: 4
strengths: 1. The paper addresses a well-motivated and underexplored problem in the medical domain.
2. The benchmark creation process is robust, with manual evaluation by a radiologist enhancing its credibility.
3. The experiments are comprehensive, demonstrating that MediConfusion presents substantial challenges to current VLMs, with promising potential for inspiring future research.
summary: This paper evaluates reliability of medical VLMs. The authors introduce MediConfusion, a benchmark comprising 352 pairs of “confusing” medical images. Each pair consists of images that are visually distinct yet similar in feature space. The authors leverage GPT-4 to generate questions for which the answers differ between the two images. The benchmark data is further evaluated by a radiologist. Experimental results indicate that MediConfusion poses significant challenges to several existing VLMs, highlighting areas for improvement and setting the stage for future research.
weaknesses: While this work provides valuable insights, it appears to closely follow the Multimodal Visual Patterns (MMVP) benchmark framework, potentially limiting its novelty. Many findings align with known challenges in general VLMs.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Does the proposed benchmark dataset only contain 352 image pairs?

- The authors only evaluate the models on the similar pair images. Can the models work well on different image pairs? For example, to use different sim_med and sim_gen thresholds and compare the model performance.
soundness: 3
strengths: - The idea of using the same question with similar images could offer a interesting way to probe model weaknesses.

- With the help of an expert-in-the-loop pipeline, the paper thoroughly examines why models struggle to differentiate between visually similar medical images.

- The evaluation metrics are comprehensive, particularly accommodating models unable to directly answer multiple-choice questions.
summary: This paper evaluates the reliability of medical Multimodal Large Language Models in healthcare. The authors present MediConfusion, a new VQA benchmark that highlights distinguishing similar medical xray images. They find that all tested MLLMs, open-source and proprietary, perform below random chance on MediConfusion, indicating serious reliability issues.
weaknesses: - The motivation is not clear. The authors claimed that "existing benchmark datasets are focused on evaluating the medical knowledge of MLLMs across large evaluation sets, heavily biased towards common or typical scenarios." What are the common and typical scenarios? Could you provide a more detailed discussion about this limitation of existing benchmarks and how the proposed benchmark in this paper can overcome the limitations?
- The results show that all available models (open-source or proprietary) achieve performance below random guessing on MediConfusion. It would be better if the authors conducted more experiments to find the reasons. For example, by using different sim_med and sim_gen thresholds and compare the model performance, we can identify whether the models can differentiate easy image pairs or they even cannot differentiate any image pairs. 
- The writing style needs refinement. For instance, the phrase "stress-tests the visual capabilities of state-of-the-art models" is unclear and may confuse readers. The motivation and innovative aspects of the benchmark remain vague until "image pair" is mentioned at the end of Section 2, and even then, it is not clarified that the task is framed as multiple-choice questions. It would be easier for the readers to better understand this paper if the writer could clarify these designs in the introduction section.
- The rationale for focusing on the medical domain is unclear. The methods do not appear specific to this field, resembling a rephrasing of existing VQA datasets where answers are scrambled, and models are tasked with similar image comparisons. Could you explain what is the specific design of this benchmark considering its medical context? What is the difference between two similar medical images and two similar general images?
- The choice to format the task as multiple-choice is questionable. Have you considered open-ended QA or report generation of CXR images? Additionally, given the relatively small dataset (352 questions), this benchmark may lack the scale needed to effectively assess MLLM capabilities. Could you justify that 352 questions are sufficient for the evaluation?

---

confidence: 3
contribution: 4
initial_rating: 6
presentation: 3
questions: See the weakness.
soundness: 3
strengths: 1. The paper brings a novel perspective by focusing on the systematic failure modes of medical multimodal models—something that hasn’t been extensively explored before. The fact that state-of-the-art models performed at or below random guessing levels is a striking finding. I can see this dataset becoming a benchmark everyone wants to use as soon as the paper is out, as it offers a strong challenge to prove model reliability.

2. The presentation and writing are clear and well-structured.
summary: The paper introduces MediConfusion, a challenging medical Visual Question Answering (VQA) benchmark that exposes the vulnerabilities of current medical multimodal large language models (MLLMs) by highlighting their failure modes. Despite promising progress, existing models—open-source and proprietary—struggle significantly, even performing below random guessing in certain tests. This suggests major reliability issues in deploying these models for healthcare. The benchmark was guided by clinician experts, using adversarial examples to stress-test models, offering a valuable resource for the research community to design more trustworthy MLLMs. Personally I like this paper because it hits on a crucial gap in medical AI that’s often overlooked.
weaknesses: 1. One significant limitation is that it feels almost obvious to train a model using both public data and this new dataset to demonstrate improved performance over current models. It’s surprising this wasn’t attempted—even for a dataset paper. Adding such experiments could have strengthened the impact by showing how incorporating MediConfusion directly improves model reliability.

2. The authors might also want to discuss the importance of the image encoder in these models. For example, both LLaVA and LLaVA-med use encoders trained on everyday images, which could be a reason why LLaVA-med performs even worse than LLaVA. If the encoder isn’t well-suited for medical images, it limits the model's ability to learn effectively. Highlighting the role of specialized image encoders could have added valuable insights into improving model reliability in the medical domain.

Meta-review: This is a datasets and benchmarks paper. The authors introduced MediConfusion, a challenging medical Visual Question Answering (VQA) benchmark dataset that probes the failure modes of medical MLLMs from a vision perspective. The benchmark dataset is designed to evaluate the ability of state-of-the-art models to recognize subtle yet clinically meaningful differences between medical images. Then, the dataset was further evaluated by a radiologist. The authors find that all tested MLLMs, open-source and proprietary, perform below random chance on MediConfusion, indicating some serious reliability problems on MLLMs. In sum: this is an interesting and timely paper on hot topics - medical MLLMs, the findings are unexpected - all models from GPT-4o, Gemini 1.5 Pro to Llava, InstructBLIP, achieve performance below random guessing on MediConfusion, and the code and data were provided with the paper. After the rebuttal stage, most reviewers supported this paper.

Decision: Accept (Poster)