Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQA

Authors: Qianqi Yan, Xuehai He, Xiang Yue, Xin Eric Wang
Primary area: datasets and benchmarks
Keywords: Vision and Language, AI for Healthcare, Benchmark

Abstract:
Large Multimodal Models (LMMs) have shown remarkable progress in medical Visual Question Answering (Med-VQA), achieving high accuracy on existing benchmarks. However, their reliability under robust evaluation is questionable. This study reveals that state-of-the-art models perform worse than random guessing on medical diagnosis questions when subjected to simple probing evaluation. To address this critical evaluation problem, we introduce the Probing Evaluation for Medical Diagnosis (ProbMed) dataset to rigorously assess LMM performance in medical imaging through probing evaluation and procedural diagnosis. Particularly, probing evaluation features pairing original questions with negation questions with hallucinated attributes, while procedural diagnosis requires reasoning across various diagnostic dimensions for each image, including modality recognition, organ identification, clinical findings, abnormalities, and positional grounding. Our evaluation reveals that top-performing models like GPT-4o, GPT-4V, and Gemini Pro perform worse than random guessing on specialized diagnostic questions, indicating significant limitations in handling fine-grained medical inquiries. We further investigate the underperformance of open-source models (e.g., LLaVA, LLaVA-Med, and Med-Flamingo) through an ablation study. This study reveals that poor visual understanding is a primary bottleneck, which can be mitigated by adding visual descriptions generated by GPT-4o, leading to an average performance improvement of 9.44%. These findings underscore the urgent need for more robust evaluation methods and domain-specific expertise to ensure LMM reliability in critical medical fields.

Decision: reject
Reviewer scores: 3, 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1730500141201

thread_type:
official_review

<thread_object>
creator: reviewer_ivmy

<content>
summary:
Authors curated a large, high quality, balanced dataset of 57k close-ended (yes / no) VQA questions for  6,303 radiology medical images sourced from existing open-source datasets. The benchmark can be sliced across a number of different categories and for about half of the questions for which the answer is "yes", an "adversarial" question is also created by perturbing the entities / locations referenced in the original questions and changing the answers to "no". The dataset is used to evaluate a range of closed-source frontier MLLMs as well as open-source general purpose and domain-specific MLLMs. 

Overall the authors find that when the adversarial pair is introduced, all models saw significant drops in performance (> 20% in accuracy) although general purpose frontier models like gemini / gpt4o appear to be more robust and saw lower decrease in performance.  
The authors further find that when performance is broken down by category, general purpose models can still struggle significantly, and may perform worse on specialized domain-specific MLLMs. Amongst other findings, the authors also investigate whether COT can enhance performance, finding that while open-source models generally saw an improved performance, larger frontier models do not, and open-source models can benefit from using higher quality visual descriptions generated by larger frontier models as part of the COT process, further boosting performance.

strengths:
The dataset is large, high quality, diverse, balanced, and is curated in a methodical manner, with plenty of useful metadata (such as question categorization, location annotation, etc.), making it an incredibly valuable benchmarking resource for the community. With these metadata intact, future works can easily build on top of the released data to curate more open-ended / multiple choice style questions if needed. This in my opinion is the paper's biggest strength. Additionally, the authors share some interesting findings about how much performance degrade when simple adversarial perturbations were applied to the original questions, which shows that even the frontier models today are quite fragile and limited in their ability to perform medical decision making reliably. The investigation into COT, especially COT with external visual descriptions are also quite insightful, revealing a clear path forward to improve open-source, domain-specific MLLMs.

weaknesses:
My main complaint with the paper is that I find the presentation of the results difficult to follow at times. For example, it is not perfectly clear to me whether the "adversarial" questions are included in the tally of 57k QA pairs, and are also included as part of the evaluation benchmark by default - or if the 57k QA pairs represent the default, "clean" benchmark, with additional X number of adversarial questions (presumably = the number of questions with "yes" as answer) used to report the numbers in table 3 and the appendix only. It is also not super clear to me what the authors mean in table 3, where the authors distinguish between "Averaged Accuracy" and "Accuracy (%) with Adversarial Pairs". An illustrative example of how these metrics are computed would be helpful. 

Additionally, a limitation of the study is that the benchmark / adversarial questions are only limited to binary yes / no settings, when I can imagine the methodology easily extending to multiple choice questions, which seems like a missed opportunity. Although I believe this is something that future works can build on given the well curated nature of the dataset.

questions:
1. For table 3, my current understanding is that for questions that form a pair of original + adversarial question:
Getting the original question correct, but adversarial question wrong would equate to an accuracy of 50% in the "averaged accuracy", and 0% in the second case. But in either case, the tally of the original questions that do not have an adversarial pair remains consistent? Wouldn't this obfuscate the true impact of the adversarial examples since it's diluted by the original, non-adversarial "no" questions?
2. Are adversarial questions included in the tally of 57k QA pairs, and used in evaluation by default? e.g. in table 4, figure 4, 5, etc.
3. Besides the question + image itself, is there any additional formatting / parsing used to evaluate the various models? especially for open-source MLLMs that may not be perfect at instruction following, what happens if a model outputs a full sentence response instead of a simple yes / no answer.
4. How consistent are these results, especially for larger frontier models where it may not be possible guarantee deterministic output. Is there a way quantify the variance in the performance if e.g. gpt4o / gemini are evaluated more than once?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730590949209

thread_type:
official_review

<thread_object>
creator: reviewer_9nkq

<content>
summary:
This work curates a medical VQA dataset with pairing original questions and negation questions  and evaluations of SOTA VLMs show the poor visual understanding abilities. Furthermore, the authors show that this issue could be eliminated by adding external visual descriptions generated by GPT-4o.

strengths:
- A new dataset, PromMed, was curated for medical VQA benchmarking, which contains adversarial question pairs
- Comprehensive experiments on multiple VLMs
- Insightful findings: SOTA VLMs perform worse than random guessing on specialized diagnostic questions

weaknesses:
- Some questions are too trivial in the dataset. The ultimate goal of multimodal models is deployment in real clinical practice when the accuracy is good enough. However, no clinician will ask questions on basic modalities (e.g., CT, MR) or organs because they are too trivial. Arguments such as “CheXagent, trained exclusively on chest X-rays, achieved the highest accuracy in determining abnormalities and conditions. However, its performance in general tasks like identifying image modality and organs was lower” should be modified because ChexXagent was not designed for tasks like identifying image modality and organs. These tasks are also not clinically significant because of their trivialities. I would suggest that the authors focus their analysis and discussion on the more clinically relevant and challenging aspects of the dataset, such as identifying specific abnormalities or conditions. 

- Minor: References format is not consistent. Some preprint papers even missed arxiv id

questions:
- It is interesting that augmentation with visual descriptions generated by GPT-4o can improve the performance. Can open-sourced multi-modal models (e.g., Qwen2-VL-72b-Instruct, LLaMa 3.2 Vision) improve the performance as well?

- Sec. 4.3.2 it is not clear how CheXagent is used to enhance the model performance. Could you please provide more details on how CheXagent was integrated with other vision-language models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730621937886

thread_type:
official_review

<thread_object>
creator: reviewer_vp6d

<content>
summary:
The paper introduces the ProbMed dataset for the rigorous evaluation of large multimodal models in clinical domains. The authors show that models perform worse than random on medical diagnosis questions when subjected to adversarial questions. They also show that the poor visual understanding of LMMs is a primary bottleneck, and can be mitigated by adding visual descriptions generated by GPT-4o.

strengths:
It is a well-written paper with many experiments to evaluate the models on the proposed dataset. Several popular models have been studied and also authors propose mitigation strategies to improve the models' performance. Overall, the paper targets important questions and has the potential to be a good contribution to the field.

weaknesses:
After reading the paper, I have a few questions and concerns as follows:


1. (major) If I understood it correctly, the dataset has been curated using the publically available data on the internet. My main concern is the possible data contamination in larger closed or open-source models. When some models have been trained on the data and some not, evaluation using this dataset loses its fairness. 

2. If I have gotten it right, all the adversarial questions are in the form of negated questions and their response is "no". Having a model that has been previously trained on the data, can we ensure that it does not cheat?

3. (major) I strongly suggest adding a distribution of the responses that are "yes" or "no". how many questions have an answer of "no" and how many "yes" within each category in the dataset? This can be done through a qualitative distribution plot in the paper. 

4. (major) The adversarial question design is creative, but it has issues as well. Within clinical data. there are always cases that have co-occurrence of multiple forms of the disease, but in the original caption, we only have one of them as according to a clinician it is the important one. In this regard, when we create adversarial questions, this important fact has been ignored. So, each question actually needs to be validated by a medical expert. I have seen that 100 samples were examined by experts, but that number is significantly small compared to the size of the dataset. In fact, the paper lacks a thorough and careful expert study to ensure correctness.

questions:
I have mentioned them in weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730632258340

thread_type:
official_review

<thread_object>
creator: reviewer_6yhr

<content>
summary:
This paper introduces ProbMed, a Med-VQA dataset with adversarial negative samples to reveal multimodal models' limitations in complex medical tasks. Chain-of-thought reasoning and GPT-4-generated visual descriptions are shown to enhance fine-grained diagnostic performance.

strengths:
1. By introducing adversarial negative samples, the dataset tests model robustness, offering a more challenging evaluation standard.
2. ProbMed includes various diagnostic tasks across different imaging modalities and body organs, providing a evaluation setting for models.
3. The model’s performance improvement strategies are validated through methods like chain-of-thought reasoning and visual description enhancement, providing a foundation for future advancements.

weaknesses:
1. The dataset has a significantly higher number of chest X-ray images than other types, which may lead to poorer model performance on other organs. Has the model's performance across different modalities been experimentally verified?
2. How does the study prevent hallucinations when using GPT-4 for caption analysis, abnormality identification, and positional descriptions through few-shot prompting?
3. ProbMed uses closed-ended questions and lacks open-ended tasks, limiting the dataset's comprehensiveness for evaluation.
4. Constructing adversarial samples by selecting random entities (such as alternative organs, modalities, or hallucinated conditions with "no" answers) may increase testing difficulty but could also introduce mistakes, resulting in samples that aren’t truly adversarial.

questions:
1. Is the construction of adversarial negative samples reasonable? Could additional strategies be introduced to ensure the validity of these adversarial samples?
2. Does class imbalance in the dataset affect the model's generalization ability? How can better diagnostic performance be achieved for organs beyond the chest? Additionally, validation across multiple modalities is recommended to assess the impact of imbalanced data distribution on each modality.
3. What is the impact of adding open-ended tasks on the model's performance? How can report generation or open-ended question-answering be incorporated into ProbMed? Clearly, open-ended questions are more representative of common clinical scenarios.
4. Could GPT-4-generated visual descriptions introduce bias or hallucinations?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces ProbMed, a curated Med-VQA dataset with 57k balanced yes/no questions and adversarial negatives to assess multimodal models' diagnostic capabilities, revealing their poor visual understanding and showing that GPT-4-generated visual descriptions and chain-of-thought reasoning can improve performance.


Reviewers commended the ProbMed dataset for its high quality, diversity, and adversarial samples that test model robustness, along with comprehensive experiments and insights into improving diagnostic accuracy using chain-of-thought reasoning and GPT-4-generated visual descriptions. On the other hand, the reviewers identified shared concerns about the dataset's balance and comprehensiveness, particularly the overrepresentation of chest X-rays, the focus on binary closed-ended questions, and insufficient validation of adversarial samples by medical experts (6yHr, vP6d, ivmY). Potential data contamination from publicly available sources raised fairness concerns, and the inclusion of clinically trivial questions reduced the dataset’s practical relevance (vP6d, 9nkQ). Additionally, unclear presentation of results, such as the distinction between clean and adversarial data, and the lack of multiple-choice questions were noted as missed opportunities (ivmY). Collectively, these issues limit the dataset's robustness, fairness, and clinical applicability.


During the discussion, the authors emphasized dataset balance with binary answers across twelve [organ, modality] combinations and defended binary closed-ended questions for their scalability and robustness, though they acknowledged the lack of open-ended tasks as a limitation. Expert validation showed high accuracy but was limited in scale, and potential data contamination concerns were addressed with proposed mitigation measures.  The reviewers acknowledged some clarifications but shared concerns about the dataset's limited focus on binary yes/no questions, which 9nkQ and ivmY found restrictive and lacking clinical relevance. vP6d remained unconvinced about data contamination mitigation, noting potential biases favoring pre-trained models, while 6yHr criticized the absence of open-ended tasks as impractical for clinical use. While some reviewers raised the score, they also pointed out the unresolved concerns about fairness, clinical applicability, and dataset comprehensiveness.


While this paper introduces a well-curated dataset and provides insights into LLM-based Med-VQA models, several limitations reduce its impact and broader applicability. The emphasis on binary close-ended questions narrows its relevance, as these tasks are generally easier to address than open-ended ones, which are more reflective of real-world clinical scenarios. This focus undermines the significance of the work, particularly given that close-ended questions in Med-VQA are typically treated as classification problems, where traditional non-LLM-based models could potentially outperform LLM-based approaches. The absence of these traditional non-LLM-based methods in the evaluation leaves the study less comprehensive and balanced. Additionally, reviewers raised concerns about potential data contamination, the limited clinical relevance of some questions, and insufficient validation of adversarial samples, which were only partially addressed.