SURE-VQA: Systematic Understanding of Robustness Evaluation in Medical VQA Tasks
Authors: Kim-Celine Kahl
Selen Erkan
Jeremias Traub
Carsten T. Lüth
Klaus Maier-Hein
Lena Maier-hein
Paul F Jaeger
Venue: Accepted by TMLR ()

Abstract: Vision-Language Models (VLMs) have great potential in medical tasks, like Visual Question Answering (VQA), where they could act as interactive assistants for both patients and clinicians. Yet their robustness to distribution shifts on unseen data remains a key concern for safe deployment. Evaluating such robustness requires a controlled experimental setup that allows for systematic insights into the model's behavior. However, we demonstrate that current setups fail to offer sufficiently thorough evaluations. 
To address this gap, we introduce a novel framework, called SURE-VQA, centered around three key requirements to overcome current pitfalls and systematically analyze VLM robustness:
1) Since robustness on synthetic shifts does not necessarily translate to real-world shifts, it should be measured on real-world shifts that are inherent to the VQA data; 2) Traditional token-matching metrics often fail to capture underlying semantics, necessitating the use of large language models (LLMs) for more accurate semantic evaluation; 3) Model performance often lacks interpretability due to missing sanity baselines, thus meaningful baselines should be reported that allow assessing the multimodal impact on the VLM.
To demonstrate the relevance of this framework, we conduct a study on the robustness of various Fine-Tuning (FT) methods across three medical datasets with four types of distribution shifts. 
Our study highlights key insights into robustness: 1) No FT method consistently outperforms others in robustness, and 2) robustness trends are more stable across FT methods than across distribution shifts. Additionally, we find that simple sanity baselines that do not use the image data can perform surprisingly well and confirm LoRA as the best-performing FT method on in-distribution data. 
Code is provided at https://github.com/IML-DKFZ/sure-vqa.

Published in Transactions on Machine Learning Research (07/2025)
SURE-VQA: Systematic Understanding of Robustness Evaluation in Medical VQA Tasks
Kim-Celine Kahl1.2.3*, Selen Erkan1.2,4* Jeremias Traub1224 Carsten T. Lüth2 Klaus Maier-Hein4.2...77 Lena maier-Hein2.:..2 Paul F. Jaeger
German Cancer Research Center (DKFZ) Heidelbery, Division of Medical Image Computing, Germany Helmholtz Imaging, German Concer Research Center (DKFZ), Heidelberg, Germany
9 Faculty of Mathematics and Computer Science, University of Heidelberg, Germany
4 German Concer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Germany
GGrran Concer Research Center (DKFZ) Heidelberg, Interactive Machine Learning Group, Germany
Pallerm Analysis and Learning Group, Department of Radiation Oncology, Heidelberry Universily Hospital, Germany "National Center for Tumor Discases (NCT) Heidelbery, Germany
ka.kahl, selen.erkan) sddm.erranm/dlhhshheeelelee @ddfi--iiddeeeeeer.eee
Reviewed on OpenReview: https: openrevee.. net/ forum? id- qjNdOpgpVV
Abstract
Vision-Language Models (VLMs) have great potential in medical tasks, like Visual Question Answering (VQA), where they could as interactive assistants for both patients and clinicians. Yet their robustness to distribution shifts unseen data remains a key concern for safe deployment. Evaluating such robustness requires controlled experimental setup that allows for systematic insights into the model's behavior. However, we demonstrate that current setups fail to offer sufficiently thorough evaluations. To address this gap, we introduce a novel framework, called SURE- VQA, centered around three key requirements to overcome current pitfalls and systematically analyze VLM robustness: 1) Since robustness on synthetic shifts does not necessarily translate real-world shifts, should be measured real-world shifts that are inherent to the VQA data; 2) Traditional token-matching metrics often fail to capture underlying semanties, necessitating the use of large language models (LLMs) for more accurate semantic evaluation; 3) Model performance often lacks interpretability due to missing sanily baselines, thus meaningful baselines should be reported that allow assessing the multinodal impact on the VLM. To demonstrate the relevance of this framework, we conduct study on the robustness of various Finc- Tuning (FT) methods across three medical datasets with four types of distribution shifts. Our study highlights key insights into robustness: 1) No FT method consistently outperforms others in robustness, and 2) robustness trends are more stable across FT methods than across distribution shifts. Additionally, we find that simple sanity basclines that do not use the image data can perform surprisingly well and confirm LoRA as the best-performing FT method on in-distribution data. Code is provided at
com/ www.ccommmmo./e/... IME//MMLLDDD/DA DDF-DDDre/uuueeaaa
1 Introduction
Recent advancements in Vision-Tanguaae Models (VLMs) have seen increasing potential for application in the medical domain, with one key area being Visual Question Answering (VQA). In this task, VLMs could assist clinicians or function as medical chatbots for patient inquiries. Several general medical pretrained VLMs, like LLa (Li al. 2202)) and Med-Flas fled-ilamnngo (Moor et al. (2023) ). have already been developed.
"Thee authors contributed equally to this work

---

Published in Transactions on Machine Learning Research (07/202) (025)
Pl: Robustness on synthetic shifts does not imply robustness real-worid shifts
P2: Traditional metrics do not capture the underlying semantics
P3: Model performance lacks interpretabllity due missing sanity baselines
image
There a rodds Prod natoo nodkle
Groand Trush
S
What the seansing prane imggr?
Protte makil
Fred Auid
LA. Mari
Traditiona Matrice
Frod Asi!
Modal performe beet! The task seams
Model does not really conaider der the image really
Scor
RT: Realistic Shifts
R2: Appropriate Metrics
R3: Relevant Sanity Baselines
Figure 1: Pitfalls and Requirements for Systematic Evaluating the Robustness of VLMs in VQA Tasks. We aim to overcome pitfalls (P1-P3) in the current evaluation of VLM robustness by satisfying the three requirements (RI-33): We define a diverse set of realistic shifts (R1). We use appropriate metrics for evaluation by using an LLM as evaluator of the VLM output (R2). Finally, we compare the results of the VLM with relevant sanity baselines to see the performance gains over such baselines like considering the text of the question only (R3).
However, a crucial question remains: how robust are these models when faced with variations in data distribution during application? The datasets used for training or fine-tuning may not fully capture the variations in real-world clinical data. As RRberrss et al. (2021) highlights how the urgency of the COVID-19 pandemic led to many studies utilizing datasets that insufficiently represent pediatric patients, introducing bias into the analyses. These shifts, whether through unseen disease variations, variations in image acquisition, or different question subjects, may cause performance degradation. Understanding how robust VLMs are to these changes is key to ensuring clinical reliability.
Roberts
Despite the importance of this research question, existing benchmarks fail to offer an adequate framework to address it effectively. While several benchmarks exist for evaluating the robustness of VLMs under artificial image text corruptions (Zhang et al. (2024); Chen et al. (202)),, there remains notable gap in benchmarks that account for more realistic data shifts. We address this gap in the medical domain by utilizing existing medical VQA datasets and setting them to test VLM robustness against realistic shifts inherent to the VQA data. This is crucial, as prior research has shown that robustness to synthotic shifts does not necessarily translate to robust ness under real-world conditions (Taori el al. (2020)). Additionally many current benchmarks rely on traditional metrics, which use token matching between the ground truth and the model's predictions. We highlight common flaws in those metrics and instead propose the use of large language models (LLMs) evaluators, validating this approach through human rater study. Finally, current benchmarks often overlook simple baselines, such as testing model's ability to answer questions based solely on text. Including such sanity baselines can reveal language priors in the dataset, where questions might be casily answered by their content or by predicting the most common answer seen during training. To overcome these three apparent pitfalls the current literature, we define three key requirements. Based on these requirements we present a ficxible framework, callod SURE- VQA, that enables mcaningful evaluation of VLM robustness in the medical domain.
To showcase the relevance of SURE-VOR VQA, we conduct study comparing the robustness of various fine- tuning (FT) approaches, similar to Chen et al. (2023). but with focus on the medical domain. We focus on FT methods, including full FT and parameter-efleient FT (PEFT) because fine-tuning VLMs crucial and common practice for precision in specialized tasks like medical VQA (Li et al. (0023); Wu et al. (2024): Singhal al. (2023)). Our study provides valuable insights to the following questions: How does medical vs. non-medical pretrained VLM perform on the medical VQA task after fine-tuning? How do the FT methods perform in comparison to sanity baselines that, for instance, omit. image content? How does the
2

---

Published in Transactions on Machine Learning Research (07/)025)
performance of different FT methods vary across datasets? How to FT methods compare in terms of i.i.d. performance and robustness? Which shift is most severe regarding model robustness?
In summary, our contributions are:
1. Systematically analyze current pitfalls and derive key requirements for meaningful robustness eval- uation of VLMs in the medical domain.
Provide a flexible open-source framework, named SURE- VQA, hosted at: https://github. com/ IML-DKFZ/ surevvaa
3. Provide empirical evidence for the derived requirements, including human rater study that confirms the importance of LLM-based metrics.
4. Show the relevance of SURE-VOR by performing a meaningful comparison of the robustness of FT methods in the medical domain leading to valuable insights for the community.
2 Requirements for a systematic evaluation of the robustness of VLMs
We identify several key pitsalls in the current evaluation of VLM robustness, addressed by our framework in Figure 1 In the following section, detail these pitfalls (P1-P3) and formulate requirements (R1-R3) to overcome them. Finally, we outline our setup to fulfill these requirements.
Pl: Robustness on synthetic shifts does not imply robustness on real-world shifts. Many existing benchmarks that focus on the robustness of VLMs, such as those in Chen et al. (2023); Shirnin ct al. (2024); Qiu et al.] (2022). primarily introduce artificial perturbations notable exception is Raaffodd
the image or text content. A
al. (2021), where the robustness of natural distribution shifts is explored in the context of their proposed CLIP model. They find that CLIP is quite effective in being more robust natural distribution shifts of ImageNet. However, their analysis is limited by CLIP's non-generative tasks and does not consider the generative VQA task. In the medical domain, Nan et al. (2024] conduct a multimodal robust benchmark. Their study focuses on medical VQA tasks, but remains limited to testing image corruptions, leaving the crucial question of how realistic distribution shifts impact model performance open. Another study by Jensen Plank (2022) takes a step toward addressing more realistic data shifts by examining the performance of VLMs on different versions of the VQA dataset, when training them from scratch vs. fine-tuning pretrained model. However, their focus is primarily on linguistic variations, leaving shifts in image content under-explored. Similarly, though the benchmark by ZZang et al. 2024h uses artificial image and text corruptions, their content bias might be one step lowards more realistic shifts.
However, while the robustness benchmarks in the VLM domain rarely address any realistic shifts, in the unimodal domain there is evidence that models are not robust against natural distribution shifts (Taori et al. (2020 Miller al. (2020)). This issue has also been proven for fine-tuned, domain-specific models (Yuan ct al. 2023 ). Furthermore, there is evidence that artificial shifts do not necessarily translate to rcalistic shifts
Taori al. (2020)).
+ R1: Evaluate VLMs under a diverse set of realistic shifts.
We utilize three datasets from the medical VQA domain, including SLAKE (Liu al. (202T)), OVQA (Huang et al. (2022)), and MIMIC-CXR-VQA (Bae et al. (2023) On these datascts, we define several realistic shifts, spanning a range of subtletics, with some having more pronounced impact, such as modality shifts, while others, like gender shifts, are more subtle. Furthermore, certain shifts primarily affect the image content, such as changes in the body location being imaged, whereas others influence the text input, such as shifts in question types.
Implementation in SURE- VQA:
P2: Traditional metrics do not capture the underlying semantics. Many studies in the VLM field
continue to rely on traditional metrics like BLEU and CIDEr (Sung et al. (2022) : Chen et al. 20233 Quu
ct al. (002)),, ), or accuracy-based metrics (Li et al. (2023): Jensen & Plank (2022); Qiu et al. (2022) ) which
are dependent on word- or n-gram matches. We refer to these as "traditional metrics" throughout the paper.
Recent research, however, has begun to adopt a more sophisticated approach by employing LLMSs to evaluate
3

---

Published in Transactions on Machine Learning Research (07/2025)
the output of VLMs (or other LLMs) (Wang et al. (2024): Ostmeier et al. (2024) 1 Liu et al. (2023b): Chiang
& Lee 2023); Fu et al. (2024); Kocmi & Federmann (2023
The primary limitation of traditional metrics is their inability to capture the underlying semantics of a sentence. They fail to recognize synonyms or account for negation, often misjudging sentences that differ from the ground truth by a single token, such as 'not.' ! Examples of these failures are shown in Appendix A.I similar Ostmeier et al. (2024). While we are not the first ones to propose using an LLM as an evaluator, the contimued reliance on traditional metrics despite their flaws highlights the need to establish this an explicit requirement for VLM evaluation.
+ R2: Evaluate VLMs with appropriate metrics that capture the underlying semantics of the output.
Implementation in SURE- VQA: We employ the Gemma model (Team Gemma et al. 2024) as an evaluator, utilizing three distinct prompts tailored for different question types: open-ended, closed-ended binary, and closed-ended multilabel. Details on the question types can be found infsection 3.1 To optimize computational efficiency and reduce potential errors from the LLM evaluator, we implement a hybrid metrie. Specifically, when the answer exactly matches the ground truth, we assign the highest score without invoking the LLM, saving computational resources and minimizing the risk of evaluation failures. Additionally, we assess the feasibility of this evaluation by conducting human rater study, where we empirically validate its performance in comparison to traditional metrics and other LLMss
P3: Model performance lacks interpretability due to missing sanity baselines. Currently, the performance of VLMs is typically reported either in isolation or in comparison to other VLMs. This is evident in papers that introduce new VLMs (T. et al. (2023): ; Moor et al. (2023)) ), and in benchmark studies Chen et al. (2023); Zhang et al. (024);; Qiu et al. [0022] : Nan et al. (2024)). An exception is the work of Liu et al. (2023a). where the performance of a language-only GPT-4 is evaluated. However, their focus is to improve the model by ensembling with the LLM rather than to highlight a potential issue in current VQA datasots that many questions can be answered using only text information. Further, they do not provide a no image sanily baseline of their own model, leaving its multimodal usage unexplored. Another st study by |Parcalabescu & Frank (2023) contextualizes the multimodal use of VLMs by employing Shapley values to assoss the contribution of cach modality to the output. However, this method is computationally more complex during inference, since multiple forward passes are needed to estimate the Shapley values.
The problem with many VQA datasets is that they tend to contain hidden patterns, allowing models to exploit shortcuts (Geirhos et al. (2020)) rather than using all available information, including the image content (aflee & Kanan 2011);; Chen et al. (2024)); Kervadee et al. 2021 Goyal et al. (2017); Dancette et al. (021h). This means that high performance on these datasets does not guarantee that the model is actually utilizing the visual input to answer the question; instead, it might be exploiting patterns in the questions (Kaffe & Kanan (2017); Kervadee et al. (001))) As demonstrated by Chen et al. (2024a)), in many cases, the visual content in VQA datasets is unnecessary, and models can achieve high performance simply by relying on the text. This indicates that the models are leveraging hidden biases in the question-answer pairs rather than solving the task as intended.
+ R3: Provide relevant sanity baselines to conteztualize the benefits of VLM fine-tuning and multimodal information usage.
We propose that using relevant sanily baselines to reveal such dataset biases can be beneficial to explore how the models solve the given task and how the datasets are structured. Thereby, WC put the performance of the finc-tuned model into context by comparing it to basclines in two aspects: (1) Not using the image information: Here, we a) choose the most frequent answer to the question in the training set and answer the same question in the test set with this (most. frequent baseline) and b) train the model without using any image information, which should lead to learning shortcuts based on the language (no image baseline). (2) Not fine-tuning the model: Use the plain VLM in zero-shot setting without any fine-tuning and report the performance on the test. set. This serves at the same time baseline to see if or how much the shifts are inherently different between i.i.d. and OoD when not finc-tuning.
Implementation in SURE- VQA:
4

---

Published in Transactions on Machine Learning Research (07 (07/2025)
3 Empirical Study
To provide evidence of the relevance of our framework, we perform experiments to (1) empirically validate the necessity of requirements R1-R3, and (2) show that our framework allows valuable insights into robustness by conducting an extensive empirical study on the robustness of FT methods. In this section, we first present the datasets and experimental setup used our studies, followed by a comprehensive analysis and discussion of the empirical findings.
3.1 Utilized Datasets
An overview of the utilized datasets provided in Figure 2 with further details regarding the datasets, preprocessing steps, and split sizes available in Appendix C.1 In total, we use three different medical VQA datasets, each with closed-ended and open-ended questions. Closed-ended questions fixed set of options is given as part of the question (e.g. "yes"/"no,, (CTT/MMRIY) while open-ended questions are not constrained to certain options in the answer. Each dataset incorporates a variety of realistic shifts to meet R1 The laxonomy for shift categories is thereby partially laken from Castro et al. (0200), also used in related work such as Bungert et al. ) Roschewitz et al. (0023) Choi et al. (0023).
questions where
2021 with two different shifts: 1) Modality shift: Rep- resenting an acquisition shift (Castro et al. (2020)) we train the model exclusively on CT and MRT images (2D slices) and then test it on X-ray images. 2) Question type shift: During training, the model is exposed to questions about imaae content such as shape and color but excludes questions related to the size of organs. These size-related questions are introduced in the OoD test set.
SLAKE
We use the SLAKE dataset Liu et al.
OVQA The OVQA dataset (Huang et al. 2022 )) is used with two shifts: 1) Body part shift: Representing & manifestation shift (Castro al. (2020)); ) we train the model on images of the hand, chest, and head, and
test it on images of the leg. 2) Question type shift: In the training set, the model is exposed questions about various image contents like abnormalities and conditions, but questions related to the organ system are reserved for the OoD test set.
MIMIC-CXR-VQA We use the MIMIC-CXR-VES VQA (Bac ct al. (2023) dataset with three different shifts, all representing population shifts (Castro el al. (2020): 1) Gender shift: The model is trained on male patients and tested on female patients. 2) Population shift: Training is conducted using data from white patients, with testing on patients from other cthnicities.
3) Age shift: The model is trained on patients over the age of 60 and tested on patients under 40. A gap is intentionally introduced between the i.i.d. and OoD groups to make the shift more explicit.
3.2 Study Design
We utilize above mentioned VQA datasets, splitting them so that the training and testing distributions differ. We employ two different base models: LLaVA-Med 1.5, state-of-the-art medical VLM (Li et al. (2023)), and its corresponding non-medical base model, LLaVA 1.6 (Liu et al. (2023ah). For fine-tuning, we use the following methods: full FT, prompt tuning [Lester et al. (2021)), LoRA (Hu al. (2021)) )), and (IA)² (Li) al. (2022)). Hyperparameters for the PEFT methods are selected based on the full training set and corresponding validation set for each dataset. Details regarding the hyperparameter search can be found in Appendix C.2 To measure robustness,
split the data into i.idd. training and i.i.d. and OoD test sets, outlined in sscction 3.1. thereby fulfilling RI We then measure the performance of the VLMs using the language model Gemma, fulfilling R2 The selection of Gemma as an evaluator is further justified in section 3.3.2 For robustness measurement, we calculate the relative robustness (RR) (Chen et al. (2023) ) defined as RR OoD test performance. For better interpretation of the results, we compare them against relevant sanity baselines described in R3
AP/P,, where AP (P, Po), and P, the i.i.d test performance and Po the
5

---

Published in Transactions on Machine Learning Research (07/ (025)
SLAKE
OVDA
MIMIC
Acquistion Shill:
Modality
Quastion Type Shift
Manifestatin Shift Body Part
Question Type Shift
Population Shift Gender
Population Shift: Ethricity
Populition Shift: Age
Shope, Plani....
Hand. Head
Abnormality Plane
Black
X-Ray
Lutino
What plane of the image?
Does cordin excood half of thorox wide??
done belong to?
organ in the pootue??
fraciures present?
acen?
picheeed?
denormuitios apperett
witth moro than half the thoras withh??
Chesl
Plane
Liver
Yea
OoD
bataser
Cutegorics
Split Cutagorica
Exampli:
Irage
Quecatin
Arreveer
Figure 2: Datasets and Shifts Used in the Study. We use three datasets with four different types of shifts, resulting in seven different settings for robustness analysis. Shifts that are mainly focused on changes in the image content are shown by a change images between i.id. and OoD and shifts that focus on the question content are shown by a change of the question and answer between i.idd and OoD shifts. The taxonomy for the shift category partially taken from Castro al. (2020).
3.3 Empirical Confirmation R1-R3
3.3.1 R1: Comparison between Realistic Shifts and Corruption Shifts
Study Design To compare the effect realistic shifts [1 against artificial shifts (corruption shifts), we perform robustnnss study the SLAKE dataset, using a simplified setup that involves only the LLaVA- Med model fine-tuned on SLAKE with (IA)². This study aims to support the claims outlined in (R1) within the context of our dataset sctup. As illustrated in Figure 20] the model is first fino-tuned on an i.i.d. training set. Then, we test it on the i.i.d. test set and two different OoD test sets-ore reflecting realistic shifts, and the other containing artificially corrupted images. For realistic shifts, we use the original i.i.d. and OoD splits defined in section 3.1 For artificial (corruption) shifts, the OoD test set is generated by applying image corruptions like blur, brightness, and noise varying severity levels to the i.idd test images. Each corruption applied with a probability of 0.5, and at least one corruption is applied per image.
As seen in Figure 3 the realistic modality and question type shifts exhibit lower relative robust- ness compared to artificial shifts (medium strength), meaning artificial shifts fail accurately capture the challenges presented by realistic shifts. This finding highlights the importance of ovaluating robustness under realistic conditions, adopted in our study. While corruption-based shifts may offer a valuable complemen- tary perspective, these results emphasize that they should not be considered the only source for evaluating robustness. Further details and experiments on other corruption levels are provided in Appendix D
Results
3.3.2
R2: Human Rater Study
Study Design To ensure that the scores assigned by the LLM align with human judgment, we conduct a human rater study. We use this to compare various LLMs for rating and select the one that aligns best with human judgment. For each dataset, we randomly selected 100 open-ended questions where the prediction did not exactly match the ground truth, as exact matches would automatically score highest by the hybrid metric [22. Five human raters evaluate the questions, and we calculate the correlation between humans and the LLM using Kendall's Tau [Kendall (1945)). Additionally, we report the correlation between humans and traditional metrics, and the inter-rater variability.
Results Among the LLMs evaluated, Gemma demonstrates the highest correlation with human ratings, as detailed in Figure 10 (Appendix B.1). Thus, Gemma selected as the evaluator, and we present the results of the human rater study using Gemma in the following section. The correlations shown in

---

Published in Transactions on Machine Learning Research (07 (07/2025)
SLAKE
2.8
Modaiiy Shift
Realstic Dob Hay
2.6
Medium Comrpption
0.14
Question Type
Rodistic OcD (Sibe)
Comypttio
Open Fruded
Choed Endeed
Choed
open
Figure 3: Comparison between Realistic shifts and Corruption shifts on SLAKE Datasct Finc- tuned on (IA) For realistic shifts, use the same i.i.d. and OoD training and test sets described in Section 3.1. For corruption shifts, the iidd training and test sets remain the same as in the realistic setup. The only difference the OoD test set, which is created by adding one or more corruptions such as blur, brightness changes, or noise the i..d lest images at varying levels of severity.
SLAKE
OVQA
Kondall Co Comolation
Kencial Correlation
Haman
Hummn
0.37
0.41
Humin
Mutch
Peeciaion
secad
HUman
Gemmia
Match
Peedsaion
MIMIC
Kondall Correlation
0.71
35
0.76
0.79
0.74
Humtn
BEXU
Muiich
Prechioo
MICHI
Figure 4: Results of the Human Rater Study. Human interrater correlation is calculated between five raters. We use Kendall's Tau Kendall (9455) correlation mcasure.

---

Published in Transactions on Machine Learning Research (07/2025)
Figure 4 with detailed results available in Appendix B.3 Gemma demonstrates the highest correlation with human ratings on the SLAKE and OVQA datasets, with differences between Gemma and the best traditional metrics being 0.15 and 0.21, respectively. Interestingly, on these datasets, the human interrater correlation is not the highest compared to correlation with other metrics. This suggests that the metrics fall between the evaluations of the human ralers, resulting in a stronger correlation with the average human rating than between the individual raters themselves. On the MIMIC dataset, traditional metrics perform slightly better, with the performance gap between Gemma and the traditional metrics ranging from 0.03 to 0.08. This variation can be attributed to several factors. In the OVQA dataset, for example, the structure of the questions and answers makes traditional metrics more prone to failure. Tokens like 'fracture," "left' /'right', or specific bone names often match between the ground truth and predictions, despite significant differences in other Lokens (see Figure 11a for an example). On the other hand, MIMIC answers are highly structured, particularly for open-ended questions, where a fixed set of classes is listed in a comma- samma--eparated format. In these cases, traditional metrics perform well because token matching tends to be more accurate especially when the classes have distinct tokens.
Despite these limitations, Gemma's performance on the MIMIC dataset remains strong, and its correlation with human ratings is even higher than on the other two datascts. Morcover, the failures of Gemma on MIMIC are not complete failures, as shown in Figure 11b In summary, although there are instances where traditional metries show higher correlations with human scores, Gemma proves to be generally more robust and less prone to complete failures. Its correlation remains consistently high across all datasets, confirming it as a reliable metric for evaluation in VQA tasks. However, even though Gemma shows high correlation with human judgments, it is important to note that LLMs may carry internal biases. These biases might come from underreprosentation of detailed medical knowledge, recent medical advancements, and diverse socio-demographie groups in LLMs training procedure and data. As result, such factors can influence the model's scoring behavior in evaluation tasks. That's why it is crucial to conduct human-rater studies and choose an LLM that aligns closely with human judgment to minimize these effects especially when applied to a new use case.
3.3.3 R3: Performance of Sanity Baselines
Study Design To evaluate the performance of sanity baselines, we compare a VLM not using the image information (no image sanity baseline) with a VLM using the image information. Specifically, we investigate how often the model using the images actually outperforms the model that is not using the images. This allows us to evaluate the model's susceptibility to shortcut learning, where it may rely predominantly on language cues while neglecting visual information. While this section analyses the no image baseline in detail, the general confirmation of the other sanity baselines is part of our large-scale study in seccion 3.4.
Results As seen in Figure 5] overall, the sanity baseline of not using the image information performs surprisingly well. Idcally, in the absence of shorteut learning, the model utilizing image information should almost consistently outperform its image-agnostic counterpart or at least lie on the individual questions. However, this is not the case on any of the datasets. While the win-to-loss ratio is overall positive for the model using the image information, the margin is narrower than anticipated. Most notably, on the MIMIC dataset, the model using the image information is even outperformed on up to 25% of the questions. While the SLAAKE dataset, the win-to-loss ratio is highest for the model using the image information, it is the worst on the MIMIC dataset, and on the OVQA dataset, the models tie the most. Further results the other sanity basclines as well as potential reasons for performance differences between datasets are provided in Section 3.4
3.4 Empirical Study on the Robustness of Fine- MuninT Methods
To show the relevance of SURE-VQA we perform an empirical study comparing the robustness of various FT methods under realistic shifts in medical VQA. This important since practitioners should be informed about the differences between the methods not only in terms of their performance but also how robust they are when selecting method. The results of the FT robustness study can be seen in Figure Figure 7 and
8

---

Published in Transactions on Machine Learning Research (07 (07/202) (025)
COO
TM
42%
21.5
17%
Chios CL.ok 062
Opes A4
02106
Cland Coaed Qu:1-.
Figure 5: Comparison between the Model using the Images and the Model not using the Images. For each PEFT method, the answer scores of one model using the images are compared the corresponding answers of the model not using the images. Lose Model using the images loses, Tie Both answers are equally good, Win Model using the images wins. Closed ii.d. / OoD Closed-Ended questions, Open i.idd OoD= Open-Ended questions. Figure layout inspired by Jeong et al. (2024)
Figure 8 Detailed tables with the results are shown in Appendix C.5
Further, an ablation on the impact
of multimodal shifts is shown in Appendix F
Comparison between Medical and Non-Medical Base Models. F Figure 6 shows the comparison between the medical (LLaVA-Med 1.5) and the non-medical base model (LLaVA 1.6), aggregated over all FT methods. The results indicate that medical and non-medical models often tie in their answers, with lies ranging from 64% Lo 90%. On SLAKE and OVQA, the medical model generally achieves a slightly higher number of wins. On the MIMIC dataset, the medical model slightly outperformed on closed-ended questions but shows marginally better performance on open-ended ones. However, the differences in wins and losses between the two models are mostly minor. This suggests that there is no clear winner between medical and non-medical base models when fine-tuning. Similar results have been shown by Jeong et al. (002g) in zero- and few-shot settings, however, in their experiments, the medical model performed slightly worse. To make the discussion about the following results easier to parse, we detail the results of the medical base model in the main paper and show detailed results on the non-medical base model in Appendix C.4
Comparison Between Datascts Considering the Sanity Basclines Generally, the PEFT models outperform the no fone-tuned models, the most frequent baseline, and their respective no image baselines on the i.i.d. datasets. This is contrast full FT, which often does not outperform the most frequent baseline. Notably, while the no image baseline and the most frequent baseline both mostly perform better than random classifier, this effect is particularly noticeable for the closed-ended questions OVQA, where average 72%, and the most frequent baseline 73.6%. Consequently, the
the image baseline achieves gap between the no image baselines and the PEFT models is smaller for OVQA (18%) compared to SLAKE (33%). This indicates that the model can often rely on learned question-answer correlations without needing the images, likely because the OVQA dataset has the least unique questions (see Table 4 Appendix C.1.4 For the MTMTC dataset, the gap between the no image baseline and the PEFT methods is even smaller than for OVQA, with only an improvement of 11% for the PEFT methods over the baselines. However, as Table 4 suggests, MIMIC does not have many repetitive questions. Thus, the small gap between PEFT models and the no image baseline might mean that image encodings contribute minimally to the VQA task. This is likely because the MIMIC dataset focuses on detailed chest X-ray images, and the image encoder may lack this fine-grained expertise. Regarding the no-finetuned model baseline, PEFT models perforrorm the no-finetuned model on the closed-ended questions by the largest margin on OVQA, with a 43% improvement on average. However, this trend differs on the open-ended questions, where the difference between the fine- tuned models and the no-finetuned model seems larger on SLAKE and MIMIC To summarize, the models overall to learn most the SLAKE dataset, which also shows the highest i.i.d. performance the

---

Published in Transactions on Machine Learning Research (07 (07/202) (025)
SLAKE
OVQA
MIMIC
Win
27%
30%
64%
08-1 64-16
13%1%
Ulaed
Opentic:
Utorl 0se1i
Figure 6: Comparison between the Medical and the Non-Medical Base Model. For each FT method, the answer scores of one medical experiment are compared to the corresponding answers of the non-medical experiment, Lose = Medical model loses, Tie = Both answers are equally good, Win = Medical model wins. Closed i.dd. OoD Closed-Enae Ended questions, Open i.id. / /oD== Open- Ended questions. Figure layout inspired by Jeong et al. (0024)
PEFT methods with an average accuracy of 86.2% and a Gemma score of 4.3. In contrast, the closed-ended performance the MTMIC dataset seems generally insufficient for practical use (61%).
Comparison Between FT Methods. When comparing full with PEFT methods, we see that full FT neither outperforms PEFT methods in terms of performance nor robustness, aligning with the observations of Dutt et al. (2024) that PEFT methods are particularly well-suited for medical and low-data scenarios. From the PEFT methods, accuracy of 80.6% closed-ended and Gemma score of 3.6 on open-ended questions across different datasets and shifts. However, the performance differences between LoRA and other PEFT methods are quite small, especially considering that these other methods require fewer parameters to train. Overall, robustness within methods is more homogencous, while there is more variance across dataset shifts. This suggests that the lype of shift has a greater impact on robustness than the choice the fine-tuning method as visualized in Figure 8p). Only on the OVQA dataset, the inter-method variability is higher than the inter- shift variability, but this is due to the failure in robustness of full FT. Only comparing the PEFT methods, also for OVQA the inter-shift variability is lower, shown in Appendix C.5 Figure 18 Besides that, none of the fine-tuning methods consistently outperforms the others in terms of robustness as seen in Figure 8p). where the rank of each method is depicted with respect to their RR. As one exceptional outlier of the PEFT methods, on closed-ended questions in SLAKE, LoRA demonstrates significantly lower robustness compared other methods, with an RR of 48%, compared 65% for prompt. tuning and 68% for (IA)
the i.i.d. set, LoRA is consistently the best PEFT method, achicving average
Comparison Between Shifts. The robustness trends vary between datasets, as shown in Figure 8a). On SLAKE, models demonstrate greater robustness on open-ended questions compared to closed-ended ones, while the opposite true for OVQA. This behavior results from the related question patterns between OoD questions and the training data. Although the OoD questions are not part of the training set, similar training questions offer enough context for generalization, seen in SLAKE for open-ended questions and in OVQA for closed-ended ones. Similar analysis on conducted ablation study is provided in Apppendix E] This pattern is especially clear in question type shifts, where RR SLAKE is 54% for closed-ended questions and increases to 98% for open-ended. In contrast, on OVQA, RR is 94% for closed-ended questions and drops to 63% for open-ended ones. This means that the question type shift seems most severe if the model performance drops. The population shifts on MIMIC did not affect the models' robustness, iee. the models seem robust against such shifts since they show over 100% RR. However, since the performance on the MIMIC dataset is generally insufficient, it is questionable if this observation would hold on higher i.i.d. performance. Future experiments could investigate whether the low performance or the kind of shift. is causing this behavior.
10

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Clesad-Ended
Open-Ended
SLAKE
Camrra Score
Gueetoon Type
lid
Oco
Mest Free
Boaed rocoliineoe (ud))
20000001
No ling Boseline
Prssuuin
LoWA
MAP
Closed-Ended
Open-Ended
Body
OVQA
Cemrra Score
Trea
Gu Question Type
Centeess
Ldd
Cop
Basoiine (dd)
No ling Boselino
hoop!
Closad-Endied
Open-Ended
Candde
id
MIMIC
OuD
0000000
Noiig Soceino
No TGt Yha Val60
1994
607
werrt
607
Figure 7: Results of the FT Robustness Study on the i.i.d. and OoD Test Set. Reported results show the mean over three seeds (exception: no FT, full FT) with the standard deviation for the non- baselines. Gemma Accuracy refers to the accuracy being rated by Gemma. For MIMIC, the most frequent sanity baseline can not be calculated as too few questions match the training set.
11

---

Published in Transactions on Machine Learning Research (07/202) /025)
(a]
SLAKE
OVGA
MIMIC
Solave
Chief
detber Giees
Edeecy Corees
Cheony Oiees
Greed
Dhen
Cheetio
hrmeet
(c)
SLAKE
OVQA
MIMIC
1921
Merfo 1o86-15
435
Chemme faeetwe
clomy
Aosee
SLARE
ONDA
MNC
Figure 8: Results of the FT Robustness Study Focusing on the Relative Robustness (RR). Since the study focuses on comparing the FT methods and our definition of OoD holds for these methods, the no FT baseline is excluded. (a) RR on the three datasets for the FT methods. Results show the mean and standard deviation over three seeds (exception: full FT). (b) RR Ranking of all FT methods. (c) Standard deviation between shifts standard deviation between FT methods.
4 Conclusion and Take-aways
We present a framework that allows testing the robustness of VLMs medical VQA tasks. Thereby, we especially focus on three key requirements for a meaningful evaluation of robustness.
Empirical Confirmation of R1-R3. While in section 2 we derived R1-R3 from flaws in the current literature, our study provides empirical evidence for their importance. R1: In study (section 3.3. and Appendix D), wo show that corruption shifts do not necessarily translate to real-world shifts, thereby justify- ing our claim also work with more real-world shifts. R2: We present several critical failures of tradit ional ooken-matching metrics and prove the applicability of our LLM evaluation setup by a human rater study in section 3.3.2. R3: We show that some sanity baselines that do not use the image information already perform surprisingly well dsection 3.3.3 and section 3.4.. This highlights two aspects: 1) As stated in R3, reporting such sanity baselines is crucial for understanding the true multi-modal performance of VLM,, beyond its language-only capabilities. 2) This observation further suggests that we need more elaborate datasets and tasks in medical VQA that minimize the potential for shorteut learning based solely on the language content, Achieving this may involve incorporating greater linguistic variability in questions, as seen in Bac al. (2023), where questions were rephrased using GPT-4, and ensuring a broad range of semantic differences in the questions. Progress can be tracked as low performance of the no-image sanity baseline.
Genoralizability of the Framework. SURE-VQA serves as starting point for a comprehensive eval- uation of robustness and it can be flexibly extended to new datasets, methods, and domains. Additionally, SURE- VQA can support method development aimed at enhancing the robustness of VLMs. In our study, we define OoD as data shift w.r.t the FT data. However, our framework also allows to compare VLMs zero-shot setting without any FT, when simply redefining OoD as data shift w.r.t the pre-training data. However, such definition becomes increasingly challenging to validate with foundation models, since the
12

---

Published in Transactions on Machine Learning Research (07/2025)
exact training data used is often not known. Notably,
R2 and R3 go beyond robustness analysis and should
be integral to any well-designed VLM study.
Main Insights from the FT Robustness Study. Our exemplary study which compares the robustness of various FT methods reveals several key insights. With regard to robustness, we find that no single FT method consistently outperforms the others in terms of robustness. Furthermore, robustness trends appear to be more consistent within FT methods than across different dataset shifts, indicating that the type of shift has greater impact on robust than the choice of FT method. This suggests that robustness alone is not decisive factor when choosing FT method. Additionally, the models are generally robust to population shifts. However, further investigation is needed to determine whether this is due to low ii.d. performance because of the nature of the shift. Besides these insights into robustness, our comparison of medical and non-medical base models shows that both perform similarly when fine-tuned. Finally, in line with findings of Dutt et al. (2024), we find that PEFT methods are more efficient than full FT. Specifically, we confirm LoRA as the best-performing FT method on the i.i.d. dalaset.
Future Work. As mentioned above, SURE- VQA starting point serving as a foundational framework and is not yet intended climically deployable benchmark for medical VQA systems. Instead, it functions a research tool to inform and guide future developments in the field. Future work can include the investigation of more datascts, shifts, methods, and models. Particularly, the exemplary study in this paper mainly focuses on radiological VQA datasets. Other medical image types like histopathology, surgical, or dermatology images could be investigated in futurre work as they may have different characteristics and thus may reveal other findings. Further, while this paper focuses on simple, computational efficient basclines that omit image input, future work could use Shapley values to gain additional insights into the role of images in the VQA task (Parcalabescu & Frank (2023)). Another key research direction is the development of additional VQA datasets, particularly in the medical domain. This would cover two needs: 1) The need for more diverse question types, mitigating shorteut learning during fine-tuning based on the language, and improving the generalization on unseen data and 2) improve the clinical relevance. While current datasets cover various question types, they might offer limited value to clinicians. For example, clinicians might find questions on prognosis and tumor spread more beneficial than scanning modalities. Therefore, collaborating with clinicians can help incorporate more clinically meaningful questions. Finally, the underperformance of two state-of-the-art VLMs, VL and LLaVA-Med the MIMIC-CXR-VQL dataset underscores the critical nced for advancing the development of medical VLMs, particularly given that LLaVA-Med is specifically designed for the medical domain. Even though recent work by Chen el al. (2024b) has focused on developing foundation models specifically for chest X-ray data, there is still a need to develop more robust models capable of handling multiple modalities across a wider range of clinical scenarios.
Acknowledgements
This work was partly funded by Helmholtz Imaging (HI), platform of the Helmholtz Incubator on Infor- mation and Data Science. We thank Dasha Trofimova for her help in the human rater study as well as insightful discussions and feedback.
13

---

Published in Transactions on Machine Learning Research (07/ (025))
References
Scongsu Bac, Dacun Kyung, Jachee Ryu, Eunbycol Cho, Gyubok Lee, Sunjun Kweon, Jungwoo Oh, Lei Ji, Eric Chang, Tackeun Kim, and Edward Choi. EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Images. Advances in Neural Information Processing Systems, 36:3867 3880, December 2023.
Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. SCARF: Sclf-Supervised Contrastive Learning using Random Feature Corruption, March 2022.
Till J. Bungert, Levin Kobelke, and Paul F. Jäger. Understanding Silent Failures Medical Image Classi- fication. In Hayit Greenspan, Anant Madabhushi, Parvin Mousavi, Septimiu Salcudean, James Duncan, Tanveer Syeda-Mahmood, and Russell Taylor (eds.), Medical Image Computing and Computer Assisted Intervention MICCAI 2023, pp. 400-410, Cham, 2023. Springer Nature Switzerland. ISBN 978-3-031- 43898-1. doi: 10. 1.11077978--3-01-433999 39.
Daniel C. Castro, Ian Walker, and Ben Glocker. Causality matters in medical imaging. Nature Communi- cations, 11(1):3673, July 2020. ISSN 2041-1723. doi: 10.1038/sA 10.1038/s41467-0200 114476000-17777--1
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are We on the Right Way for Evaluating Large Vision- Laiion-Lnguuge Models?, April 2024a.
Shuo Chen, Jindong Gu, Zhen Han, Yunpu Ma, Philip Torr, and Volker Tresp. Benchmarking robustness of adaptation methods on pre-trained vision-language models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 51758 51777. Curran Associates, Inc., 2023.
Zhihong Chen, Maya Varma, Jean- Man-ienoitt Delbrouck, Magdalini Paschali, Louis Blankemcier, Dave Van Veen, Jeya Maria Jose Valanarasu, Alaa Youssef, Joseph Paul Cohen, Eduardo Pontes Reis, Emily B. Tsai, Andrew Johnston, Cameron Olsen, Tanishq Mathew Abraham, Sergios Gatidis, Akshay S. Chaudhari, and Curtis Langlotz. CheXagent: Towards a Foundation Model for Chest X-Ray Interpretation, January 2024b.
Cheng-Hap Han Chiang and Hung-yi Lee. Can Large Language Models Be an Alternative to Human Evaluations? In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Popers), pp. 15607 15631, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1 200653/////22....oooogg....
Youngwon Choi, Wenxi Yu, Mahesh B. Nagarajan, Pangyu Teng, Jonathan G. Goldin, Steven S. Raman, Dieter R. Enzmann, Grace Hyun J. Kim, and Matthew S. Brown. Translating Al to Clinical Practice: Overcoming Data Shift with Explainability. RadioGraphics, 43(5)::220105,, May 2023. ISSN 0271-5333. doi: 10. 2.1148/rg.22011
Corentin Dancette, Remi Cadene, Damien Tency, and Matthieu Cord. Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answering. In 2021 IEEE/CVI International Conference on Computer Vision (ICCV), pp. 1554-1563, Montreal, QC, Canada, October 2021. IEEE. ISBN 978-1-66542-812-5 doi: 4T1109/TCCV48992..2000 2021. .00160.
Raman Dutt, Linus Ericsson, Pedro Sanchez, Sotírios A. Tsaftaris, and Timothy Hospedales. Parameter- Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunity, June 2024.
Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Penglei Liu. GPTScore: Evaluate as You Desire. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 6556 6576, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 10.18653/v1 h/1/3//vv//// w10.11/222.2e// www.-laaaaa........
14

---

Published in Transactions on Machine Learning Research (07/2025)
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaclis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shorteut learning in deep neural networks. Nature Machine Intelli- gence, 2(11):665 673, November 2020. ISSN 2522-5839, doi:
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6904-69111 2017.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models, October 2021.
Yefan Huang, Xiaoli Wang, Feiyan Liu, and Guofeng Huang, OVQA: A Clinically Generated Visual Ques- tion Answering Dataset. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22, pp. 2924-2938, New York, NY, USA, July 2022. Association for Computing Machinery. ISBN 978-1-4503-8792-3. doi: 10.1145/347749553536
Kristian Norgaard Jensen and Barbara Plank. Finc-tuning From Scratch: Do Vision & Language Models Have Similar Capabilities Out-of-Distribution Visual Question Answering? In Nicoletia Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hi- toshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 1496-1508, Marseille, France, June 2022. European Language Resources Association.
Daniel P Jeong, Saurabh Garg, Zachary Chase Lipton, and Michael Oberst. Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 12140, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 00.18653/vl /ttp//11122222mm// 2022..emll-mmann...
Alistair E. W. Johnson, Tom J. Pollard, Seth Berkowitz, Nathaniel R. Greenbaum, Matthew P. Lungren, Chih-ying Deng, Roger G. Mark, and Steven Horng. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Dala, 6(1):317, December 2019. ISSN 2052-4463. doi: 10.1038/4415977-09900322
Alistair E. W. Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J. Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, Li-wei H. Lehman, Leo A. Celi, and Roger G. Mark. MIMIC-IV, a freely accessible electronic health record dataset. Scientific Data, 10(1):1, January 2023.
ISSN 2052-4463. doi: 10.1038/s41597-00 000-559772222-2----
Kushal Kafle and Christopher Kanan. Visual question answering: Datasets, algorithms, and future chal- lenges. Computer Vision and Image Understanding, 163:3-200 20, October 2017. ISSN 1077-3142. doi:
(..10166..cr122211..6...0
M. G. Kendall. The Treatment of Ties in Ranking Problems. Biometrika, 33(3):239-255 1945. ISSN 0006-3444. doi: 10.2307 /333223333
Corentin Kervadec, Grigory Antipov, Moez Baccouche, and Christian Wolf. Roses are Red, Violets are Blue... But Should VQA expect Them To? In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2775 2784, Nashville, TN, USA, June 2021. IEFE. TSBN 978-1-66544- 509-2. doi: 10. 0BB0B//CCCPRA4442222..0000
Tom Koemi and Christian Federmann. Large Language Models Are Statc-of-the-Art Evaluators of Transla- tion Quality. In Mary Nurminen, Judith Bremner, Maarit. Koponen, Sirkku Latomaa, Mikhail Mikhailov, Frederike Schierl, Tharindu Ranasinghe, Eva Vanmassenhove, Sergi Alvarez Vidal, Nora Aranberri, Mara Nunziatini, Carla Parra Escartín, Mikel Forcada, Maja Popovic, Carolina Scarton, and Helena Moniz (eds.), Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pp. 193 203, Tampere, Finland, June 2023. Furopean Association for Machine Translation.
15

---

Published in Transactions on Machine Learning Research (07/2025)
Brian Lester, Rami Al-Rfou, and Noah Constant. The Power of Scale for Crrameter-Efficient Prompt Tuning, September 2021.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang. Tristan Naumann, Hoifung Poon, and Jianfeng Gao. LLaVA-Med: Training Large Language-and-Visin Assistant for Biomedicine in One Day. In Advances in Neural Information Processing Systems, volume 36, pp. 28541- 28564, December 2023.
Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A Semantically- Ghowledge-Enhancee Dataset For Medical Visual Question Answering. In 2021 JEEE 18th International Symposium on Biomedical Imaging (ISBI), pp- 1650-1654, April 2021. 9434010.
doi: 10.1109 1111099SBB18211... 1.2021.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raf- fel. Few-shot parameter-effin efficieen fine-tuning is better and cheaper than in-context learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (ods.), Advances in Neural Information Pro- cessing Systems, volume 35, pp. 1950 1965. Curran Associates, Inc. 2022.
Haotian Liu, Chunyuan Lú, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. Tn Advances in Newral Information Processing Systems, volume 36, pp. 34892 34916, December 2023a.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-Eval: NLG Evaluation using Gpt-4 with Better Human Aligument. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2511-2522, Singapore, December 2023b. Association for Computational Linguistics. doi: 10.18653/v1/ 2023.emnlp-main.155.
John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. The Effect of Natural Distribution Shift on Question Answering Models. In Proceedings of the 37th International Conference Machine Learning, pp. 6905 6916. PMLR, November 2020.
Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. Med-Flaamino: A Multimodal Medical Few-shot Learner. In Proceedings of the 3rd Machine Learning for Health Symposium, pp. 357-367. PMLR, December 2023.
Yang Nan, Huichi Zhou, Xiaodan Xing, and Guang Yang. Beyond the Hype: A dispassionate look vision-language models in medical scenario, August 2024.
Sophie Ostmeier, Justin Xu, Zhihong Chen, Maya Varma, Louis Blankemeier, Christian Bluethgen, Arne Ed- ward Michalson, Michael Moseley, Curtis Langlotz, Akshay S. Chaudhari, and Jean-Benoit Delbrouck. GREEN; Generative Radiology Report Evaluation and Error Notation, May 2024.
Letitia Parcalabescu and Anette Frank. MM-SHAP: A Performaance-agnostic Metric for Measuring Multi- modal Contributions in Vision and Language Models & Tasks. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4032-4059, Toronto, Canada, July 2023. Association for Com- acl.//..ccll
putational Linguistics. doi:
Jielin Qiu, Yi Zhu, Xingjian Shi, Zhiqiang Tang, Ding Zhao, Bo Li, and Mu Li. Benchmarking Robustness under Distribution Shift of Multimodal Image-Text Models. In NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications, October 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Nya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference Machine Learning, pp. 8748 8763. PMT.R July 2021.
16

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Michael Roberts, Derck Driggs, Matthew Thorpe, Julian Gilbey, Michael Yeung, Stephan Ursprung, An- gelica I. Aviles-Rivero, Christian Etmann, Cathal McCague, Lucian Beer, Jonathan R. Weir-MeCall, Zhongzhao Teng, Effrossyni Serraia-KKlosar James H. F. Rudd, Evis Sala, and lieb. Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans. Nuture Machine Intelligence, 3(3):199 217, March 2021.
Schön-
ISSN 2522-5839. doi: (00088//2255-021000000
Mélanie Roschewitz, Galvin Khara, Joe Yearsley, Nisha Sharma, Jonathan J. James, Éva Ambrózay, Adam Heroux, Peter Keeskemethy, Tobias Rijken, and Ben Glocker. Automatic correction of performance drift under acquisition shift in medical image classification. Nature Communications, 14(1),6608, October 2023.
ISSN 2041-1723. doi:
Alexander Shirnin, Nikita Andreev, Sofia Potapova, and Ekaterina Artemova. Analyzing the Robustness of Vision & Language Models. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32: 2024. ISSN 2329-9304.
doi: (LL1109/TASLP.02220
Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulezyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schackermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Nenad Tomasev, Yun Liu, Renee Wong, Christopher Semturs, S. Sara Mahdavi, Joelle Barral, Dale Webster, Greg S. Corrado, Yossi Matias, Shekoofch Azizi, Alan Karthikesalingan, and Vivek Natarajan. Towards Expert-Level Medical Question Answering with Large Language Models, May 2023.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. VI--Appter: Parameter-Ffficien Transfer Learning for Vision- and-Language Tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5227 5237, 2022.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt, Mea- suring Robustness to Natural Distribution Shifts in Image Classification. In Advances in Neural Informa- tion Processing Systems, volume 33, pp. 18583-18599.
Curran Associates, Inc., 2020.
Team Gemma, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent. Sifre, Morgane Rívière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Roc Ambrose Slone, Amélic Hélion, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquetle-Choo, Clément Crepy, Daniel Cer, Daphue Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Chrisee Christian Mu- raru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean- Caan-Bapisste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel L. Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, So- ham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu-hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alck Andreev, and Kathleen Kencaly. Gemma: Open Models Based on Gemini Research and Technology, April 2024.
Zilong Wang, Xufang Luo, Xinyang Jiang, Dongsheng Li, and Lili Qiu. LLM-RadJudge: Achieving Radiologist- Evaluation for X-Ray Report Generation, April 2024.
Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Weidi Xie, and Yanfeng Wang. PMC-LLaMA: Toward building open-source language models for medicine. Journal of the American Medical Informatics Association, 31 19):18333 1843, September 2024. ISSN 1527-974X. doi: 111033/amma//caeeU0
17

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Joy T. Wu, Nkechinyere N. Agu, Ismini Lourentzon, Arjum Sharma, Joseph A. Paguio, Jasper S. Yao, Edward C. Dee, William Mitchell, Satyananda Kashyap, Andrea Giovannini, Leo A. Celi, and Mehdi Moradi. Chest ImaGenome Dataset for Clinical Reasoning, July 2021.
Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fangyuan Zou, Xingyi Cheng, Heng Ji, Zhiyuan Liu, and Maosong Sun. Revisiting out-of-distribution robustness in nlp: Benchmarks, analysis, and LLMs evaluations. Aduances in Neural Information Processing Systems, 36:58478 58507, 2023.
Hao Zhang, Wenqi Shao, Hong Liu, Yongqiang Ma, Ping Luo, Yu Qiao, and Kaipeng Zhang. AVIBench: Towards Evaluating the Robustness of Large Vision-Language Model Adversarial Visual-nstructiion. March 2024.
18

---

Published in Transactions on Machine Learning Research (07/202) 2025)
A Evaluation Details
A.1 Failures of traditional metrics
Examples of failures of traditional token-matching metrics are shown in Figure 9
image
ao ling image
image
Prediction:
imoge shows lungs
Expected Rattig:: (Human Scores)
Expected Rating:
High
Gemma Score
BLEU Score
Evact Match
Score
Precesion
Recall
Gemma Score
BLEU
Exact Match
Score
Precisonn
Recall
0
5/5
0.13
0.1311
0.22
0.3
1/l
X
X
X
X
Ground Truth:
This lung image
Ground Truth:
No
Prediction:
This heart imago
Prediction:
Yes, this normal image
Expected Rating:
Low
Expected Rating:
(Human Scores)
(Human Scores)
BLEU
Exact Match
Score
Precisoon
Gemma Score
Accuracy
1/5
0.76 :1
0.75 il
0.55 1
075 il
0.75 il
6
I/l
X
X
X
X
Figure Failures of Traditional Metries
19

---

Published in Transactions on Machine Learning Research (07 (025))
A.2 Prompts for Evaluation
Listing 1: Gemma Prompt for Evaluating Open-Ended Questions
You are a helpful evaluator to evaluate answers to questions about biomedical images.
Score the following anawer to question about an image with reapect to the ground truth answer with one to five stars.
Where the stars have the following meaning:
1. One Star: "Incorrect"
The answer does not match the ground truth and contains significant
inaccuracies.
Demonsttrates clear nisunderstanding misinterpretation of the question.
2. Two Stars: "Partially Correct"
The answer
elements that match ground truth, but there
notable discrepancies.
partial understanding question.
lacks overall accuracy addressing
3. Three Stare: "Mostly Correct" The answer aligns with the ground truth to a reasonable extent, but there are some insccuracies or gaps.
Demonstrates a moderate understanding but may lack
4. Four Stars: "Correct with Kinor Deviations"
-The answer is largely accurate and corresponds closely to the ground truth. Minor deviations or omissions are present but do not significantly impact the overall correctness.
5. Pive Stars: "Perfect Match"
- The answer exactly matches the ground truth with discrepancies. Demonstrates precise and complete understanding of the question, providing flavless response.
Here some instructions the input and output format:
The input will be passed json format with the folloving fields that important:
question about image -gtt": ground truth
"question":
question
"pred": the predicted answer to the question
The output should be in json format and look the following:
{ score: <xxx>}
where <xxx> is the number of stars you give to the ansver Do not add anything else to the anawer.
Input:
Listing Gemma Prompt for Evaluating Closed-Ended Questions
You are helpful evaluator evaluate ansvers questions about biomedical images.
question about image with respect the The questions all close-ended, the
Score the following with zero
ground truth
either as correct variations phrasing or acceptable synonyms should still meaning remains unchanged.
incorrect,
Evaluate whether
prediction (pred) accurately matches the ground truth (gt) based on meaning and relevance.
The stars have the following meaning:
0. Zero Star: "Incorrect"
The predicted ansuer incorrect.
The main entity or concept from the ground truth is not correctly
identified in the prediction
1. One Star: "Correct"
The predicted ansver is correct.
The nain entity concept from the ground truth is correctly identified in the prediction
the prediction provides the same information identifies the same entity/ concept ground truth if it includes additional, irrelevant details
Ensure that unrelated phrases
descriptions the prediction do distract the evaluation correctness.
instructions input output
20

---

Published in Transactions on Machine Learning Research (07/2025)
The input will pasaed jaon format with folloving fields that important:
"question": question about image
"gt": ground truth
question
"pred": the predicted answer to the question
The output should be in json format and look the following:
score: <xxx>}
where <xxx> is the number of stars you give to the answer. Do not add anything else to the anawer.
Input:
Listing 3: Gemma Prompt for Evaluating Closed-Ended Multilabel Questions
helpful evaluator evaluate
questions about biomedical 0, 0.5 or star.
images.
Score following answer
question
an image respect to
ground truth answer question asks for two options in the image the ansver can either be one of the options, both of the options or none. The questions are all close close-ended, so the ansuer is either correct or incorrect, but minor variations in phrasing or acceptable synonyms should still count as correct if the core neaning remains unchanged.
Evaluate whether the prediction (pred) accurately matches the ground truth (gt) based on zeaning and relevance.
The stars for rating have the following meaning:
Star: "Incorrect"
The predicted anaver is incorrect.
The main entity
concept from the ground truth is not correctly identified in the prediction.
This is the case if Option is the ground truth answer, Option B is the ground truth ground
prediction is Option "both", but prediction Option prediction is "none"
The ground truth ansuer "none",
prediction is "both"
0.5 Star: "Partially Correct"
The predicted anewer is partially correct.
The main entity or concept from the ground truth is partially correctly identified the prediction.
This is the case if Option A/B is the ground truth answer but the prediction is "both" Option A/B is the ground truth answer. but the prediction is "none"
The ground truth is "both", but the prediction is option A/B The ground truth in "none", but the prediction is option A/B Star: "Correct"
The predicted answer is correct.
The main entity concept from the ground truth correctly identified in
the prediction
The prediction provides the information identifies the entity/
concept ground truth if includes additional, irrelevant details
This
Option A is the ground truth ansver the prediction Option Option B is the ground truth anever and the prediction Option The ground truth is "both" and the prediction is "both" The ground truth is "none" and the prediction is "none"
Especially for the "none" Cases:
When the ground truth is "none":
If the prediction is "none", the score should be star.
If the prediction is "both", the score should be stars
If the prediction is Option A or B, the score should be 0.5 stars. When the prediction is "none":
If the ground truth is "none", the score should be 1 star.
If the ground truth is "both", the If the ground truth Option B, the should be 0.5
should be 0 stars.
Especially "both" Cases: When ground truth
21

---

add Published in Transactions on Machine Learning Research (07 (07/2025)
If the prediction "both",
should star should B, the score should be
prediction "none", prediction Option When prediction "both":
the ground truth is "both", the score should star. the ground truth is "none", the score should If the ground truth is Option or B, the score should be 0.5 stare.
stars.
Ensure that unrelated phrasea or extra descriptions in the prediction do not distract from the evaluation of its correctness.
Here are some instructions on the input and output format:
The input will be passed as json format with the following fields that are important:
"question"; the question about the image to the question
"gt": the ground truth
"pred": the predicted answer to the question
The output should be in json format and look the folloving:
score: <xxx>}
where
the number stars you give the
snything
Input:
22

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
B Human Rater Study Details
B.1 Comparison of different LLMs
Figure 10 compares various LLMs and traditional metrics based on their correlation with human raters. On the SLAKE and OVQA datasets, the best-performing LLMs surpass traditional metrics in aligning with human judgments. However, on the MIMIC dataset, traditional metrics slightly outperform the LLMs. Interestingly, models specialized for the medical domain (Biomistral and Biomistral DARE) or designed evaluators (Prometheus) do not demonstrate any advantage and even rank among the worst-performing models. On average, Gemma, Qwen, and Mistral v0.3 achieve the highest correlation with human judgments, with Gemma notably ranking in the top three across all datasets. Among the traditional metrics, the Fl score exhibits the best mean performance across datasets.
SLAKE
Mean
OVDA
0.21
6:8
0.23
0.33
MIMIC
Human LLM LMseaaed Traditional
Figure 10: Comparison various LLMs and Traditional Metrics in the Human Rater Study, Human interrater correlation calculated between five human raters. Shown are the Kendall correlation between the human rating and LLM metrics as well as the traditional metrics. Gold, silver, and bronze medal show the best LLMs on the respective datasets. Left side shows the individual datascts and right side the mean over all datasets.
23

---

Published in Transactions on Machine Learning Research (07 (025)
B.2 Qualitative Results of the Human Rater Study
I Figure 11 shows qualitative results for the human rater study. Thereby, F Figure 11a shows failures of the traditional metric on the OVQA dataset, where the LLM metrics clearly outperform the traditional metrics in terms of correlation to humans. contrast Figure 11b show failures of Gemma on the MIMIC datasct, where the traditional metrics show a slightly higher correlation with human judgment.
Question
What donormaliees are in the carpal?
Question
Which bones are obnormal in this image?
Ground Truth
Fracture of the fitth metocorpo pai bone of left hand.
Ground Truth
onklee
Prediction
Fracture the of the first metacorpal bone ggtt
Prediction
medial media mollekssseeerrrrrr ma leolusand
Expected Rating (Human
012111110
Expected
Scores)
44/418/1111
Precision
(a) Failures of traditional metrics on the OVQA dataset
ocint
findings
Could you paint the anatomical associated any assessmerss?
mediostinum.
Ground Truth
cardloc pacer and wires eniarged cardiac ethouette
Ground Truth
none
Prediction
cardla poier and wires
Prediction
laft lung right king
Expected Rating
Expected Ratting
(Human Scores)
(Human Scores)
Gemma Score
BLEU Score
Exact Match
F1
Precision
Recall
Gemmo Score
BLEU Score
Exact Match
Precision
Recall
0:47
0.66
0
X
(b) Failures of Gemma on the MIMTC dataset.
Figure 11: Qualitative Results of the Human Rater Study. For each sample, the question, ground truth, prediction, and expected score by human ratings are shown. On the bottom, for each automated metric, the absolute value shown with an indication if is high or low in the metrics range and an indication if the expectations from the human ratings are met. Gemma scores range from 1-5 and traditional metrics from 0-1.
B.3 Detailed Quantitative Results of the Human Rater Study
The following figures show detailed results of the human rater study. Figure 12, Figure 13 and Figure 14 show scatter plots with the correlation between the human ratings and the other metrics. Figure 15 shows detailed correlation results, including the correlation between Gemma and the other metrics.
24

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
vs
HurrAn
pertect conmation mescured comelstion
perfect cone neaoured comelation
1.0
Ozetre
09
02
SLAKE Human Gemma Correlation
(b) SLAKE Human BLEU Correlation
SLAEE Human ve Exact Match
SLAKE Human vSF1
perfect condiction
perfoct conciit en measured conclation
mecsurel conclation
1.0
10
00
0.
02
02
cc) SLAKE Human Exact Match Correlation
(d) SLAKE Human Correlation
St AKE Human vs Precicion
AXF Human vs Rerall
partect roneation
pertict ronee be mescured comelation
mescured comelation
1.0
07-
02
Human
(e) SLAKE Human Precision Correlation
(f) SLAKE Human Recall Correlation
Figure 12: Scatter plots showing the correlation between the human ratings and respective other metrics on the SLAKE dataset. Size of the dots indicates the number of ratings that correspond that point.
25

---

Published in Transactions on Machine Learning Research (07/2025)
(VVJ Germaa
perfect Human
pertect conmation meccure comelstion
neasured comelstion
1.0
Ozeire
02
(a) OVQA Human Gemma Correlation
(b) OVQA Human BLEU Correlation
UVLJ Human ys Exect Match
CVVJA Human 11
perfect correittion recisuel conclation
perfoct can on mecsured conclation
1.0
1.0
0.6
02
02
(c) OVQA Human Exact Match Correlation
(d) OVQA Human F1 Correlation
DVDA Human vs Procision
DVQA Human us Rerail
partect roneation
pertiet mone be on mescured comelaton
mescured comelation
10
02-
Human
(e) OVQA Human Precision Correlation
(f) OVQA Human Recall Correlation
Figure 13: Scatter plots showing the correlation between the human ratings and respective other metrics on the OVQA dataset. Size of the dots indicates the number of ratings that correspond that point.
26

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
Gemma
pertect conmation mescured comelstion
perfect coneee
neaoured comeestton
1.0
Dzmtre
0
02
(a) MTMIC Human Gemma Correlation
(b) MTMTC Human BLEU Correlation
MIV IC itumnn vs Exact Katch
MIMIC Human 11
perfect corriitiin mecisuel conclation
perfoct can on measured conclation
1.0
100
0
02
02
(c) MIMIC Human Exact Match Correlation
(d) MIMIC Humnan Correlation
M MIC Human vs Pracision
MINIC Humar us Recail
partect ronwation
pertiet ronne
mescured comelation
mescured comelstton
1.0
00
07-
02
(e) MIMIC Human Precision Correlation
(f) MIMIC Human Recall Correlation
Figure 14: Scatter plots showing the correlation between the human ratings and respective other metrics on the MIMIC dataset. Size of the dots indicates the number of ratings that correspond that point.
27

---

Published in Transactions on Machine Learning Research (07 (07/202) (025)
SLAKE
Ceedt Complatton
Speatmann Ctrmeee on
0.9
0.8
(4.00:40)
0228-12
16-20-011
13 1.0
12.00 DAT2
13-90-421
1240:
40c-7)
0.735-42 42
15.4% 221
0000
6666c 400
+4.3% 301
1.00 33)
15 Bic 631
0.66
HARL
OVQA
0.9
Noridal Comptation
Spearren Con
0.8
17:30e011
12
17.00-251
3.646-20
13:220-331
006-0111
G220-212
01-88-20
(4:48-41
13.20-211
0.7
0.6
12 12-000
T1 131
13 33+
11.5%
10500 T1
800 241
27:40 321
13 13
0.53 43)
85
04
MIMIC
1.00
0.95
Cends E Cormeatton
Speatman Cornelt on
0.90
12
0.458 8X)
(S446
16.20
(LAC0-JL
22026-221
244
12W0 JE11
0.536 227
0.80
0.75
0.236-201
6.00 641
17 50eWWi
0.539
11.39-011
13 13
28226-91 93)
(4.002111)
12 1:400-C2
12409:200
GrtD Math
0.65
Figure 15: Extended Results of the Human Rater Study. Human interrater correlation is calculated between five human raters. Shown are the Kendall and Spearman correlation between the humnn rating and metrics as well as the correlation between Gemma and the traditional metrics.
28

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
C Robustness Study Details
C.1 Dataset Details
C.1.1 SLAKE
The SLAKE datasct (Liu ct al. (2021)) is bilingual radiological VQA dataset, containing English and Chinese questions. We use the English subset of the SLAKE dataset. The dataset is composed of MRI, CT, and X-ray images. All images are 2D, so for the MRT and CT images, single slices are extracted. For each question, metadata information about the location, the modality, and the content provided. Overall, the images are split into different body locations, 11 different content types (question types), and the mentioned three modalities.
The exact sizes of the dataset splits are listed in Table Note that for the modality shift, merged the test set with the OoD cases from the training set, since the images distinct, and thus, the same image cannot appear in the training and test set. As this is not the case for the question type shift, we only use the OoD cases from the test set here.
Table 1: Size of the SLAKE dataset for the different splits.
Split
Li.d./OoD/all
# Cases
Whole Dataset
Train
all
4568
Validate
all
1043
Modality Shift (OoD: X-Ray)
f.1dd
Test
L.dd.
689
Test
OoD
1779
Question Type Shift (OoD: Sixe)
Train
L....
4581
Test
f..d.
994
Test
OnD
56
C.1.2 OVQA
The OVQA dataset (Hang et al. (2022) is orthopodic VQA dataset, containing CT and X-Ray images. All images 2D, so for the CT images, either 3D rendering is shown 2D image single plane. For each question, metadata information is provided about the imaged organ (like the "location" in the SLAKE dataset), and the question type (like the 'content' in SLAKE). The dataset contains 6 different question types and diffcrent body parts.
The exact sizes of the dataset splits are listed in Table 2 We removed closed-ended questions with more than two categories
choose from and closod-ended questions where the categories to answer were not exactly contained in the question. As for the SLAKE dataset, we merged the questions from the training set to the OoD test set for the organ shift, but not for the question type shift.
Table Size of the OVQA dataset for the different splits.
Split
Lidd//OD/all
# Cases
Whole Dataset
Train
all
134922
Validate
1645
Organ Shift
(OoD: Leg)
i.i.d.
Test
L..d.
1041
Test
OoD
5350
Question Type Shift (OoD: Organ System)
Train
i..dd
11924
i.idd
Trstt
OnD
237
29

---

Published in Transactions on Machine Learning Research (07 (07/202) (025)
C.1.3 MIMIC-CXR-VQA
The MIMIC-CXR- VQA dataset (Bae et al. 2023 is a chest X-ray dataset, which is built based on the MIMIC-CXR dataset (Johnson et al. (2019) the MIMIC-IV dataset (Johnson al. (2023): ), and the Chest ImaGenome dataset (Wu et al. (2021)) ). For each question, the semantic type is specified. Three different semantic types are specified, which are "choose", "query', and "verify". For 'choose", the task is to choose between two options provided in the answer, but also both or none of the oplions can be correct. For "query', the task is to list all the categories that match the questions, e.g. all anatomical findings. Lastly, "verify" are nes/no questions. All the questions can be answered based on a fixed set of classes, where the dataset overall contains 110 answer labels. The answers are given as a list of the correct classes. We preprocess the questions differently, based on their semantic type: For the "choose" questions, whenever the list of answers contains both options, we change the answer to 'both", and whenever the list of answers empty, we change the answer to 'none". For the 'query" questions, concatenate the list of answers to string, with the answer labels being comma-separann separated. For the "verify' questions, we do not apply any specific preprocessing. The information for the patient's 's gender, ethnicity, and age is taken from the MIMIC-IV dataset. Whenever the metadata information of a subject ID is not unique, we set it to 'none". In the respective shifts, we exclude questions where the corresponding metadata field is not known, which includes all fields with 'none', and for the ethnicity shift also the value 'unknown/other The exact sizes of the dataset splits are listed Table
Table 3: Size of the MIMIC datasct for the different splits.
Split
Li.d./OoD/all
#
Whole Dataset
Train
all
290031
Validate
all 73567
Gender (OnD:
Female)
Train
L..d.
147790
Test
ii.d.
7277
Test
OoD 6120
Ethmicity Shift (OnD:
Non-white)
Train
1..d.
171593
Test
L.d.
8101
Test
OnD
Age Shift (OoD: Young)
Train
Li.dd
135941
Test
L...
6686
Test.
OnD
2076
C.1.4 Ratio of unique questions in the datasets
Table 4: Ratio of unique questions in the datasets
Overall
Unique
Ratio
MIMIC
290031
132387
0.46
Train
SLAKE
4866
579
0.12
OVQA
13492
960
0.07
MIMIC
73567
31148
0.42
Val
SLAKE
OVQA
1645
266
0.16
MIMIC
13793
7565
0.55
Test
SLAKE
1050
313
0.3
OVQA
30

---

Published in Transactions on Machine Learning Research (07 (025)
C.2 Hyperparameter Search
We performed several hyperparameter sweeps for each dataset and PEFT method in order to find suitable setups for the experiments in the FT robustness study, For the hyperparameter sweeps, we trained on the whole training set for cach dataset and PEFT mothod and ran inference on the validation set. Training ran for epochs and 3 seeds for each experiment.
C.2.1 Prompt Tuning
For prompt tuning, we performed the following hyperparameter sweeps:
Number of tokens: [40, 60, 80, 100]
Learning rate: [3e - 2e 1]
The results for SLAKE can be found in Table for OVQA in Table and for MIMIC in Table
Table 5: Hyperparameter sweep for prompt tuning on the SLAKE dataset. Selected hyperparameters for the final FT robustness study are highlighted. Mean and standard deviation are reported for three seeds.
(a)
Hyperparameter sweep for the medical base model.
# Tokens
Learning Rate
Closed-Ended (Gemma Aceuracy)
Open-Ended (Gemma Score)
40
3e-2
0.79 +/- 0.02
4.21 +/- 0.03
40
3e-1
0.02
4.24
0.06
60
3e-2
0.76 -/- 0.04
4.22 +/- 0.03
+/-
+/-
80
3e-2
0.78 -/- 0.05
4.21 +/- 0.02
80
3e-1
0.82 -/- 0.01
4.23 +/- 0.02
100
3e-2
0.77 0.06
4.21 +/- 0.05
100
3e-1
0.81 +/- 0.0
4.23 +/- 0.04
(b) Hyperparameter sweep for the non-medical base model.
# Tokens
Learning Rate
Endee (Gemma Accuracy)
Epen-Ended (Gemma Score)
40
3e-2
0.76 +/- 0.04
4.15
0.02
40
3e-1
0.79 - 0.01
4.18 +/- 0.02
+/-
+/-
60
3e-1
0.79 /- 0.01
4.17 +/- 0.02
3e-2
0.76 -/-
+/-
80
3e-1
0.78 0.01
4.17 +/- 0.01
100
3e-2
0.77 /- 0.02
4.17 +/- 0.02
100
3e-1
0.79 +/- 0.02
4.17 +/- 0.06
31

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 6: Hyperparameter sweep for prompt tuning on the OVQA dataset. Selected hyperparameters for the final FT robustness study highlighted. Mean and standard deviation reported for three seeds.
(a) Hyperparameter sweep for the medical base model.
# Tokens
Learning Rate
Closed-Ended (Gemma Aceuracy)
Open-Ended (Gemma Score)
0.85 +/-
3.05 +/-
40
3e-1
0.85 0.01
3.05 +/- 0.04
60
3e-2
0.85 /- 0.01
3.06 +/- 0.03
60
3e-1
0.85 0.0
3.05 0.04
80
3e-2
0.82 -/- 0.03
3.05 +/- 0.05
-/-
+/-
100
3e-2
0.83 /- 0.02
3.1 +/- 0.03
3e-1
+/- 0.0
+/-
(b) Hyperparameter sweep for the non-medical base model.
# Tokens
Learaing Rate
Closed-Ende (Gemma Accuracy)
Opda-Ended (Gemma Score)
40
3e-2
0.8 0.03
2.97 +/- 0.02
40
3e-1
0.84 -/- 0.01
2.98 +/- 0.03
60
3e-2
0.81 0.02
2.97 0.06
60
3e-1
0.83 +/- 0.0
3.01 +/- 0.05
+/-
80
3e-1
0.84 +/- 0.0
3.0 +/- 0.05
100
3e-2
+/- 0.01
3.0 +/-
100
3e-1
0.84 0.0001 0.01
3.03 +/- 0.07
Table 7: Hyperparameter sweep for prompt tuming on the MIMIC dataset. Selected hyperparameters for the final FT robustness study are highlighted. Mean and standard deviation reported for three seeds.
(a) Hyperparameter sweep for the medical base model.
# Tokens
Learning Rate
Cloard-Ended (Gemma Acsuracy)
Open-Endhen Ender (Gemma Score)
+/-
3.21 +/-
40
3e-1
0.67 -/- 0.01
3.2 +/- 0.05
60
3e-2
0.68 -/- 0.01
3.18 +/- 0.01
60
3e.1
0.69 0.01
3.24 0.02
80
3e-2
0.66 -/- 0.05
3.21 +/- 0.02
80
-/-
3.21 +/-
100
3e-2
0.68 /- 0.02
3.23 +/- 0.03
3e-1
+/-
3.21 +/-
(b) Hyperparameter sweep for the non- non-medical base model.
# Tokens
Learning Rate
Closed-Ended (Gemma Accuracy)
Open-Ended (Gemma Score)
40
3e-2
0.69 +/- 0.0
3.16 +/- 0.03
40
3e-1
0.69 +/- 0.0
3.18 +/- 0.03
60
3e-2
0.69 // 0.0
3.22 0.04
60
3e-1
0.68 -/- 0.03
3.19 +/- 0.02
0.68 -/-
3.2 +/-
80
3e-1
0.7 /- 0.0
3.19 +/- 0.04
0.01
3.21 +/-
100
3e-1
0.68 0.01
3.17 +/- 0.02
32

---

Published in Transactions on Machine Learning Research (07 (07/2025)
C.2.2 LoRA
For LoRA, we performed the following hyperparameter sweeps:
Rank: [16, 32, 64, 128, 256]
Learning rate: [3e 5, 4]
for OVQA in Table 9 and for MIMIC in Table 10 Note that some of the hyperparameter configurations led to instabilities during training loss, indicated by 'NaN'.
is set to 2 x Rank. The results for SLAKE can be found in Table 8
Table 8: Hyperparameter sweep for LoRA on the SLAKE dataset. Selected hyperparameters for the final FT robustness study are highlighted. Mean and standard deviation are reported for three seeds. Rows with "NaN' showed instabilities in the loss during training.
(a) Hyperparameter sweep for the medical base model.
Rank
Learning Rate
Closed-Fnde Ended (Gemma Accuracy)
Open-Ended (Gemma Score)
16
3e-5
0.83 +/- 0.01
4.29 +/- 0.02
+/-
+/-
32
3e-5
0.85 +/- 0.01
4.31 +/- 0.03
3e-1
0.73 +/- 0.07
4.25 +/- 0.01
3e-5
0.84 +/- 0.01
4.33 +/- 0.04
61
3e-1
0.51 +/- 0.07
2.92 +/- 1.41
+/-
+/-
128
3e-4
NaN
NaN
+/-
+/- 0.01
256
3e-1
NaN
NaN
(b) Hyperparameter sweep for the non-medical base model.
Learning
Closed- (Gemma Accuracy)
Open-Ended (Geuma Score)
3e-5
0.84 +/- 0.01
4.25 +/- 0.02
16
3e-4
0.82 +/- 0.01
4.26 +/- 0.02
36-5
0.84 +/- 0.0
4.33 +/- 0.02
3e-4
+/-
+/-
64
3c-5
0,86 +/- 0.01
4.34 +/- 0.02
3o-4
+/- 0.09
+/- 0.78
128
3e-5
0.86 +/- 0.01
4.32 +/- 0.04
128
3e-4
NaN
NaN
256
3e-5
0.84 +/- 0.01
4.31 +/- 0.03
256
3e-4
NaN
NaN
33

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 9: Hyperparameter sweep for LoRA on the OVQA dataset. Selected hyperparameters for the final FT robustness study are highlighted. Mean and standard deviation reported for three seeds. Rows with "NaN' showed instabilities in the loss during training.
(a) Hyperparameter sweep for the medical base model.
Rank
Learning Rate
Closed Ended (Gemna Accuracy)
Open-Ended (Geruma Score)
3e-5
+/-
3.13 +/- 0.06
16
3e-4
0,83 +/- 0.02
3.18 +/- 0.02
+/-
+/-
32
3e-4
0.82 +/- 0.01
3.08 +/- 0.04
3e-5
0.85 +/-
3.2 +/- 0.02
64
3e-4
0.66 +/- 0.01
2,03 +/- 0.13
128
3e-5
0.85 +/- 0.0
3.18 +/- 0.03
NaN
NaN
256
3e-5
0.85 +/- 0.0
3.18 +/- 0.02
NaN
(b) Hyperparaneter sweep for the non-medical base model.
Rank
Learning Rate
Closed-Ended (Gemma Accuracy)
Open-Ended (Gemmn Score)
16
3e-5
0.84 + 0.0
3.08 +/- 0.03
3e-1
0.81 +/- 0.02
3.17 +/- 0.02
3e-5
0,84 +/- 0.01
3.12 +/- 0.04
3e-4
0.81 +/- 0.03
3.07 +/- 0.03
3e-5
0.84 +/-
3.15 +/- 0.01
64
3e-4
0.61 +/- 0.07
2.12 +/- 0.11
+/-
+/- 0.04
128
3e-4
NaN
NaN
256
3e-5
0.84 0/-001 001
3.2 +/- 0.05
256
3e-4
NaN
NaN
Table 10: Hyperparameter sweep for LoRA on the MIMIC dataset. Selected hyperparameters for the final FT robustness study are highlighted. Mean and standard deviation are reported for three seeds. Rows with "NaN" showed instabilities in the loss during training.
(a) Hyperparameter sweep for the medical base model.
Rank
Learning Rate
Closed-Eaded (Gemma Accuracy)
Open-Ended (Gemms Score)
3e-5
0.71 +/- 0.01
3.33 +/- 0.02
16
3e-1
0.68 +/- 0.01
3.23 +/- 0.05
+/-
+/- 0.01
32
3e-4
0.42 +/- 0.18
2.43 +/- 0.07
36-5
+/- 0.01
3.35 +/- 0.02
64
3e-4
NaN
NaN
128
36-5
0.71 +/-
3.37 +/- 0.02
3e-1
NaN
NaN
256
3e-5
NaN
NaN
NaN
NaN
(b) Hyperparameter sweep for the non medical base model.
Rank
Learning Rate
Closed-Ended (Gemma Accuracy)
Open-Ended (Gemma Score)
16
3e-5
0.7 +/- 0.01
3.28 +/- 0.03
36-4
+/- 0.01
3.25 +/-
32
3e-5
0.7 +/- 0.0
3.31 +/- 0.04
3e-4
0.44 +/- 0.07
2.43 +/- 0.05
3e-5
+/- 0.01
3.31 +/- 0.02
64
3e-4
NaN
NaN
+/-
3.36 +/- NaN
128
3e-4
NaN
256
36-5
0.7 +/-0.02 0.02
3.37 +/- 0.02
256
3e-4
NaN
NaN
34

---

Published in Transactions on Machine Learning Research (07 (07/2025)
C.3 (IA)3
For (IA)², performed the following hyperparaaeeer sweeps:
Learning rate: [3e - 3, 3e - 2, 1]
The results for SLAKE can be found in Table 11 for OVQA in Table 12 and for MIMIC in Table 13
Table 11: Hyperparameter sweep for (I the SLAKE dataset. Selected hyperparameters for the final FT robustness study are highlighted. Mean and standard dcviation are reported for three seods.
(a) Hyperparameter sweep for the medical base model.
Learning Rate
Closed- Ended (Gemma Accuracy)
Open-Ended (Gemma Score)
1r3e-3
0.63 +/- 0.02
3.72 +/- 0.01
+/- 0.01
+/-
Ir3e-1
0,65 +/- 0.01
4.24 +/- 0.04
(b) Hyperparameter sweep for the non-medical base model.
Learning Rate
Closed Blosed-Ended (Gemmaa Accuracy)
Open-Euded Ended (Gemma Score)
1r3-3
0.75 +/- 0.01
3.61 +/- 0.01
Ir3e-2
0.86 +/- 0.01
4.27 +/- 0.02
Ir3e-1
0.66 +/- 0.03
4.18 +/- 0.04
Table 12: Hyperparameter sweep for (I the OVQA dataset. Selected hyperparameters for the final FT robustness study are highlighted. Mean and standard deviation are reported for three seeds.
(a) Hyperparameter sweep for the medical base model.
Learning
Closed-Ended Ended (Gemma Accuracy)
Closed
Open-Endor (Gemma Score)
1ree-3
0.75 + 0.01
2.91 +/- 0.04
Ir3e-2
0.84 0.0
3.15 +/- 0.03
Ir3e-1
0.8 +/- 0.04
3.08 +/- 0.04
(b) Hyperparameter sweep for the non-medical base model.
Learning Rate Ended (Gemma Accuracy)
Open-Ended (Gemma Score)
1r3e-3
0.75 +/- 0.02
2.65 +/- 0.03
Ir3e-2
0.84 + /- 0.0
3.1 +/- 0.05
Ir3e-1
+/- 0.02
3.05 +/-
Table 13: Hyperparameler sweep for (IA) the MIMIC dataset. Selected hyperparameters for the final FT robustness study are highlighted. Mean and standard deviation are reported for three seeds.
Hyperparameter sweep for the medical base model.
Learning Rate
Closed-Ended (Gemma Accuracy)
Open-Ended (Gemma Score)
+/- 0.7 +/- 0.01
2.92
Ir3e-2
3.33 /- 0.02
1r3e-1
0.61 +/- 0.05
3.1 +/- 0.03
(b) Hyperparameter sweep for the non-medical hase model.
Learning Rate
Closed Mldeed-Ended (Gemma Accuracy)
Open Euded (Gemma Score)
1r3e-3
0.6 +/- 0.01
2.7 +/- 0.02
Ir3e-2
0.7 + /- 0.0
3.33 +/- 0.02
Ir3o-1
0.6 +/- 0.11
+/- 0.06
35

---

Published in Transactions on Machine Learning Research (07 (07/2025)
C.4 Robustness Results for the
Base Model
The robustness study results for the non-medical base model are presented in Figure 16 and Figure 17 corresponding to Figure 7 [andFigure for the medical model also apply to the non-medical base model, as indicated by Figure 6 which suggests similar behavior across the two models.
Figure 8 for the medical base model. The main trends analyzed in section 3.4
While the overall trends are consistent, some minor differences are observed. For example, LoRA is no longer an outlier for closed-ended questions in SLAKE. However, a similar trend appears for the non-medical model on OVQA, where LoRA shows lower RR compared to Prompt and (IA)², primarily due the question type shift. Additionally, while the question type shift continues to cause significant failures, this effect is now limited to the OVQA dataset, affecting both open-ended and closed-ended questions. Finally, while the trends for full FT are similar on the SLAKE and the OVQA dataset between the medical and the non non-medical base model, the non non-medical base model performs worse on the MIMIC dataset than its medical counterpart.
Despite these differences, the results confirm the main insights from the robustness study: None of the methods consistently outperforms others in terms of robustness, and robustness trends are more consistent across FT methods than across shifts. LoRA remains the best-performing PEFT method overall on the i.idd data.
Closed-Ended
Open-Encied
SLAKE
Dhu free
Cemera
Question Typa
Cennon
udd
CiO
Wort
No ling Basuline
10
TGl wwm.baoleee
Prss
Closed-Ended
Open-Endee
Body
OVQA
Cemmno Soore
Most Frog
Queertor
Det
Howdirm
No Etzeeine
DA7
1060
Closed-Endes
Open-Endled
Cender
lid
MIMIC
OkP
Gonno
No kmng Booel ne
Peak0ong Teallete woth
6
146 Pralerttng
Figure 16: Results of the FT Robustness Study the i.i.d. and OoD Test Set for the Non- Medical Base Model. Reported results show the mean over three seeds (exception: with the standard deviation for the non-baselines. Gemma Accuracy refers to the accuracy being rated by Gemma. For MIMI, the most frequent sanity baseline can not be calculated as too few questions match the training set.
FT, full FT)
36

---

Published in Transactions on Machine Learning Research (07 (07/202) /025)
[a]
SLAKE
OVGA
MIMIC
Ealave
Credd
Menoy
Dosthen
Checcaa
Fat Comd
BaSy Pappeer Ogen
Spe Apr
datber
Edeecy Corees
Cheeny Oiees
Creed
Dhen
home! EXD
(c)
OVGA
MIMIC
Merfor lok6-1y
Mesionp Fraees
dep (ESwAN)
Bed the copee)
conver cotbee
crosn
Ape Copnt
SLARE
OVOA
MNC
Figure 17: Results of the FT Robustness Study Focusing on the Relative Robustness (RR) for the Non-Medical Base Model. Since the study focuses on comparing the FT method and our definition of OoD holds for these methods, the no FT bascline is excluded. (a) RR on the three datasets for the mothods. Results show the mean and standard deviation over three seeds (exception: full FT). (b) RR Ranking of all FT methods. (c) Standard deviation between shifts standard deviation between FT methods.
C.5 Detailed Results of the Robustness Study
Tables 19 show the detailed results of the robustness study. Further, Figure 18 shows the inter-method and inter-shift variability of the different PEFT methods for the medical base model, so not including full FT.
Table 14: Robustness Results on the SLAKE Dataset. Results with + indicate the mean and standard deviation over three seeds. Note that the most frequent bascline can only be calculated for the i.i.d. set as for OoD too few questions match the training set. RR: Relative Robustness.
(a) Results for the medical base model.
Medility Shift
Question Type Shin
Closed- Rexted
Open Ktebed
Clased Claeed-Srete Steted
Open Kidded
Col
OcD
No Finstuning
0.5t
2.57
2.1
EX01
0.54
(1.11
235
Finetuning
6.27
0.54
3.25
0.56
0.30
6.14
4.30
6.45
435
3:
0:40
0.57
4:0
1/13
LalkA
45
0151
02
4:36-7 9/.
3.0 1/-
1.:2 02:
1103
0:7
97 0.6
0:00
02:
010
0.30 8/-
07.
0m
0:17
02-
Moet Proqurt
0.59
3.20
31
(b) Results for the non-medical base model.
OcD: X-Hay
OoD: Sin
Likel.
Owf
RR
Li-t.
OuiT
Lidl.
Oui)
RR
Oui
RR
Pulll Piretiiing
8.15
0.
60
2.10
1:21
0.75
0.:7
0.11
2.07
4.00
1:15
Prosupt Tuning
0.23
4.2
6/+
+/-
0.20
+2-
0.28
+/+
0/-0.22
6,58
0.82
0.66
3/2
0.00
0.00
0.55
4.8
1
6.56
+2-
0.22
0.57 4 0106
+2-
L.1 +/-
3.28
+/- 3/01
+/-
0.57
+/-
0,01 +/-
4.55
1/-0100
4.20 0/1-0.13
Most Prequent
3,22
0.666
37

---

Published in Transactions on Machine Learning Research (07/ 2205))
Table 15: No Image Baseline on the SLAKE Dataset. Results with + indicate the mean and standard deviation over three seeds. The model was trained with the same methods as Table 14 just without seeing the image content. RR: Relative Robustness.
(a) Results for the medical base model.
Midilliy Shift
Quitiion Type Shih
Cloosd Redeel
Openn Rediel
Open Resiol
Onl3
No Flnnttning
0:45
0.75
1.75
1.05
1
0.46
0.50
1.77
2455
Tuming
8:23
+/-
522 +/
LottA
0.54
0.47 4LU7
1.78
0.11
0.85
285 4/- 0.0
0,82 +2- 0012
+ 8.83
0.8
0133
0.79
0.23
5.35 0.2
290
0.34
2/11
1AB
6.55
m7 6:28
6.47 by w/0e
0.65 T7 0/0
6.3 m/ 6.86
1B0 4/ 0.66
0.6 47 0.0
+7 6.66
6:50 ey 0.00
0.00
+7 0.00
8.66 0.00
060 +/ 0.03
0:41
6/ 3/17
(b) Results for the non-medica base model.
Modalit slin Oui. X Ray
Quidtion Shift Oufh. Sia
L.1.t.
Out
HH
11.4
Ohil
JL
Oet
HRI
1d.d.
OoD
H0L
Promut Tusing
0.64 + B.33
0u65 4-- 0.00
Lot +-- 0.00
237 +/- R.l
2 + 0.00
0.02 + 0--
D.65 +/- 6.88
0.61 +2- 0.00
0.01 +2 0.13
5:1T +/5 0.00
2:0 e 0.0
0:33 m/ 3.01
IA0
0.67 +7 0.
0:6 47 0.00
0.00 0.1
83- 4/ 0.64
3.0 4/ 0.00
0.6 4/ 0.0
0.66 +/ 0.68
0.57 47 0.00
0.80 47 0.11
3.17 4/ 0.01
0.0 4/ 0.00
0.21
6/ 3.17
Table 16: Robustness Results on the OVQA Dataset. Results with + indicate the mean and standard deviation over three seeds. Note that the most frequent baseline can only be calculated for the i.i.d. set as for OoD too few questions match the training set. RR: Relative Robustness.
(a) Results for the medical base model.
Gal, Ieg
OuD. Orgin System
Cluseds Staded
Open-Ennded
Eloeed-Endedd
Open-Ended
Onl
Hi
11:b
Okl
HRI
5.6..
Onl
No. Finstuning
0.4
0.00
2-27
6.83
2:30
3.1
6.75
0.90
0.76
0.00
2.66
Promut Tuning
-/96-/
0,82
+2
8.18
+/
0.2
66
+/ 0.0
0:22
+/ 0.tl
0.85
+7 0.54
0.00
5.60
+/
0.80
1.0
0:07
3.1
6.66
6:31
0.0
0.58
0.15
1A3
0.54 +1-
8.23
0.75 4:- 0.01
03 4/- 002
325
4/-
0.85
248
0.20 02-0001
RL 4/- 0.20
0.22
42-4111
0.1 43-66.3
3D 4/-006
1.77
4/- 000
0.39 +/--
(b) Results for the non- non-medical base model.
Body Part Shilt Lag
Quertion Type Shift Orgin Systeme
Classd Redeel
Openn Rediel
Closed-
Open Rudiel
No Finstuning
0:43
11:00
LIM
1.5
2.45
114
1.85
1.0
Finetuming
2.5
0.3
Prompt
0.82
0.22
6.2
4-5 11.0
5.14
0,5
25 4/
6,71
4/1 6.0
4/- B.56
01. 4-221.2
0.03 4. 0.51
2 4/1
187
4.6 0.17
0.35
LettA
0:22
B/22
0.6
03 4/--.1
4/-
0.75
+/4
US +/4
0.2
033
41- 0.00
0.31
+3- 4.00
4/-
131
4/- 010
0:43
5.10-1
0.0
3.03
Mosd Propecn
6.75
3.6
6.75
5.50
Table 17: No Image Bascline on the OVQA Datasct. Results with + indicate the mcan and standard deviation over three seeds. The model was trained with the same methods as Table 16 just without seeing the image content. RR: Relative Robustness.
(a) Results for the medical base model.
Questioo Shift (hiD) Organ System
(DDD Lang
Open-
L.4.
Oet
11.4.
Ocl
JUL
1...
Oct
JUI
114.
Ocl
JUL
Promutt Tuning
6.35 +7
8:08
0.01
0.85
1/ 8.l
5:6
47
011
30003 4/ 0.00
0.29 e/
0.
047 +7 868
0:44 42 6.i
0.05
am
638
1:20
0.07
3.0.
1/
074
+7. 0.
0.60 0.0
0/0 9/ am
435
47 0.6
10 o0
0.77 87-000
07
0.44
0.30 92
0.00
0.50
0.
038
846
1:3
0/
0.00
0.37
5.03
(b) Results for the non-medical base model.
OuD:
OoD: Organ System
Closeds Rldddd
Open-Ended
Clioeeds Eaded
Open-Ended
L...l
Out
Lidb
Ouib 1.1
L.S.l.
OuD
Li..
Oub
Nu Finstusing
0.65
1:21
0.60
0130
1:30
1:15
6.73
6.28
8.0
9.34
R.04
0.47
5:30
0.66
0,08
IaRA
6.73
B:38
0/00 17
8.00
540 +7
6.14
18 +/
0.29 12
ons
675-47 17
0-4 72
0.00
0:30
//
0.34 +7 3.03
38

---

Published in Transactions on Machine Learning Research (07 (075)
Table 18: Robustness Results on the MIMIC Dataset. Results with + indicate the mean and standard deviation over three seeds. Note that the most frequent. baseline can not be calculated as too few questions match the training set, RR: Relative Robustness.
(a) Results for the medical base model.
M-42
1440
Ohi
14
Deo
(b) Results for the con-medical base model.
Classi
Ureae Model
Shoue
Sate
14:2
14.
Table 19: No Image Baselinc on the MIMIC Datasct. Results with indicate the mean and standard deviation over three seeds. The model was trained with the same methods as Table 18 just without seeing the image content. RR: Relative Robustness.
(a) Results for the medical base model.
Cordae Hinch
Chicaal Socded
Upen Rodel
Opec
Chaes Babee
Open Seard
0VD
(b) Results for the non-medical base model.
Thang
Open
Inter-Shift Variability
Inter-Method Variability
0.200
0.175
Standard Daviation
0.150
0.125
0.100
0.075
0.050
0.025
0.000
SLAKE
OVQA
MIMIC
Figure 18: Standard deviation between shifts vs. standard deviation between PEFT methods for the medical base model, not including full FT. The type of shift has higher impact on the robustness than the PEFT method.
39

---

Published in Transactions on Machine Learning Research (07/202) 2025)
Full FT
13
20
Prompt
22
16
14
LoRA
20
10
Sum
(Method
outperformed
23
17
18
by others)
4
Full
Prompt
LoRA
IA
Sum
(Method autperforming others) 1
Figure 19: Win/Los, Matrix for pairwise comparison between FT methods. The entries show cases where one FT method outperforms the other significantly, measures by Welch's t-test with p 0.05. A good method would achieve high values in its row (often outperforming others) and low values in its column (not oflen outperformed by others).
C.5.1 Statistical analysis
In this section, we confirm two of the main findings in our robustness study by detailed statistical analysis.
No FT method consistently outperforms others in terms of robustness. To prove this in addi- tion to the rank analysis in Figure 8p), run pairwise Welch's t-test between the FT methods across the different shifts and correct for multiple testing. Each test-set is bootstrapped such that we have 100 robustness measures per experiment. We then report the Win/Loss Matrix with a p value of = 0.05 (Bahri et al. (202)) to see which FT method significantly outperforms another FT method significantly on a shift. The results are shown in Figure 19 While there is slight trend indicating that Full is less likely to outperform other methods and is more frequently outperformed by them, the remaining methods show very similar performance levels. Overall, this analysis suggests that there
clear winner of method that more often outperforms others and is at the same time not often outperformed.
Robustness trends are more stable across FT methods than across distribution shifts. For this comparison, we conduct a one-way ANOVA on the robustness scores to compare the variability across FT methods versus across distribution shift lypes. The corresponding F-values and p-values are shown in Table 20 Confirming Figure 8e), we see that generally, the variance between the shifts significantly higher than between methods (F I.P 0.05). The only outlier is OVQA with a one shift being instable for Full FT, but this trend changes when excluding Full FT as also seen in Figure 18
Table 20: One way ANOVA to analyze the variance between shifts and variance between methods.
Dataset
All FT methods
w Full FT
F-value
p-value
F- F-Value
p-value
SLAKE
19.16
7.17 x 10
16.83
0.0008
OVQA
1.11
0.38
21.71
0.0003
MIMIC
5.92
0.002
5.84
0.006
40

---

Published in Transactions on Machine Learning Research (07 (07))
D Corruption Study
This section discusses the details of the corruption study showing the empirical confirmation of R1 sce section 3.3.1p. Figure 20 shows the visualization of the corruption study pipeline, illustrating how artilicia and realistic OoD test sets are generated and used to evaluate model robust We use OpenCV for image corruptions with the settings shown in Table 21
Table 21: Corruption settings for the artificial shifts. Brackets indicate the altered parameter for each corruption, [......]] indicate ranges for the corruption where randomly a value in that range chosen.
Blur (Kernel Sixe)
Gaussian Noise (Mean)
Brightness (Alpha)
Low
[0,
1.1.
Medium
7
00.09, 0.15
2.5.
High
(0. 18, 0.25
[4.5,
Table 22 shows the relative robustness results for both artificial and realistic shifts. The results show that both modality shift and question type shift exhibit lower relative robustness compared to all artificial shifts at low, medium, and high strengths. This suggests that artificial shifts, such as image corruption, fail to accurately represent the challenges posed by real-world, realistic shifts. The most prominent example here is the relative robustness of closod-ended questions under the artificial corruption shift which is up 96% compared to the realistic shift (question type) which only has 61%. The only exception where the realistic shift shows higher robustness is the question type shift on the open-ended questions.
Table 22: Robustness results for the artificial and realistic shifts on SLAKE dataset
Comupticn Stift (OoD Cocrupted 1idd Enihel
Corrapcion Slift (OaD: Comppted LLd lmages)
Open Emdhal
Eucked
M.d.
R.R.
L.d.
M.d.
Onl
Corruption
Carruption
6.94
4.06
0.01
High Corrattion
0:01
601
0.87
0.05
581
0.38
0.02
0.00
0.03
0.01
3.94
808
082
(Ohl) X-Ray)
Type Shil
Ended
Open
Open
0.01
0.06
41

---

Published in Transactions on Machine Learning Research (07/2025)
Train
Inference
gred
Compphi comppted
RR
Teet
Mode Fnctunig
Apply Roallstte Shif
CoD Teat Sal ShiHled
Ocu lest Sat Shifted
Figure 20: Overview of the Corruption Study Pipeline. The left side of the figure illustrates the training phase, where the model fine-tuned depicts how two types of OoD test sets (artificial and realistic) are generated and used to evaluate robustness. For both the artificial and realistic shifts, the same i.i.d. test set is used. The corrupted OoD test set is generated by applying described artificial corruptions to the i.i.d. test images. The realistically shifted OoD test set is constructed using the data shifts introduced in section 3.1. Robustness is then mcasured by comparing the model's performance on the i.i.d. test set to its performance on each respective OoD set.
i.i.d. training set. On the right, the inference phase
42

---

Published in Transactions on Machine Learning Research (07 (075)
E Swapped Shifts
We conduct an ablation study on the SLAKE datasct to sec how changing the i..d. and OoD set influences the robustness. Thereby, we use the same type of shifts as described in section 3.1 (Modality and Question Type), but with different i.i.d. and OoD sets. The concrete realization of the swapped shifts are shown in Table 23
Table 23: Original and Swapped Shifts on the SLAKE Dataset.
Shift Type
Original Shift
Swapped Shift
L.d.
OoD
L.tdd.
OoD
Modality Question Type
CT/MRI
X-Ray
CT/X-Ray
All questions except Size
Size
All questions except Position
Position
The results shown in Figure 21
Overall, there is swap in closed-ended and open-ended robustness: While for the original shifts, the models showed less robustness on the closed-ended questions, in the swapped shifts, they show less robustness on the open-ended questions. As also noted in section 3.4 this arises from the related question patterns between certain OoD questions and the ii.d training data. Although the OoD questions are not explicitly seen during training, similar questions in the training set can provide enough contextual information for effective gencralization. In this case, performance on the closod-endod OoD questions is higher because training examples like "Is this study of the [body region]?' implicitly support OoD questions such as "Do the organs in the image exist in the [body region]?" This observation confirms one of our main findings: the type of shift has a greater impact robustness than the choice of fine-tuning method. To achieve more reliable and generalizable performance, it is therefore more effective to focus on increasing the diversity of training data so that it better reflects the target population- rather than solely optimizing the fine-ture tunetuning strategy.
43

---

Published in Transactions on Machine Learning Research (07/202) 2025)
Closed-Ended
Open-Endee
Modkilly
Owt 36-ty
Original Shifts
lid gundonn monerapnpn Sow CsC Suo
dosaayy (Coodd
Modaliy coee
Questoo Twee cheudd
deeettrr "gpo Cayea
Prompt DAA
ktt--G-th Uaresiily
Closad- Ended
Opan- Endaadi
Swapped Shifts
Questton Type
alligund
CoC Foutinn
LOKK
LPA
Noesohyy (Deed
Meselty Cheed
Donatin Sye sheend
Grestarn toar Chees
SLAKE
Prompr LatA
Inte---th Wanshilly inwe-lerhoo Wesconiity
1A7
Figure 21: Results the SLAKE dataset with swapped shifts, meaning the i..d. and OoD splits are changed. For each shift (Original Shifts, Swapped Shifts) the top row shows the i.i.d. and OoD performance Open- and Closed-Ended questions, lower left shows the Relative Robustness, and lower right shows the Inter-Shift vs. Intor-Method variability.
44

---

Published in Transactions on Machine Learning Research (07 (07/2025)
F Multimodal Shifts
We conduct an ablation study the OVQA datasct to evaluate the impact of multimodal shift compared to the previously introduced unimodal shifts. Similar to the corruption study (Appendix DD, this ablation study is performed on the medical base model with (IA) PEFT method. This multimodal shift combines the manifestation (Body Part) and question type shifts reported in our experiments. Specifically, we define the OoD set as samples featuring body part "Leg" and question type 'Organ System', with all other samples classified i.i.d. As shown in Figure 22 the multimodal shift demonstrates the lowest robust compared to unimodal shifts, which is expected given that multimodal shifts represent extreme divergence than the unimodal ones.
OVQA
Open-Ended
Open-Ended
07
Wecathe Ribb atteeetees
Type
Body Por
Lid
Queston Type
OcD
Nullimodk
OoD
00
Quaaton
Vitimodal
Image Organ
Qann Ty:M
Errada
Figure 22: Performance results on OVQA dataset with image organ shift. question type shift, and multimodal shift which combines image organ shift and question type shift.
45


Reviews:
[{"review_id": "WR2JfuMsYB", "reviewer": "Reviewer_Euh2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- Targets real clinical pain‑points. The authors don’t settle for toy corruptions like random noise. Instead, they probe shifts clinicians actually face—different imaging devices, diverse patient populations, and varied question phrasings—so the lessons carry straight to the bedside.\n- By mixing three datasets, seven shift categories, and four fine‑tuning strategies, the study chases patterns showing that no single tweak makes a model bullet‑proof.\n-   All code, split scripts, and prompts are on GitHub, lowering the barrier for follow‑up work and making the paper a community touchstone. \n\n### Weaknesses\n-  Narrow imaging scope. Everything is radiology, so generality remains an open question\n-  Single‑grader limitation. The study leans on Gemma alone to score answers; human spot‑checks help, but Gemma’s medical know‑how still has blind spots.\n- Shallow sanity check. The “image‑blind” baseline tells us vision isn’t strictly required, yet there’s no Shapley‑style breakdown to show how much each modality (vision vs. text) really contributes.\n- Lightweight statistics. Results come without confidence intervals or multiple‑comparison corrections, so it’s hard to judge which performance gaps are truly significant.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces SURE‑VQA, a three‑part evaluation framework designed to probe how well medical vision‑language models handle real world distribution shifts rather than the synthetic corruptions commonly used today.\n1. Diverse, realistic splits (R1). The authors carve seven out‑of‑distribution scenarios—modality, body part, question type, sex, ethnicity, age, and combined shifts—across three public datasets. \n2. Semantics‑aware scoring (R2). They replace surface‑form metrics like BLEU with a Gemma‑based rubric that assigns 1‑to‑5‑star ratings and falls back to exact match when answers coincide verbatim; a five‑rater study shows Gemma’s scores correlate best with human judgment. \n3. Sanity baselines (R3). “Image‑blind” and “majority‑answer” baselines reveal how far a model can get on language priors alone, anchoring the robustness scores.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fQcPhZFbpP", "reviewer": "Reviewer_hWeX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper addresses an important problem—how to mitigate common pitfalls in medical VQA evaluation—by proposing a robustness-oriented benchmark, which has practical value.\n2. The experiments are comprehensive and large-scale. The resulting takeaways derived from the evaluations offer meaningful insights that can guide future research.\n3. The authors provide open-source code, which facilitates reproducibility and adoption of the proposed Super-VQA benchmark.\n\nWeakness:\n\n1. The evaluation relies solely on Gemma for semantic-level assessment using LLMs. Due to the limitations of Gemma itself, the reliability of the evaluation results may be affected to some extent.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper propose Super-VQA, a medical VQA benchmark designed to systematically evaluate the robustness of vision-language models on medical VQA tasks. Specifically, Super-VQA is constructed with three key design principles: (1) leveraging naturally distributed data, (2) employing large language models for semantic-level answer evaluation, and (3) establishing robustness-focused baselines. Using this benchmark, the paper conduct a comprehensive robustness analysis across seven evaluation settings on three widely used medical VQA datasets (SLAKE, OVQA, and MIMIC-CXR-VQA) and four fine-tuning strategies (Full Fine-tuning, LoRA, Prompt Tuning, and (IA)³), yielding several key takeaways.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Q1**: Have the authors considered using more powerful LLMs, such as GPT-4o or Gemini 2.5 Pro, for the semantic-level evaluation?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rfVOk1O6ZP", "reviewer": "Reviewer_SaK8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper provides thorough details on all the experiments and clearly explains the motivation behind them.\n- The SURE-VQA method tackles the problem of determining robustness of a medical VQA system which is very important before real world deployment.\n\nWeaknesses:\n- The paper does not clearly explain the difference between open and close questions.\n- The authors do not provide a reason for the consistent better performance on open oods.\n- The authors do not explain how they decide on iids for training. For example, why use CT and MRI for training SLAKE and X-Ray for testing.\n- The authors do not explicitly mention if they are testing the robustness for certain VLM architecture of if they are testing the robustness of VLMs fine-tuned on a certain dataset. In the second case the robustness also depends on the dataset and calls for creating better datasets (the most frequent baseline performing well is a problem of dataset for example). If the robustness of pretrained VLMs is measured here then that requires better training methods and pretraining data.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper mainly studies the robustness of Medical Visual Question Answering systems on realistic distribution shifts through the SURE-VQA framework. The framework introduces:\n\n- Measuring Medical VQA performance on real world shifts (through modality shifting, body part shifting, question type shifting and population shifting) instead of synthetic corruption of adding noise, blur and brightness.\n- The framework introduces a LLM based evaluation criteria for Medical VQA and shows its correlation with Human raters.\n- The framework introduces basic sanity check baselines like no image baseline and most frequent baseline to determine how effectively the Medical VQA models use the input information or if they just use shortcuts in the dataset to get the answers.\n\nThe paper used this framework to observe the robustness of different fine-tuning methods across different shifts and datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The authors should clearly define what they mean by open and close OoD.\n- The authors should a reason of why they think the models perform well on Open ended question OoD.\n- The authors say \"Another study by Parcalabescu & Frank (2023) contextualizes the multimodal use of VLMs by employing Shapley values to assess the contribution of each modality to the output.\" Can the authors explain why this is not good enough to understand the contribution of images in answering the questions?\n- In figure 3 the authors mention that they used iid with one of noise, blur or brightness corruption for medium corruption results. However for OoD shifts they use iid with one or more corruptions along with OoD samples. However, wouldn't using only OoD samples give us a better idea of the models performance on OoD samples? Also why is the corruption in iid samples increased to create OoDs? Doesn't this create similar synthetic OoDs in addition to the actual OoDs?\n- The authors should provide an ablation on at least on one dataset where they change the training iids. For example in SLAKE train on CT and X-Ray and test on MRI. Or the authors should provide some explanation on what basis they chose the combination.\n- The authors should clarify which fine-tuning method they are using for the initial experiments. (Fig 3 results)\n- The authors should explicitly mention if they are testing the robustness of pre-trained VLMs or robustness of VLMs fine-tuned on certain datasets. For example, the most frequent baseline performing well is also dataset issue which leads to the model not getting trained to use the images effectively.", "broader_impact_concerns": "The authors should mention that the framework should not be used as a measure of robustness to deploy Medical VQAs in real world situations yet.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)