MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Models

Authors: corentin  royer
Venue: MIDL_io_2024_Conference
Published: 2024-01-25 10:56:49

Abstract: We introduce MultiMedEval, an open-source toolkit for fair and reproducible evaluation of large, medical vision-language models (VLM). MultiMedEval comprehensively assesses the models’ performance on a broad array of six multi-modal tasks, conducted over 23 datasets, and spanning over 11 medical domains. The chosen tasks and performance metrics are based on their widespread adoption in the community and their diversity, ensuring a thorough evaluation of the model’s overall generalizability. We open-source a Python toolkit (https://anonymous.4open.science/r/MultiMedEval-C780) with a simple interface and setup process, enabling the evaluation of any VLM in just a few lines of code. Our goal is to simplify the intricate landscape of VLM evaluation, thus promoting fair and uniform benchmarking of future VLMs.

Main paper: 
 introduction: 
 Large language models (LLM) and vision-language Models (VLM) are text generators capable of tackling a multitude of tasks based on textual or textual-and-visual prompts, e.g. question answering, machine translation, summarization, visual-question answering, image captioning, image classification, etc. Typically, assessing the performance of these models means evaluating them over a variety of tasks (mentioned above) on diverse datasets. This enables a reliable tracking of their progress and generalizability. General-purpose language models are therefore popularly benchmarked on toolkits such as OpenAI Evals, Huggingface LLM leaderboard (Beeching et al., 2023) , and OpenVLM Leaderboard (Contributors, 2023) . Such leaderboards offer a common platform for comparing open-access (Llama 2 (Touvron et al., 2023) and Flamingo (Alayrac et al., 2022) ), and oftentimes, closed-source models (GPT-4V (Achiam et al., 2023) and Gemini (Team et al., 2023 )) based on their performance.Adapting VLMs to the medical domain proves challenging, primarily due to the domainspecific hurdles posed by proprietary datasets, fine-grained knowledge requirements, and the overall difficulty to generalize across medical domains and tasks. Despite these challenges, recent efforts culminated in truly capable medical VLMs. For instance, LLaVA-Med (Li et al., 2023) , and PMC-VQA (Zhang et al., 2023b) build VLM assistants for medical VQA, while MAIRA-1 (Hyland et al., 2023) focuses on radiology report generation, specifically chest x-rays (CXR). RadFM (Wu et al., 2023) proposes a versatile VLM, with a focus on radiology. Circumscribing the capabilities of the above-mentioned VLMs, MedPaLM M (Tu et al., 2023) and BiomedGPT (Zhang et al., 2023a) are pitched as generalist models capable of performing a wider array of tasks such as image classification, text summarization, etc.Among these plethora of models, evaluation has been highly non-uniform. For instance, RadFM is evaluated on seven tasks while MedPaLM M is evaluated on five. Even among the tasks that both of them have been evaluated on, discrepancies exist in terms of either the datasets (for report generation MedPALM is evaluated on MIMI-CXR while RadFM is evaluated on three more datasets) or the metrics (BLEU and Recall in RadFM; BLEU and F1 in MedPALM) . Similarly, LLAVA-Med reports six metrics, while RadFM reports 61 metrics; however, they share no common metrics, hindering a direct comparison of both approaches. Thus, there is a need for a unified benchmark, the lack of which has been consistently acknowledged (Wu et al., 2023; Li et al., 2023) . Our work aims to build such a unified benchmark. Closely related to our work are the works by (Wu et al., 2023) and (Tu et al., 2023) . However, the benchmark of (Wu et al., 2023) is specific to radiologyrelated tasks and doesn't include important domains such as general medicine. While the evaluation performed by (Tu et al., 2023) is holistic, it is closed-source, preventing a fair replication.Summarizing the topic of the evaluation of medical VLMs, we identify three key issues that make evaluation difficult and eventually slow down medical VLM research. First, as stated above, models are benchmarked using different metrics, methodologies, and tasks preventing fair comparison to other models. Second, the scope of generalist models is so wide that every new model is required to benchmark against all the prior work. However, re-implementing the metrics and recomputing benchmarks is time-consuming when models are open-access and impossible for closed-source models. Third, even if one plans to reimplement the benchmark, the evaluation pipelines (from data to metrics) are complex and cumbersome. Addressing these issues, we introduce MultiMedEval, an open-source, Python-based evaluation toolkit for medical VLMs. Our toolkit is designed to be user-friendly, offering reproducible evaluation capabilities across six distinct medical tasks implemented on 23 datasets spanning over 11 medical image and text modalities. Our benchmark encompasses a broader spectrum of medical domains than any model mentioned above. Additionally, we reimplemented two open-source models (RadFM and LLAVA-Med) and compared them with closed models according to the metrics reported therein. Our comprehensive comparison serves as a foundational baseline for future medical VLM research, with future evaluations seamlessly unified into this benchmark through MultiMedEval, resulting in its continued growth. 
tasks and evaluations: 
 The typical VLM is designed to take an interleaved image-text prompt as input and generate a textual response as output. For example, the prompt <image> What is the modality of this image? might generate a response This image shows an MRI of the brain. Any evaluation primarily focuses on the model's ability to respond to a variety of question prompts, which in turn defines the variety of tasks the model can perform. For a comprehensive evaluation of the VLM's capabilities, we evaluate it on six tasks: image classification, question answering (QA), visual QA, report summarization, report generation, and natural language inference (NLI). In every task, we detail the prompt design, datasets, and performance metrics that MultiMedEval employs during evaluation. We let the user decide if they want to do few-shot inference (i.e. prepend the prompt with examples of prompts and responses) or zero-shot inference. We use the official split for every dataset except MIMIC-III and Pad-UFES-20 which do not have an official split. We propose such a split for Pad-UFES 20 where we use 20% of the dataset for testing and we use the split proposed by (Delbrouck et al., 2022) for MIMIC-III. Fig. 1 gives an overview of the dataset, task, and modality distribution in MultiMedEval. For every task, example prompts for all the datasets are listed in the appendix.Multi-class and multi-label image classification. We use a total of 15 datasets spanning nine modalities (c.f. Table 1 ). MIMIC-CXR is a multi-label classification task while the others are multi-class classification. The input prompt for image classification is constructed by the image followed by the classes and then a question, e.g. for OrganMNIST the full prompt will be: <img> Options:1:bladder 2:femur-left ... 11: spleen Which options correspond to the image?.For all datasets, except MIMIC-CXR, the model's predicted answer is determined by calculating the BLEU score between the model's response and each class, selecting the class with the highest score. For the MIMIC-CXR dataset, we use CheXBert (Smit et al., 2020) labeler on the response. CheXBert is a report-labeling tool that extracts 14 conditions of which we keep five conditions (Atelectasis, Cardiomegaly, Consolidation, Edema, and Pleural Effusion) for computing the metrics. Once the classes are extracted, we report the classification performance using macro F1, macro AUROC, and macro accuracy.Table 1 : List of the image classification datasets and the different modalities they cover.Dataset name Classes SizeOrganSMNIST (Xu et al., 2019; Bilic et al., 2023) 11 8827 OrganCMNIST (Xu et al., 2019; Bilic et al., 2023) 11 8216Dermatology Pad-UFES 20 (Pacheco et al., 2020) 7 2298 DermaMNIST (Tschandl et al., 2018; Codella et al., Table 2 : List of the QA datasets and the different modalities they cover.MedQA (Lau et al., 2018) 1273 MedMCQA (He et al., 2020) 4183 PubMedQA (Liu et al., 2021) 500Table 3 : List of the VQA datasets and the different modalities they cover.Radiology VQA-Rad (Lau et al., 2018) 451 SLAKE (Liu et al., 2021) 1061 Pathology Path-VQA (He et al., 2020 ) 6719Question answering. For QA, we evaluate on three datasets (c.f. Table 2 ): MedQA and MedMCQA consist of multi-choice questions (MCQ) and PubMedQA consists of closeended questions (yes-no questions). For the MCQs, the model is prompted with the question followed by all the options and finally the phrase What is the correct answer?. For Pub-MedQA, the question is prepended with Answer the question with yes, no, or maybe.To determine the model's predicted answer from its response, we utilize the BLEU metric (Papineni et al., 2002) to compare the predicted answer to each option, selecting the one with the highest BLEU score. For PubMedQA, we check if the answer contains the words yes, no, or maybe. Once the model's answers have been generated, we report answering performance using accuracy.Visual question answering. For evaluating the performance of a VLM on VQA, we use three datasets (c.f. Table 3 ) containing a mix of open-ended and close-ended questions: Path-VQA, SLAKE, and VQA-Rad. The prompt for VQA is constructed by concatenating the image and the question, e.g. <img> What is the main organ in the image?. Similar to QA, the close-ended questions are prepended with Answer the question with yes or no.Since there are no MCQs in VQA, the evaluation differs from QA. Specifically, the correct answer and the predicted one are tokenized and the resulting sets are used to compute precision and recall. We also differentiate between close-ended and open-ended questions to report close-ended accuracy, open-ended accuracy, and open-ended recall (Nguyen et al., 2019) . Additionally, we also report overall recall and F1 score. Finally, we compute the BLEU score from the non-tokenized texts (Wu et al., 2023; Li et al., 2023) . To calculate the accuracy, close-ended questions are considered correct if their recall is at least 0.5, while open-ended questions require a recall of at least 0.75.Report generation. We include the MIMIC-CXR dataset (Johnson et al., 2019) containing de-identified radiology reports with the associated CXRs to evaluate report-generation capabilities. The task is to generate the findings section of the report based on the radi-ology images. The input prompt for the model is constructed with the image (or multiple images pertaining to one case) followed by a sentence asking for the report, as in, <img> <img> Please caption this scan with findings and impressions.Following common practices, the generated reports are evaluated using n-gram-based methods: ROUGE-L, BLEU-1, BLEU-4, and METEOR (Banerjee and Lavie, 2005) . Additionally, we compute F1-RadGraph, CheXBert vector similarity, F1-bertscore, and RadCliQ to capture the subtleties of radiological language. F1-RadGraph is the F1 score between the entities extracted from the reference report and generated one using RadGraph (Jain et al., 2021) . F1-BertScore employs CheXBert to label the reference and generated reports, as introduced in the image classification task above. CheXBert vector similarity (Yu et al., 2023) computes the cosine similarity between the embedded reference and generated reports. Lastly, RadCliQ (Yu et al., 2023 ) is a composite metric composed of the previous four metrics, said to closely match the practitioners' feedback on report quality.Report summarization. We evaluate report summarization on MIMIC-III (Johnson et al., 2016) . Following (Van Veen et al., 2023) , the VLM has to generate the impressions section of a radiology report based on the findings section. So, the model is prompted with the free-text findings (e.g. Intracranial vessels are normal... there is mild venẗ riculomegal... the subarachnoid hemorrhage noted in the right sylvian fissure) followed by the task prompt, Summarize the findings. We use the same metric as for the report generation task to compare the ground truth impressions to the generated summary.Natural language inference. Gauging the logical reasoning capabilities of the VLM, Natural language inference (NLI) involves the categorization of pairs of sentences into three classes: contradiction, entailment, or neutral, effectively resembling a 3-class classification task. We evaluate NLI using MedNLI (Romanov and Shivade, 2018) , a dataset consisting of pairs of medical statements. We prompt the model with the two sentences from a pair followed by a question asking to classify the logical relationship between them, as in <sentence-1> <sentence-2> Determine the logical relationship between these two sentences.To extract the VLM's predicted answer from its response, we check the presence of either of the three terms (contradiction, entailment, or neutral) . For the answer to be valid, only one of the three classes must be present; if none of the three or several of them are present, the answer is deemed invalid. The performance of the model is then reported using accuracy. 
multimedeval setup and utilization: 
 Ideally, once a VLM is developed, the entire suite of evaluations mentioned above needs to be conducted. Typically, this involves downloading the datasets, implementing the data pre-processing, implementing the computation of the metrics, running inference through the VLM, and finally recording the performance. We design MultiMedEval to abstract this entire pipeline, only exposing APIs for setting up the data and for evaluating on them. In this section, we briefly describe the usage of MultiMedEval and strongly encourage the reader to peruse the official documentation (currently hosted at https://github.com/ corentin-ryr/MultiMedEval).Parameters. MultiMedEval exposes two parameter classes, SetupParams and EvalParams . The setup parameters control the data download. For the datasets hosted on PhysioNet, the setup process also requires appropriate credentials. The evaluation parameters control the evaluation configuration such as batch_size, device (GPU-id), etc.Batcher. The only code that the user needs to implement is a Callable, batcher, which wraps around the user's VLM inference module. Every call to batcher takes, as input, a batch of conversation prompts and returns, as output, the decoded model responses. The input prompts are constructed according to HuggingFace's conversation style.Evaluation. Once the parameters and the batcher are ready, the evaluation can be run using the eval API, which takes as input the list of datasets that the user wants to benchmark on, along with the batcher and the evaluation parameters. The results of the evaluation are saved as a JSON file. Below, we provide the pseudocode for one such evaluation:  
baselines: 
 As previously indicated, MultiMedEval's purpose is to enable a comprehensive assessment of any VLM. To demonstrate this, we benchmark two recent, publicly-available models, RadFM (Wu et al., 2023) and LLaVA-Med (Li et al., 2023) . In this benchmark, we also include the performance reported by two closed models, MedPALM M (Tu et al., 2023) and Maira-1 (Hyland et al., 2023) , as well as one very recent public model, BiomedGPT (Zhang et al., 2023a) . In Tables 4 and 5 , we report the complete picture of the model performances across six tasks, 23 datasets, and 81 metrics. The performance is grouped by tasks and color-coded in green. The brighter the color, the superior the performance (i.e. highest accuracy or lowest RadCliQ score is the brightest). A gray cell indicates that the performance was never reported.Owing to a holistic picture provided by MultiMedEval, we make five crucial observations: First, there is not a single task or metric that every medical VLM, to date, has been evaluated on. This speaks to the non-uniformity in the existing evaluation regimes. Second, closed models such as MAIRA-1 (on report generation) and MedPaLM M (on QA, image classification, etc.) show superior performance compared to open-source models. Third, we can see an improvement in VLMs' performance on QA and image classification. In both cases, MedPaLM M outperforms the others by a significant margin. Fourth, the most recent open-source model (BiomedGPT) seems to be very promising. On metrics reported by both MedPaLM M and BiomedGPT, the latter shows a competitive performance. At image classification (Macro-F1 on CBIS-DDSM), BiomedGPT even outperforms MedPaLM M, showing an encouraging prospect for open-source models. Fifth and finally, the number of empty cells clearly showcases the need for a standardized evaluation protocol for medical VLMs as well as an easy-to-use toolkit that performs the evaluation so that researchers can cater more focus on the development of the models. 
conclusion: 
 Medical vision-language models are just gathering momentum, they already show interesting generalizable capabilities, and their capabilities are bound to expand. Therefore, this is an opportune moment to establish a standard evaluation protocol based on community consensus. Addressing this, we presented MultiMedEval, a Python toolkit to comprehensively assess the performance of any VLM model on multiple medical tasks. Using this, we benchmarked RadFM and LLaVVa-Med and compared their results to the reported performances of state-of-the-art medical VLMs.Future work. MultiMedEval will be released to the community and will be actively maintained by adding new tasks, metrics, and datasets. To this end, we will work with open-source medical imaging libraries such as MONAI (MONAI Consortium, 2023) and MLCommons (MLCommons Consortium) to increase community adoption.  
 
 Appendix: intracranial vessels are all normal in appearance including the carotid arẗ eries and circle of without any aneurysm identified. again noted is the right parafalcine subdural hematoma extending over the right tentorium withö ut significant change from recent prior exam. no gross reaccumulation of the left sided subdural hematoma is appreciated, though evaluation for subtle hemorrhage is difficult as this was a contrast-enhanced study. there has been interval placement of a right sided ventriculostomy tube terminating in the posterior aspect of the right lateral ventricle. again noted is some high attenuation material in the left lateral ventricle, third ventricle and fourth ventricle which likely represents clot adherent to choroid plexus. there is mild ventriculomegaly which is not significantly changed from the prior exam. intraparenchymal high attenuation surrounding the ventriculostomy tube in the right frontal lobe likely represents a small amount of intraparenchymal hemö rrhage. the subarachnoid hemorrhage noted in the right sylvian fissure on the prior exam is difficult to appreciate due to contrast-enhancement, but is likely not significantly changed. limited views through the cervical spine demonstrate multilevel spondylosis with mild anterolisthesis of c3 upon c4, likely on the basis of facet degenerative change as no spondylolysis is idenẗ ified. there is no significant central canal stenosis. incidentally noted is enlargement of the right lobe of the thyroid gland which contains a foc us of high attenuation possibly representing calcification. Summarize the findings.<img> <img> <img> Please caption this scan with findings and impression. Table A6 : Prompt example for each of the Natural Language Inference datasets.Sentence 1: Labs were notable for Cr 1.7 (baseline 0.5 per old records) and lactate 2.4.\n Sentence 2: Patient has elevated Cr\n Determine the logical relationship between these two sentences.Does the second sentence logically follow from the first (entailment), conẗ radict the first (contradiction), or if there is no clear logical relationship between them (neutral)? 


Decision: The authors have addressed the majority concerns of reviewers after rebuttal. The reviewers all confirm the merits of this paper (1 Borderline accept, 1 Weak accept and 1 Strong accept after rebuttal). Given the consensus of reviewers, a decision of accept is recommended.

Reviews: review: paper_summary: paper_summary: The paper introduces MultiMedEval, a Python-based evaluation toolkit designed for medical VLMs. MultiMedEval provides reproducible evaluation functions for six different medical tasks, applied to 23 datasets covering 11 medical image and text modalities.
main_review: main_review: Considering that the majority of my concerns have been adequately addressed and recognizing the positive potential of the proposed framework, I am inclined to adjust my final rating to "Borderline accept."
strength_weakness: strength_weakness: The primary concern raised by the reviewer pertains to the accuracy of implementation, as depicted in Tables 4 and 5.

Whether the Toolkit is user-friendly and easy to use is another concern from reviewers.
questions: questions: The concerns from the above comments should be carefully addressed.
paper_summary: paper_summary: The paper introduces an open-source benchmark toolkit, namely MultiMedEval, for fair and comprehensive evaluation of medical vision-language models (VLM). The benchmark consists of six multi-modal tasks with 23 datasets, spanning 11 medical domains. Two open-source models (RadFM and LLAVA-Med) are evaluated using MultiMedEval and compared with their reported performances, as well as the reported performances of three closed models (MedPaLM M, MAIRA-1, and BiomedGPT). MultiMedEval establishes a standard evaluation protocol for the community.
main_review: main_review: Thank you to the authors for addressing my doubts in the rebuttal. 
The authors claim they have thoroughly verified the correctness of their implementation via four aspects. They also corrected the implementation of RadFM and LLaVA-Med and updated the evaluation results.
All of my concerns have been addressed.
strength_weakness: strength_weakness: 1. The primary concern regarding the proposed open-source toolkit is its reliability. Tables 4 and 5 reveal significant disparities between the performance scores of RadFM and LLAVA-Med evaluated by MultiMedEval and their reported scores. For instance, in the MIMIC-CXR Report Generation task, MultiMedEval yields considerably lower BLEU-1 and ROUGE-L scores (0.006 and 0.065, respectively) compared to the reported scores of 0.194 and 0.262. Similarly, in image classification tasks, a majority of the Macro-AUC and Accuracy scores are either equal to or less than 0.5, casting doubt on the credibility and reliability of MultiMedEval.

2. The paper's explanation of the evaluation method for visual question answering lacks clarity. It mentions that "the correct answer and the predicted one are tokenized, and the resulting sets are used to compute precision and recall." However, it is essential to clarify how the answers are tokenized and which model is employed for this tokenization process. Additionally, it is necessary to elaborate on what constitutes the resulting sets used for computing precision and recall. Providing details on the tokenization process, including any specific algorithms or techniques utilized, would enhance understanding and reproducibility of the evaluation methodology.
questions: questions: In the rebuttal, the authors must elucidate the reasons behind the poor performance scores assessed by MultiMedEval. Additionally, further evidence is required to bolster the reliability of the proposed toolkit. Furthermore, a detailed clarification of the evaluation methodology for Visual Question Answering (VQA) is essential.
paper_summary: paper_summary: This paper introduces MultiMedEval, an open-source toolkit for fair and reproducible evaluation of large, medical vision-language models (VLM). MultiMedEval comprehensively assesses the models’ performance on a broad array of six multi-modal tasks, conducted over 23
datasets, and spanning over 11 medical domains. This is a meaningful endeavor that requires sustained long-term efforts for maintenance and success.
main_review: main_review: Thanks to the author's positive improvements, I believe several issues that were concerning to me have been addressed. 
They added Diff-VQA to MultiMedEval.
They are concerning fairness.
As a result, I've decided to adjust my rating accordingly.
strength_weakness: strength_weakness: 1. diff-VQA[1] is missing.
[1] Expert Knowledge-Aware Image Difference Graph Representation Learning for Difference-Aware Medical Visual Question Answering
2. Introducing additional fairness evaluation metrics is important. The authors have not addressed the fairness of large models and its metric assessment.
questions: questions: Taking into account the aforementioned weaknesses and detailed comments,

Comments: comment: Dear Reviewers,

Thank you for your valuable feedback on our submission. As you saw, we addressed the concerns you raised and made significant improvements to our work. We are open to further discussion and collaboration to ensure the robustness and effectiveness of MultiMedEval as a benchmarking tool.

Please feel free to engage with us during the discussion period and we will do our best to provide further details.
We appreciate your review and we are glad you recognize the scope and work that went into MultiMedEval. We tried to address your concerns.


## Concern 1: Addition of Diff-VQA


Thank you for pointing us towards Diff-VQA. We have now added Diff-VQA to MultiMedEval [Diff-VQA pull request](https://github.com/corentin-ryr/MultiMedEval/pull/4). The full evaluation is currently underway and the results will be updated on GitHub.


Your comment also gave us an idea for a new feature: to allow the user to add any new dataset. Just format the dataset as a JSON file and treat it as one of the task “families” of MultiMedEval (VQA, QA, Report Comparison, Image classification). The tool will then apply the same metrics as the other tasks in the "family" (e.g. accuracy for QA, BLEU, accuracy, recall for VQA). More details can be found on the GitHub repository.


## Concern 2: Stratified Analysis and Fairness




This is indeed good feedback, the option to perform a more detailed evaluation per task. Especially concerning fairness, we are contemplating analysis of existing datasets (Pad-UFES 20) as well as using additional datasets (MIMIC and Genderbias_CheXnet; https://arxiv.org/pdf/2310.09507.pdf). Given that this is a longer exercise, we track the progress on this issue: [GitHub issue](https://github.com/corentin-ryr/MultiMedEval/issues/6). Any other ideas in this regard are more than welcome as feature requests on the repository.


## Concern 3: Long-term maintenance


We fully agree with you that such a tool is as good as it is maintained. In the near term, the tool will be actively maintained, as evidenced by the commits performed during this MIDL review process. Simultaneously, we plan to develop features that will eventually render large-scale development unnecessary (e.g. possibility of using any new dataset). Simultaneously, once the tool is stable, we plan to reach out to open-source libraries such as MONAI and MLCommons and transition the maintenance to the community.
Thank you for your review and for appreciating the scope and goals of our work. We would like to address the three concerns you expressed and we kindly ask you to look at the revised version of our manuscript.


## Concern 1: Accuracy of the evaluation


The accuracy and reliability of the benchmark are paramount. To ensure this, we provide assurances for the 4 potential sources of error: (A) data download, (B) prompt construction, (C) raw answer processing, and (D) metric computation.


For (A), (C), and (D) we implemented unit tests and a continuous integration process in the GitHub repository. Specifically, we have:
* Correctness of the data download ([Unit test file](https://github.com/corentin-ryr/MultiMedEval/blob/main/tests/test_loading_all.py))
* Correctness of the tokenizer for VQA tasks ([Unit test file](https://github.com/corentin-ryr/MultiMedEval/blob/main/tests/test_vqa_preprocessing.py))
* Accuracy of compound metrics such as RadCliQ ([Unit test file](https://github.com/corentin-ryr/MultiMedEval/blob/main/tests/test_report_comparison.py)). Simpler metrics like accuracy and F1 score are taken from libraries like sklearn and pytorch.

As for the source of error (B), we base our prompt construction on previous work whenever possible and construct a suitable prompt for every other case. We provide a new table (Table A7) in the appendix of the revised manuscript to clarify this process. Our goal is to propose a standard prompt for each task to minimize the disparities due to prompt sensitivity (a problem acknowledged by previous works [RadFM issue](https://github.com/chaoyi-wu/RadFM/issues/10))


## Concern 2: Result discrepancies

The low numbers in Table 4 (image classification and NLI) are simply explained by the absence of these modalities in the training data of RadFM and LLaVA-Med. For example, the image classification tasks cover a large range of different modalities (dermatology, pathology...) whereas RadFM was only trained on radiology tasks leading to wrong or even invalid answers and low performance in these tasks.

We want to support MultiMedEval in the same open-source spirit as similar to tools like [SentEval](https://github.com/facebookresearch/SentEval/issues). Thanks to your feedback, we realized that there was a discrepancy in the report generation task. This was due to the prompting setup that we used compared to the one in the other papers. Specifically, the implementation of this benchmark in RadFM and MedPaLM M included the indication section of the report in the prompt and we only included the images. We addressed this issue and reevaluated RadFM and LLaVA-Med. You can find the updated results below and in the revision of our manuscript.


|Dataset|Metric|RadFM|RaFM(reported)|LLaVA-Med|LLaVA-Med(reported)|
|---------|---------------|-----|---------------|---------|---------|
|MIMIC-III|BLEU-1|0.127|\-|0.129|\-|
||BLEU-4|0.016|\-|0.030|\-|
||ROUGE-1|0.221|\-|0.222|\-|
||ROUGE-L|0.157|\-|0.164|\-|
||F1-RadGraph|0.149|\-|0.211|\-|
||RadCliQ|1.396|\-|1.315|\-|
||CheXbertvector|0.602|\-|0.596|\-|
||METEOR|0.191|\-|0.314|\-|
|MIMIC-CXRReportGeneration|BLEU-1|0.127|0.194|0.109|\-|
||BLEU-4|0.004|\-|0.000|\-|
||ROUGE-1|0.208|0.262|0.149|\-|
||ROUGE-L|0.128|\-|0.103|\-|
||F1-RadGraph|0.108|\-|0.021|\-|
||RadCliQ|1.994|\-|2.415|\-|
||CheXbertvector|0.214|\-|0.156|\-|
||METEOR|0.178|\-|0.123|\-|


The difference between our numbers and the reported numbers is also explained by the stochastic generation method (sampling from the token distribution) which we adopted from the original RadFM and LLaVA-Med implementation.

We would like to stress that successful reproduction in MultiMedEval indicates that the metric computation is accurate. However, RadFM has not yet made its evaluation code publicly available making it difficult to pinpoint the exact differences. Once released, we will examine the code and incorporate any necessary improvements, thereby emphasizing the importance of community consensus.


## Concern 3: VQA tokenization

In the interest of conciseness, we did not go in-depth into the details of the tokenization and in light of your feedback on the performance discrepancy, we decided to align our tokenization procedure with the one used by LLaVA-Med. The updated procedure consists of first cleaning the generated string by removing punctuation, articles ("a", "an", "the"), turning all numbers into numerals ("one" -> "1"), and expanding contractions that lack apostrophes. We then split the sentence at spaces and end up with a list of words that we turn into a set (removing duplicates). After these changes, we reproduced LLaVa-Med's performance, as shown above in the table (also included in the revised manuscript), and we added the tokenization details in the appendix section of the manuscript.
Thank you for appreciating the need for this benchmark and especially for identifying the two areas where we could improve tremendously. Below we address your three major concerns and are happy to hear back from you.

### Concern 1: Accuracy of the evaluation

We aim to address the broader concern regarding the overall accuracy of our implementation. MultiMedEval encompasses several tasks, each with potential for errors: data download, prompt construction, VLM response decoding, and metric computation. Specifically, the following errors are possible: (A) incorrect data download, (B) inaccurate model prompting, (C) incorrect decoding of the VLM's response, and (D) incorrect metric computation.

To ensure the validity of the evaluation and address errors A, C, and D, we implemented a suite of unit tests and a continuous integration pipeline. These tests verify the following:

* Correctness of the data download ([Unit test file](https://github.com/corentin-ryr/MultiMedEval/blob/main/tests/test_loading_all.py))
* Correctness of the tokenizer for VQA ([Unit test file](https://github.com/corentin-ryr/MultiMedEval/blob/main/tests/test_vqa_preprocessing.py))
* Accuracy of compound metrics such as RadCliQ ([Unit test file](https://github.com/corentin-ryr/MultiMedEval/blob/main/tests/test_report_comparison.py)). Simpler metrics like accuracy and F1 score are taken from libraries like sklearn and pytorch.

Regarding error B (input prompt construction), MultiMedEval uses prompts from prior work whenever possible. We provide this information in a new table (Table A7) in the supplementary material of the revised manuscript. In cases where the prompt is modified, it is due to disagreements in the community. MultiMedEval proposes using an adapted version as the future standard to mitigate the well-known concern surrounding VLMs' sensitivity to input prompts [RadFM issue](https://github.com/chaoyi-wu/RadFM/issues/10).

### Concern 2: Re-implemented models consistently exhibit lower performance than the reported values.

Language generation is probabilistic, and additional stochasticity is introduced by previous work. For example, during inference, models typically set sampling=True for token prediction. Non-standard prompting, as mentioned earlier, also introduces randomness. RadFM, for example, randomly selects a prompt for report generation ([RadFM report generation prompts](https://github.com/chaoyi-wu/RadFM/blob/main/src/Dataset/dataset/report_prompt.json)). We have attempted to minimize this stochasticity, which contributes to discrepancies.

In line with open-source principles, MultiMedEval will continue to improve based on user feedback, similar to tools like SentEval (https://github.com/facebookresearch/SentEval/issues). Your feedback helped us identify a discrepancy in text normalization between LLaVa-Med and MultiMedEval, with the latter using a more generic routine. After updating the routine, we reproduced LLaVa-Med's performance, as shown below (also included in the revised manuscript in Tables 4 and 5).

|Dataset|Metric|RadFM|RaFM(reported)|LLaVA-Med|LLaVA-Med(reported)|
|-|-|--|-|-|--|
|VQA-Rad|BLEU-1|0.443|0.522|0.058|\-|
||closedQaccuracy|0.581|\-|0.621|0.614|
||openQrecall|0.326|\-|0.335|0.282|
||recall|0.468|0.428|0.485|\-|
||openQaccuracy|0.260|\-|0.255|\-|
||F1|0.454|\-|0.102|\-|
|SLAKE|BLEU-1|0.707|0.786|0.051|\-|
||closedQaccuracy|0.715|\-|0.515|0.522|
||openQrecall|0.720|\-|0.408|0.392|
||recall|0.718|0.744|0.444|\-|
||openQaccuracy|0.684|\-|0.361|\-|
||F1|0.718|\-|0.090|\-|
|VQA-Path|BLEU-1|0.254|\-|0.029|\-|
||closedQaccuracy|0.500|\-|0.511|0.541|
||openQrecall|0.019|\-|0.079|0.123|
||recall|0.259|\-|0.290|\-|
||openQaccuracy|0.008|\-|0.037|\-|
||F1|0.256|\-|0.052|\-|

It is important to note that successful reproduction here indicates that the metric computation is accurate. However, RadFM has not yet made its evaluation code publicly available. Once released, we will examine the differences and incorporate any necessary improvements, thereby emphasizing the importance of community consensus.

### Other concerns
*Poor performance on MedNLI*: The metric used for MedNLI has been thoroughly unit-tested. The suboptimal performance can be primarily attributed to the current limitations of VLM's capability in language inference. With the introduction of MultiMedEval, we encourage future VLMs to also assess their performance on LLM tasks.

*Ease of Use*: We respectfully differ from the reviewer's concern regarding the ease of use. The most significant challenges in evaluating VLMs lie in consolidating the datasets and implementing the performance metrics. This is handled seamlessly in the background, simplifying VLM evaluation into a few lines of code. Suggestions on improving the ease of use are very welcome. 

*Data Pre-processing*: Data pre-processing is more of a model-specific requirement than a dataset-specific one. MultiMedEval only handles dataset-side processing, such as file system operations.
Thank you for taking the time to review our paper and for providing valuable feedback. Based on your insights we made some adjustments to our work. Here's a summary of the changes we made based on your suggestions:
## Accuracy of Evaluation: 
We have thoroughly addressed concerns regarding potential errors in our implementation, including data download, prompt construction, VLM response decoding, and metric computation. To ensure accuracy, we implemented a suite of unit tests and a continuous integration pipeline, and we added more details on our tokenization process. We also gave more details on our prompt construction approach and provided a new table in the supplementary material to clarify this process.

## Result Discrepancies: 
We clarified the origin of the discrepancies between our results and the reported ones, particularly regarding image classification and report generation. By aligning our tokenization procedure with previous works, we have achieved more consistent and reliable performance metrics. Additionally, we await the release of RadFM's evaluation code to further ensure alignment and foster community consensus.

## Incorporation of Diff-VQA and Flexibility: 
We have integrated Diff-VQA into MultiMedEval, acknowledging the importance of accommodating diverse datasets. Additionally, we have introduced a new feature allowing users to easily add any new dataset to MultiMedEval, streamlining the benchmarking process and enhancing its flexibility.

## Fairness:
In response to concerns raised about fairness in evaluation, we are actively exploring options for more detailed, stratified analysis per task. This includes considerations for evaluating datasets such as Pad UFES 20 and potentially incorporating additional datasets like MIMIC and Genderbias_CheXnet. We recognize the importance of fairness in benchmarking, especially in the context of medical applications where biases can have significant implications.


We would greatly appreciate any further feedback you may have, particularly on the revised manuscript. Your insights are valued in our efforts to improve MultiMedEval and ensure its effectiveness as a benchmarking tool.