Exploring Compositional Generalization of Multimodal LLMs for Medical Imaging
Authors: Zhenyang Cai, Junying Chen, rongshengwang, Weihong Wang, Yonglin Deng, Dingjie Song, Yize Chen, Zixu Zhang, Benyou Wang
Accepted at: acl25

Exploring Compositional Generalization of Multimodal LLMs for Medical Imaging

Zhenyang Cai†, Junying Chen†, Rongsheng Wang†, Weihong Wang, Yonglin Deng, Dingjie Song, Yize Chen, Zixu Zhang, Benyou Wang*
The Chinese University of Hong Kong, Shenzhen
wangbenyou@cuhk.edu.cn

Abstract

Medical imaging provides essential visual insights for diagnosis, and multimodal large language models (MLLMs) are increasingly utilized for its analysis due to their strong generalization capabilities; however, the underlying factors driving this generalization remain unclear. Current research suggests that multi-task training outperforms single-task as different tasks can benefit each other, but they often overlook the internal relationships within these tasks. To analyze this phenomenon, we attempted to employ compositional generalization (CG), which refers to the models’ ability to understand novel combinations by recombining learned elements, as a guiding framework. Since medical images can be precisely defined by Modality, Anatomical area, and Task, naturally providing an environment for exploring CG, we assembled 106 medical datasets to create Med-MAT for comprehensive experiments. The experiments confirmed that MLLMs can use CG to understand unseen medical images and identified CG as one of the main drivers of the generalization observed in multi-task training. Additionally, further studies demonstrated that CG effectively supports datasets with limited data and confirmed that MLLMs can achieve CG across classification and detection tasks, underscoring its broader generalization potential. Med-MAT is available at https://github.com/FreedomIntelligence/Med-MAT.

1 Introduction

Medical imaging provides essential visual insights into the structures of the human body, making it a critical tool for medical diagnosis. Recently, multimodal large language models (MLLMs) (Liu et al., 2023; Li et al., 2024; Chen et al., 2024b) have been employed to analyze these images due to their strong interpretability and generalization capabilities. In this paper, we focus on the latter: generalization of MLLMs in medical imaging. Current research (Mo and Liang, 2024; Ren et al., 2024) has demonstrated that models trained on multiple tasks outperform those trained on a single task as they can leverage potential knowledge from other tasks. Yet, the underlying factors that contribute to this generalization remain insufficiently explored.

To this end, we take the perspective of composition generalization (CG) (Li et al., 2019; Xu et al., 2022; Tang et al., 2024) to investigate the generalization phenomenon of mutual improvement in MLLMs’ understanding of medical images. Specifically, CG is the model’s ability to learn fundamental elements and recombine them in novel ways to understand unseen combinations (e.g., learning Cat from White Cat and Black from Black Dog, then generalizing to Black Cat, as shown in Figure 1).

In this paper, we categorize each image to three elements: Modality, Anatomical area, and medical Task, presenting numerous natural opportunities for CG. We defined these three elements as the MAT-Triplet and collected 106 medical datasets, subsequently merging those that share the same MAT-Triplet to create the Med-MAT dataset.

![Examples of Compositional Generalization: The model is required to understand unseen images by recombining the fundamental elements it has learned.](page_1012_370_388_246.png)

Figure 1: Examples of Compositional Generalization: The model is required to understand unseen images by recombining the fundamental elements it has learned.
Ultimately, Med-MAT comprises 53 subsets, encompassing 11 modalities, 14 anatomical regions, and 13 medical tasks, providing a foundation for investigating CG and other generalization methods.

To verify the existence of CG, we designated specific datasets as Target data and selected all Related data from Med-MAT that shared the same MAT-Triplet with the Target data. Using these data combinations, we accessed the generalization performance of MLLMs and observed that they could leverage CG to understand unseen medical images. To further validate this finding, we repeated the experiments on different MLLMs and obtained consistent results, confirming the universality of CG.

Building on these insights, we expanded the number of combinations and observed the changes in model generalization performance after deliberately disrupting CG, ultimately revealing that CG is a key factor driving the generalization of MLLMs. Furthermore, we explored the potential applications of CG and its performance across classification and detection tasks, finding that CG enhances MLLMs’ ability to handle medical scenarios with limited training data and improves their capacity for spatial awareness.

Here are the key contributions of our work: 1) A VQA dataset, Med-MAT, has been constructed, providing a platform to explore the generalization of MLLMs on medical images. 2) Through this dataset, we observed that MLLMs in different architectures can utilize compositional generalization to understand unseen images and demonstrated that this is one of the main forms of generalization for medical MLLMs. 3) Finally, the real-world applicability of CG, along with its presence across detection and classification tasks, has been further explored, highlighting its potential to enhance data-efficient training and its broad applicability.

2 A Pilot Study on Generalization

2.1 Data Collection (Med-MAT)

Most existing datasets for MLLMs (Zhang et al., 2023c; Li et al., 2024; Chen et al., 2024b), primarily VQA datasets, provide broad coverage but lack attribute annotations for individual samples, which are not suitable for CG exploration. To address this gap, we curated a large collection of image-text pairs to develop Med-MAT, ensuring that each sample is explicitly defined by MAT-Triplet.

Data Construction  Med-MAT contains a total of 106 image-label pair medical datasets, sourced from various medical public challenges or high-quality annotated datasets. All datasets are categorized according to their MAT-Triplet, with data having identical elements grouped into a single subset (Figure 2). Labels are manually clustered to ensure that annotations with the same meaning are not repeatedly used. In total, Med-MAT covers 11 medical modalities \( \mathbb{M} \), 14 anatomical areas \( \mathbb{A} \), and 13 medical tasks \( \mathbb{T} \), hoping that it can spread across various medical tasks like a mat. (Data lists are shown in Appendix B)

Data Distribution  All subsets are divided into training and test sets following their original distributions or using a 9:1 ratio. To ensure a fair comparison, each training set is limited to 3,000 samples \( ^1 \), with label balance maintained as much as possible. Any subset that cannot meet this requirement is treated as an OOD (out-of-distribution) dataset. For the test sets, we strictly balance the number of samples per label to ensure that the accuracy metric reliably reflects model performance.

QA Pairs Construction  To enable MLLMs to directly train and test on Med-MAT, all image-label

\( ^1 \) Most datasets contain around 3,000 samples.
<table>
  <tr>
    <th>Subset No.</th>
    <td>02 03 07 08 09 11 13 14 15 16 18 19 21 22 23 25 26 28 30 31 32 33 35 36 37</td>
  </tr>
  <tr>
    <th>Baseline</th>
    <td>21 47 40 25 26 27 28 24 22 24 25 23 49 26 25 24 49 30 49 21 49 20 25 23 19</td>
  </tr>
  <tr>
    <th>Single-task Training</th>
    <td>24 49 50 68 65 76 83 53 61 32 29 57 53 28 24 57 64 89 60 97 54 29 51 49</td>
  </tr>
  <tr>
    <th>Multi-task Training</th>
    <td>96 89 80 80 79 97 92 88 76 57 88 74 87 86 93 52 98 72 94 61 100 72 75 60 50</td>
  </tr>
</table>

Table 1: Accuracy(%) of different models on In-Distribution datasets (each dataset contains over 3,000 samples, with 3,000 selected for training). Within each segment, bold highlights the best scores, and underline indicates the second-best. Baseline represents the results without any training, Single-task Training refers to the results after training on a single dataset, and Multi-task Training represents the results after training on all datasets.

<table>
  <tr>
    <th>Subset No.</th>
    <td>01 04 05 06 10 12 17 20 24 27 29 34</td>
  </tr>
  <tr>
    <th>Baseline</th>
    <td>32 25 33 33 48 27 33 13 34 37 31 20</td>
  </tr>
  <tr>
    <th>Multi-task Training</th>
    <td>39 26 70 31 58 38 61 40 35 41 55 50</td>
  </tr>
</table>

Table 2: Accuracy(%) of different models on Out-Of-Distribution Dataset (each dataset contains fewer than 3,000 samples and is used only for testing). Bold highlights the best scores. Multi-task Training represents the results after training on all datasets.

paired data were converted into a visual question-answering (VQA) format (Figure 3). Specifically, each subset was manually assigned 6 instructions to guide the MLLM in answering the subset task. For convenience, all samples were converted into single-choice questions with up to four options, and the remaining distractor options were randomly drawn from other labels within the subset. To mitigate potential evaluation biases arising from varying option counts, the ImageWikiQA dataset (Zhang et al., 2024b), a non-medical dataset consisting of single-answer, four-option questions, was incorporated during the training.

![An example of formatting a raw classification sample into a Question-answering sample in Med-MAT.](page_101_1042_646_340.png)

Figure 3: An example of formatting a raw classification sample into a Question-answering sample in Med-MAT.

2.2 Observation

Experiment Setup We chose LLaVA-v1.5-7B-Vicuna (Liu et al., 2023) as the base model due to its transparent pretraining process and minimal use of medical data, reducing the risk of knowledge leakage. Leveraging MLLM’s flexibility, we enabled task switching and generalization by adjusting prompts, streamlining generalization studies. Each experiment ran for 5 epochs on 8 A800 GPUs with a batch size of 32 and a learning rate of 5e-6.

Analysis To access the generalization of MLLMs, we trained the baseline on all ID datasets to simulate Multi-task Training and separately trained on individual ID datasets to establish the Single-task Training as the control group. We then evaluated the models on all datasets. The results in Table 1 and 2 confirm that Multi-task Training outperformed Single-task Training on specific tasks and improved OOD prediction, suggesting certain data combinations enhance classification and identifying valuable combinations for medical tasks warrants further research. This observation leads to a research question (RQ):

What drives the generalization observed in MLLMs during Multi-task Training?

To address it, we aim to explore the generalization mechanism of MLLMs from the perspective of compositional generalization (CG).

3 Proof of Concept on CG

This section will prove the existence of CG in MLLMs, offering preliminary insights to address the RQ and providing support for further analysis.

3.1 Experiment Setup

To explore the existence of CG from a finer perspective, this section focuses on CG with only two
<table>
  <tr>
    <th rowspan="2">Related Combination</th>
    <th colspan="2">Target Subset</th>
    <th colspan="3">Baseline Baseline+ Trained | CG Helps</th>
  </tr>
  <tr>
    <th>Brain</th>
    <th>Cancer</th>
    <th>Lung</th>
    <th>Cancer</th>
    <th>Lung</th>
    <th>Cancer</th>
  </tr>
  <tr><td>Lung</td><td>COVID</td><td>Brain</td><td>Cancer</td><td>Lung</td><td>Cancer</td><td>25</td><td>25</td><td>27</td><td>✓</td></tr>
  <tr><td>Lung</td><td>Cancer</td><td>Brain</td><td>State</td><td>Lung</td><td>State</td><td>47</td><td>46</td><td>50</td><td>✓</td></tr>
  <tr><td>Brain</td><td>Cancer</td><td>Lung</td><td>State</td><td>Brain</td><td>State</td><td>33</td><td>50</td><td>57</td><td>✓</td></tr>
  <tr><td>Bones</td><td>Level</td><td>Lung</td><td>State</td><td>Bones</td><td>State</td><td>49</td><td>53</td><td>51</td><td>✓</td></tr>
  <tr><td>Bones</td><td>Level</td><td>Brain</td><td>State</td><td>Bones</td><td>State</td><td>49</td><td>53</td><td>72</td><td>✓</td></tr>
  <tr><td>Bones</td><td>Level</td><td>Breast</td><td>Diseases</td><td>Bones</td><td>Diseases</td><td>37</td><td>33</td><td>39</td><td>✓</td></tr>
  <tr><td>Bones</td><td>Level</td><td>Lung</td><td>Diseases</td><td>Bones</td><td>Diseases</td><td>37</td><td>33</td><td>43</td><td>✓</td></tr>
  <tr><td>Bones</td><td>Level</td><td>Chest</td><td>Diseases</td><td>Bones</td><td>Diseases</td><td>37</td><td>31</td><td>43</td><td>✓</td></tr>
  <tr><td>Bones</td><td>Level</td><td>Breast</td><td>Diseases</td><td>Bones</td><td>Diseases</td><td>37</td><td>37</td><td>43</td><td>✓</td></tr>
  <tr><td>Bones</td><td>Level</td><td>Lung</td><td>Diseases</td><td>Bones</td><td>Diseases</td><td>37</td><td>37</td><td>43</td><td>✓</td></tr>
  <tr><td>Bones</td><td>Level</td><td>Chest</td><td>Diseases</td><td>Bones</td><td>Diseases</td><td>37</td><td>37</td><td>41</td><td>✓</td></tr>
  <tr><td>Lung</td><td>COVID</td><td>Breast</td><td>Lung</td><td>Lung</td><td>Diseases</td><td>49</td><td>48</td><td>51</td><td>✓</td></tr>
  <tr><td>Lung</td><td>COVID</td><td>Bones</td><td>Lung</td><td>Lung</td><td>Diseases</td><td>49</td><td>48</td><td>52</td><td>✓</td></tr>
  <tr><td>Lung</td><td>COVID</td><td>Chest</td><td>Lung</td><td>Lung</td><td>Diseases</td><td>49</td><td>48</td><td>51</td><td>✓</td></tr>
  <tr><td>CT</td><td>Cancer</td><td>X-ray</td><td>COVID</td><td>CT</td><td>COVID</td><td>47</td><td>46</td><td>72</td><td>✓</td></tr>
  <tr><td>X-ray</td><td>Diseases</td><td>CT</td><td>COVID</td><td>X-ray</td><td>COVID</td><td>30</td><td>21</td><td>49</td><td>✓</td></tr>
  <tr><td>X-ray</td><td>Diseases</td><td>CT</td><td>State</td><td>X-ray</td><td>State</td><td>30</td><td>21</td><td>46</td><td>✓</td></tr>
  <tr><td>CT</td><td>State</td><td>X-ray</td><td>Cancer</td><td>CT</td><td>Cancer</td><td>33</td><td>28</td><td>28</td><td>✓</td></tr>
  <tr><td>X-ray</td><td>Bones</td><td>CT</td><td>Brain</td><td>X-ray</td><td>Brain</td><td>49</td><td>49</td><td>91</td><td>✓</td></tr>
  <tr><td>X-ray</td><td>Bones</td><td>CT</td><td>Brain</td><td>X-ray</td><td>Brain</td><td>49</td><td>50</td><td>81</td><td>✓</td></tr>
  <tr><td>X-ray</td><td>Bones</td><td>CT</td><td>Brain</td><td>X-ray</td><td>Brain</td><td>25</td><td>51</td><td>74</td><td>✓</td></tr>
  <tr><td>X-ray</td><td>Bones</td><td>CT</td><td>Brain</td><td>X-ray</td><td>Brain</td><td>49</td><td>52</td><td>52</td><td>✓</td></tr>
  <tr><td>CT</td><td>Lung</td><td>X-ray</td><td>Brain</td><td>CT</td><td>Brain</td><td>33</td><td>50</td><td>60</td><td>✓</td></tr>
  <tr><td>CT</td><td>Lung</td><td>X-ray</td><td>Lung</td><td>CT</td><td>Lung</td><td>25</td><td>25</td><td>36</td><td>✓</td></tr>
  <tr><td>CT</td><td>Lung</td><td>X-ray</td><td>Lung</td><td>CT</td><td>Lung</td><td>47</td><td>50</td><td>81</td><td>✓</td></tr>
  <tr><td>CT</td><td>Lung</td><td>X-ray</td><td>Lung</td><td>CT</td><td>Lung</td><td>47</td><td>50</td><td>71</td><td>✓</td></tr>
  <tr><td>X-ray</td><td>Bones</td><td>CT</td><td>Lung</td><td>X-ray</td><td>Lung</td><td>30</td><td>32</td><td>28</td><td>✓</td></tr>
  <tr><td>X-ray</td><td>Bones</td><td>CT</td><td>Lung</td><td>X-ray</td><td>Lung</td><td>30</td><td>32</td><td>35</td><td>✓</td></tr>
  <tr><td>X-ray</td><td>Bones</td><td>CT</td><td>Lung</td><td>X-ray</td><td>Lung</td><td>30</td><td>32</td><td>41</td><td>✓</td></tr>
  <tr><td>X-ray</td><td>Bones</td><td>CT</td><td>Lung</td><td>X-ray</td><td>Lung</td><td>30</td><td>32</td><td>42</td><td>✓</td></tr>
  <tr><td>Der - Skin</td><td>Cancer</td><td>FP - Fundus</td><td>Diseases</td><td>Der - Skin</td><td>Diseases</td><td>25</td><td>29</td><td>33</td><td>✓</td></tr>
  <tr><td>Der - Skin</td><td>Cancer</td><td>OCT - Retina</td><td>Diseases</td><td>Der - Skin</td><td>Diseases</td><td>25</td><td>29</td><td>33</td><td>✓</td></tr>
  <tr><td>Der - Skin</td><td>Diseases</td><td>DP - Mouth</td><td>Cancer</td><td>Der - Skin</td><td>Cancer</td><td>40</td><td>33</td><td>63</td><td>✓</td></tr>
  <tr><td>Der - Skin</td><td>Diseases</td><td>Mic - Cell</td><td>Cancer</td><td>Der - Skin</td><td>Cancer</td><td>40</td><td>33</td><td>63</td><td>✓</td></tr>
  <tr><td>DP - Mouth</td><td>State</td><td>Der - Skin</td><td>Cancer</td><td>DP - Mouth</td><td>Cancer</td><td>48</td><td>50</td><td>52</td><td>✓</td></tr>
  <tr><td>DP - Mouth</td><td>State</td><td>Der - Skin</td><td>Cancer</td><td>DP - Mouth</td><td>Cancer</td><td>48</td><td>50</td><td>55</td><td>✓</td></tr>
  <tr><td>FP - Fundus</td><td>Diseases</td><td>Mic - Cell</td><td>Level</td><td>FP - Fundus</td><td>Level</td><td>33</td><td>36</td><td>42</td><td>✓</td></tr>
  <tr><td>Mic - Cell</td><td>Recognition</td><td>FP - Fundus</td><td>Level</td><td>Mic - Cell</td><td>Level</td><td>23</td><td>33</td><td>32</td><td>✓</td></tr>
  <tr><td>Mic - Cell</td><td>Recognition</td><td>Der - Skin</td><td>Cancer</td><td>Mic - Cell</td><td>Cancer</td><td>49</td><td>50</td><td>50</td><td>✓</td></tr>
  <tr><td>Mic - Cell</td><td>Recognition</td><td>DP - Mouth</td><td>Cancer</td><td>Mic - Cell</td><td>Cancer</td><td>49</td><td>51</td><td>62</td><td>✓</td></tr>
  <tr><td>Mic - Cell</td><td>Level</td><td>Der - Skin</td><td>Cancer</td><td>Mic - Cell</td><td>Cancer</td><td>49</td><td>51</td><td>52</td><td>✓</td></tr>
  <tr><td>Mic - Cell</td><td>Level</td><td>DP - Mouth</td><td>Cancer</td><td>Mic - Cell</td><td>Cancer</td><td>49</td><td>51</td><td>58</td><td>✓</td></tr>
  <tr><td>Mic - Cell</td><td>Cancer</td><td>FP - Fundus</td><td>Level</td><td>Mic - Cell</td><td>Level</td><td>23</td><td>24</td><td>27</td><td>✓</td></tr>
</table>

Table 3: Generalization results on classification datasets: Related Combination is the training set, Target Subset is the goal. Baseline, Baseline+, and Trained represent the model’s accuracy(%) without training, trained on randomly sampled unrelated data, and trained on related data, respectively. ✓ in CG Helps indicates successful generalization, while ✗ denotes failure. The 4 segmented areas represent different Direction Types: fixed modality ⧫, fixed area ⧫, fixed task ⧫, and modality-area paired combinations ⧫. Although some combinations share the same name, they differ because they fix different elements.

MAT-Triplet elements varying while the third remains constant. Additionally, we identified specific Modality-Area pairs ⧫, such as dermoscopy paired consistently with skin, which were treated as a special category. These 4 different fixed formats were classified into distinct Direction Types.

We adhered to the training setup described in Section 2.2 and evaluated the model’s performance on the Target data. Baseline refers to the model without any training, while Trained refers to the model trained solely on Related data. To ensure that our conclusions are not influenced by the amount of training data, we randomly sampled an equal number of data from the Unrelated subsets, and this configuration is referred to as Baseline+.

3.2 Results

Results are shown in Table 3 and it can be observed that almost all CG combinations are able to generalize to downstream tasks, highlighting that MLLMs can leverage CG to generalize Target data across all Direction Types. Besides that, since this experiment focused solely on two-element tuples, we further investigated three-element tuples in Appendix A.4, where we also observed similarly strong generalizations when obtaining MAT-Triplet elements from three different datasets.

Take-away 1: MLLMs can leverage CG to understand unseen medical images.

In the Baseline+ setting, we removed all datasets sharing any MAT-Triplet element with the Target
<table>
  <tr>
    <th>Related Combination</th>
    <th></th>
    <th>Target Subset</th>
    <th></th>
    <th>Qwen</th>
    <th>Llama</th>
  </tr>
  <tr>
    <td>Bones</td>
    <td>State</td>
    <td>Breast</td>
    <td>Diseases</td>
    <td>Bones</td>
    <td>Diseases</td>
    <td>+4</td>
    <td>+7</td>
  </tr>
  <tr>
    <td>Lung</td>
    <td>COVID</td>
    <td>Bones</td>
    <td>Diseases</td>
    <td>Lung</td>
    <td>Diseases</td>
    <td>+11</td>
    <td>+11</td>
  </tr>
  <tr>
    <td>X-ray</td>
    <td>Diseases</td>
    <td>CT</td>
    <td>COVID</td>
    <td>X-ray</td>
    <td>COVID</td>
    <td>+5</td>
    <td>+5</td>
  </tr>
  <tr>
    <td>X-ray</td>
    <td>Diseases</td>
    <td>CT</td>
    <td>State</td>
    <td>X-ray</td>
    <td>State</td>
    <td>+8</td>
    <td>+8</td>
  </tr>
  <tr>
    <td>CT</td>
    <td>Brain</td>
    <td>X-ray</td>
    <td>Lung</td>
    <td>CT</td>
    <td>Lung</td>
    <td>+1</td>
    <td>-2</td>
  </tr>
  <tr>
    <td>CT</td>
    <td>Brain</td>
    <td>X-ray</td>
    <td>Lung</td>
    <td>CT</td>
    <td>Lung</td>
    <td>+7</td>
    <td>+8</td>
  </tr>
  <tr>
    <td>FP - Fundus</td>
    <td>Diseases</td>
    <td>Mic - Cell</td>
    <td>Level</td>
    <td>FP - Fundus</td>
    <td>Level</td>
    <td>-3</td>
    <td>+6</td>
  </tr>
  <tr>
    <td>Mic - Cell</td>
    <td>Recognition</td>
    <td>FP - Fundus</td>
    <td>Level</td>
    <td>Mic - Cell</td>
    <td>Level</td>
    <td>+7</td>
    <td>+22</td>
  </tr>
</table>

Table 4: Result of Qwen2-VL and Llama-3.2-Vision on selected classification datasets in Med-MAT. Qwen and Llama represent the accuracy(%) gains they achieved on the respective backbones through CG.

data. Consequently, Baseline+ models perform at near-random levels on the test set, indicating they failed to acquire target-relevant knowledge. This suggests that only datasets related through the MAT-Triplet can help the model learn and generalize to new target tasks.

Take-away 2: Generalization arises in medical datasets in which at least partial MAT elements pre-exist during training.

3.3 Extending CG to other Backbones

LLaVA was selected as the baseline because its training data and processes are publicly available, ensuring minimal exposure to medical images and preventing bias in the integration of medical image knowledge into the MLLM. To ensure that the results are not affected by the training data or the visual encoder of LLaVA, we randomly sampled two combinations from each Direction Type to investigate CG on Qwen2-VL-7B (Wang et al., 2024a) and Llama3.2-11B-Vision (Meta AI, 2024).

Qwen2-VL undergoes additional training on proprietary data based on ViT and incorporates a strategy to adjust the number of vision tokens according to resolution. Llama3.2-Vision, on the other hand, pretrains its own vision encoder from scratch using proprietary data. Thus, both models serve as a means to assess whether MLLMs with different training data and vision encoders can still leverage CG to understand unseen images, ensuring that CG is not merely an artifact of LLaVA’s data fitting or specific to its vision encoder.

Table 4 presents the experimental results, showing that both selected backbones exhibit a certain degree of generalization across most tasks. This suggests that despite differences in pre-train data and vision encoders, different MLLMs can still leverage CG to understand unseen images.

Take-away 3: CG persists across different MLLM backbones.

4 Scaling Combination in CG

After confirming that CG is indeed a form of generalization in MLLMs, we expanded the number of participating combinations to explore the generalizability of CG and examine its relationship with the generalization exhibited by Multi-task Training to address the RQ.

4.1 Experiment Setup

Two sub-questions have been defined to verify the applicability of CG in multiple data combinations and examine its role in Multi-task Training.

• (Q1) While previous experiment on CG indicated that Unrelated combinations provide no benefit to Target data, can generalization arise when training incorporates more Unrelated combinations, simulating a multi-task scenario?

• (Q2) Previous studies suggest that Multi-task Training generally promotes better generalization than single-task training. If the CG conditions in Multi-task Training are deliberately disrupted, will the resulting generalization effect be affected?

Selection Strategy To ensure a balanced evaluation of Related and Unrelated combinations, Subset 03 and Subset 28 were chosen as Target datasets because they exhibit the most balanced ratios of Related to Unrelated subsets (13:11 for Subset 03 and 11:13 for Subset 28), making them ideal for providing a diverse range of compositions in the scale-up experiments.

The baseline was trained on all subsets excluding the Target data to evaluate the claim that mixing multi-task data enhances generalization (All Data). To construct multiple comparative experiments, models were further trained on either Related or Unrelated subsets (All Related / All Unrelated ) to address Q1. For
Figure 4: Accuracy(%) results on the Target dataset for various models. All Related/Unrelated models are trained on all the related or unrelated datasets of the Target Data. w/o Modality/Area/Task are trained on All Related datasets but omit those sharing the same element as the Target Data, to intentionally disrupt CG. All Data uses all available training sets. (Note: The Target Data is excluded from training to observe generalization.)

Q2, individual MAT-Triplet elements were systematically removed from the Related subsets (Related w/o Modality / Area / Task ), disrupting CG and assessing the ability to maintain generalization. To ensure consistency, the total data volume in all experiments was limited to 15,000 samples, aligning with the number of ID subsets available after excluding related tasks from Subset 03.

4.2 Analysis of Scaling Experiment

Figure 4 illustrates the results. It can be observed that even when we expanded the Unrelated combination volumes and increased task diversity, the performance of All Unrelated remains close to the Baseline , indicating that these datasets can not support MLLMs to understand the Target data.

Take-away 4: Datasets without MAT-Triplet overlap offer limited benefit for generalization even in the multi-task training scenario (Q1).

Besides, w/o Modality / Area / Task showed significant accuracy drops compared to All Related , despite holding the training data volume constant. This indicates that if the CG combinations are forcibly disrupted, MLLMs will lose a significant amount of generalization capability for the target data.

Take-away 5: Disrupting CG leads to a significant decline in generalization ability. (Q2).

Notably, All Related achieves a performance level comparable to All Data , where all datasets are included in training. This suggests that CG plays a crucial role in enhancing the generalization effect of Multi-task Training. Therefore, in conclusion:

Take-away 6: CG plays an important role in generalization for MLLMs in medical imaging.

5 Potential Applications of CG

As MLLMs can use CG to generalize unseen medical images, this section attempts to explore its potential applications in training medical MLLMs.

5.1 Generalization without Target Data

In medical tasks, new and unpredictable conditions, like COVID-19, can emerge at any time. Exploring how to use CG to help MLLMs enhance their ability to identify unknown diseases in the absence of specific datasets is both important and meaningful.

We selected some Target datasets and trained the MLLMs using Related and Unrelated data to observe their generalization to the Target data. The generalization trend was assessed by progressively increasing the size of the combination datasets.

Selection Strategy To highlight the generalization trends, the combinations with strong generalization results were selected from the main experiments. For fairness, we chose the combinations across four types where Trained results exceed both Baseline and Baseline+ by at least 10. If multiple combinations meet the criteria, a random seed of 42 was used to determine the selection.

Analysis The experimental results are shown in Figure 5, where the red line represents the accuracy curve for Related combinations, and the purple line shows the gain from Unrelated combinations. The Related combinations group significantly outperformed the Unrelated combinations in terms of generalization across all tasks, with this ability continuing to improve as the data size increased. This suggests that Related combinations, leveraging CG, enhance the model’s ability to understand unknown medical tasks.

Take-away 7: CG might enable MLLMs to handle tasks without dedicated training data.
Figure 5: The accuracy curve reflects the impact of gradually increasing the composition dataset size without using Target data in training. The green and red lines represent training with Related and Unrelated Data, respectively.

Figure 6: The accuracy curve shows the impact of increasing the composition dataset volume while incorporating Target data in training. The green and red lines represent training with Related and Unrelated Data, respectively.

5.2 Generalization with Limited Target Data
This section investigates the benefit of CG for tasks with limited data, e.g. processing medical images in rare conditions.

Selection Strategy To assess generalization in limited data scenarios, we select combinations with poor generalization from Table 3. Specifically, for each Direction Type, we randomly choose a CG combination with weak generalization (i.e., rows marked with \( X \) in the last column of Table 3). For these combinations, we introduce an additional 2,000 examples from the Target data.

Analysis Figure 6 shows the results. It can be seen that as we gradually expand the training volume of Target data, adding the Related combination for training enabled the model to reach the peak performance more quickly. This suggests that leveraging CG to assist low-data medical scenarios can lead to more data-efficient training, even when CG does not directly result in significant generalization gains in these scenarios.

Take-away 8: Although CG might not provide direct generalization gains, it helps data efficiency for MLLM training.

6 CG across Detection and Classification
Previous studies (Ren et al., 2024; Wang et al., 2025) have shown that jointly training classification and detection tasks can mutually enhance their performance. Building on this, we investigate whether MLLMs can leverage classification data (e.g., visual knowledge) and detection data (e.g., spatial information) through CG to improve downstream classification (\( Q1 \)) or detection tasks (\( Q2 \)).

6.1 Experiment Settings

Training Setup Each generalization combination used for training in this experiment includes one detection dataset and one classification dataset to examine the generalization relationship between these two vision tasks. The detailed training parameters can be found in Appendix A.6.

Model Selection Next-Chat (Zhang et al., 2023a) and MiniGPT-v2 (Chen et al., 2023a) are selected as baselines, representing the two main approaches MLLMs use for detection tasks. The former treats bounding boxes as embeddings and decodes them into coordinates using a visual decoder, while the latter processes coordinate points as special text tokens and generates bounding box coordinates directly as output text.

Data Processing Med-MAT includes both detection and segmentation datasets. If a segmentation dataset provides object localization using masks, we extract the outermost coordinates of the corresponding mask to construct a bounding box, facilitating generalization experiments for detection. Subsequently, to streamline the experiments, we structured the dataset following the official data formats of Next-Chat and MiniGPT-v2.

6.2 Benefits for Classification (Q1)
In this experiment, all possible CG combinations were selected and the CG-trained model will be
tested on classification task. The final results in Figure 7 show that all CG combinations demonstrated the model’s successful utilization of detection data for CG to the Target data.

6.3 Benefits for Detection (Q2)

Subset 38 and 39 are selected as the objects in these datasets are relatively randomly distributed in the images, making them suitable for evaluating the model’s detection capability. Subsequently, we selected certain classification datasets to construct CG for testing and used cloU to evaluate the detection performance (follow (Chen et al., 2023a)).

Since both baselines lack localization capabilities for medical tasks, we incorporated a fixed amount of Target data into our experiments, adjusting the evaluation scenario to assess support in low-data settings. The results in Table 5 show that all selected CG combinations help MLLMs achieve better performance in detection tasks.

<table>
  <tr>
    <th>Related Combination</th>
    <th>Target Subset</th>
    <th>Next-Chat</th>
    <th>MiniGPT-v2</th>
  </tr>
  <tr>
    <td>D - Skin</td>
    <td>C - Intestine</td>
    <td>D - Intestine</td>
    <td>+3.8</td>
    <td>+4.1</td>
  </tr>
  <tr>
    <td>D - Intestine</td>
    <td>C - Skin</td>
    <td>D - Skin</td>
    <td>+8.4</td>
    <td>+7.6</td>
  </tr>
</table>

Table 5: Next-Chat and MiniGPT-v2 respectively represent the cloU gain brought by CG. C indicates classification task, D indicates detection task.

Take-away 9: MLLMs can perform CG across classification and detection tasks.

7 Related Work

Medical MLLMs Recently, adapting MLLMs to medical tasks has gained prominence due to their success in capturing complex visual features. Current MLLMs typically pair a visual encoder with a text-only LLM, aligning image data with language understanding. Such as Med-Flamingo (Moor et al., 2023) and Med-PaLM (Tu et al., 2024), fine-tuned general multimodal models and achieved notable results. Med-Flamingo enhanced OpenFlamingo-9B (Chen et al., 2024a) with medical data, while Med-PaLM adapted PaLM-E (Driess et al., 2023) using 1 million data points. Similarly, LLaVA-Med (Li et al., 2024), Med-Gemini (Saab et al., 2024), and HuatuoGPT-Vision (Chen et al., 2024b) utilized specialized datasets and instruction tuning to refine medical VQA tasks.

Generalization on Medical Imaging Generalization in medical imaging (Matta et al., 2024) has been extensively studied. Early methods utilized data manipulation techniques, such as data augmentation (Li et al., 2022; Zhang et al., 2022), to enhance model generalization on unseen medical data by adapting to varying distributions. Later approaches focused on representation learning (Le-Khac et al., 2020), preserving essential image information to enable models to handle more complex scenarios. Additionally, some studies (Ren et al., 2024) explore multiple aspects of medical image processing, examining how classification and segmentation tasks can mutually benefit each other.

Detection with MLLMs Recent studies employ various strategies to equip MLLMs with the capability to handle detection tasks, such as encoding regions as features to allow models to accept regions as input (Zhang et al., 2023b), representing object bounding box coordinates with text tokens (Wang et al., 2024c; Peng et al., 2023; Chen et al., 2023b), and employing unique identifiers for task instructions to improve learning efficiency. Additionally, some approaches introduce special tokens to represent images and use their hidden states to decode position information (Zhang et al., 2023a, 2024a).

8 Conclusion

To investigate whether MLLMs can leverage CG to generalize to unseen medical data, we constructed the Med-MAT dataset as a research platform for generalization experiments. The results confirmed the presence of CG and identified it as a key factor of MLLMs’ generalization observed in multi-task learning. Further experiments showed that CG helps MLLMs handle limited data conditions, providing support for low-data medical tasks. Additionally, our findings showed that MLLMs can apply CG across detection and classification tasks, underscoring its broad generalization potential.
Limitations

The experiment confirms that MLLMs leverage CG for unseen medical images and data-efficient training. However, as shown in Section 4, disrupting CG reduces generalization but retains some effectiveness, indicating CG is just one aspect of MLLM generalization in medical imaging.

Potential Risks

Our research focuses on the compositional generalization of MLLMs on medical images, using data sourced from medical challenges and open-source datasets. However, further experiments are needed to mitigate potential risks when deploying this concept in real-world medical settings.

Acknowledgments

This work was supported by Shenzhen Medical Research Fund (No.C2406002) from the Shenzhen Medical Academy of Research and Translation (SMART), the Shenzhen Science and Technology Program (JCYJ20220818103001002), Shenzhen Doctoral Startup Funding (RCBS20221008093330065), Tianyuan Fund for Mathematics of National Natural Science Foundation of China (NSFC) (12326608), Shenzhen Science and Technology Program (Shenzhen Key Laboratory Grant No. ZDSYS20230626091302006), Shenzhen Stability Science Program 2023, and National Natural Science Foundation of China (NSFC) (72495131).

References

Andrea Acevedo, Anna Merino, Santiago Alférez, Ángel Molina, Laura Boldú, and José Rodellar. 2020. A dataset of microscopic peripheral blood cell images for development of automatic recognition systems. Data in brief, 30:105474.

Walid Al-Dhabayni, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy. 2020. Dataset of breast ultrasound images. Data in brief, 28:104863.

Shams Nafisa Ali, Md. Tazuddin Ahmed, Joydip Paul, Tasnim Jahan, S. M. Sakeef Sani, Nawshaba Noor, and Taufiq Hasan. 2022. Monkeypox skin lesion detection using deep learning models: A preliminary feasibility study. arXiv preprint arXiv:2207.03342.

Sharib Ali, Barbara Braden, Dominique Lamarque, Stefano Realdon, Adam Bailey, Renato Cannizzaro, Noha Ghatwary, Jens Rittscher, Christian Daul, and James East. 2020. Endoscopy disease detection and segmentation (edd2020).

MD Anouk Stein, Carol Wu, Chris Carr, George Shih, Jamie Dulkowski, kalpathy, Leon Chen, Luciano Prevedello, MD Marc Kohli, Mark McDonald, Peter, Phil Culliton, Safwan Halabi MD, and Tian Xia. 2018. Rsna pneumonia detection challenge. https://kaggle.com/competitions/rsna-pneumonia-detection-challenge. Kaggle.

Will Arevalo. 2020. Chexpert v1.0 small. https://www.kaggle.com/datasets/willarevalo/chexpert-v10-small. Kaggle.

A Asraf and Z Islam. 2021. Covid19, pneumonia and normal chest x-ray pa dataset. mendeley data v1 (2021).

Francisco José Fumero Batista, Tinguaro Diaz-Aleman, Jose Sigut, Silvia Alayon, Rafael Arnay, and Denisse Angel-Pereira. 2020. Rim-one dl: A unified retinal image database for assessing glaucoma using deep learning. Image Analysis & Stereology, 39(3):161–167.

Dev Batra. 2024. Fracture detection using x-ray images. https://www.kaggle.com/datasets/devbatrax/fracture-detection-using-x-ray-images. Kaggle.

Veronica Elisa Castillo Benítez, Ingrid Castro Matto, Julio César Mello Román, José Luis Vázquez Noguera, Miguel García-Torres, Jordan Ayala, Diego P Pinto-Roa, Pedro E Gardel-Sotomayor, Jacques Facon, and Sebastian Alberto Grillo. 2021. Dataset from fundus images for the study of diabetic retinopathy. Data in brief, 36:107068.

BenO, jjlones, Kumar H, Meg Risdal, MRao, Vadim Sherman, Vipul, Wendy Kan, and Yau Ben-Or. 2017. Intel & mobileodt cervical cancer screening. https://kaggle.com/competitions/intel-mobileodt-cervical-cancer-screening. Kaggle.

Jorge Bernal, F Javier Sánchez, Gloria Fernández-Esparach, Debora Gil, Cristina Rodríguez, and Fernando Vilariño. 2015. Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized medical imaging and graphics, 43:99–111.

Bukun. 2019. Breast cancer histopathological database (breakhis). https://www.kaggle.com/datasets/ambarish/breakhis. Kaggle.

Olivia Cardozo, Verena Ojeda, Rodrigo Parra, Julio César Mello-Román, José Luis Vázquez Noguera, Miguel García-Torres, Federico Divina, Sebastian A. Grillo, Cynthia Villalba, Jacques Facon, Veronica Elisa Castillo Benítez, Ingrid Castro Matto, and Diego Aquino-Brítez. 2023. Dataset of fundus images for the diagnosis of ocular toxoplasmosis. Data in Brief, 48:109056.
Ling-Ping Cen, Jie Ji, Jian-Wei Lin, Si-Tong Ju, Hong-Jie Lin, Tai-Ping Li, Yun Wang, Jian-Feng Yang, Yu-Fen Liu, Shaoying Tan, et al. 2021. Automatic detection of 39 fundus diseases and conditions in retinal photographs using deep neural networks. Nature communications, 12(1):4828.

Delong Chen, Jianfeng Liu, Wenliang Dai, and Baoyuan Wang. 2024a. Visual instruction tuning with polite flamingo. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17745–17753.

Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuaraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. 2023a. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478.

Junying Chen, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, Guangjun Yu, et al. 2024b. Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale. arXiv preprint arXiv:2406.19280.

Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. 2023b. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195.

Pingjun Chen. 2018. Knee osteoarthritis severity grading dataset. Mendeley Data, 1(10.17632).

Muhammad E. H. Chowdhury, Tawsifur Rahman, Amith Khandakar, Rashid Mazhar, Muhammad Abdul Kadir, Zaid Bin Mahbub, Khandakar Reajul Islam, Muhammad Salman Khan, Atif Iqbal, Nasser Al Emadi, Mamun Bin Ibne Reaz, and Mohammad Tariqul Islam. 2020. Can ai help in screening viral and covid-19 pneumonia? IEEE Access, 8:132665–132676.

Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. 2019. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv:1902.03368.

Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al. 2018. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), pages 168–172. IEEE.

Marc Combalia, Noel CF Codella, Veronica Rotemberg, Brian Helba, Veronica Vilaplana, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Allan C Halpern, Susana Puig, et al. 2019. Bcn20000: Dermoscopic lesions in the wild. arXiv preprint arXiv:1908.02288.

Will Cukierski. 2018. Histopathologic cancer detection. https://kaggle.com/competitions/histopathologic-cancer-detection. Kaggle.

Training Data. 2023. Computed tomography of the brain. https://www.kaggle.com/datasets/trainingdatapro/computed-tomography-ct-of-the-brain. Kaggle.

Coen de Vente, Koenraad A. Vermeer, Nicolas Jaccard, He Wang, Hongyi Sun, Firas Khader, Daniel Truhn, Temirgali Aimyshev, Yerkebulan Zhanibekuly, Tien-Dung Le, Adrian Galdran, Miguel Ángel González Ballester, Gustavo Carneiro, Devika R G, Hrishikesh P S, Densen Puthusserry, Hong Liu, Zekang Yang, Satoshi Kondo, Satoshi Kasai, Edward Wang, Ashritha Durvasula, Jónathan Heras, Miguel Ángel Zapata, Teresa Araujo, Guilherme Aresta, Hrvoje Bogunović, Mustafa Arikan, Yeong Chan Lee, Hyun Bin Cho, Yoon Ho Choi, Abdul Qayyum, Imran Razzak, Bram van Ginneken, Hans G. Lemij, and Clara I. Sánchez. 2023. Airos: Artificial intelligence for robust glaucoma screening challenge. arXiv preprint arXiv:2302.01738.

Danny Diress, Fei Xia, Mehdi SM Saijadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. 2023. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378.

Fernando Feltrin. 2022. Brain tumor mri images 17 classes. https://www.kaggle.com/datasets/fernando2rad/brain-tumor-mri-images-17-classes. Kaggle.

Mohammad Fraiwan, Ziad Audat, Luay Fraiwan, and Tarek Manssreh. 2022. Using deep transfer learning to detect scoliosis and spondylolisthesis from x-ray images. Plos one, 17(5):e0267851.

Huazhu Fu, Fei Li, José Ignacio Orlando, Hrvoje Bogunović, Xu Sun, Jingan Liao, Yanwu Xu, Shao-chong Zhang, and Xiulan Zhang. 2019. Palm: Pathologic myopia challenge.

Ioannis Giotis, Nynke Molders, Sander Land, Michael Biehl, Marcel F Jonkman, and Nicolai Petkov. 2015. Med-node: A computer-assisted melanoma diagnosis system using non-dermoscopic images. Expert systems with applications, 42(19):6578–6585.

Haifan Gong, Guanqi Chen, Ranran Wang, Xiang Xie, Mingzhi Mao, Yizhou Yu, Fei Chen, and Guanbin Li. 2021. Multi-task learning for thyroid nodule segmentation with thyroid region prior. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), pages 257–261. IEEE.

Haifan Gong, Jiaxin Chen, Guanqi Chen, Haofeng Li, Fei Chen, and Guanbin Li. 2022. Thyroid region prior guided attention for ultrasound segmentation of
thyroid nodules. Computers in Biology and Medicine, 106389:1–12.

Shivanand Gornale and Pooja Patravali. 2020. Digital knee x-ray images.

Matthew Groh, Caleb Harris, Luis Soenksen, Felix Lau, Rachel Han, Aerin Kim, Arash Koochek, and Omar Badri. 2021. Evaluating deep neural networks trained on clinical images in dermatology with the fitzpatrick 17k dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1820–1828.

David Gutman, Noel CF Codella, Emre Celebi, Brian Helba, Michael Marchetti, Nabin Mishra, and Allan Halpern. 2016. Skin lesion analysis toward melanoma detection: A challenge at the international symposium on biomedical imaging (isbi) 2016, hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv:1605.01397.

Saba Hesaraki. 2022. Breast ultrasound images dataset (busi). https://www.kaggle.com/datasets/sabahesaraki/breast-ultrasound-images-dataset. Kaggle.

Md Nazmul Islam, Mehedi Hasan, Md Kabir Hossain, Md Golam Rabiu Alam, Md Zia Uddin, and Ahmet Soylu. 2022a. Vision transformer and explainable transfer learning models for auto detection of kidney cyst, stone and tumor from ct-radiography. Scientific Reports, 12(1):1–14.

Towhidul Islam, Mohammad Arafat Hussain, Forhad Uddin Hasan Chowdhury, and B M Riazul Islam. 2022b. A web-scraped skin image database of monkeypox, chickenpox, smallpox, cowpox, and measles. bioRxiv 2022.08.01.502199.

Stefan Jaeger, Sema Candemir, Sameer Antani, Yi-Xiang J Wang, Pu-Xuan Lu, and George Thoma. 2014. Two public chest x-ray datasets for computer-aided screening of pulmonary diseases. Quantitative imaging in medicine and surgery, 4(6):475.

Debesh Jha, Pia H Smedsrud, Michael A Riegler, Pål Halvorsen, Thomas de Lange, Dag Johansen, and Håvard D Johansen. 2020. Kvasir-seg: A segmented polyp dataset. In Multimedia Modeling: 26th International Conference, MMM 2020, Daejeon, South Korea, January 5–8, 2020, Proceedings, Part II 26, pages 451–462. Springer.

Kai Jin, Xingru Huang, Jingxing Zhou, Yunxiang Li, Yan Yan, Yibao Sun, Qianni Zhang, Yaqi Wang, and Juan Ye. 2022. Fives: A fundus image dataset for artificial intelligence based vessel segmentation. Scientific data, 9(1):475.

JR2NGB. 2019. Cataract dataset. https://www.kaggle.com/datasets/jr2ngb/cataractdataset. Kaggle.

Nur Karaca. 2022. Nlm montgomery cxr set. https://www.kaggle.com/datasets/nurkaraca/nlm-montgomerycxrset. Kaggle.

Karthik, Maggie, and Sohier Dane. 2019. Aptos 2019 blindness detection. https://kaggle.com/competitions/aptos2019-blindness-detection. Kaggle.

Andrey Katanskiy. 2019. Skin cancer isic. https://www.kaggle.com/datasets/nodoubttome/skin-cancer9-classeisic. Kaggle.

Jakob Nikolas Kather, Niels Halama, and Alexander Marx. 2018. 100,000 histological images of human colorectal cancer and healthy tissue.

Daniel Kermany. 2018. Labeled optical coherence tomography (oct) and chest x-ray images for classification. Mendeley data.

Felipe Campos Kitamura. 2018. Head ct - hemorrhage.

Jorge F Lazo, Benoit Rosa, Michele Catellani, Matteo Fontana, Francesco A Mistretta, Gennaro Musi, Ottavio de Cobelli, Michel de Mathelin, and Elena De Momi. 2023. Semi-supervised bladder tissue classification in multi-domain endoscopic images. IEEE Transactions on Biomedical Engineering.

Trang Le, Casper F Winsnes, Ulrika Axelsson, Hao Xu, Jayasankar Mohanakrishnan Kaimal, Diana Mahdessian, Shubin Dai, Ilya S Makarov, Vladislav Ostankovich, Yang Xu, et al. 2022. Analysis of the human protein atlas weakly supervised single-cell classification competition. Nature methods, 19(10):1221–1229.

Phuc H Le-Khac, Graham Healy, and Alan F Smeaton. 2020. Contrastive representation learning: A framework and review. Ieee Access, 8:193907–193934.

Rebecca Sawyer Lee, Francisco Gimenez, Assaf Hoogi, Kanae Kawai Miyake, Mia Gorovoy, and Daniel L Rubin. 2017. A curated mammography data set for use in computer-aided detection and diagnosis research. Scientific data, 4(1):1–9.

Sangjune L Lee, Poonam Yadav, Yin Li, Jason J Meudt, Jessica Strang, Dustin Hebel, Alyx Alffson, Stephanie J Olson, Tera R Kruser, Jennifer B Smilowitz, et al. 2024. Dataset for gastrointestinal tract segmentation on serial mrIs for abdominal tumor radiotherapy. Data in Brief, page 111159.

Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Nau mann, Hoifung Poon, and Jianfeng Gao. 2024. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36.

Yuanpeng Li, Liang Zhao, Jianyu Wang, and Joel Hestness. 2019. Compositional generalization for primitive substitutions. arXiv preprint arXiv:1910.02612.
Yuexiang Li, Nanjun He, and Yawen Huang. 2022. Single domain generalization via spontaneous amplitude spectrum diversification. In MICCAI Workshop on Resource-Efficient Medical Image Analysis, pages 32–41. Springer.

Jie Lian, Jingyu Liu, Shu Zhang, Kai Gao, Xiaoqing Liu, Dingwen Zhang, and Yizhou Yu. 2021. A structure-aware relation network for thoracic diseases detection and segmentation. IEEE Transactions on Medical Imaging, 40(8):2042–2052.

Xiao Liang. 2021. Adam dataset. https://www.kaggle.com/datasets/xiaoliang2121/adamdataset. Kaggle.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning.

Jacob A Macdonald, Zhe Zhu, Brandon Konkel, and Mazurowski. 2020. Siim-acr pneumothorax segmentation. https://doi.org/10.5281/zenodo.7774566. Zenodo.

K Scott Mader. 2017. Mias mammography. https://www.kaggle.com/datasets/kmader/mias-mammography. Kaggle.

Salman Maqbool, Aqsa Riaz, Hasan Sajid, and Osman Hasan. 2020. m2caiseg: Semantic segmentation of laparoscopic images using convolutional neural networks. arXiv preprint arXiv:2008.10134.

Christian Matek, Sebastian Krappe, Christian Münzenmayer, Torsten Haferlach, and Carsten Marr. 2021. An expert-annotated dataset of bone marrow cytology in hematologic malignancies. The Cancer Imaging Archive.

Sarah Matta, Mathieu Lamard, Philippe Zhang, Alexandre Le Guilcher, Laurent Borderie, Béatrice Cochener, and Gwenolé Quellec. 2024. A systematic review of generalization research in medical image classification. arXiv preprint arXiv:2403.12167.

Teresa Mendonca, M Celebi, T Mendonca, and J Marques. 2015. Ph2: A public database for the analysis of dermoscopic images. Dermoscopy image analysis.

Meta AI. 2024. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/.

Shentong Mo and Paul Pu Liang. 2024. Multimed: Massively multimodal and multitask medical understanding. arXiv preprint arXiv:2408.12682.

Paul Mooney. 2017. Blood cell images. https://www.kaggle.com/datasets/paultimothymooney/blood-cells. Kaggle.

Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. 2023. Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (MLAH), pages 353–367. PMLR.

Loris Nanni, Michelangelo Paci, Florentino Luciano Caetano dos Santos, Heli Skottman, Kati Juuti-Uusitalo, and Jari Hytinen. 2016. Texture descriptors ensembles enable image-based classification of maturation of human stem cell-derived retinal pigmented epithelium. PLoS One, 11(2):e0149399.

Hieu T Nguyen, Ha Q Nguyen, Hieu H Pham, Khanh Lam, Linh T Le, Minh Dao, and Van Vu. 2023. Vindr-mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography. Scientific Data, 10(1):277.

Masoud Nickparvar. 2021a. Brain tumor mri dataset. https://www.kaggle.com/datasets/masoudnickparvar/brain-tumor-mri-dataset. Kaggle.

Msoud Nickparvar. 2021b. Brain tumor mri dataset.

Nikita Orlov, Wayne Chen, David Eckley, Tomasz Macura, Lior Shamir, Elaine Jaffe, and Ilya Goldberg. 2010a. Automatic classification of lymphoma images with transform-based global features. IEEE transactions on information technology in biomedicine : a publication of the IEEE Engineering in Medicine and Biology Society, 14:1003–13.

Nikita Orlov, Wayne Chen, David Eckley, Tomasz Macura, Lior Shamir, Elaine Jaffe, and Ilya Goldberg. 2010b. Automatic classification of lymphoma images with transform-based global features. IEEE transactions on information technology in biomedicine : a publication of the IEEE Engineering in Medicine and Biology Society, 14:1003–13.

Silvia Ovreiu, Elena-Anca Paraschiv, and Elena Ovreiu. 2021. Deep learning & digital fundus images: Glaucoma detection using densenet. In 2021 13th international conference on electronics, computers and artificial intelligence (ECAI), pages 1–4. IEEE.

Andre GC Pacheco, Gustavo R Lima, Amanda S Salomao, Breno Krohling, Igor P Biral, Gabriel G de Angelo, Fábio CR Alves Jr, José GM Esgario, Alana C Simora, Pedro BC Castro, et al. 2020. Pad-ufoes-20: A skin lesion dataset composed of patient data and clinical images collected from smartphones. Data in brief, 32:106221.

Sachin Panchal, Ankita Naik, Manesh Kokare, Samiksha Pachade, Rushikesh Naigaonkar, Prerana Phadnis, and Archana Bhange. 2023. Retinal fundus multi-disease image dataset (rfmid) 2.0: a dataset of frequently and rarely identified diseases. Data, 8(2):29.

Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. 2023. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824.
H Hieu Pham, T Thanh Tran, and Ha Quy Nguyen. 2022. Vindr-pcxr: An open, large-scale pediatric chest x-ray dataset for interpretation of common thoracic diseases. PhysioNet (version 1.0.0), 10:2.

Hieu Huy Pham, H Nguyen Trung, and Ha Quy Nguyen. 2021. Vindr-spinexr: A large annotated medical image dataset for spinal lesions detection and classification from radiographs. PhysioNet.

Konstantin Pogorelov, Kristin Ranheim Randel, Thomas de Lange, Sigrun Losada Eskeland, Carsten Griwodz, Dag Johansen, Concetto Spampinato, Mario Taschwer, Mathias Lux, Peter Thelin Schmidt, Michael Riegler, and Pål Halvorsen. 2017a. Nerthus: A bowel preparation quality video dataset. In Proceedings of the 8th ACM on Multimedia Systems Conference, MMSys’17, pages 170–174, New York, NY, USA. ACM.

Konstantin Pogorelov, Kristin Ranheim Randel, Carsten Griwodz, Sigrun Losada Eskeland, Thomas de Lange, Dag Johansen, Concetto Spampinato, Duc-Tien Dang-Nguyen, Mathias Lux, Peter Thelin Schmidt, Michael Riegler, and Pål Halvorsen. 2017b. Kvasir: A multi-class image dataset for computer aided gastrointestinal disease detection. In Proceedings of the 8th ACM on Multimedia Systems Conference, MMSys’17, pages 164–169, New York, NY, USA. ACM.

Praveen. 2019. Coronahack chest x-ray dataset. https://www.kaggle.com/datasets/praveengovi/coronahack-chest-xraydataset. Kaggle.

Pavle Prentasic, Sven Loncaric, Zoran Vatavuk, Goran Bencic, Marko Subasic, Tomislav Petković, Lana Dujmovic, Maja Malenica Ravlic, Nikola Budimirija, and Rašeljka Tadić. 2013. Diabetic retinopathy image database(dridb): A new database for diabetic retinopathy screening programs research. In International Symposium on Image and Signal Processing and Analysis, ISPA, pages 711–716.

Xianbiao Qi, Guoying Zhao, Jie Chen, and Matti Pietikäinen. 2016. Hep-2 cell classification: The role of gaussian scale space theory as a pre-processing approach. Pattern Recognition Letters, 82:36–43.

Raddar. 2019. Chest x-rays (indiana university). https://www.kaggle.com/datasets/raddar/chest-xrays-indiana-university?select=indiana_reports.csv. Kaggle.

Tawsifur Rahman, Amith Khandakar, Muhammad Abdul Kadir, Khandaker Rejaul Islam, Khandakar F Islam, Rashid Mazhar, Tahir Hamid, Mohammad Tariqul Islam, Saad Kashem, Zaid Bin Mahbub, et al. 2020. Reliable tuberculosis detection using chest x-ray with deep learning, segmentation and visualization. Ieee Access, 8:191586–191601.

MOHD ZAID RASHID. 2024. Oral cancer dataset. https://www.kaggle.com/datasets/zaidpy/oral-cancer-dataset. Kaggle.

Sucheng Ren, Xiaoke Huang, Xianhang Li, Junfei Xiao, Jieru Mei, Zeyu Wang, Alan Yuille, and Yuyin Zhou. 2024. Medical vision generalist: Unifying medical imaging tasks in context. arXiv preprint arXiv:2406.05565.

Manuel Alejandro Rodríguez, Hasan AlMarzouqi, and Panos Liatsis. 2022. Multi-label retinal disease classification using transformers. IEEE Journal of Biomedical and Health Informatics.

Veronica Rotemberg, Nicholas Kurtansky, Brigid Betz-Stablein, Liam Caffery, Emmanouil Chousakos, Noel Codella, Marc Combalia, Stephen Dusza, Pascale Guitera, David Gutman, et al. 2021. A patient-centric dataset of images and metadata for identifying melanomas using clinical context. Scientific data, 8(1):34.

Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulczyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, et al. 2024. Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.18416.

Salman Sajid. 2024. Oral diseases. https://www.kaggle.com/datasets/salmansajid05/oral-diseases/data. Kaggle.

F Shaker. 2018. Human sperm head morphology dataset (hushem). Mendeley Data, 3.

Julio Silva-Rodríguez, Adrián Colomer, María A Sales, Rafael Molina, and Valery Naranjo. 2020. Going deeper through the gleason scoring scale: An automatic end-to-end system for histology prostate grading and cribriform pattern detection. Computer Methods and Programs in Biomedicine, 195:105637.

Eduardo Soares, Plamen Angelov, Sarah Biaso, Michele Higa Froes, and Daniel Kanda Abe. 2020. Sars-cov-2 ct-scan dataset:a large dataset of real patients ct scans for sars-cov-2 identification. Cold Spring Harbor Laboratory Press.

Malliga Subramanian, Kogilavani Shanmugavadivel, Obuli Sat Naren, K Premkumar, and K Rankish. 2022. Classification of retinal oct images using deep learning. In 2022 International Conference on Computer Communication and Informatics (ICCCI), pages 1–7.

Summers and Ronald. 2020. Chestxray nihcc. https://nihcc.app.box.com/v/ChestXray-NIHCC/folder/36938765345. NIH.

SunneYi. 2021. Chest CT-Scan images Dataset.

Siham Tabik, Anabel Gómez-Ríos, José Luis Martín-Rodríguez, Iván Sevillano-García, Manuel Rey-Area, David Charté, Emilio Guirado, Juan-Luis Suárez, Julián Luengo, MA Valero-González, et al. 2020. Covidgr dataset and covid-sdnet methodology for predicting covid-19 based on chest x-ray images. IEEE journal of biomedical and health informatics, 24(12):3595–3605.
Yihong Tang, Ao Qu, Zhaokai Wang, Dingyi Zhuang, Zhaofeng Wu, Wei Ma, Shenhao Wang, Yunhan Zheng, Zhan Zhao, and Jinhua Zhao. 2024. Sparkle: Mastering basic spatial capabilities in vision language models elicits generalization to composite spatial reasoning. arXiv preprint arXiv:2410.16162.

Tao Tu, Shekofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, et al. 2024. Towards generalist biomedical ai. NEJM AI, 1(3):A1oa2300138.

Peking University. 2019. Odir-2019 dataset. https://odir2019.grand-challenge.org/introduction/. Grand Challenge.

Preet Viradiya. 2020. Brain tumor dataset. https://www.kaggle.com/datasets/preetviradiya/brain-tumor-dataset. Kaggle.

Haiyang Wang, Hao Tang, Li Jiang, Shaoshuai Shi, Muhammad Ferjad Naem, Hongsheng Li, Bernt Schiele, and Liwei Wang. 2025. Git: Towards generalist vision transformer through universal language interface. In European Conference on Computer Vision, pages 55–73. Springer.

Linda Wang, Zhong Qiu Lin, and Alexander Wong. 2020. Covid-net: a tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images. Scientific Reports, 10(1):19549.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024a. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024b. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. Preprint. arXiv:2409.12191.

Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. 2024c. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36.

Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. 2017. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2097–2106.

wjXiaochuangw. 2019. Covid-19-ct scan images.

Zhenlin Xu, Marc Niethammer, and Colin A Raffel. 2022. Compositional generalization in unsupervised compositional representation learning: A study on disentanglement and emergent language. Advances in Neural Information Processing Systems, 35:25074–25087.

Anna Zawacki, Carol Wu, George Shih, Julia Elliott, Mikhail Fomitchev, Mohammad Hussain, Paras Lakhani, Phil Culliton, and Shunxing Bao. 2019. Siim-acr pneumothorax segmentation. https://kaggle.com/competitions/siim-acr-pneumothorax-segmentation. Kaggle.

Yaya Zha. 2021. Rus-chn. https://aistudio.baidu.com/datassetdetail/69582/0. AI Studio.

Ao Zhang, Liming Zhao, Chen-Wei Xie, Yun Zheng, Wei Ji, and Tat-Seng Chua. 2023a. Next-chat: An lmm for chat, detection and segmentation. arXiv preprint arXiv:2311.04498.

Edward Zhang and Sauman Das. 2022. Glaucoma detection. https://www.kaggle.com/datasets/sshikamaru/glaucoma-detection. Kaggle.

Ruipeng Zhang, Qinwei Xu, Chaoqin Huang, Ya Zhang, and Yanfeng Wang. 2022. Semi-supervised domain generalization for medical image analysis. In 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), pages 1–5. IEEE.

Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, and Ping Luo. 2023b. Gpt4roi: Instruction tuning large language model on region-of-interest. arXiv preprint arXiv:2307.03601.

Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Change Loy Chen, and Shuicheng Yan. 2024a. Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding. In NeurIPS.

Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023c. Pmc-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305.10415.

Yuhui Zhang, Alyssa Unell, Xiaohan Wang, Dhruba Ghosh, Yuchang Su, Ludwig Schmidt, and Serena Yeung-Levy. 2024b. Why are visually-grounded language models bad at image classification? arXiv preprint arXiv:2405.18415.

Jinyu Zhao, Yichen Zhang, Xuehai He, and Pengtao Xie. 2020. Covid-ct-dataset: a ct scan dataset about covid-19. arXiv preprint arXiv:2003.13865.

Chuang Zhu, Wenkai Chen, Ting Peng, Ying Wang, and Mulan Jin. 2021a. Hard sample aware noise robust learning for histopathology image classification.
IEEE transactions on medical imaging, 41(4):881–894.

Chuang Zhu, Wenkai Chen, Ting Peng, Ying Wang, and Mulan Jin. 2021b. Hard sample aware noise robust learning for histopathology image classification. IEEE transactions on medical imaging, 41(4):881–894.

Xile Zhu. 2022. Lc25000. https://www.kaggle.com/datasets/xilezhu/lc25000. Kaggle.

. 2021. Augmented ocular diseases. https://www.kaggle.com/datasets/nurmukhammed7/augmented-ocular-diseases. Kaggle.

A More Experiments

A.1 Benefits for Segmentation

Segmentation-enabled LLMs, such as Next-GPT, first use the LLM to identify potential regions of the target object and then apply a SAM to decode the object mask, thereby completing the segmentation task. In this context, segmentation can be seen as an extension of detection, potentially requiring more images to achieve improved performance. We conducted additional experiments to explore whether MLLMs can still utilize CG to understand new images across both segmentation and classification tasks.

<table>
  <tr>
    <th>Related Combination</th>
    <th>Target Subset</th>
    <th>Next-Chat</th>
  </tr>
  <tr>
    <td>D - Skin</td>
    <td>C - Intestine</td>
    <td>S - Intestine</td>
    <td>+7.46</td>
  </tr>
  <tr>
    <td>D - Intestine</td>
    <td>C - Skin</td>
    <td>S - Skin</td>
    <td>+5.42</td>
  </tr>
</table>

Table 6: Next-Chat represents the clou gain brought by CG. C indicates classification task, S indicates Segmentation task.

The results in Table 6 demonstrate that, in the context of segmentation tasks, MLLMs are still able to leverage CG to understand new tasks, which is consistent with our original conclusions.

A.2 More Complex Medical Elements

While MAT-Triplet Categorization is useful, predefined categories may limit the exploration of more complex medical attributes, so we also considered integrating more flexible categorization to explore additional medical attributes.

Additional Element 1: Population Groups We selected VinDr-PCXR and MedMAT Subset 31 for the experiment, as they contain X-ray images of children and adult groups, respectively. The results are shown in Table 7.

<table>
  <tr>
    <th>Related Combination</th>
    <th>Target Subset</th>
    <th>LLaVA</th>
  </tr>
  <tr>
    <td>X-ray @ Young | Unrelated Data</td>
    <td>X-ray @ Adults</td>
    <td>+6.04</td>
  </tr>
  <tr>
    <td>X-ray @ Young | CT Children (CG)</td>
    <td>X-ray @ Adults</td>
    <td>+18.12</td>
  </tr>
</table>

Table 7: Results of using Population Groups as a CG element.

<table>
  <tr>
    <th>Related Combination</th>
    <th>Target Subset</th>
    <th>LLaVA</th>
  </tr>
  <tr>
    <td>X-ray @ Pneumonia | Unrelated Data</td>
    <td>X-ray @ COVID</td>
    <td>+11.33</td>
  </tr>
  <tr>
    <td>X-ray @ Pneumonia | CT @ COVID (CG)</td>
    <td>X-ray @ COVID</td>
    <td>+12.67</td>
  </tr>
</table>

Table 8: Results of using Finer Disease as a CG element.

Additional Element 2: Finer Disease "Finer disease" means more detailed categorization. For instance, we treat COVID and common pneumonia as distinct diseases for generalization. We split the Normal data in the training set into two parts and combined each with COVID and Pneumonia data to create new datasets. The results are shown in Table 8.

A.3 Statistical Tests of the Generalization Results

To ensure consistency and repeatability of the experiment, we performed statistical tests in this section. LLaVA is selected as the baseline, and we used the same data combinations from Section 3.3. Each experiment was repeated 3 times, and we reported the mean and standard deviation (SD) of the results.

From the results in Table 9, we can observe that the outcomes across runs show low variance, indicating overall stability, and they continue to support our original experimental conclusions.

A.4 CG with All MAT-Triplet Elements from Different Sources

In previous controlled experiments (Section 3), one element of the MAT-Triplet was kept constant while CG was explored in the remaining two elements. To ensure that all the 3 MAT-Triplet elements of the target data originated from three distinct datasets, additional experiments were conducted to further validate the effectiveness of CG. For these experiments, all possible combinations meeting the criteria in Med-MAT were selected (Selection Strategy). The results presented in Table 10 demonstrate that most combinations can effectively generalize to the Target data.

Analysis of the results The results in Table 7 and 8 indicate that the two new attributes show data leakage due to subtle visual differences in corresponding images (e.g., COVID-19 and pneumonia
<table>
  <tr>
    <th>Related Combination</th>
    <th>Target Subset</th>
    <th>Baseline</th>
    <th>1st</th>
    <th>2nd</th>
    <th>3rd</th>
    <th>Mean and SD</th>
  </tr>
  <tr>
    <td>Bones State</td>
    <td>Breast Diseases</td>
    <td>Bones Diseases</td>
    <td>Bones Diseases</td>
    <td>37.31</td>
    <td>43.28</td>
    <td>44.78</td>
    <td>43.28</td>
    <td>43.78 ± 0.87</td>
  </tr>
  <tr>
    <td>Lung COVID</td>
    <td>Bones Diseases</td>
    <td>Lung Diseases</td>
    <td>Lung Diseases</td>
    <td>49.00</td>
    <td>52.00</td>
    <td>52.00</td>
    <td>52.00</td>
    <td>52.00 ± 0.00</td>
  </tr>
  <tr>
    <td>X-ray Diseases CT</td>
    <td>COVID X-ray</td>
    <td>X-ray COVID</td>
    <td>X-ray COVID</td>
    <td>30.00</td>
    <td>47.33</td>
    <td>49.33</td>
    <td>49.33</td>
    <td>48.66 ± 1.15</td>
  </tr>
  <tr>
    <td>X-ray Diseases CT</td>
    <td>State X-ray</td>
    <td>X-ray State</td>
    <td>X-ray State</td>
    <td>30.00</td>
    <td>46.00</td>
    <td>45.33</td>
    <td>44.67</td>
    <td>45.33 ± 0.67</td>
  </tr>
  <tr>
    <td>CT Brain X-ray</td>
    <td>Lung CT</td>
    <td>Lung CT</td>
    <td>Lung CT</td>
    <td>25.00</td>
    <td>31.50</td>
    <td>32.00</td>
    <td>32.00</td>
    <td>31.83 ± 0.29</td>
  </tr>
  <tr>
    <td>CT Brain X-ray</td>
    <td>Lung CT</td>
    <td>Lung CT</td>
    <td>Lung CT</td>
    <td>47.00</td>
    <td>71.00</td>
    <td>71.00</td>
    <td>70.00</td>
    <td>70.67 ± 0.58</td>
  </tr>
  <tr>
    <td>FP - Fundus Diseases Mic - Cell Level</td>
    <td>FP - Fundus Level</td>
    <td>FP - Fundus Level</td>
    <td>FP - Fundus Level</td>
    <td>33.33</td>
    <td>42.42</td>
    <td>45.45</td>
    <td>45.45</td>
    <td>44.44 ± 1.75</td>
  </tr>
  <tr>
    <td>Mic - Cell Recognition FP - Fundus Level</td>
    <td>Mic - Cell Level</td>
    <td>Mic - Cell Level</td>
    <td>Mic - Cell Level</td>
    <td>23.00</td>
    <td>32.00</td>
    <td>32.00</td>
    <td>31.50</td>
    <td>31.83 ± 0.29</td>
  </tr>
</table>

Table 9: Statistical tests of CG experiments. The 1st, 2nd, and 3rd show the generalization results of the experiment in different runs. "Mean" and "SD" represent the average accuracy (%) and standard deviation.

<table>
  <tr>
    <th>Related Combination</th>
    <th>Target Subset</th>
    <th>Baseline</th>
    <th>Trained</th>
    <th>CG Helps</th>
  </tr>
  <tr>
    <td>Brain Cancer CT</td>
    <td>Brain Cancer</td>
    <td>28</td>
    <td>26</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>CT Brain Cancer</td>
    <td>Brain Cancer</td>
    <td>28</td>
    <td>25</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>CT Brain State</td>
    <td>Brain State</td>
    <td>33</td>
    <td>64</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Brain State CT</td>
    <td>Brain State</td>
    <td>33</td>
    <td>70</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>X-ray Lung Diseases</td>
    <td>X-ray Lung Diseases</td>
    <td>30</td>
    <td>48</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>X-ray Lung Diseases</td>
    <td>Lung Diseases</td>
    <td>30</td>
    <td>38</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>X-ray Lung Diseases</td>
    <td>Lung Diseases</td>
    <td>30</td>
    <td>44</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>X-ray Breast Diseases</td>
    <td>X-ray Breast Diseases</td>
    <td>31</td>
    <td>52</td>
    <td>✓</td>
  </tr>
</table>

Table 10: Results from 3 datasets providing different elements of MAT-Triplet. ✓ in CG Helps indicates successful generalization, while ✗ denotes failure.

have similar features). Importantly, the MLLM trained with CG combinations still shows improvements on downstream tasks, confirming that our approach remains valid for new attributes.

Reason to choose the existing three attributes (MAT-Triplet: Modality, Area, Task) We have considered additional categories such as age, gender, and finer disease classification, but we ultimately chose to focus on the MAT-Triplet categories for the following reasons.

• The boundaries between MAT-Triplet (Modality, Area, Task) are clear. Different modalities and areas correspond to distinct imaging methods and body areas, leading to significant differences between images; different tasks also require the MLLM to extract specific information, demanding varied understanding of the images.

• All datasets can be annotated using MAT-Triplet (Modality, Area, Task) easily. Other medical labels, such as gender and age, are only available in a small portion of datasets and are not suitable for large-scale annotation.

• Similar categorization strategies have been adopted in previous studies.

A.5 Details of Section 3.3: Exploring CG on different MLLM Backbones

To ensure the experiment results are not influenced by the model choice, we also tested several other models on some subsets of Med-MAT and observed similar results.

Selection Strategy: For testing, some generalized combinations were selected from classification tasks 3. Using a random seed of 42, we shuffled each Direction Type’s combinations and selected the first two compositions as test data.

Experimental Setup: We conducted experiments to evaluate the compatibility of CG across different backbone architectures. We selected two MLLMs with representative architectures, namely Qwen2-VL-7B-Instruct (Wang et al., 2024b) and Llama-3.2-11B-Vision-Instruct (Meta AI, 2024), to assess the performance of CG on these models. Each experiment involved full-parameter fine-tuning of all models over 5 epochs, utilizing 8 A800 (80GB) GPUs. The training was performed with a batch size of 32 and a learning rate set to 2e-6, ensuring that all parameters were updated to optimize the model performance.

A.6 Details of Section 6: Exploring CG across Detection and Classification

Experimental Setup: We conducted generalization experiments for detection and classification. Specifically, we performed generalization validation on Next-Chat (Zhang et al., 2023a) and MiniGPT-v2 (Chen et al., 2023a). Next-Chat models the bounding box as an embedding and utilizes a decoder for decoding, while MiniGPT-v2 treats the bounding box as a text token, which are common approaches used by existing MLLM implementations for detection. By conducting CG validation using distinct bounding box modeling methods, we further demonstrate the broad applicability of the CG approach. Each experiment was conducted on 8 A800 (80GB) GPUs.

The two backbones were trained separately in this experiment. For Next-Chat, we directly trained the model in its second training stage and fine-tuned it for 2 epochs with a learning rate of 2e-5,
<table>
  <tr>
    <th>Related Combination</th>
    <th>Target Subset</th>
    <th>Baseline</th>
    <th>Trained</th>
    <th>CG Helps</th>
  </tr>
  <tr>
    <td>Bones State</td>
    <td>Breast Diseases</td>
    <td>Bones Diseases</td>
    <td>61</td>
    <td>65</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Lung COVID</td>
    <td>Bones Diseases</td>
    <td>Lung Diseases</td>
    <td>80</td>
    <td>91</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>X-ray Diseases CT</td>
    <td>COVID X-ray</td>
    <td>COVID</td>
    <td>35</td>
    <td>40</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>X-ray Diseases CT</td>
    <td>State X-ray</td>
    <td>State</td>
    <td>35</td>
    <td>43</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>CT Brain X-ray</td>
    <td>Lung CT</td>
    <td>Lung</td>
    <td>52</td>
    <td>53</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>CT Brain X-ray</td>
    <td>Lung CT</td>
    <td>Lung</td>
    <td>65</td>
    <td>72</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>FP - Fundus Diseases Mic - Cell Level</td>
    <td>FP - Fundus Level</td>
    <td>Level</td>
    <td>48</td>
    <td>45</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>Mic - Cell Recognition FP - Fundus Level</td>
    <td>Mic - Cell Level</td>
    <td>Level</td>
    <td>34</td>
    <td>41</td>
    <td>✓</td>
  </tr>
</table>

Table 11: Result of Qwen2-VL on selected classification datasets in Med-MAT. ✓ in CG Helps indicates successful generalization, while ✗ denotes failure.

<table>
  <tr>
    <th>Related Combination</th>
    <th>Target Subset</th>
    <th>Baseline</th>
    <th>Trained</th>
    <th>CG Helps</th>
  </tr>
  <tr>
    <td>Bones State</td>
    <td>Breast Diseases</td>
    <td>Bones Diseases</td>
    <td>52</td>
    <td>59</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Lung COVID</td>
    <td>Bones Diseases</td>
    <td>Lung Diseases</td>
    <td>64</td>
    <td>75</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>X-ray Diseases CT</td>
    <td>COVID X-ray</td>
    <td>COVID</td>
    <td>33</td>
    <td>38</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>X-ray Diseases CT</td>
    <td>State X-ray</td>
    <td>State</td>
    <td>33</td>
    <td>41</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>CT Brain X-ray</td>
    <td>Lung CT</td>
    <td>Lung</td>
    <td>51</td>
    <td>29</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>CT Brain X-ray</td>
    <td>Lung CT</td>
    <td>Lung</td>
    <td>49</td>
    <td>57</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>FP - Fundus Diseases Mic - Cell Level</td>
    <td>FP - Fundus Level</td>
    <td>Level</td>
    <td>55</td>
    <td>61</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Mic - Cell Recognition FP - Fundus Level</td>
    <td>Mic - Cell Level</td>
    <td>Level</td>
    <td>10</td>
    <td>52</td>
    <td>✓</td>
  </tr>
</table>

Table 12: Result of Llama-3.2-Vision on selected classification datasets in Med-MAT. ✓ in CG Helps indicates successful generalization, while ✗ denotes failure.

<table>
  <tr>
    <th>Related Combination</th>
    <th>Target Subset</th>
    <th>Baseline</th>
    <th>Trained</th>
    <th>CG Helps</th>
  </tr>
  <tr>
    <td>Lung Lung Det</td>
    <td>Bones Diseases</td>
    <td>Lung Diseases</td>
    <td>49</td>
    <td>52</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Lung Lung Det</td>
    <td>Breast Diseases</td>
    <td>Lung Diseases</td>
    <td>49</td>
    <td>54</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Bones Spinal Error Det</td>
    <td>Breast Diseases</td>
    <td>Bones Diseases</td>
    <td>20</td>
    <td>30</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Bones Spinal Error Det</td>
    <td>Lung Diseases</td>
    <td>Bones Diseases</td>
    <td>20</td>
    <td>33</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>End Level MRI</td>
    <td>Diseases Det COVID</td>
    <td>End Diseases</td>
    <td>24</td>
    <td>27</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>X-ray Lung Det CT</td>
    <td>X-ray COVID</td>
    <td>X-ray COVID</td>
    <td>23</td>
    <td>26</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Der - Skin Cancer Det FP - Fundus Diseases</td>
    <td>Der - Skin Diseases</td>
    <td>Der - Skin Diseases</td>
    <td>24</td>
    <td>29</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Mic - Cell Cancer Det CT - Kidney Diseases</td>
    <td>Mic - Cell Diseases</td>
    <td>Mic - Cell Diseases</td>
    <td>24</td>
    <td>26</td>
    <td>✓</td>
  </tr>
</table>

Table 13: Result of NEXT-Chat on CG by using detection and classification tasks to generalize classification Target dataset. Generalization results on classification datasets: Related Combination is the training set, Target Subset is the goal. Baseline and Trained represent the model’s accuracy without training and trained on related data, respectively. ✓ in CG Helps indicates successful generalization, while ✗ denotes failure.

<table>
  <tr>
    <th>Related Combination</th>
    <th>Target Subset</th>
    <th>Baseline</th>
    <th>Trained</th>
    <th>CG Helps</th>
  </tr>
  <tr>
    <td>Lung Lung Det</td>
    <td>Bones Diseases</td>
    <td>Lung Diseases</td>
    <td>41</td>
    <td>47</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Lung Lung Det</td>
    <td>Breast Diseases</td>
    <td>Lung Diseases</td>
    <td>41</td>
    <td>49</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Bones Spinal Error Det</td>
    <td>Breast Diseases</td>
    <td>Bones Diseases</td>
    <td>31</td>
    <td>35</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Bones Spinal Error Det</td>
    <td>Lung Diseases</td>
    <td>Bones Diseases</td>
    <td>31</td>
    <td>37</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>End Level MRI</td>
    <td>Diseases Det End X-ray</td>
    <td>End Diseases X-ray COVID</td>
    <td>24</td>
    <td>26</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>X-ray Lung Det CT</td>
    <td>COVID</td>
    <td>X-ray COVID</td>
    <td>22</td>
    <td>23</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Der - Skin Cancer Det FP - Fundus Diseases</td>
    <td>Der - Skin Diseases</td>
    <td>Der - Skin Diseases</td>
    <td>27</td>
    <td>30</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Mic - Cell Cancer Det CT - Kidney Diseases</td>
    <td>Mic - Cell Diseases</td>
    <td>Mic - Cell Diseases</td>
    <td>20</td>
    <td>24</td>
    <td>✓</td>
  </tr>
</table>

Table 14: Result of MiniGPT-v2 on CG by using detection and classification tasks to generalize classification Target dataset. Generalization results on classification datasets: Related Combination is the training set, Target Subset is the goal. Baseline and Trained represent the model’s accuracy without training and trained on related data, respectively. ✓ in CG Helps indicates successful generalization, while ✗ denotes failure.

keeping all other training parameters at their default settings. Similarly, for MiniGPT-v2, we trained the backbone model from the second stage, starting with a learning rate of 2e-5 and gradually reducing it to 2e-6 over 3 epochs.

A.7 CG with Medical Multimodal LLM

In previous experiments, general MLLMs are selected to prevent the MLLM’s inherent medical knowledge from affecting CG results. Our experiments focus on how MLLMs leverage CG to interpret unseen medical images. If the model has
<table>
  <tr>
    <th>Related Combination</th>
    <th>Target Subset</th>
    <th>HuatuoGPT</th>
  </tr>
  <tr>
    <td>Bones State</td>
    <td>Breast Diseases</td>
    <td>Bones Diseases +6.12</td>
  </tr>
  <tr>
    <td>Lung COVID</td>
    <td>Bones Diseases</td>
    <td>Lung Diseases +15.00</td>
  </tr>
  <tr>
    <td>X-ray Diseases CT</td>
    <td>COVID X-ray</td>
    <td>X-ray COVID +38.00</td>
  </tr>
  <tr>
    <td>X-ray Diseases CT</td>
    <td>State X-ray</td>
    <td>State +40.67</td>
  </tr>
  <tr>
    <td>CT Brain</td>
    <td>X-ray Lung</td>
    <td>Lung +1.5</td>
  </tr>
  <tr>
    <td>CT Brain</td>
    <td>X-ray Lung</td>
    <td>Lung +18.00</td>
  </tr>
  <tr>
    <td>FP - Fundus Diseases Mic - Cell Level</td>
    <td>FP - Fundus Level</td>
    <td>Level +12.12</td>
  </tr>
  <tr>
    <td>Mic - Cell Recognition FP - Fundus Level</td>
    <td>Mic - Cell Level</td>
    <td>Level +10.50</td>
  </tr>
</table>

Table 15: Result of HuatuoGPT-Vision on selected classification datasets in Med-MAT. HuatuoGPT represent the accuracy(%) gains the model achieved through CG.

learned some fundamental elements of the Target data, it would compromise the fairness of the experiments.

To demonstrate that our results still work on medical LLMs, we employed the same data combinations from Section 3.3 to investigate CG on medical MLLMs (we selected HuatuoGPT-Vision as the baseline).

The results in Table 15 demonstrate that the medical-expert MLLM can still leverage CG to enhance their performance on novel tasks, further supporting the validity and consistency of our findings.

B The Dataset: Med-MAT

This section provides an overview of Med-MAT. First, a detailed explanation of MAT-Triplet will be presented in B.1. Next, the methods for constructing the QA formatting will be discussed in B.2. Finally, the data composition details and open-source specification will be provided in B.3.

B.1 Details of MAT-Triplet

MAT-Triplet stands for Medical Modality, Anatomical Area, and Medical Task. We define all samples in Med-MAT using these three components and integrate datasets with identical triplets into subsets.

Medical Modality refers to different types of techniques or methods used in medical imaging or data acquisition. Each modality is designed to present the human body’s structures or pathological features in unique ways, providing auxiliary support for clinical diagnosis and treatment. Most modalities exhibit significant visual differences, making them easily distinguishable. Med-MAT encompasses 11 modalities, including common ones such as Computed Tomography (CT), Magnetic Resonance Imaging (MRI), X-ray, Fundus Photography (FP), Endoscopy (End), Optical Coherence Tomography (OCT), and Ultrasound (US), as well as rare and specialized modalities like Colonoscopy (Co), Dermoscopy (Der), Digital Pathology (DP), and Microscopy (Mic).

Anatomical Area refers to specific anatomical structures or regions within the human body or other organisms, defined by distinct anatomical characteristics to describe various body parts, their functions, and relative positions. Med-MAT encompasses 14 anatomical areas, including the cervix, kidney, lung, brain, intestine, bladder, fundus, retina, breast, bones, and chest. To further facilitate data description, additional categories such as skin, mouth, and cell are included as specialized anatomical areas.

Medical Task refers to the specific detection task that needs to be performed on the dataset. Med-MAT includes 13 distinct tasks, with classification tasks encompassing Quality Identification (image quality analysis), COVID Diagnosis, Cancer Diagnosis (determining the presence of a specific disease), State (such as identifying brain hemorrhage), Level Identification (assessing disease severity), and Multiple Classification (classifying multiple diseases or cell types). Given the limited options of COVID Diagnosis and Cancer Diagnosis, these tasks can be interpreted as identifying whether a patient is in a diseased state. To enhance generalization and provide more diverse examples, these tasks are grouped under the broader category of State. In addition, we have 16 datasets defining segmentation or classification tasks with different objectives.

B.2 QA construction method

A large amount of image-label datasets was collected to build the Med-MAT dataset. To ensure compatibility with MLLM training inputs and outputs, all data is transformed into a question-answering format. Questions are formulated based
on modality, anatomical area, and medical task, with 6 question prompts applied to each subset.

The labels within each data subset will be clustered to prevent redundant definitions of the same condition. Then, all training set and test set will be converted into multiple-choice questions following the template in Table 8. Each question will have up to four options, with distractor options randomly selected from the corresponding subset.

B.3 Data composition and Open-source Specification

Med-MAT is composed of multiple datasets. After being transformed into different QA formats, the new data is organized into several subsets to support generalization experiments in medical imaging.

Table 17 shows all of our subset datasets, which are separated based on different combinations in MAT-Triplet. The specific MAT-Triplets are listed, along with the labels corresponding to the image-label datasets for each subset. Correspondingly, all the image-label datasets are also displayed in Table 18, which includes their names, descriptions of the tasks performed, download links, and the level of accessibility.

All question-answering text datasets in Med-MAT will be publicly available. To accommodate varying access permissions, we will release datasets based on their respective licenses: openly accessible datasets will be directly available, while restricted datasets can be accessed by applying through the links provided in this paper. We hope this dataset will support and advance future generalization experiments on medical imaging.

B.4 Data Sources and Distribution

All Med-MAT data are sourced from public medical image challenges or widely used, high-impact datasets previously applied in deep learning training, ensuring reliable annotations. Before inclusion in Med-MAT, all datasets underwent label averaging where possible; test sets, in particular, were strictly balanced to ensure accuracy reliably reflects model performance. Each Med-MAT training subset contains 3,000 samples, while test sets maximize size under label balance constraints.

C Bad cases analysis and solutions

C.1 Bad case analysis

Some Trained models show minimal gains or even performance declines in Table 3, with classification accuracy lower than either the Baseline or Baseline+. After a thorough examination, we found that these Target datasets require more fine-grained medical condition classification. Beyond disease presence, they need detailed assessments, such as severity grading (e.g., bone age estimation, cancer staging) or distinguishing similar conditions (e.g., differentiating COVID-19 from pneumonia).

• The Related combinations lack suitable fundamental elements: For CG, the training data must include the Target task’s core elements. Here, we use other "level classification/grading" tasks for generalization, but their criteria differ significantly, misaligning with the Target task’s needs.

• Without defined grading standards, MLLMs lacking relevant knowledge can’t perform fine-grained tasks: Tasks like bone age assessment and cancer staging vary by criteria, and without this knowledge, MLLMs can’t accurately classify them.

C.2 Possible solutions

Few-shot prompting As we illustrated before, most of the bad cases involve fine-grained tasks needing specialized knowledge. So, in order to minimize the effect of a lack of relevant knowledge, we also conducted few-shot experiments to add some target images in the prompts. Subset X-ray, Lung, Normal-COVID-Pneumonia was chosen for its simple structure, with LLaVA as the baseline. We randomly sampled n images per label for n-shot inference and repeated each experiment 3 times.

<table>
  <tr>
    <th>Model</th>
    <th>0-shot</th>
    <th>2-shot</th>
    <th>3-shot</th>
    <th>4-shot</th>
  </tr>
  <tr>
    <td>LLaVA</td>
    <td>30.00</td>
    <td><b>28.83 ± 0.85</b></td>
    <td>29.33 ± 1.25</td>
    <td>29.83 ± 1.31</td>
  </tr>
  <tr>
    <td>LLaVA + CG</td>
    <td>28.00</td>
    <td>28.67 ± 0.94</td>
    <td><b>37.00 ± 0.82</b></td>
    <td><b>36.67 ± 0.47</b></td>
  </tr>
</table>

Table 16: Results of Few-shot prompting.

The results in Table 16 demonstrate that training with CG combinations can improve the few-shot performance of MLLMs on downstream tasks, even when direct CG generalization is not effective.

Adding some Target data in training As described in Section 5.2, we selected cases where CG alone couldn’t achieve satisfactory results and augmented their training sets with target data. The results in this section indicate that while CG may not directly enhance generalization, it accelerates the model’s adaptation to downstream tasks.
Multiple-choice Questions Template

<question>
A. <option_1>
B. <option_2>
C. <option_3>
D. <option_4>
Answer with the option’s letter from the given choices directly.

Figure 8: The Template of multiple-choice questions.

Q: Assess this brain CT image and reply 'Hemorrhage' if visible, or 'Normal' if no hemorrhage is present.
A. Hemorrhage   B. Normal

Q: Please examine this dermoscopic image and identify the condition present.
A. Benign skin lesion   B. Nevus   C. Malignant sinus   D. Genodermatosis

Q: Please examine this bladder image taken during endoscopy and determine the type of cancer present.
A. No tumor lesion   B. Low-grade carcinoma   C. High-grade carcinoma   D. Tumourous bladder

Q: You are viewing a mammogram. Kindly assess the conditions shown in the image, separating multiple conditions by commas. If no condition is present, return 'Normal'.
A. Normal   B. Asymmetry   C. Nipple retraction   D. Mass

Q: This is a spine X-ray image. Kindly assess the type of spinal pathology.
A. Surgical implant   B. Vertebral spondylosis   C. Other lesions   D. Osteophytes

Q: Shown here is an X-ray of a bone. Please determine the appropriate age category.
A. 7   B. 6   C. 9   D. 11

Q: This image contains human protein cells. Please list their types, separating each one with a comma.
A. Centromere, cytosol   B. Centromere, nucleoplasm   C. Filaments   D. Cytoplasm

Q: Review this chest X-ray image and specify the type of pneumonia if found, or return 'Normal' if no condition is present.
A. Secondary pnb   B. Covid   C. Left pnb   D. Right upper pnb

Q: Please examine this X-ray image of a bone and rate the severity of arthritis from 0 to 4.
A. Level 1   B. Level 2   C. Level 3   D. Level 4

Q: This is an OCT image. Please identify any condition present.
A. Drusen   B. Choroidal neovascularization   C. Normal   D. Age-related macular degeneration

Q: This is an image of cells taken under a microscope. Please identify all the cells present (separate multiple cell types with a comma).
A. Eosinophil, lymphocyte   B. Neutrophil, eosinophil   C. Lymphocyte   D. Neutrophil

Q: Please examine this microscopic image and determine the cancer grade of the prostate.
A. G3   B. No cancer   C. G4   D. G5

Q: Please assess this fundus image and list all conditions present, using commas to separate them. If no conditions are detected, return 'Normal'.
A. Retinal venous thrombosis   B. Central serous retinopathy   C. Choroidal neovascularisation   D. Media haze, optic disc pallor

Q: Review this image from the endoscopy and specify any intestinal diseases detected.
A. Polyp   B. normal result   C. Dyed lifted polyps   D. Dyed resection margins

Q: This is an image taken during an endoscopy. Please assess the cleanliness level of the intestine, rating it from 0 to 3.
A. 1   B. 1   C. 0   D. 0

Figure 9: Illustration of diverse samples with varying numbers of candidate options in the Med-MAT dataset.
<table>
  <tr>
    <th>Subset No.</th>
    <th>Modality</th>
    <th>Anatomical Area</th>
    <th>Task</th>
    <th>Datasets No.</th>
  </tr>
  <tr><td>01</td><td>Co</td><td>Cervix</td><td>Cervical Picture Quality Evaluation</td><td>1</td></tr>
  <tr><td>02</td><td>CT</td><td>Kidney</td><td>Kidney Diseases Classification</td><td>2</td></tr>
  <tr><td>03</td><td>CT</td><td>Lung</td><td>COVID-19 Classification</td><td>3,4,6</td></tr>
  <tr><td>04</td><td>CT</td><td>Lung</td><td>Lung Cancer Classification</td><td>5</td></tr>
  <tr><td>05</td><td>CT</td><td>Brain</td><td>Brain Hemorrhage Classification</td><td>7</td></tr>
  <tr><td>06</td><td>CT</td><td>Brain</td><td>Brain Cancer Classification</td><td>8</td></tr>
  <tr><td>07</td><td>Der</td><td>Skin</td><td>Melanoma Type Classification</td><td>10</td></tr>
  <tr><td>08</td><td>Der</td><td>Skin</td><td>Skin Diseases Classification</td><td>9, 11-15, 71, 72, 74</td></tr>
  <tr><td>09</td><td>DP</td><td>Mouth</td><td>Teeth Condition Classification</td><td>16</td></tr>
  <tr><td>10</td><td>DP</td><td>Mouth</td><td>Oral Cancer Classification</td><td>17</td></tr>
  <tr><td>11</td><td>End</td><td>Intestine</td><td>Intestine Cleanliness Level</td><td>18</td></tr>
  <tr><td>12</td><td>End</td><td>Bladder</td><td>Cancer Degree Classification</td><td>19</td></tr>
  <tr><td>13</td><td>End</td><td>Intestine</td><td>Intestine Diseases Classification</td><td>20</td></tr>
  <tr><td>14</td><td>FP</td><td>Fundus</td><td>Eye Diseases Classification</td><td>21-23, 26-28, 31, 32, 75</td></tr>
  <tr><td>15</td><td>FP</td><td>Fundus</td><td>Multiple-labels Eye Diseases Classification</td><td>24, 25, 68</td></tr>
  <tr><td>16</td><td>FP</td><td>Fundus</td><td>Blindness Level</td><td>29</td></tr>
  <tr><td>17</td><td>FP</td><td>Fundus</td><td>Retinal Images Quality Evaluation</td><td>30</td></tr>
  <tr><td>18</td><td>Mic</td><td>Cell</td><td>Cell Type Classification</td><td>33, 36-38, 39-41, 44, 65, 70</td></tr>
  <tr><td>19</td><td>Mic</td><td>Cell</td><td>Prostate Cancer Degree Classification</td><td>34</td></tr>
  <tr><td>20</td><td>Mic</td><td>Cell</td><td>Multiple-labels Blood Cell Classification</td><td>35</td></tr>
  <tr><td>21</td><td>Mic</td><td>Cell</td><td>Cancer Classification</td><td>42, 67</td></tr>
  <tr><td>22</td><td>MRI</td><td>Brain</td><td>Head Diseases Classification</td><td>44, 45</td></tr>
  <tr><td>23</td><td>OCT</td><td>Retina</td><td>Retina Diseases Classification</td><td>46, 47</td></tr>
  <tr><td>24</td><td>US</td><td>Breast</td><td>Breast Cancer Classification</td><td>48</td></tr>
  <tr><td>25</td><td>X-ray</td><td>Bones</td><td>Degree Classification of Knee</td><td>49, 53</td></tr>
  <tr><td>26</td><td>X-ray</td><td>Bones</td><td>Fractured Classification</td><td>50, 51</td></tr>
  <tr><td>27</td><td>X-ray</td><td>Bones</td><td>Vertebrae Diseases Classification</td><td>52</td></tr>
  <tr><td>28</td><td>X-ray</td><td>Lung</td><td>COVID-19 and Pneumonia Classification</td><td>54-57, 60, 62, 81</td></tr>
  <tr><td>29</td><td>X-ray</td><td>Breast</td><td>Breast Diseases Classification</td><td>58, 78</td></tr>
  <tr><td>30</td><td>X-ray</td><td>Lung</td><td>Tuberculosis Classification</td><td>59, 79</td></tr>
  <tr><td>31</td><td>X-ray</td><td>Chest</td><td>Multiple-labels Chest Classification</td><td>61, 73, 76, 77, 80, 85, 87</td></tr>
  <tr><td>32</td><td>X-ray</td><td>Brain</td><td>Tumor Classification</td><td>63</td></tr>
  <tr><td>33</td><td>Mic</td><td>Cell</td><td>Multi-labels Diseases</td><td>84</td></tr>
  <tr><td>34</td><td>FP</td><td>Fundus</td><td>Level Identification</td><td>66</td></tr>
  <tr><td>35</td><td>X-ray</td><td>Bones</td><td>Level Identification</td><td>69</td></tr>
  <tr><td>36</td><td>X-ray</td><td>Bones</td><td>Spinal lesion Classification</td><td>86</td></tr>
  <tr><td>37</td><td>X-ray</td><td>Breast</td><td>Multi-labels Diseases</td><td>82</td></tr>
  <tr><td>38</td><td>Der</td><td>Skin</td><td>Lesion Det/Seg</td><td>88-91</td></tr>
  <tr><td>39</td><td>End</td><td>Intestine</td><td>PolyP Det/Seg</td><td>92-93</td></tr>
  <tr><td>40</td><td>End</td><td>Intestine</td><td>Surgical Procedures Det/Seg</td><td>94</td></tr>
  <tr><td>41</td><td>End</td><td>Intestine</td><td>Multi-labels Det/Seg</td><td>95</td></tr>
  <tr><td>42</td><td>Mic</td><td>Cell</td><td>Cancer Cell Det/Seg</td><td>96</td></tr>
  <tr><td>43</td><td>US</td><td>Chest</td><td>Cancer Det/Seg</td><td>97</td></tr>
  <tr><td>44</td><td>US</td><td>Thyroid</td><td>Thyroid Nodule Region Det/Seg</td><td>98</td></tr>
  <tr><td>45</td><td>MRI</td><td>Intestine</td><td>Multi-labels Det/Seg</td><td>103</td></tr>
  <tr><td>46</td><td>MRI</td><td>Liver</td><td>Liver Det/Seg</td><td>104, 105</td></tr>
  <tr><td>47</td><td>X-ray</td><td>Lung</td><td>Lung Det/Seg</td><td>99</td></tr>
  <tr><td>48</td><td>X-ray</td><td>Lung</td><td>Pneumothorax Det/Seg</td><td>106</td></tr>
  <tr><td>49</td><td>X-ray</td><td>Bones</td><td>Spinal Anomaly Det</td><td>100</td></tr>
  <tr><td>50</td><td>X-ray</td><td>Chest</td><td>Multi-labels Det</td><td>101, 102</td></tr>
  <tr><td>51</td><td>FP</td><td>Fundus</td><td>Vessel Seg</td><td>107</td></tr>
  <tr><td>52</td><td>FP</td><td>Fundus</td><td>Optic Disc and Cup Seg</td><td>108</td></tr>
  <tr><td>53</td><td>FP</td><td>Fundus</td><td>Optic Disc Seg</td><td>109</td></tr>
</table>

Table 17: The details of subset. In particular, Co stands for Colposcopy, CT represents Computed Tomography, DP refers to Digital Photography, FP is for Fundus Photography, MRI denotes Magnetic Resonance Imaging, OCT signifies Optical Coherence Tomography, Der refers to Dermoscopy, End stands for Endoscopy, Mic indicates Microscopy Images, and US represents Ultrasound. The blue section represents the classification dataset and the green section represents the detection
<table>
  <tr>
    <th>No.</th>
    <th>Name</th>
    <th>Description</th>
    <th>Citation</th>
  </tr>
  <tr>
    <td>1</td>
    <td>Intel & MobileODT Cervical Screening</td>
    <td>Cervix Type in Screening</td>
    <td>(BenO et al., 2017)</td>
  </tr>
  <tr>
    <td>2</td>
    <td>CT Kindney Dataset</td>
    <td>Normal or Cyst or Tumor</td>
    <td>(Islam et al., 2022a)</td>
  </tr>
  <tr>
    <td>3</td>
    <td>SARS-COV-2 Ct-Scan</td>
    <td>COVID19, Classification Dataset</td>
    <td>(Soares et al., 2020)</td>
  </tr>
  <tr>
    <td>4</td>
    <td>COVID CT COVID-CT</td>
    <td>COVID19, Classification Dataset</td>
    <td>(Zhao et al., 2020)</td>
  </tr>
  <tr>
    <td>5</td>
    <td>Chest CT-Scan</td>
    <td>Cancer Classification</td>
    <td>(SunneYi, 2021)</td>
  </tr>
  <tr>
    <td>6</td>
    <td>COVID-19-CT SCAN IMAGES</td>
    <td>COVID19, Classification</td>
    <td>(w)Xiaochuangw, (2019)</td>
  </tr>
  <tr>
    <td>7</td>
    <td>Head CT</td>
    <td>Head Hemorrhage</td>
    <td>(Kitamura, 2018)</td>
  </tr>
  <tr>
    <td>8</td>
    <td>CT of Brain</td>
    <td>Head Cancer</td>
    <td>(Data, 2023)</td>
  </tr>
  <tr>
    <td>9</td>
    <td>MED-NODE</td>
    <td>Melanoma or Naevus</td>
    <td>(Giotis et al., 2015)</td>
  </tr>
  <tr>
    <td>10</td>
    <td>ISIC 2020</td>
    <td>Melanoma, Benign or Malignant</td>
    <td>(Rotenberg et al., 2021)</td>
  </tr>
  <tr>
    <td>11</td>
    <td>PAD-UFES-20</td>
    <td>Skin Multi Classification</td>
    <td>(Pacheco et al., 2020)</td>
  </tr>
  <tr>
    <td>12</td>
    <td>Web-scraped Skin Image</td>
    <td>Skin Desease Multi Classification</td>
    <td>(Islam et al., 2022b)</td>
  </tr>
  <tr>
    <td>13</td>
    <td>ISBI 2016</td>
    <td>Skin Lesion Classification</td>
    <td>(Gutman et al., 2016)</td>
  </tr>
  <tr>
    <td>14</td>
    <td>ISIC 2019</td>
    <td>Skin Desease Multi Classification</td>
    <td>(Combalia et al., 2019)</td>
  </tr>
  <tr>
    <td>15</td>
    <td>Skin Cancer ISIC</td>
    <td>Skin Cancer Multi Classification</td>
    <td>(Katanskiy, 2019)</td>
  </tr>
  <tr>
    <td>16</td>
    <td>Dental Condition Dataset</td>
    <td>Teeth condition classification</td>
    <td>(Sajid, 2024)</td>
  </tr>
  <tr>
    <td>17</td>
    <td>Oral Cancer Dataset</td>
    <td>Oral cancer Classification</td>
    <td>(RASHID, 2024)</td>
  </tr>
  <tr>
    <td>18</td>
    <td>The Nerthus Dataset</td>
    <td>Cleanliness level</td>
    <td>(Pogorelov et al., 2017a)</td>
  </tr>
  <tr>
    <td>19</td>
    <td>Endoscopic Bladder Tissue</td>
    <td>Cancer Degree Classification</td>
    <td>(Lazo et al., 2023)</td>
  </tr>
  <tr>
    <td>20</td>
    <td>Kvasir</td>
    <td>Multi Disease Classification</td>
    <td>(Pogorelov et al., 2017b)</td>
  </tr>
  <tr>
    <td>21</td>
    <td>ACRIMA</td>
    <td>Glaucoma</td>
    <td>(Ovreiu et al., 2021)</td>
  </tr>
  <tr>
    <td>22</td>
    <td>Augmented ocular diseases AOD</td>
    <td>Multi Classification of eye diseases</td>
    <td>(, 2021)</td>
  </tr>
  <tr>
    <td>23</td>
    <td>JSIEC</td>
    <td>Multi Classification of eye diseases</td>
    <td>(Cen et al., 2021)</td>
  </tr>
  <tr>
    <td>24</td>
    <td>Multi-Label Retinal Diseases</td>
    <td>Multi Classification of eye diseases</td>
    <td>(Rodríguez et al., 2022)</td>
  </tr>
  <tr>
    <td>25</td>
    <td>RFMiD 2.0</td>
    <td>Multi Classification of eye diseases</td>
    <td>(Panchal et al., 2023)</td>
  </tr>
  <tr>
    <td>26</td>
    <td>ToxoFundus(Data Processed Paper)</td>
    <td>Ocular toxoplasmosis</td>
    <td>(Cardozo et al., 2023)</td>
  </tr>
  <tr>
    <td>27</td>
    <td>ToxoFundus(Data Raw 6class All)</td>
    <td>Ocular toxoplasmosis</td>
    <td>(Cardozo et al., 2023)</td>
  </tr>
  <tr>
    <td>28</td>
    <td>Adam dataset</td>
    <td>Age-related Macular Degeneration</td>
    <td>(Liang, 2021)</td>
  </tr>
  <tr>
    <td>29</td>
    <td>APTOS 2019 Blindness</td>
    <td>Blindness Level Identification</td>
    <td>(Karthik et al., 2019)</td>
  </tr>
  <tr>
    <td>30</td>
    <td>DRIMDB</td>
    <td>Quality Testing of Retinal Images</td>
    <td>(Prentasici et al., 2013)</td>
  </tr>
  <tr>
    <td>31</td>
    <td>Glaucoma Detection</td>
    <td>Glaucoma Classification</td>
    <td>(Zhang and Das, 2022)</td>
  </tr>
  <tr>
    <td>32</td>
    <td>AIROGS</td>
    <td>Glaucoma Classification</td>
    <td>(de Vente et al., 2023)</td>
  </tr>
  <tr>
    <td>33</td>
    <td>ICPR-HEp-2</td>
    <td>Multi Classification</td>
    <td>(Qi et al., 2016)</td>
  </tr>
  <tr>
    <td>34</td>
    <td>SICAPv2</td>
    <td>Cancer Degree Classification</td>
    <td>(Silva-Rodriguez et al., 2020)</td>
  </tr>
  <tr>
    <td>35</td>
    <td>Blood Cell Images</td>
    <td>Blood Cell Classification</td>
    <td>(Mooney, 2017)</td>
  </tr>
  <tr>
    <td>36</td>
    <td>BreakHis</td>
    <td>Cell type and beginormag</td>
    <td>(Bukun, 2019)</td>
  </tr>
  <tr>
    <td>37</td>
    <td>Chaoyang</td>
    <td>Multi Classification of pathologists</td>
    <td>(Zhu et al., 2021a)</td>
  </tr>
  <tr>
    <td>38</td>
    <td>HuSheM</td>
    <td>Sperm Head Morphology Classificaion</td>
    <td>(Shaker, 2018)</td>
  </tr>
  <tr>
    <td>39</td>
    <td>Bone Marrow Cell Classification</td>
    <td>Bone Marrow Cell Classification</td>
    <td>(Matek et al., 2021)</td>
  </tr>
  <tr>
    <td>40</td>
    <td>NCT-CRC-HF-100K</td>
    <td>Multi Classification</td>
    <td>(Kather et al., 2018)</td>
  </tr>
  <tr>
    <td>41</td>
    <td>Malignant Lymphoma Classification</td>
    <td>Multi Classification</td>
    <td>(Orlov et al., 2010a)</td>
  </tr>
  <tr>
    <td>42</td>
    <td>Histopathologic Cancer Detection</td>
    <td>Cancer Classification</td>
    <td>(Cukierski, 2018)</td>
  </tr>
  <tr>
    <td>43</td>
    <td>LC25000</td>
    <td>Multi Classification of Lung and Colon</td>
    <td>(Zhu, 2022)</td>
  </tr>
  <tr>
    <td>44</td>
    <td>Brain Tumor 17 Classes</td>
    <td>Multi Classification</td>
    <td>(Felrin, 2022)</td>
  </tr>
  <tr>
    <td>45</td>
    <td>Tumor Classification</td>
    <td>Pituitary or Glioma or Meningioma or Notumor</td>
    <td>(Nickparvar, 2021a)</td>
  </tr>
  <tr>
    <td>46</td>
    <td>Malignant Lymphoma Classification</td>
    <td>Multi Classification of eye diseases</td>
    <td>(Orlov et al., 2010b)</td>
  </tr>
  <tr>
    <td>47</td>
    <td>Retinal OCT-C8</td>
    <td>Multi Classification of eye diseases</td>
    <td>(Subramanian et al., 2022)</td>
  </tr>
  <tr>
    <td>48</td>
    <td>BUSI</td>
    <td>Breast Cancer</td>
    <td>(Al-Dhabyani et al., 2020)</td>
  </tr>
  <tr>
    <td>49</td>
    <td>Digital Knee X-Ray Images</td>
    <td>Degree Classification of Knee</td>
    <td>(Gornale and Patravali, 2020)</td>
  </tr>
  <tr>
    <td>50</td>
    <td>Bone Fracture Multi-Region X-ray Data</td>
    <td>Fractured Classification</td>
    <td>(Nickparvar, 2021b)</td>
  </tr>
  <tr>
    <td>51</td>
    <td>Fracture detection</td>
    <td>Fractured Classification</td>
    <td>(Batra, 2024)</td>
  </tr>
  <tr>
    <td>52</td>
    <td>The vertebrae X-ray image</td>
    <td>Vertebrae</td>
    <td>(Fraiwan et al., 2022)</td>
  </tr>
  <tr>
    <td>53</td>
    <td>Knee Osteoarthritis Dataset</td>
    <td>Knee Osteoarthritis with severity grading</td>
    <td>(Chen, 2018)</td>
  </tr>
  <tr>
    <td>54</td>
    <td>Shenzhen Chest X-Ray Set</td>
    <td>COVID19, Classification Dataset</td>
    <td>(Jaeger et al., 2014)</td>
  </tr>
  <tr>
    <td>55</td>
    <td>Chest X-ray PD</td>
    <td>COVID and Pneumonia</td>
    <td>(Asraf and Islam, 2021)</td>
  </tr>
  <tr>
    <td>56</td>
    <td>COVID-19 CHEST X-RAY DATABASE</td>
    <td>COVID and Pneumonia</td>
    <td>(Chowdhury et al., 2020)</td>
  </tr>
  <tr>
    <td>57</td>
    <td>COVIDGR</td>
    <td>COVID19, Classification</td>
    <td>(Tabik et al., 2020)</td>
  </tr>
  <tr>
    <td>58</td>
    <td>MIAS</td>
    <td>Multi Classification of Breast</td>
    <td>(Mader, 2017)</td>
  </tr>
  <tr>
    <td>59</td>
    <td>Tuberculosis Chest X-Ray Database</td>
    <td>Tuberculosis</td>
    <td>(Rahman et al., 2020)</td>
  </tr>
  <tr>
    <td>60</td>
    <td>Pediatric Pneumonia Chest X-Ray</td>
    <td>Pneumonia Classification</td>
    <td>(Kermamy, 2018)</td>
  </tr>
</table>

Table 18: The details of the medical datasets are provided
<table>
  <tr>
    <th>No.</th>
    <th>Name</th>
    <th>Description</th>
    <th>Citation</th>
  </tr>
  <tr>
    <td>61</td>
    <td>Random Sample of NIH Chest X-Ray Dataset</td>
    <td>Multi Classification of Chest</td>
    <td>(Wang et al., 2017)</td>
  </tr>
  <tr>
    <td>62</td>
    <td>CoronaHack-Chest X-Ray</td>
    <td>Pneumonia Classification with Virus type</td>
    <td>(Praveen, 2019)</td>
  </tr>
  <tr>
    <td>63</td>
    <td>Brain Tumor Dataset</td>
    <td>Tumor Classification</td>
    <td>(Viradiya, 2020)</td>
  </tr>
  <tr>
    <td>64</td>
    <td>Fitzpatrick 17k (Nine Labels)</td>
    <td>Multi Classification</td>
    <td>(Groh et al., 2021)</td>
  </tr>
  <tr>
    <td>65</td>
    <td>BioMediTech</td>
    <td>Multi Classification</td>
    <td>(Nanni et al., 2016)</td>
  </tr>
  <tr>
    <td>66</td>
    <td>Diabetic retinopathy</td>
    <td>Diabetic Retinopathy Level</td>
    <td>(Benitez et al., 2021)</td>
  </tr>
  <tr>
    <td>67</td>
    <td>Leukemia</td>
    <td>Cancer Classification</td>
    <td>(Codella et al., 2019)</td>
  </tr>
  <tr>
    <td>68</td>
    <td>ODIR-5K</td>
    <td>Multiple Labels Classification</td>
    <td>(University, 2019)</td>
  </tr>
  <tr>
    <td>69</td>
    <td>Arthrosis</td>
    <td>Bone Age Classification</td>
    <td>(Zha, 2021)</td>
  </tr>
  <tr>
    <td>70</td>
    <td>HSA-NRL</td>
    <td>Multi Classification of pathologists</td>
    <td>(Zhu et al., 2021b)</td>
  </tr>
  <tr>
    <td>71</td>
    <td>ISIC 2018 (Task 3)</td>
    <td>Multi Classification</td>
    <td>(Codella et al., 2019)</td>
  </tr>
  <tr>
    <td>72</td>
    <td>ISIC 2017 (Task 3)</td>
    <td>Multi Classification</td>
    <td>(Codella et al., 2018)</td>
  </tr>
  <tr>
    <td>73</td>
    <td>ChestX-Det</td>
    <td>Multi Classification</td>
    <td>(Lian et al., 2021)</td>
  </tr>
  <tr>
    <td>74</td>
    <td>Monkeypox Skin Lesion Dataset</td>
    <td>Only Monkeypox</td>
    <td>(Ali et al., 2022)</td>
  </tr>
  <tr>
    <td>75</td>
    <td>Cataract Dataset</td>
    <td>Multi Classification</td>
    <td>(JR2NGB, 2019)</td>
  </tr>
  <tr>
    <td>76</td>
    <td>ChestX-rays IndianaUniversity</td>
    <td>Multi-label Classification</td>
    <td>(Raddar, 2019)</td>
  </tr>
  <tr>
    <td>77</td>
    <td>CheXpert v1.0 small</td>
    <td>Multi-label Classification</td>
    <td>(Arevalo, 2020)</td>
  </tr>
  <tr>
    <td>78</td>
    <td>CBIS-DDSM</td>
    <td>Multi Classification</td>
    <td>(Lee et al., 2017)</td>
  </tr>
  <tr>
    <td>79</td>
    <td>NLM-TB</td>
    <td>Tuberculosis</td>
    <td>(Karaca, 2022)</td>
  </tr>
  <tr>
    <td>80</td>
    <td>ChestXray-NIHCC</td>
    <td>Multi-label Classification</td>
    <td>(Summers and Ronald, 2020)</td>
  </tr>
  <tr>
    <td>81</td>
    <td>COVIDx CXR-4</td>
    <td>COVID19, Classification</td>
    <td>(Wang et al., 2020)</td>
  </tr>
  <tr>
    <td>82</td>
    <td>VinDr-Mammo</td>
    <td>Multi-label Classification</td>
    <td>(Nguyen et al., 2023)</td>
  </tr>
  <tr>
    <td>83</td>
    <td>PBC dataset normal DIB</td>
    <td>Multi Classification</td>
    <td>(Acevedo et al., 2020)</td>
  </tr>
  <tr>
    <td>84</td>
    <td>Human Protein Atlas</td>
    <td>Multi-label Classification</td>
    <td>(Le et al., 2022)</td>
  </tr>
  <tr>
    <td>85</td>
    <td>RSNA Pneumonia Detection Challenge 2018</td>
    <td>Multi-label Classification</td>
    <td>(Anouk Stein et al., 2018)</td>
  </tr>
  <tr>
    <td>86</td>
    <td>VinDr-SpineXR</td>
    <td>Multi Classification of Bones Diseases</td>
    <td>(Pham et al., 2021)</td>
  </tr>
  <tr>
    <td>87</td>
    <td>VinDr-PCXR</td>
    <td>Multi-label Classification</td>
    <td>(Pham et al., 2022)</td>
  </tr>
  <tr>
    <td>88</td>
    <td>PH2</td>
    <td>Melanoma Segmentation</td>
    <td>(Mendonca et al., 2015)</td>
  </tr>
  <tr>
    <td>89</td>
    <td>ISBI 2016 (Task3B)</td>
    <td>Melanoma Segmentation</td>
    <td>(Gutman et al., 2016)</td>
  </tr>
  <tr>
    <td>90</td>
    <td>ISIC 2016 (Task 1)</td>
    <td>Melanoma Segmentation</td>
    <td>(Gutman et al., 2016)</td>
  </tr>
  <tr>
    <td>91</td>
    <td>ISIC 2017</td>
    <td>Melanoma Segmentation</td>
    <td>(Codella et al., 2018)</td>
  </tr>
  <tr>
    <td>92</td>
    <td>CVC-ClinicDB</td>
    <td>Polyp Segmentation</td>
    <td>(Bernal et al., 2015)</td>
  </tr>
  <tr>
    <td>93</td>
    <td>Kvasir-SEG</td>
    <td>Polyp segmentation</td>
    <td>(Jha et al., 2020)</td>
  </tr>
  <tr>
    <td>94</td>
    <td>m2cseg</td>
    <td>Surgical Instrument Segmentation</td>
    <td>(Maqbool et al., 2020)</td>
  </tr>
  <tr>
    <td>95</td>
    <td>EDD 2020</td>
    <td>Multiple Diseases Segmentation in Intestine</td>
    <td>(Ali et al., 2020)</td>
  </tr>
  <tr>
    <td>96</td>
    <td>SICAPv2</td>
    <td>Cancer Cells Segmentation</td>
    <td>(Silva-Rodriguez et al., 2020)</td>
  </tr>
  <tr>
    <td>97</td>
    <td>BUSI</td>
    <td>Cancer Segmentation</td>
    <td>(Hesaraki, 2022)</td>
  </tr>
  <tr>
    <td>98</td>
    <td>TN3K</td>
    <td>Thyroid Nodule Segmentation</td>
    <td>(Gong et al., 2022)</td>
  </tr>
  <tr>
    <td>99</td>
    <td>NLM-TB</td>
    <td>Lung Segmentation (With left or right)</td>
    <td>(Gong et al., 2021)</td>
  </tr>
  <tr>
    <td>100</td>
    <td>VinDr-SpineXR</td>
    <td>Spinal X-ray Anomaly Detection</td>
    <td>(Pham et al., 2021)</td>
  </tr>
  <tr>
    <td>101</td>
    <td>VinDr-PCXR</td>
    <td>Multiple Diseases Segmentation in Chest</td>
    <td>(Pham et al., 2022)</td>
  </tr>
  <tr>
    <td>102</td>
    <td>ChestX-Det</td>
    <td>Multiple Diseases Segmentation in Chest</td>
    <td>(Lian et al., 2021)</td>
  </tr>
  <tr>
    <td>103</td>
    <td>UW-Madison GI Tract Image Segmentation</td>
    <td>Surgical Instrument Segmentation</td>
    <td>(Lee et al., 2024)</td>
  </tr>
  <tr>
    <td>104</td>
    <td>Duke Liver Dataset MRI v1</td>
    <td>Liver Segmentation</td>
    <td>(Macdonald et al., 2020)</td>
  </tr>
  <tr>
    <td>105</td>
    <td>Duke Liver Dataset MRI v2</td>
    <td>Liver Segmentation</td>
    <td>(Macdonald et al., 2020)</td>
  </tr>
  <tr>
    <td>106</td>
    <td>SHIM-ACR Pneumothorax Segmentation</td>
    <td>Pneumothorax Segmentation</td>
    <td>(Zawacki et al., 2019)</td>
  </tr>
  <tr>
    <td>107</td>
    <td>FIVES</td>
    <td>Fundus Vascular Segmentation</td>
    <td>(Jin et al., 2022)</td>
  </tr>
  <tr>
    <td>108</td>
    <td>RIM-ONE DL</td>
    <td>Optic Disc and Cup Segmentation</td>
    <td>(Batista et al., 2020)</td>
  </tr>
  <tr>
    <td>109</td>
    <td>PALM19</td>
    <td>Optic Disc Segmentation</td>
    <td>(Fu et al., 2019)</td>
  </tr>
</table>

Table 19: Continued from Table 18.

Reviewer paper summaries: This paper investigates compositional generalization (CG) as a key mechanism behind the generalization performance of MLLMs in medical imaging tasks. The authors introduce Med-MAT, a comprehensive dataset assembled from 106 existing datasets, structured around modality, anatomical area, and clinical task (MAT-triplets). Through rigorous experiments, the paper demonstrates that MLLMs effectively leverage CG to generalize across unseen MAT-triplet combinations, highlighting its importance in multi-task training scenarios.
Summary of strengths: The Med-MAT dataset is extensive and thoughtfully designed, which is valuable for future research.
The experimental design is robust and systematic, including ablation studies clearly demonstrating the role of CG.
Summary of weaknesses: Limited details are provided on dataset harmonization, raising concerns about heterogeneity, annotation quality, and task imbalance.
The selected MLLMs are general-purpose and not specifically tailored to the medical domain, which may limit the clinical relevance.
The theoretical contribution of CG itself is incremental; the study primarily validates an existing concept within the medical domain.
There is a minimal analysis of cases where CG failed, reducing the interpretability.
Comments, suggestions and typos: Incorporating statistical tests across multiple experimental runs would strengthen the confidence in the findings.
Please perform detailed qualitative or quantitative analyses of unsuccessful CG cases to identify the limits and potential improvements for CG.
Overall assessment scores: 4
Metareview score: 3.5
Metareview: The paper addresses compositional generalization in multimodal large language models (MLLMs) for medical imaging by introducing Med-MAT, a dataset created from 106 medical datasets organized into modality, anatomical area, and task (MAT-triplets). It shows that models like Llava, Llama, and Qwen can effectively combine learned elements to generalize to new MAT-triplet combinations, especially under multi-task training. The experiments highlight the role of compositional generalization in enabling data-efficient learning and better performance across classification and detection tasks, which offers a valuable framework for understanding and improving MLLM generalization in medical applications.
Summary of reasons to publish: - The paper constructs a dataset with explicit attribute annotations that enable the study of compositional generation (CG) in the medical domain. 
- The authors demonstrate how multimodal large language models (MLLMs) can recombine learned components to generalize to unseen images, providing robust empirical analyses through ablation studies and cross-task evaluations. 
- The paper highlights the effectiveness of CG in both classification and detection tasks and offers a valuable framework that can guide future medical MLLM research