Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks
Authors: Lehan Wang
Haonan Wang
Honglong Yang
Jiaji Mao
Zehong Yang
Jun Shen
Xiaomeng Li
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Large Language Model
Biomedicine
Region-Text
Primary area: foundation or frontier models, including LLMs
TL;DR: We present an interpretable, region-aware bilingual multimodal large language model for biomedicine with a large-scale region-centric dataset.

Abstract: Several medical Multimodal Large Languange Models (MLLMs) have been developed to address tasks involving visual images with textual instructions across various medical modalities, achieving impressive results. 
Most current medical generalist models are region-agnostic, treating the entire image as a holistic representation. However, they struggle to identify which specific regions they are focusing on when generating a sentence.
To mimic the behavior of doctors, who typically begin by reviewing the entire image before concentrating on specific regions for a thorough evaluation, we aim to enhance the capability of medical MLLMs in understanding anatomical regions within entire medical scans.
To achieve it, we first formulate \textbf{Region-Centric tasks} and construct a \textbf{large-scale dataset, MedRegInstruct,} to incorporate regional information into training. Combining our collected dataset with other medical multimodal corpora for training, we propose a \textbf{Region-Aware medical MLLM, MedRegA}, which is the first bilingual generalist medical AI system to simultaneously handle image-level and region-level medical vision-language tasks across a broad range of modalities. Our MedRegA not only enables three region-centric tasks, but also achieves the best performance for visual question answering, report generation and medical image classification over 8 modalities, showcasing significant versatility. Experiments demonstrate that our model can not only accomplish powerful performance across various medical vision-language tasks in bilingual settings, but also recognize and detect structures in multimodal medical scans, boosting the interpretability and user interactivity of medical MLLMs. The codes and model will be made publicly available.

Published as a conference paper at ICLR 2025
INTERPRETABLE BILINGUAL MULTIMODAL LARGE LANGUAGE MODEL FOR DIVERSE BIOMEDICAL TASKS
Lehan Wang', Haonan Wang', Honglong Yang", Jiaji Mao", Zehong Yang", Jun Shen?, Xiaomeng Li
ABSTRACT
Several medical Multimodal Large Languange Models (MLLMs) have been de- veloped to address tasks involving visual images with textual instructions across various medical modalities, achieving impressive results. Most current medical generalist models are region-agnostic treating the entire image holistic rep- resentation. However, they struggle to identify which specific regions they are focusing when generating sentence. To mimic the behavior of doctors, who typically begin by reviewing the entire image before concentrating on specific regions for thorough evaluation, we aim enhance the capability of medi- cal MILMS in understanding anatomical regions within entire medical scans. To achieve it, first formulate Region-Centric tasks and construct large- scale dataset, MedRegInstruct, to incorporate regional information into train- ing. Combining our collected dataset with other medical multimodal corpora for training, we propose a Region-Aware medical MILM, MedRegA, which is the first bilingual generalist medical AI system to simultaneously handle image-level and region-level medical vision-language tasks across broad range of modali- ties. Our MedRegA not only enables three region-centric tasks, but also achieves the best performance for visual question answering, report generation and medi- cal image classification over modalities, showcasing significant versatility, Ex- periments demonstrate that model can not only accomplish powerful per- formance across various medical vision-language tasks in bilingual settings, but also recognize and detect structures in multimodal medical scans, boosting the interpretability and user interactivity of medical MLLMs. Our project page is htns://meedrega.githhiiii
bon
Figure 1: MedRegA, an interpretable bilingual generalist model for diverse biomedical tasks, repre- sented by its outstanding ability to leverage regional information. MedRegA can perceive 8 modal- ities covering almost all the body parts, showcasing significant versatility.
*Corresponding to Xiaomeng Li (cexmli @ ust.hkk.. 'The Hong Kong University of Science and Technology. "Sun Yat-Sen Memorial Hospital, Sun Yat-Sen University.

---

Published as a conference paper at ICLR 2025
Plaase locate and describ the lesion scan.
w/ Region w/o Region
Locate Region
Cannot
Reglen-ggostlc -ernn loutorr
VnDD-Mamma
MedDr
Matastadis the left tem of
MedOr with surrounding edema, piocurate dugg
MedRegA Output
Region-Centee Centric
The lesian (resion) the right temporal and Hight besal gangla regdon, with high Heelhood of a highrgrade glome.
VnDD-SpinnXR
MedRooaA
Ground Truth
A mass in the right tempore occipital regioniadd and the posterior the right basal ganglai observed, suggesting malignant occupying lesion, with high possibiltt of glioblastoms.
CC Doctor
VinDr-CKR
Vinbn PCXR
(a)
(b)
Figure 2: The significance of Region-Centriy ability. (a) Comparison between the region-agnostic model (MedDr) and the region-centric MedRegA in analyzing lesion area within the medical scan. (b) Performance comparison of prompting the model with and without regional information on five benchmarks of Visual Question Answering (VQA) and classification tasks.
INTRODUCTION
In recent years, Multimodal Large Language Models have witnessed rapid advancements (Ope- nAI, 2023; Liu et al., 2024; Reid et al., 2024; Bai et al., 2023; Chen et al., (124b),which also appear promising for adaption into the healthcare field. By integrating visual and textual modalities, MLLMs can address diverse medical needs, such as patient consultations, medical report genera- tion, and disease diagnosis. Several medical Artificial Intelligence (AI) models have been proposed to tackle different tasks involving visual images with textual instructions across multiple medical modalities and achieved impressive results (Moor al., 2023; Li et al., 2024; Tu al., 2024; Wu et al., 2023; He et al., 2024; Zhang et al., 2024a). However, most current medical generalist models are region-agnostic, tailored to treat the entire image as a holistic representation, but struggling to detect or reason about specific regions within a medical scan. This lack of spatial awareness often leads to inaccurate descriptions that negatively affect the quality of the generated reports. As illus- trated in Figure 2 (a), region-agnostic models, such as MedDr (He et al., 2024), mistakenly identified the lesion area the left part of the brain, while the ground truth is located in the right part. This incorrect localization led MedDr to classify the lesion as surrounding edema instead of glioma, as radiologists noted. Consequently, this makes it difficult for clinicians to verify, trust, or revise the generated report, thereby hindering the model's interpretability and clinical usability.
The primary challenge arises because previous methods were not trained on region-centrie tasks. Specifically, medical reports often consist of descriptions of multiple regions, making it unclear which regions the model is addressing for each generated sentence. This ambiguity reduces relia- bility and does not align with the clinical workflow. Radiologists typically begin by reviewing the entire image and then concentrating on specific regions to ensure a thorough evaluation. Inspired by this process, we aim integrate region-centric information into medical MLLMs, mirroring the diologist's 's workflow. This approach is crucial as it ensures the model focuses on the correct regions being described, whether normal, abnormal, uncertain, or in any other state.
To address the above limitations, we first introduce three Region-Centrie tasks: (1) Region-to- Text Identification aims to identify structures, organs, or any abnormalities within given bounding box region as input: (2) Text-to-Region Detection aims to accurately locate the positions of structures or abnormalities described in the instruction by providing bounding boxes; (3) Grounded Report Generation aims to generate detailed reports with corresponding bounding boxes for associated anatomies in the medical image. To enable MIL.Ms to perform these three tasks, we constructed large-scale dataset, MedRegInstruct, encompassing all the above tasks. We collect real clinical data from Sun Yat-Sen Memorial Hospital, including approximately 25K Chinese scan-report pairs of X-ray, CT, and MRI modalities from 15K patients. Then, propose automatic labeling system to curate grounded reports, lowering the expense of manually annotating one-grained organs within each medical scan.
Combining our collected dataset MedRegInstruct with other medical multimodal corpora for train- ing, we propose Aware medical MILMM, MedRegA, which is the first bilingual general-
2

---

Published as a conference paper at ICLR 2025
ist medical AI system to simultaneously handle image-level and region-level vision- vision-language tasks across a broad range of modalities, such as radiology. pathology, dermatology, and ophthalmology; see Figure 1. Experiments presented in Figure 2 (b) show that our MedRegA can precisely out- put the bounding box of the specific region being focused on, while other existing best-performing medical MILMMs cannot. This significantly helps the model to correctly describe the condition of certain areas (Figure 2-a) and further enhances other downstream tasks such as visual question an- swering and diagnosis (Figure 2-b). In summary, our MedRegA outperforms MedDr by 3.91% % and 8.03% (BLEU-1) on MIMIC-CXR and IU-Xray in English report generation, with an additional im- provement of 27.34% for generating Chinese reports. Besides, our MedRegA not only enables three region-centric tasks but also achieves the best performance for visual question answering, report generation, and medical image classification over modalities, showcasing significant versatility.
To summarize, our contributions can be concluded as follows:
(1) We establish Region-Centric tasks with a large-scale dataset, MedRegInstruct, where each sample is paired with coordinates of body structures or lesions. This would expand the model's functionality to perceive regions within the medical scan, encouraging the model to focus on critical areas and improving interpretability. To benchmark the regional ability of medical MLLMs in per- forming those tasks, we propose a EeggionAAligned evaluation framework to assess the quality and alignment of the output text and regions.
(2) Based on the proposed dataset, we develop a Region-Aware medical MLLM, MedRegA, as a bilingual generalist medical AI system to perform both image-level and region-level medical vision- language tasks, demonstrating impressive versatility. In the inference stage, we introduce Regional CoT (Chain-of-Thought) to further enhance model performance with its spatial knowledge.
(3) We evaluate our model on comprehensive medical tasks, including visual question answering, report generation, image classification, region identification, region detection, and grounded report generation. MedRegA outperforms SOTA methods by large margin on traditional visual- visual-language and region-centric tasks.
2 RELATED WORK
Vision-Language Datasets for Medicine. In the advancement of Multimodal Large Language Models, there strong demand for image-text datasets that serve as pretraining data sources of MLLMs, as these datasets play crucial role in impacting model performance. The most universally applied vision-language datasets medicine including those for visual question answering (VQA) tasks (Lau et al., 2018; Liu et al., 2021; He et al., 2020; Zhang et al., 2023a; Chen et al., 2024a) and medical report generation (MRG) tasks (Johnson al., 2019; Hamamci et al., 2024). However, these datasets lack region annotations relevant to the texts, impeding MLLMs from further understanding structural information within medical scans. Huang et al. (2024); Xie et al. (2024) has attempted to close the gap by incorporating region grounding into multimodal conversations, but had limited access to manually written descriptions from real clinical scenarios. Although works of Zhang et al. (2024b); Lei et al. (2024) constructed region-wise reports paired with anomaly masks from clinical data, they concentrated on chest CT and brain MRI respectively, still leaving a blank space for other body structures such as colon, liver, and pancreas. As consequence, existing datasets have not met the requirements in developing generalist MLLM in healthcare, especially to comprehend regional information and accommodate diverse multimodal inputs from a wide range of medical systems. This poses significant challenges in creating a model that can effectively assist clinicians across various specialties.
Medical Multimodal Large Language Models. The rapid progression in Multimodal Large Lan- guage Models has driven the integration of MILMSs into the medical field to enhance diagnostic processes, medical research, and patient care. The development of large-scale medical models fol-
lows the trend of progressively scaling up the application scope. Several AI assistants are specif- ically designed for single modality. Given the abundant public datasets for Chest X-ray (CXR) images, Thawkar et al. (2023); Pellegrini et al. (2023); Chen et al. (2024c); Hyland et al. (2023) have facilitated MLLMs' ability understand CXR scans. Specifically for pathology, Lu et al. (2023) curated a large pathological image dataset and presented a pathological AI assistant. Since the large-scale parameters of MLLMS enable them to store knowledge across various modalities,
3

---

Published as a conference paper at ICLR 2025
extensive works enlarge the coverage of medical imaging to deploy more general and applicable medical MLLMs (Wu et al., 2023; Chaves et al., 2024; Bai et al., 2024; Zhang et al., 2023a; Moor et al., 2023; Li et al., 2024: Liu et al., 2023; Tu et al., 2024; He et al., 2024; Zhang et al., 2024a). However, without the capability of recognizing and detecting anatomies within medical scans, pre- vious models are still confronted with challenges in providing expert-like detailed descriptions of specific structures or lesions, which notably limits their reliability and interpertability.
Multimodal Large Language Models with Regional Ability. Enhancing the regional ability to perceive particular objects and ground responses with related areas is crucial for facilitating visual understanding, making a prevailing target in training MLLMs for natural images (Wang et al., 2023; Peng et al., 2023; Guo et al., 2024). However, enabling MLLMs to detect and ground fine- grained structures in medical scans remains largely underexplored. Current research primarily fo- cused on limited set of region-related tasks involving only a few modalities. For instance, models proposed by Huang et al. (2024); Alkhaldi et al. (2024); Zhou et al. (2024) are capable of detecting single structures based on the given prompts, but failing to delineate fine-grained descriptions for each region, as a clinician could. Bannur et al. (2024) introduced a grounded report generation task and developed MLLM to generate descriptions of all findings in medical image accompanied with corresponding location, but the model was restricted to receiving merely chest X-ray input. Based on these limitations, we aimed to fully integrate the function of recognizing, detecting and describing regions for a wide range of modalities into our proposed model.
3 REGION-CENTHN DATASET CONSTRUCTION
In this section, we introduce the MedRegInstruct dataset, which is composed of two subsets: (1) the Region-Text Dataset containing 550K triplets of image. question-answer pair, and corresponding regions, (2) the report annotated with regions and their fine- grained descriptions. Data distribution details can be found in Appendix A. Compared with the previous vision- languaee datasets for medicine (Liu et al., 2021; Johnson et al., 2019; Hamamci et al., 2024), our dataset not only places an extra empha- sis on cultivating the regional abilities of medical MLLMs, but also substantially enhances diversity by integrating a wider range of anatomical structures and bilingual setting for report generation.
Dataset including 240K instances where each image combines a
3.1 REGION-TERE TEXT DATASET
The creation of our Region-Text dataset stems from SA-Med2D-20M (Ye et al., 2023), which is large-scale segmentation dataset containing 2D medical images of almost all the body parts. We filter approximately 285K images from the original dataset to maintain the diversity of anatomical structures and anomaly lesions. The original labeled segmentation masks are converted into bound- ing boxes to represent the highlight region. The image-region pairs were formulated into two forms: (1) Region-to- Text: the model outputs the information of highlighted region given the bounding box; (2) Text-to-Region: the model locates the specific regions by generating bounding boxes. To obtain question-answer pairs, we fill the predefined templates with region names and coordinates. In statistics, the Region-Tex. dataset contains 550K data items, with approximately a half designated as Region-to- Text and the other half as Region respectively.
3.2 REPORT-GROUNDEI DATASET
The report-grounder dataset is sourced from two databases, MIMIC-CXR dataset (Johnson et al., 2019), and our in-house clinical data containing 25K X-ray, MRI and CT scans from 15K patients. The in-house data covers multiple regions, including the brain, abdomen, chest, spine, and pelvis. In this section, we will illustrate the automatic data processing procedure to curate grounded reports following three steps: Image-Report Pair Construction, Report Refinement and Structure Detection.
Image-Report Pair Construction. For the MIMIC-CXR dataset, we follow previous works (Wu et al., 2023) to utilize both frontal and lateral images, and include findings and impressions in the report. For our in-house dataset, we extract central slices from each 3D scan to formulate the 2D inputs, which typically provide the most representative views of the anatomical structures. In total, we construct 95K image-report pairs from the collected clinical data, which can also be employed to stimulate the bilingual abilities of MILM by enhancing it to generate structured reports in Chinese.
4

---

Published as a conference paper at ICLR 2025
Image-Repott Pair Construction
Image-Report Construction
FEEE.,
WEEBS: 0ah-40
Sign of onetmtatio addo and moderate plewnl
Rit. HEEA
H RERS. W
zort No paeunooess
1WX MINTH
W0 5. 2oh. Ba
Structure Detection
Report Refinement
kang Cuntie diicuent
Ingaess on: RIFAAR $88.9A #
Structure Detection
Organ Litt
Report Refinement
Gull beter
Gog
Lett beng Right lang ineleng sokumes uvere fuctinnet roddaets offexsory
Sagwe
Doctor
Finetuned MILM
proceet chube
Chest ImaGenome
persel Na supeerrenes
descending.omm
Organ
P.
www. Tean
EED000 BERLHE
Lesion Amotation
Organ Detection
WaReS Av BRRZHERS.
(R) MIMIC-CXR Detrest
(b) Our house Datreet
Figure 3: Data Construction Pipeline for Report-Grounded Dataset. The automatic data pro- cessing procedure composed of three steps: image-Report Pair Construction, Report Refinement, and Structure Detection.
Report Refinement. Our objective to decompose raw report data into fine- fine-grained descriptions for each organ mentioned in medical scans. We observe that the open-source InternLM model (Cai et al., 2024) demonstrates impressive textual comprehension ability for Chinese medical reports. To better leverage the LLM's information extraction ability, first implement rule-based strategy to summarize a coarse overview of all the organs referred in the report. Subsequently, we employed InternLM to segment each report into detailed descriptions given the organ list the prompt.
Structure Detection. For the MIMIC-CXR dataset, which has been labeled with bounding box coordinates of 29 anatomical locations by Chest ImaGenome (Wu et al., 2021), we select 12 stan- dardized ones in chest. Then, we combine the selected bounding boxes with the corresponding description in the segmented reports form the region-groundet reports for chest X-ray scans. Overall, 220K grounded reports are constructed from the MIMIC-CXR dataset. For our in-house dataset, certain scans are manually labeled with the lesion area, such as brain tumor, lung cancer, and other abnormalities. Nevertheless, the region labels of anatomical structures are still in demand. To automatically locate the observed organs within the medical scan, we first finetune an MLLM based on the Region-Text dataset, which contains scans of the same modalites and organs our collected data. Then, we input the collected images into the finetuned MLLM, prompted with the organs cov- ered in the reports, in order to acquire the corresponding bounding boxes. Subsequently, match the annotated abnormality coordinates with descriptions for the lesion part and the detected organ regions with corresponding parts in the segmented report.
Human Validation. To evaluate the quality of the automatically annotated part of our dataset and present quantitative assessment, randomly select 50 samples and asked experts to create manual labels for comparison. For Report Refinement, the sentence-level accuracy is 93.33%. For Structure Detection, the accuracy of generated bounding boxes achieves 72% compared with the human annotations. We also conducted visual evaluation and found that although most bound- ing boxes are slightly larger, they can still encompass the target region. This is sufficient for our approach since we only require localization rather than a tight and accurate bounding box.
4 REGION-AWARE MEDICAL MLLM
The Region-Aware Medical MLLM MedRegA served as an interpretable bilingual medical AI sys- tem to tackle variety of vision-language tasks, including visual question answering, report gener- ation and classification, with a specialized regional ability for Region-Centrr tasks. In this section, first introduce the formulation of Region-Centria tasks in model. Secondly, illustrate the training process the model. Finally, we propose a Region- Guided generation strategy aimed at further improving the inference quality.
5

---

Published as a conference paper at ICLR 2025
www.eerrerreeeebeeeeeeaae ass sorticess
Esd vnstiom begouett
cashaerdae Reeata
siwoeeee con pua oo duth cosose
Large Language Model
cadlogg heedd setraggnn beflehooor di ieneeralllpgggp panublly consoldaroos
Alignment
Alignment
Alignment
Vision Encoder
Tokenizer
Vision Encader
Tokenizer
Vision
Tokenizer
Image
Encoder
Text Token
Instruction Can me the rome of the
Intructior: where
Coorcinate Tokan
object 2b2x 482,
wit ventricle and right AUlI
reglon decortze conditon of each stnecture
740fleboss
<refz Token
witlin theches Kityy scan.
<hoo> Token
Region
Region Detection
Report
Figure 4: An illustrative example of performing region-centrie tasks with MedRegA.
Instractions
Insruction
reglonanes indidd dhooj 414, 386, 438) (016, 200, 687, 204jphea
Anewar Indan X
Arswer Ostesplytes,
impler O
MedRegA
MedRegA
Stage
Stage
Thathional Inlane
80 Angionall Oet
Figure 5: Comparison of traditional inference and generation pipeline with Regional CoT.
Region-Centrr Task Formulation. We define the Region-Centric tasks from three perspectives: (1) Region-to-Text Identification, where the model outputs the name of the specified region; (2)
Text-to-Region Detection, where the model detects the area of the given organ or anomaly; (3) Grounded Report Generation, where the model generates report for the medical scan, aligning each description with the corresponding region. To enable the model to encode the regional information, represented regions in the format of bounding boxes [I1, V1; 22, 1/3. where (æ1, indicate the
top left point of the box and (xz, y2) denotes the bottom right one. All the values are normalized into integers within the range of [0, 1000). These bounding boxes were inserted into the sentences surrounded by pair of special tokens, like <box> 11,, , 22, ///</boxxx while the correlated object was marked with <ref></ref>, as illustrated in Figure
Model Training. To train a general medical MLLM not limited to understanding regions, we pro- cess and integrate various public datasets targeted at different tasks, including visual question an- swering, report generation, organ classification, and disease diagnosis. We design task-specific instructions to prompt the model to recognize and address different objectives. Considering the competitive performance of the open-source model InternVL 1.2 (Chen et al., 2024b), utilize as the
as our general-domait foundation to begin training. which is composed of vision encoder, and Hous-Hermes--2Yi. 2-i--34B the language model. Our training process is divided into two steps: alignment training and instruction tuning. During the alignment training phase, we freeze the vision encoder and language model, only fine-tuning the alignment module with medical image captioning datasets. In the instruction tuning stage, we apply both public datasets and our Region-Centie Centrie datasets, MedRegInstruct, optimize the language model, while keeping the other components unchanged. The language model loss is applied as the loss function. More training details can be found Appendix B.
Regional CoT. After training on Region-Centria tasks, the model has been equipped with region identification and detection abilities, which have the potential to further enhance model performance on other general tasks. As proved in Zhang et al. (2023b); Yang et al. (2023), textualizing additional guidance such as spatial coordinates into the prompt would unleash the language model's ability to process multimodal information. Following these insights, incorporate regional CoT into the generation pipeline to better leverage these regional skills, as illustrated in Figure 5. The model is required to initially detect critical regions in the input image, followed by generation prompted by the detected regions. This approach could encourage the model to attend more to internal structures within the medical scan when answering patient consultations diagnosing diseases.

---

Published as a conference paper at ICLR 2025
Table I: Performance comparison on general medical tasks and region-centric tasks. 'N/A' means the scores are not reported. indicates the model cannot generate valid outputs. x denotes the model cannot tackle corresponding tasks. **' means that the model is fine-tuned on the dataset.
Task
Metrics
Med-Flaninno
Lea RadFM
MedDr
BlomedGPT
InternVL
MedRegA
Visnal Onestion Amswrinng
BLEU-1
65.72
English Visual
Score
22.28
$1.81*
51.36
66.87
NIA
65.05
68.33
Question Answering
AlosrAccuracy
OpenRecall
41.91
12,03
63,51
43.73*
61.01
42.63
49.81
$4.03
86,40"
57.73*
44.17
64.99
83.25
53.35
BenSoore
BLEU-T
10.28
22.17
11.30
37.76
39.73
611.99
Chinese Visual
Question Answering
Accurcc
FI Score
11.64
22.32
12.32
11.63
39.86
10.95
62.66
Recall
16.85
23.23
14.43
38.77
40.20
62.01
BetScore
59.10
28.21
65.75
87.55
87.58
91.63
Generazion
BLEU-1
21.30
17.47
29.06
34,49
NIA
13:12
40.46
BLEU-1
2,63
0.28
1.30
10.02
NIA
0.53
12.60
English Report Generation
METEOR
17.30
12.68
19.38
27.70
13.55
14.26
31.94
ROUGE-L
BerScare
49.81
44.34
53.43
55.12
NIA
49.26
62.54
BLBU-1
3.59
4.91
3,66
11.99
10.71
40.76
BLEU-1
Chinese Report Generation
METEOR
2.59
4.05
10.71
NIA
9:68
34,03
ROUGE-L
2.98
3.58
4.46
8.44
12.90
22.98
BettSeure
58.16
57.22
62.54
61.39
61.14
71.11
Imagge Claszification
Single-Label Classlfcation
Multi-Labe Classification
FI Score
Score
16.99
2.53
22.81
5.27
18.29
8.32
32.65
9.38
NIA
21.13
5.16
47.97
13.32
Region
BLEU-T
0.13
0.13
0.35
0.15
0.13
69.72
F1
0.72
1.15
0.80
1.27
0.21
70.43
Region-te to-ttt---eee Identification
3.52
Accuracy
0.01
1.74
1.88
1.36
66,24
BenScore
24.87
34.53
37.07
50.28
49.85
87.13
Fl
56.60
77.93
Text-to-Resee Re--ein Detection
Region
38.24
Alignment F1
X
X
X
X
X
5.45
36,53
JoU
12:28
23.43
Keport BLEU.T
22.32
27.43
Grounded Report Generation
Region Acc
X
X
0.57
76.59
lignment Ace
X
X
0.12
62.29
lol
0.38
$2,07
EXPERIMENT
5.1 PERFORMANCE ON GENERAL MEDICAL TASKS
To evaluate the ability of MedRegA on general medical tasks, namely visual question answering. report generation and classification, we comprehensively compare our model with the base model InternVL. (Chen et al., 2024c) and open source medical MLLMs in general domain, including Med- Flamingo (Moor et al., 2023), LLaVA-Med (Li et al., 2024), RadFM (Wu al., 2023), MedDr (He et al., 2024), and BiomedGPT (Zhang et al., 2024a) by reproducing their released checkpoints with official prompting instructions. An overall performance comparison is shown in Table with aver- aged results on each task. It should be notified that the existing baselines cannot manage to locate highlighted areas within the medical scan and provide valid regional outputs, exposing huge limi- tation in differentiating body structures and detecting fine-grained abnormalities.
Performance Visual Question Answering. In Visual Quesion Answering task, the model needs to answer questions involving the modality, visible structures, and possible diseases of the given medical scan. We perform comparison on several medical VQA benchmarks, such as English and Chinese versions of SLAKE (Liu et al., 2021), VQA-RAD (Lau et al., 201 8), and PathVQA (He al., 2020). The averaged results on these datasets revealing that MedRegA outperforms all the baselines in the overall metrics by approximately 2% to 40% in English and over 10% in Chinese.
Performance on Medical Report Generation. Medical Report Generation task requires the model to generate a detailed report based on the provided medical scan. For English report generation, we evaluate our model on the report generation task for chest X-ray datasets MIMIC-CXI (Johnson Hrmmner-Fushman et al., 2016). Moreover, to evaluate the bilingual ability
al., 2019) and IU-Xray of our model in report generation, we collect a Chinese report generation test dataset covering brain, chest, spine, abdomen and pelvis from the hospital. Table 1 shows that MedRegA excels all other

---

Published as a conference paper at ICLR 2025
Table Performance on the region-to-text identification task.
Subtask
Metries
Med-Flamingo
LLaVA-Med
RadFM
MedDr
InternVL
MedRegA
BLBU-1
0.24
0.35
0.22
0.93
0.13
78.34
1.33
78.95
Structure
Identification
Recall
9.4
7.33
2.00
4.66
0.52
79.02
Accuracy
1.3
2.62
2.71
73.06
BertScore
26.58
34.88
38.28
51.18
51.78
91.63
BLEU-1
0.51
0.47
61.09
0.10
1.35
0.20
61.9
Lesion
Identifccation
Recall
0.39
14.05
7,49
2.37
0.51
63.07
Accuracy
0.02
2.18
1.14
59,42
BertSoore
23.16
34.18
35.86
49.38
47.92
82.62
Multi-bjjet Singe-redon
Multi-object Multireggon
Region
Loject
Reglon-Oby Alignment
Object-Level
Region Object Alignment
not detected
Object Region aigned
small howel denected
doodenen
Dedendemum
Region not detected
Region
detected
Banones
pancress
Ciject &. Sagion alinned
Teart Object sligned
Y.stomach
C.dunderum
VLaptam
object d region not detected
Precision #Detected/ AAlI, Recall -DDetected ##Predicted
Figure 6: Definition of Region-Level, Object-Level, and Object-Region Alignment evaluation. Boxes labeled with roman numerals denote groundtruths and those with capital letters denote pre- dictions.
baselines in generating reports with both English and Chinese, achieving 5.97% and 28.77% higher averaged BLEU-1 score compared with MedDr, respectively.
Performance Medical Image Classification. In Medical Image Classification task, the model is expected to either classify the medical structure shown in the image or diagnose the specific diseases indicated by the scan. We conduct experiments on both single-label classification and multi-label classification with a wide range of datasets across radiology, ultrasound, ophthalmology, and der- matology. We report the averaged Fl score in Table 1. For single-label classification, MedRegA outperforms existing models by large margin from 15.32% to 30.98%. Multi-label classification appears more challenging for MLLMs due to the difficulties in decoupling subtle symptoms and relating them to corresponding diagnoses. To enhance the model's focus on disease lesions, we employ Regional CoT into multi-label classification, and present the results in Figure 2 (b). Specif- ically, with Regional COT. the model achieved Fl score of 61.75% VinDr-SpineXR dataset, surpassing MedDr and MedRegA without Regional CoT by and 34.95% and 31.59%, respecitvely. The improved results indicate that incorporating regional information assists medical MLLMs in establishing explicit relationships between local regions and each class label, rather than consider- ing the entire global image with all labels (as illustrated in Figure 5), which further enhances image perception and handles multi-label classification.
5.2 REGION- -ELGNED EVALUATION ON REGION-CENTRES TASKS
To evaluate regional ability of MedRegA, we implement our model on the proposed Region- Centric Tasks. To quantitatively evaluate the regional perception and comprehension capabilities of medical MLLMs, we introduce a Region-Aligned evaluation framework to measure the model performance these tasks. In this section, first present the assessment metries for each task and demonstrate the experimental results. Due to the inability existing open-souce medical MLLMs generate precise coordinates, apply MILM with impressive regional ability natural images, Intern VL. (Chen et al., 2024b), as the compared baseline.
5.2.1 EVALUATION ON REGION-TO- TEXT IDENTIFICATION
Settings. In the Region-to- Text Identification task, the model is required to identify the name of the specified region based the corresponding bounding box. Since the output for this task is in the form pure texts, we adopt Natural Language Generation (NLG) metrics for performance measurement, including BLEU-1, F1 score, Recall, Accuracy and BertScore.
8

---

Published as a conference paper at ICLR 2025
Table 3: Performance on the text-to-region detection task. 'N/A' means that the task does not correspond to the evaluation dimension.
Subtask
Vodel
Object-Leve Metries
Region-I.evv Level Metries
Object-Region Alignment Metries
ToU
Single-ubject Single-region
Accuracy 1.2
InternVL MedRegA
N/A
NA
16.61
45.11
42.70
Single-ohject
Precision
Recall
F1 Score
Single-region
InternVL. MedRegA
5,40
5.45
13.99
31.15
25.48
24.78
31,63
Multi-ohject
Precision
Precision
Recall
Score
Precision
Score
Siugle-reglon
InternVL MedRegA
85002
71.87
67.13
7.86
TO.56
8.69
6.28
7.1
6.38
14.48
87.85
48.11
48.13
48.12
44,91
44,93
Multi-object Mulli-region
Precision
Recall
FT
Precision
Rocall
FT Score
Precision
Recall
FT Score
InternVI.
43.42
53.40
46.06
4.90
7.49
5.44
4.08
5.04
4.32
4.05
MedRegA
62.90
8182
68.03
26.15
27.04
34.94
29.17
33.51
28.14
15.23
Table 4: Performance Chest X-ray Grounded Report Generation. x denotes the model
cannot tackle corresponding tasks.
NLG Metries ROUGE-L
CE Metries RadGraph
Region-Aligned Metries
Model
IoU
BLEU-1
BertScore
CheXBert 25.37
RadCliq
Object
Region
Aligument
MedDr
27.43
18.23
53.00
14.76
1.14
X
X
X
InternVI.
19.46
12.20
16.07
26.75
9.33
1.42
0.23
0.57
0.12
0.38
MedRegA
33.18
21.64
55.37
39.00
25.23
0.96
62.65
76.59
62.29
$2.07
MedBr Pradictio
Metrics
MedDr
Intern
MedRegA
RE ZAL Wis 2
BRAKERRS
BLEU-T
11.79
3.76
20.04
ROUGE-
8.38
4.52
14.01
8M: 838 R3 RaI d:3 2tt.://88iiee0.......... (it1 2007
BertScore
66.59
63.22
74.11
Region Ace
X
25,80
Seoouna
Arl: TIIEEEEEEERRRR BESTER
IoU
X
2.06
35.63
CRR 10N AARORAAAATT I
Table 5: Performance on Lesion Grounded Report Generation. 'x' denotes the model cannot tackle corre- sponding tasks. '-' indicates the model cannot generate valid outputs.
nustel
tawearceed
Figure 7: Examples on Organ Grounded Report Generation. The predicted bounding boxes are labeled with the same color as the detected organ generated texts.
Results. We categorize the task into (1) structure identification for identifying anatomics such as structures within the brain, heart, lung, abdomen spine, and (2) lesion identification focusing abnormalities like tumors and cancers. Results of these two subtasks are reported in Table 2. can be observed that MedRegA is able to accurately recognize the regions delineated by bounding boxes and associate them with the corresponding areas with 39.85% higher BertScore than Intern VL. while previous methods fail comprehend region encodings medical scans. Notably, the model demonstrates a stronger capability of identifying body structures compared to lesions, possibly be- cause anatomies tend to be larger, whereas lesions are subtler and exhibit more variation in shape.
5.2.2 EVALUATION ON TEXT-TO-REGION DETECTION
Settings. In the Text-to-Region Detection task, the model should detect the region corresponding to specific organs or anomalies. The input textual instruction contains the name of single multiple objects, and the model is expected to output bounding boxes for all the relevant regions. We classify the task into four categories according to the number of detected objects and the number of regions per object: single-object single-region. singlc-object multi-region, multi-object single-region, and multi-ohject multi-region, as illustrated Appendix 1. We evaluate the performance across three dimensions: (1) Object-Level assessing the correctly identified objects; (2) Region-Level: as- sessing accurately detected regions; (3) Object-Region Alignment: assessing whether the detected boxes are correctly aligned with the corresponding object. For single-object detection, all the out- put boxes are aligned with the given object, where evaluating the detection performance from the region-level is sufficient. The region is correctly detected if its Intersection over Union (IoU) score exceeds the threshold of 0.5. For multi-object detection, the alignment of text and box must also be considered. Figure 6 illustrates the three-dimensional evaluation in the multi-object setting.

---

Published as a conference paper at ICLR 2025
Results. Table 3 demonstrate the model performance for fext-to-Region Detection. When detecting objects corresponding to a single region, our model achieves relatively high scores, with detection accuracy approaching 50%. However, when multiple regions associated with the given objects, the detection task becomes more complex for MLLMs. In such cases, recall is higher than precision, inferring that the model suffers from difficulty in detecting all the required regions thoroughly and tends to conservatively generate fewer bounding boxes.
5.2.3 PERFORMANCE ON GROUNDED REPORT GENERATION.
Chest X-ray Grounded Report Generation. For the grounded reports constructed from MIMIC- CXR, we follow the same training-test split as the report generation task and evaluate the model with those accompanied by organ bounding boxes, consisting of 3,022 samples total. The model per- formance on region detection is assessed as the multi-region single-object sub-task defined in $5.2.2. Furthermore, we integrate NLG and CE metrics to estimate the quality of the descriptions for each region. As shown in Table 4, model can detect the chest structures and generate descriptions simultaneously with impressive scores on both report generation and region detection.
Lesion Grounded Report Generation. To evaluate our model performance on generating grounded reports that concentrate on the lesion area, we sample 438 pairs of tumor coordinates and corre- sponding Chinese descriptions annotated by doctors from our collected data. The evaluation metrics can be referred to as the single-region single-object sub-task defined in $5.2.2 since each medical scan associated with a single area of abnormality. NLG metrics are applied evaluate the de- scriptions. Table 5 demonstrates that our model outperforms established baselines, especially with 33.57% higher IoU than Intern
Organ Grounded Report Generation. As illustrated in $3.2, we enlarge the collected Chinese image-report data with automatically-annotaa self-training with these data, MedRegA has the capability to yield grounded reports with detailed descriptions on organs, as the examples Figure 5. Without training on region-centric tasks, MedDr can only generate broad description for each organ and fail to relate fine-grained descriptions with organ coordinates. contrast, MedRegA managed to ground generated sentences with correspond- ing organs, and generated more detailed description there are subtle deviation in the nodule position, MedRegA still provided a thorough insight on the medical scan, exhibiting remarkable ability to leverage regional information.
organs to enrich the report- report-grounded data. Through
the internal condition of the organ. Though
CONCLUSION
In this paper, we present an interpretable bilingual generalist medical Al system with an addi- tional intention to enhance the ability of medical MLLMs to investigate critical regions within the given medical scan. From data aspect, we first formulate Region-Centric tasks tailored to re- gion identification, detection, and fine-grained grounded report generation for each recognized re- gions. Specifically based on these tasks, we construct a large-scale dataset, MedRegInstruct, with a semi-automatic labeling system. In terms of model, we propose Region-Aware medical MLLM, MedRegA, by leveraging our constructed dataset and multimodal medical corpora across diverse tasks for training. Experiments demonstrate that MedRegA achieves promising results on both im- age and region-level tasks. We believe that region-centric capability is essential medical MILM development, since establishing relations between specific regions and generated texts would not only encourage the model to focus on critical areas, but also facilitate interpretability and clinical interactivity.
ACKNOWLEDGEMENT
This work was supported by grants from the National Natural Science Foundation of China (NSFC) and the Research Grants Council (RGC) of Hong Kong under the Joint Research Scheme (JRS), Project No. N HKUST654. was also supported by the Research Grants Council of the Hong Kong Special Administrative Region, China (Project Reference Number: T45-401/22-N).
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
The in-house dataset used in our research consists of existing retrospective data collected from the hospital, and is merely intended to advance research in healthcare. All data have been anonymized. Personal identifiers such names, addresses, and contact details have been removed to protect the privacy and confidentiality of individuals. The data is ensured to be analyzed and interpreted responsibly avoid any potential negative impact patients or individuals. We will only release the constructed data derived from public datasets. We understand that we are solely responsible for any legal violations with respect our benchmark, and accept all the risks associated with research and any data we release.
REFERENCES
Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy. Dataset of breast ultra- sound images. Data in brief, 28: 28:104863, 2020.
Asma Alkhaldi, Raneem Alnajim, Layan Alabdullatef, Rawan Alyahya Jun Chen, Deyao Zhu, Ahmed Alsinan, and Mohamed Elhoseiny. Minigpt-med: Large language model as a general interface for radiology diagnosis. arXiv preprint arXiv:2407.04106 2024.
Fan Bai, Yuxin Du, Tiejun Huang, Max Q-H Meng, and Bo Zhao. M3d: Advancing 3d medical image analysis with multi-modal large language models. arXiv preprint arXiv:2404.00578, 2024.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond. 2023.
Shruthi Bannur, Kenza Bouzid, Daniel C Castro, Anton Schwaighofer, Sam Bond-Taylor, Maxi- milian Ilse, Fernando Pérez-Gareía, Valentina Salvatelli, Harshita Sharma, Felix Mcissen, et al. Maira-2: Grounded radiology report generation. arXiv preprint XXX::2406.004499 2024.
ButterflyNetworklne Butterfly network ultrasound dataset. Website, 2018. https: /
com/
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint urXiv:2403. 17297. 2024.
Juan Manuel Zambrano Chaves, Shih-Cheng Huang, Yanbo Xu, Hanwen Xu, Naoto Usuyama, Sheng Zhang, Fei Wang. Yujia Xie, Mahmoud Khademi, Ziyi Yang, et al. Training small multi- modal models to bridge biomedical competency gap: A case study in radiology imaging. arXiv preprint ar Xrxi:22403.0R 08000, 2024.
Junying Chen, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji. Guangjun Yu, al. Huatuogpt-visionn towards injecting medical visual knowledge into multimodal Ilms at scale. arXiv preprint arXiv:2406. 0114406.192800 19280, 2024a.
Yixiong Chen, Chunhui Zhang, Li Liu, Cheng Feng, Changfeng Dong, Yongfang Luo, and Xiang Wan. Uscl: pretraining deep ultrasound image diagnosis model through video contrastive rep- resentation learning. In Medical image computing and computer assisted intervention-MICC. 2021: 24th International conference, Strashourg, France, September 27-October 1, 2021, Pro- ceedings, Part VIII 24, pp. 627-633 Springer, 2021.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, gpt-4v? closing the gap com-
Kongzhi Hu, Jiapeng Luo, Zheng Ma, al. How far mercial multimodal models with open-source suites. arXiv preprint arXiv: 16821, 2024b.
Zhihong Chen, Yan Song. Tsung- Hui Chang, and Xiang Wan. Generating radiology reports via memory-driven transformer. arXiv preprint arXiv:2010.16055. 2020.

---

Published as a conference paper at ICLR 2025
Zhihong Chen, Maya Varma, Jean-Benoi Delbrouck, Magdalini Paschali, Louis Blankemeier, Dave Van Veen, Jeya Maria Jose Valanarasu, Alaa Youssef, Joseph Paul Cohen, Eduardo Pontes Reis, et al. Chexagent: Towards a foundation model for chest x-ray interpretation. arXiv preprint 20v:24001122008 2024c.
Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gut- man, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv: 09020033....,, 2019.
Mare Combalia, Noel CF Codella, Veronica Rotemberg, Brian Helba, Veronica Vilaplana, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Allan C Halpern, Susana Puig, et al. Bcn20000: Dermo- scopic lesions the wild. arXiv preprint arXiv: 02288, 2019.
Dina Demner-Fushman, Mare D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. Preparing collection of radiol- ogy examinations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2):304-310, 2016.
Jevgenij Gamper, Navid Alemi Koohbanani, Ksenija Benet, Ali Khuram, and Nasir Rajpoot. Pan- nuke: an open pan-cancer histology dataset for nuclei instance segmentation and classification. In Digital Pathology: 15th European Congress, ECDP 2019, Warwick, UK, April 10-13, 2019,
Proceedings 15, pp. 1 11-19. Springer, 2019.
Qiushan Guo, Shalini De Mello, Hongxu Yin, Wonmin Byeon, Ka Chun Cheung, Yizhou Yu, Ping Luo, and Sifei Liu. Regiongpt: Towards region understanding vision language model. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13796- 13806, 2024.
Ibrahim Ethem Hamamci, Sezgin Er, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Dogan, Muhammed Furkan Dasdelen, Bastian Wittmann, Enis Simsar, Mehmet Simsar, et al. A foundation model utilizing chest et volumes and radiology reports for supervised-level zero-shot detection of abnormalities. arXiv preprint arXiv:2403. 20024417..4444.. 2024.
Sunan He, Yuxiang Nie, Zhixuan Chen, Zhiyuan Cai, Hongmei Wang, Shu Yang, and Hao Chen. Meddr: Diagnosis-guided bootstrapping for large-scale medical vision-language learning. arXiv preprint arXiv:2404. 15127, 2024.
Xuehai He, Zhuo Cai, Wenlan Wei, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathological visual question answering. arXiv preprint arXi::2010.12433, 2020.
Gregory Holste, Ziyu Jiang, Ajay Jaiswal, Maria Hanna, Shlomo Minkowitz, Alan C. Legasto, Joanna Escalon, Sharon Steinberger, Mark Bittman, Thomas C Shen, et How does pruning impact long-tailes multi-label medical image classifiers? In International Conference Medical Image Computing and Computer-Assinn Intervention, pp. 663-673. Springer, 2023.
Xiaoshuang Huang, Haifeng Huang, Lingdong Shen, Yehui Yang, Fangxin Shang. Junwei Liu, and Jia Liu. refer-and-groume multimodal large language model for biomedicine. arXiv preprint arXiv:2406. 18146,675 2024.
Stephanie L Hyland, Shruthi Bannur, Kenza Bouzid, Daniel C Castro, Mercy Ranjit, Anton Schwaighofer, Fernando Maira-1: A specialised large multimodal model for radiology report generation. arXiv preprint arXiv:2311. 13668. 2023.
Valentina Salvatelli, Shaury Srivastav, Anja Thieme, al.
Wisdom Ikezogwo, Saygin Seyfioglu, Fatemeh Ghezloo, Dylan Geva, Fatwir Sheikh Mohammed, Pavan Kumar Anand, Ranjay Krishna, and Linda Shapiro. Quilt- lm: One million image-text pairs for histopathology. Advances in neural information processing systems, 36, 2024.
Jeremy Irvin, Pranay Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-lleus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 590-599,99999 2019.
12

---

Published as a conference paper at ICLR 2025
Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lun- gren, Chih-ying Deng. Roger G Mark, and Steven Horn.. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317, 2019.
Jakob Nikolas Kather, Johannes Krisam, Pornpimol Charoentong, Tom Luedde. Esther Herpel, Cleo-Aron Weis, Timo Gaiser, Alexander Marx, Nektarios A Valous, Dyke Ferber, et al. Pre- dicting survival from colorectal cancer histology slides using deep learning: A retrospective mul- ticenter study. PLoS medicine, 16(1) 16())ee100233) 2019.
Daniel S Kermany, Michael Goldbaum, Wenjia Cai, Carolina CS Valentim, Huiying Liang, Sally L Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, et al. Identifying medical diag- noses and treatable diseases by image-based deep learning. cell, 172(5):1 121-))1122-1311,,, 2018.
Rajpurkar Lab. Rexrank leaderboard. attpss//raappurkaaaa...gg ioxrerank/ July 2024.
Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1-10, 2018.
Jiayu Lei, Xiaoman Zhang, Chaoyi Wu, Lisong Dai, Ya Zhang, Yanyong Zhang, Yanfeng Wang, Weidi Xie, and Yuehua Li. Autorg-brain: Grounded report generation for brain mri. arXiv preprint arXiv:2407. 2024.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang. Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-visiee assis- tant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2024.
Ning Li, Tao Li, Chunyu Hu, Kai Wang. and Hong Kang. A benchmark of ocular disease intelligent recognition: One shot for multi-disease detection. In Benchmarking. Measuring, and Optimiz- ing: Third BenchCouncil International Symposium, Bench 2020, Virtual Event, November 15-16, 2020, Revised Selected Papers 3, pp. 177-193. Springer, 2021.
Weixiong Lin, Ziheng Zhao, Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-clip: Contrastive imngua pre-training using biomedical documents. In In- ternational Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 525-536. Springer, 2023.
Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically- labeled Enowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium Biomedical Imaging (ISBI), pp. 1650-1654. IEEE, 2021.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Junling Liu, Ziming Wang. Qichen Ye, Dading Chong, Peilin Zhou, and Yining Hua. Qilin- med-vl: Towards chinese large vision-language model for general healthcare. arXiv preprint arXiv:2310. /7956, 2023.
Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard Chen, Kenji Ikamura, Georg Gerber, Ivy Liang, Long Phi Le. Tong Ding, Anil V Parwani, et al. A foundational multimodal vision language ai assistant for human pathology. arXiv preprint arXiv:23/2.078/4 2023.
Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovee, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. Med-flaming multimodal medical few- shot learner. In Machine Learning for Health (ML4H), pp. 353-367. PMLR, 2023.
Luis Filipe Nakayama. Mariana Goncalves, L Zago Ribeiro, Helen Santos, Daniel Ferraz Fernando Malerbi, Leo Anthony Celi, and Caio Regatieri. A brazilian multilabel ophthalmological dataset (brset). PhysioNet https://doi. org/10, 13026, 2023.
Ha Q Nguyen, Khanh Lam, Linh T Le, Hieu H Pham, Dat Tran, Dung B Nguyen, Dung D Le, Chi Pham, Hang TT Tong. Diep H Dinh, et al. Vindr-exr: An open dataset chest x-rays with radiologist's annotations. Scientific Data, 9(1):499, 2022.
13

---

Published as a conference paper at ICLR 2025
Hieu T Nguyen, Hieu H Pham, Nghia T Nguyen, Ha Q Nguyen, Thang Q Huynh, Minh Dao, and Van Vu. Vindr-spinexr: A deep learning framework for spinal lesions detection and classification from radiographs. In Medical Image Computing and Computer Assisted Ditevention-MICCA. 2021: 24th International Conference, Strasbourg, France, September 27-October I, 2021, Pro- ceedings, Part V 24, pp. 291-301. Springer, 2021.
Hieu T Nguyen, Ha Q Nguyen, Hieu H Pham, Khanh Lam, Linh T Le, Minh Dao, and Van Vu. Vindr-mammo: large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography. Scientific Data, 10(1):277 2023.
OpenAL. Gpt-4v(ision) system card, 2023. URL settps://ed.oopnaaat com/ /amer/pprs//////
GPTV_ @rr___yytem__Crdd.2
Andre GC Pacheco, Gustavo R Lima, Amanda S Salomao, Breno Krohling, Igor P Biral, Gabriel G de Angelo, Fábio CR Alves Jr, José GM Esgario, Alana C Simora, Pedro BC Castro, et al. Pad- ufes-20: A skin lesion dataset composed of patient data and clinical images collected from smart- phones. Data in brief, 32: 52:106221, 2020.
Chantal Pellegrini, Ege Özsoy, Benjamin Busam, Nassir Navab, and Matthias Keicher. Radialog: A large vision-language model for radiology report generation and conversational assistance. arXiv preprint arXiv:2311. 2023.
Zhiliang Peng, Wenhui Wang, Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
H Hieu Pham, T Thanh Tran, and Ha Quy Nguyen. Vindr-pexr: An open, large-scale pediatric chest x-ray dataset for interpretation of common thoracic diseases. PhysioNet (version 1.0. 0), 10:2, 2022.
Pranav Rajpurkar, Jeremy Irvin, Aarti Bagul, Daisy Ding, Tony Duan, Hershel Mehta, Brandon Yang, Kaylie Zhu, Dillon Laird, Robyn L Ball, et al. Mura: Large dataset for abnormality detec- tion musculoskeletal radiographs. arXiv preprint arXiv: 111111722999777 2017.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittiieser, al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint MrXiv:2403.05560 2024.
Tim Tanida, Philip Müller, Georgios Kaissis, and Daniel Rueckert. Interactive and explainable region-guided radiology report generation. Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 7433-7442 2023.
Omkar Thawkar, Abdelrahman Shaker, Sahal Shaji Mullappilly, Hisham Cholakkal, Rao Muham- mad Anwer, Salman Khan, Jorma Laaksonen, and Fahad Shahbaz Khan. Xraygpt: Chest radio- graphs summarization using medical linguuel models. arXiv preprint (XXiv:330600977, 2023.
Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, et al. Towards generalist biomedical ai. NEJM AI, Aloa2003301 2024.
Weiyun Wang, Min Shi, Qingyun Li, Wenhai Wang, Zhenhang Huang, Linjie Xing, Zhe Chen, Hao Li, Xizhou Zhu, Zhiguo Cao, et al. The all-seeing project: Towards panoptic visual recognition and understanding of the open world. arXiv preprint arXiv:2308. 01908,0110,,.. 2023.
Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards generalist foun- dation model for radiology. arXiv preprint arXiv:2308.0244 2023.
Joy T Wu, Nkechinyere N Agu, Ismini Lourentzou, Arjun Sharma, Joseph A Paguio, Jasper S Yao, Edward C Dee, William Mitchell, Satyananda Kashyap, Andrea Giovannini, et al. Chest imagenome dataset for clinical reasoning. arXiv preprint arXiv: 108 003 2021.
14

---

Published as a conference paper at ICLR 2025
Yunfei Xie, Ce Zhou, Lang Gao, Juncheng Wu, Xianhang Li, Hong- Yu Zhou, Sheng Liu, Lei Xing, James Zou, Cihang Xie, et al. Medtrinit-225m: A scage-scale multimodal dataset with multigran- ular annotations for medicine. arXiv preprint arXiv:2408.c 0et1:24000002000,, 2024.
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381 2023.
Jin Ye, Junlong Cheng, Jianpin Chen, Zhongying Deng, Tianbin Li, Haoyu Wang, Yanzhou Su, Ziyan Huang, Jilong Chen, Lei Jiang, et al. Sa-med2d-20m dataset: Segment anything in 2d medical imaging with 20 million masks. arXiv preprint arXiv:2311.11966 2023.
Kai Zhang, Rong Zhou, Eashan Adhikarla, Zhiling Yan, Yixin Liu, Jun Yu, Zhengliang Liu, Xun Chen, Brian D Davison, Hui Ren, et al. A generalist vision language foundation model for diverse biomedical tasks. Nature Medicine, pp. 1-13, 2024a.
Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pme-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305.10415, 2023a.
Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Jiayu Lei, Ya Zhang, Yanfeng Wang, and Weidi Xie. Rudgenome-chest ct: A grounded vision-language dataset for chest et analysis. arXiv preprint 16754 2024b.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.0009 2023b.
Hong- Yu Zhou, Subathra Adithan, Julián Nicolás Acosta, Eric J Topol, and Pranav Ra- jpurkar. A generalist learner for multifaceted medical image interpretation. arXiv preprint 2rX7v:2405.07988, 2024.
15

---

Published as a conference paper at ICLR 2025
APPENDIX OF "INTERPRETABLE BILINGUAL MULTIMODAL LARGE LANGUAGE MODEL FOR DIVERSE BIOMEDICAL TASKS"
A DATASET DETAILS
A. MEDREGINSTRUCT DATASET
The motivation of creating the MedRegInstruct Dataset lies in the lack of region-centris abilities and interoperability in existing medical MLLMs. In other words, existing medical MLLMs are global- centrie, designed to treat the entire image as holistic representation and struggling to detect or reason about specific regions within medical scan. As result, current medical generalist models cannot manage to concentrate on critical region within the given image, thus leading to inaccurate descriptions specifically in the lesion location. Besides, merely generating diagnostics or reports without fine-grained locations is less interpretable, making hard for clinicians to trace the source of the generated text.
Thus, we first establish Region-Centric tasks, aimed at improving medical MLLMs' capability to encode regional information. Then, we develop large-scale dataset, MedRegInstruct, covering all those tasks order to train an interpretable generalist medical MLLM with region-centric abilities.
A.1.1
TASK
Region-to Identification
Region
Structre Mardificaton
Seigtt Single Reghas
Multiple Ofjects Singlt Rlggtnn
Intrection::
Whet mhis obeect
Intraciion Paase wellubr sntho VIT soar.
Intruston Where lift vartiide and right ventride
thik eass
Proporse Leftoadngg
Willore
bostaiin dure (5% 599 47% bill)-bse
soal cheas, wrliigh wonk wwwkliineeeee 225 405
Singht Oipect Mehyye Regiens
Mi Prpli Otject Multgre fiagen
Instructin: Where brain the 1011 sow
inpouation Plote bork steraass oden DE CTat
nemool dboo (206, 660)
453, 2enel- Mill scan!
772 fuss
Iiee 35% 417. 4051 [d69, 308, 750
50L 7211 1260 420. dit 47071 wwssieeeeee www.-r------eees------
686 361, 7450
1838
750 454 794 20000
Grounded Report Generation
MIMIC EXB Grounded Report Cenerstion
Leilon Crounded Ruport Ganeratin addn
Organ Grounded Report Generation
intruction Plaase la the agan and Sauohh the coecition of each
Iratrcctio: th AaA 1..
Tratuuttore in Prpli
at wttesttisessteeeeeeee00 521, 755, 6000seoox
dod)) Baleo codddett www.eeeedddeoeoooo (500,
2.25% R- Rm m.G.e. Aea "
BSS, Tiljeibeas ling
6000-boor
uetoartbe dhouttteee bi 236, 43R, 738,
6 -SR 2 0.5) Mi2hirmmm
you hases Lankerecie houatter amaata
RT A.h. 2:
loft
indhoaatt hww.hhhheeehhhhemmmmmm aemiiethaaeee... ot sholeeo 138, 583,
FM 9 7 F1
light
and -idf--ooo-oo-oe 40%, 5000]-
hersdiggrage by contimatin offeutos atalettock conool intee
Figure 8: Region-Centric Task Fomulation. In the Region-to-Text Identification task, each question-answer pair contains one region to recognize. The detected region composed of two types, body structure and disease lesion. In the Text-to-Region detection task, the model would be asked to simultaneously locate multiple regions for an object or multiple objects. Based on the number of detected objects and the number of regions per object, the detection task can be catego- rized into four subtasks, namely single-object single-region, single-object multi-region, multi-objes ringie-region, and multi-object multi-region. In the Groubded Report Generation task, each region should be matched with a detailed description, discussing the condition of the located organ or le- sion.
As illustrated 8, we propose three types of Region-Centricn tasks, (1) Region-to-Texx identification: recognizing the name condition of given region; (2) Text-to-Region Detection: locating the position structures mentioned in the instruction; (3) Grounded Report Generation: providing detailed reports for all highlighted anatomies along with their corresponding regions in the medical scan.
Based on the specific tasks in mind, we construct the MedRegInstruct dataset, consisting of the Region-Tess Text Dataset for Region-to- Text Identification task and Text-to-Ree -oetion detection task, and the Seport-Grrunded Dataset for Grounded Report Generation task.
16

---

Published as a conference paper at ICLR 2025
16
(8)
Orgar
Sengi-mple
18G
Sagerregon
(c)
(db
(e)
(8
Figure 9: RedRestnttruct dataset overview. (a) and (b) demonstrate the distribution over modal- ities and datasets for Region-to-Texx and Text-to-Region data, respectively. (c) and (d) presents the subtask distribution and training-ter test splits for Region-Tex: Dataset. (e) shows the body structure distribution of the collected report data. (f) exhibits the statistics of Report-Grounded dataset.
A.1.2 REGION- -EXT DATASET
The Region-Text dataset sourced from SA-Med2D-20M (Ye et al., 2023), from which approxi- mately 285K images filtered construct into image-text-region triplets. To formulate the data into eegion-to-texs identification and exx-to--rgioon detection task, we first employed GPT-4 to pre- define 50 templates for each. Template examples are shown in Figure 10.
Esomple vemple bite for constroting Reglen- Tear Dota
Example semplates for constmeting Test to- R--giinn Data
Quedion:
locaiinn of this medical scan!?
Question: What lacnsl
molical
[CBJECT]
[OBJRCT]is
Quedion:: proside
Question: Cin you identity stroture present [aaGGce]] in inage?
bonading (0BJBCT)] (RBCION).
Asswer: strusture [REGION]] (cBMEC)).
Quodimn: [0o.TcC) proitionsl image?
Question: What is Frund in FASUSTON
Answer: [cBaEce)] proctionad within (S86ION).
Answer: [RSCIC6] contrine the (06.1977).
Quetioo: Mark the area where [Geasct]] appears in this scon.
Question: What segan laisen oecupics LRmGTON in seau??
Aaswer: [SBJECC) present (P8SG1ON)
Answer: [OBJECT] is (R39ION].
Quedion: Can you highlight the houndaries of [0BJBBCT) in the son?
Question: Please zame the object located [RSccOoN) is the medical scan.
Answer: The bonaddari is [osusccze are (MSSGICM)
Amswer: The object [A3G1ON) (emmmmen))
Figure 10: Template examples to formulate triplets of (image, text, region) into region-to-text and text-to-region tasks.
In statistics, the region-text dataset is composed 278,923 samples for region region-to-text to-text task and 262,031 samples for text-to-region task, respectively.
17

---

Published as a conference paper at ICLR 2025
A.1.3 REPORT- GROUNDED DATASET
The Report-Groundee dataset sources from MIMIC-CXR and a clinical in-house dataset collected from the hospital, containing approximately 240K samples in total. Examples for the grounded reports can be found in Figure 8.
In-house dataset for Chinese report generation. Here we will provide detailed overview of the collected clinical data. Our in-house clinical data containing approximately 25K X-ray, MRI and CT scans from 15K patients, each with a piece of report written in Chinese. The in-house data covers multiple regions, including the brain, chest, abdomen, spine, and pelvis, as shown in Figure 9, Also, various lesion types are considered, such as brain tumor, ischemic stroke, cerebral hemorrhage, na- sopharyngeal carcinoma, vestibular schwannoma, lung cancer, pul- monary embolism, liver cancer, colon cancer, kidney cancer, pancreas cancer, prostate cancer, lum- bar degenerative disease, disc space narrowing, spondylolisthesis, etc. For scans in 3D volume, we select the central 2D slice that provides the most representative views of the anatomical structures. In total, our in-house dataset is composed of 27, 2,7357 image-report pairs. Examples are shown in Figure 11.
KO BERRER APRLLE GRAE WIT BARE LIT AUR BAR MKGMMKRRRAAATS SEMMITIA anti WaRe. R0ITE
MIBN: Mr. 2WW.CEERSTTIN COILLL WiE.E
CT
Hoal&Neck
RXs ENIT AAPEAAEE aeveeeee is MINEE Bi pou BODE
FROR:
ANNE SREAREK BAREERMENE SITM IORRR
J. 733601 Se Par EEOENTON BURARRB C BENE FirE F2 CHARRO B4TTO86AAX ME- ISRRE BEEEE 2.AAA
FlLS
AV WB PEREAAA RORAKK WF WESSEEL GORAAIPMEN SOU RESA
MRAISIN WER-MERNOTIC. 19022
16 KRNE. AR 1EEE
*O RWIISR *ERAG299NN MIS PERRER PAGennPine
roun: THeu REX JuNitSemm deem Stenes guo BUA XnLAs
NOR
19255 Ex SparrarE FERRAAR I Dinen der 1POAE AUENTE
memerrn 2 LHFIR 23 BORRO WOOIA BEWO
SUREWAS FUAUVE
203229 MONNE IN 20N MIMEIAA
kanR: TTUMM HOHOO ATBITU 1y 75-55 FR
KAMS RETRIS A AARR IF GURTINN JOM
Spine X-ray
Spine
BO'E TEAR ACRERO KIIVERRMa--- B
BOON PEAURAXE BEUR DONLEEWER:: 6И PHETESI LECATR #J.00.
LEEE LERTER T TI
RELPOEEE BLREIOI -5.018
40Y JON
RAPLER
UFS LN/9 IASIRRERT.
REBCH WITH LEAAR MORLER 128AA17
Figure 11: Data examples of the Chinese image-report pairs in our collected dataset.
Grounded report construction from MIMIC-CXR dataset. We apply Chest-ImaGenome (Wu al., 2021), which includes bounding box annotations for 29 structures in the frontal chest x- ray images from MIMIC-CXR dataset. From these atanomies, we select 12 standardized ones in chest create the report-grounded dataset for chest x-ray examination, including left lung, right lung, mediastinum, cardiac silhouette, left hilar structures, right hilar structures, left clavicle, right clavicle, left hemidiaphragm. right hemidiaphragm, right atrium, and abdomen. We filtered the chext x-ray scans paired with annotation boxes, and obtained 225,610 data samples in total.
Grounded report construction In-house dataset. The constructed report-grounded dataset from our in-house dataset contains two parts: (1) Lesion Grounded Report and (2) Organ Grounded Re- port. In our in-house dataset, certain scans are manually labeled with the lesion area by experts. This lesions include 12 types in total, namely, brain tumor, ischemic stroke, cerebral hemorrhage, nasopharyngeal carcinoma, vestibular schwannoma. lung cancer, pulmonary embolism, liver can- cer, colon cancer, kidney cancer, pancreas cancer, prostate cancer. For these scans, match the annotated abnormal region coordinates with specific descriptions for the lesion part, acquiring 6,881 grounded reports specifically for abnormalities. For the rest cases, follow the semi-automatical
18

---

Published as a conference paper at ICLR 2025
pipeline illustrated in $3.2 construct organ grounded reports. The organ grounded report data covers body parts including heart, lung, liver, colon, kidney, spleen, and pancreas. We selected slices from each original 3D scan to obtain more diverse views, and obtain 10,907 data items for organ grounded reports.
A.2
TRAINING DATASETS ON TRADITIONAL. MEDICAL TASKS
A.2.1 IMAGE CAPTIONING
PMC-OA (Lin et al., 2023) is biomedical dataset with 1.6M image-caption pairs collected from PubMedCentral's OpenAccess subset which covers diverse modalities or diseases. We leveraged the dataset for alignment training.
QUILT (Ikezogwo et al., 2024) is a large-scale vision-language dataset consisting of 802, 144 image and text pairs, which was curated from video frames and corresponding subtitles on YouTube. We leveraged the dataset for alignment training.
A.2.2 VISUAL QUESTION ANSWERING
SLAKE (Liu et al., 2021) is bilingual radiology VQA dataset consists 642 radiology images and over 7000 diverse QA pairs annotated by experienced physicians. Following the official split, used both English and Chinese versions for training, which contains 4919 and 4916 question-answer pairs repectively.
(Lau et al., 2018) consists of 3.5K question-answering pairs on 314 radiology images, where clinicians asked naturally occurring questions about radiology images and provided reference answers. Following the official split, we use 3,064 question-answer answer pairs for training.
VQA-RAD
Path (He et al., 2020) consists of 32,799 open-ended questions from 4,998 pathology images where each question is manually checked to ensure correctness. Every image is paired with several questions related multiple aspects such shape, color and location. Following the official split, we use 19,755 question-ansee answer pairs for training.
PMC-VQA (Zhang al., 2023a) consists of 1.6 million question-answer pairs, which is a large- scale medical visual question-answeriin dataset generated from PMC-OA. We combined two ver- sions of PMC-VQA and use 329,551 question-ansee ansser pairs for training.
MedPix is collected from MedPix website', which is free open-access online database for medical usage. RadFM (Wu et al., 2023) separate the dataset into MPx sixx-single and MPx-multi. We apply the MPX-single part and use 92,282 question-answer pairs for training.
A.2.3 REPORT GENERATION
MIMIC-CXR (Johnson et al., 2019) is a large-scale chest image-report dataset that consists of 371,920 chest X-rays associated with 227,943 reports from 65,079 patients. Following RadFM (Wu al., 2023), we use 354,569 cases for training.
IU-Xray (Demner- Fushman et al., 2016) consists of 7,470 images and 3,955 reports collected from Demmner-Fushman the Indiana Network. Following R2Gen (Chen et al., 2020), we use 4,720 cases for training.
In-house Dataset. We split the collected reports at the patient level, ensuring that the training and test sets do not contain overlapping patients. we use approximately 90% of all the data for training. which contains 24,608 image-repon pairs.
A.2.4 MEDICAL IMAGE CLASSIFICATION
VinDr-CXR (Nguyen et al., 2022) consists of 18,000 images that were manually annotated by a total of 17 experienced radiologists with 22 local labels of rectangles surrounding abnormalities and 6 global labels suspected diseases. The training set contains 15,000 scans, and radiologists independently label each image. Following the official split, 45,000 samples for training.
hhtps:/mmeppix..c http:///mmddix...mmm......g.
19

---

Published as a conference paper at ICLR 2025
VinDr-PCXR (Pham et al., 2022) is pediatric CXR dataset of 9, 9,125 studies that were retrospec- tively collected from a major pediatric hospital in Vietnam between 2020-2021. Each sean was manually annotated by an experienced radiologist for the presence of 36 critical findings and 15 diseases. Following the official split, we use 4,585 samples for training.
VinDr-SpineXRR (Nguyen et al., 2021) is large-scale annotated medical image dataset for spinal lesion detection and classification from radiographs. The dataset contains 10,466 spine X-ray images from 5,000 studies, each of which is manually annotated with 13 types of abnormalities by an experienced radiologist with bounding boxes around abnormal findings. Following RadFM (Wu al., 2023), we use 8,389 samples for training.
VinDr-Mammo (Nguyen et al., 2023) is large-scale full-field digital mammography dataset of 5,000 four-view exams. Following the official split, we use 16,391 samples for training.
CheXpert (Irvin et al., 2019) is a large public dataset for chest radiograph interpretation, which retrospectively collected the chest from Stanford Hospital, performed between October 2002 and July 2017. The dataset contains 224,316 chest radiographs of 65,240 patients. Following the official split, we use 223,414 samples for training.
MURA (Rajpurkar et al., 2017) is large-scale dataset of musculoskeletal radiographs containing 40,561 images from 14,863 studies, where each study manually labeled by radiologists as either normal or abnormal. Following the official split, we use 36,808 samples for training.
ISIC2018 (Codella et al., 2019) is a skin lesion dataset acquired with 7 dermatoscope types. Fol- lowing the official split, we use 10,015 samples for training.
ISIC2019 (Combalia et al., 2019) is skin lesion dataset labeled with 8 different categories. Fol- lowing the official split, we use 25,331 samples for trianing.
PAD-UFES (Pacheco et al., 2020) is a skin lesion dataset composed of clinical images collected from smartphone devices and a set of patient clinical data containing up 22 features. The dataset consists of 1,373 patients, 1,641 skin lesions, and 2,298 images for six different diagnostics. We randomly sample 80% of the dataset for training, which includes 1,838 samples.
Kather colon dataset (Kather et al., 2019) a dataset of 100,000 non-overlapping image patches from hematoxylin & cosin (H&E) stained histological images of human colorectal cancer (CRC) and normal tissue, covering tissue classes in total.
BRSET (Nakayama al., 2023) is multi-labeled ophthalmological dataset onsisting of 16,266 im- ages from 8,524 Brazilian patients. Multi-labels are included alongside color fundus retinal photos. We randomly sample 80% of the dataset for training, containing 13,012 samples.
ODIR-5K (I.i et al., 2021) is a structured ophthalmic database of 5,000 patients with age, color fun- dus photographs from left and right eyes and doctors' diagnostic keywords from doctors. Following the official split, we use 6,392 samples for training.
OCT2017 (Kermany al., 2018) includes 83,484 OCT images of 4,686 patients, consisting of 4 categories, normal, drusen, choroidal neo Vascularisation (CNV), and Diabetic Macular Edema (DME). Following the official split, we use 82,484 samples for training.
Butterfly Network ultrasound dataset Nettorrllyet...klnn.. 2018) is a large dataset containing 9 different classes of ultrasound images acquired with the Butterfly IQ on 31 individuals. Follow- ing Chen et al. (2021), 34,325 images are applied for training.
BUSI (Al-Dhabyani et al., 2020) includes breast ultrasound images among women between 25 and 75 years old. The number of patients is 600 females, patients. The dataset consists of 780 images that are categorized into three classes, namely, standard, benign, and malignant. We randomly sample 80% of the dataset for training, which includes 630 images.
A.3 TRAINING DATASETS ON REGION-CENTEN TASKS (MEDREGINSTRUCT)
A.3.1 REGION- -EXIT TASKS
SA-Med2D-20M (Ye et al., 2023) is a large-scale segmentation dataset of 2D medical images built upon numerous public and private datasets. The dataset consists of 4.6 million 2D medical images
20

---

Published as a conference paper at ICLR 2025
and 19.7 million corresponding masks, covering almost the whole body and showing significant di- versity, We filter approximately 285K images from the original dataset, and construct 242,268 and 229,340 training samples for Region-to-Tar Text Identification and Text-to-Region Detection respec- tively.
VinDr Series Dataset is a large-scale classification composed of VinDr-CXR, VinDr-PCXR, VinDr- SpineXR, VinDr-Mammo. The datasets provide radiologist's bounding-bol annotation for abnormal areas. We follow the official split and apply the samples with bounding boxes.
ISIC Challenge Dataset contains lesion segmentation data where the original image is paired with manually annotated lesion boundaries. We follow the official split and convert the segmentation map into bounding boxes.
PanNuke (Gamper et al., 2019) semi-automatically generated nuclei instance segmentation dataset. We follow the official split and convert the segmentation map into bounding boxes to for- mulate region-text pairs.
A.3.2 DEPORT-GROUNDED TASKS
Chest-ImaGenome (Wu et al., 2021) applied a CXR bounding box detection pipeline to automati- cally label frontal chest x-ray images from MIMIC-CKK dataset with 29 annotations, from which selected 12 standardized structures in the chest. Following the split of MIMIC-XR in RadFM Wu al. (2023), we filtered the chext X-ray scans paired with annotation boxes, and obtained 222,588 samples for training.
In-house dataset. The constructed report-grounded dataset from in-house dataset contains two parts, Lesion Grounded Report and Organ Grounded Report. We applied 6,443 lesion grounded reports and 10,907 organ grounded reports for training.
A.4 TEST DATATSETS
A.4.1 VISUAL QUESTION ANSWERING
SLAKE. Following the official split, we use 1,061 and 1,033 quesion-answer pairs for test on the English and Chinese version, respectively.
VQA-RAD. Following the official split, we use 451 quesion-answe pairs for evaluation.
Path VQA. Following the official split, we use 6.761 quesion-answer pairs for evaluation.
PMC-VQA. Following the official split, the two versions of PMC-VQA test set contain 50,000 and 33,430 quesion-answer pairs, respectively. We test the model on both versions and report the averaged result.
A.4.2 REPORT GENERATION
MIMIC-CXR. Following RadFM (Wu et al., 2023), we use 4,710 cases for test.
IU-Xray. Following R2Gen (Chen al., 2020), we use 1,180 cases training.
In-house Dataset. We split the collected reports at the patient level, ensuring that the training and test sets do not contain overlapping patients. we use approximately 10% of all the data for test, which includes 2,749 image-report pairs.
A.4.3 MEDICAL IMAGE CLASSIFICATION
MURA. Following the official split, we use 3.193 X-ray images for test.
PneumoniaMNNIST. Following the official split, we use 624 X-ray samples for test.
OrganCMNIST. Following the official split, use 8,216 CT samples for test.
OrganAMNIST. Following the official split, we use 8,827 CT samples for test.
ISIC2016. Following the official split, we 379 skin lesion images for test.
21

---

Published as a conference paper at ICLR 2025
ISIC2018. Following the official split, use 1,512 skin lesion images for test.
PAD-UFES-20 Since no official split is provided, we randomly split 20% of the data for evaluation, which contains 460 images.
Kather Colon Dataset. The original dataset is split into "NCT-CRC-HE- 100K" and "CRC-VAL- HE-7K" subsets, which share overlap with each other. Since the "NCT-CRC-HE- 100K" subset applied for training, we use "CRC-VAL BRC-VVL--E-7K3" subset for test, which contains 7,180 pathology images.
Messidor-2. The Messidor-2 dataset is a collection of Diabetic Retinopathy (DR) examinations. We treat this task a binary classification to detect DR disease. Since official split is provided, randomly split 20% of the data for evaluation, which includes 378 fundus images.
OCT2017. Following the official split, use 968 OCT scans for test.
OCTMNIST. Following the official split, we use 1,000 OCT scans for test.
BUSI. Since no official split is provided, randomly split 20% of the data for test, which includes 150 ultrasound images.
BreastMNIST. Following the official split, we 156 ultrasound scans for test.
CheXpert. Following the official split, we use 234 chest x-ray images for test.
ChestMNIST. Following the official split, we use 22,433 chest x-ray images for test.
CXR14. We follow the official split in Holste et al. (2023) and apply 21,081 chest x-ray images for test.
VinDr-CXR. Following the official split, we use 3,000 chest x-ray images for evaluation.
VinDr-PCXR. Following the official split, use 1,397 chest x-ray images for evaluation.
VinDr-SpineXR. Following the official split, we use 2,077 spine x-ray images for evaluation.
VinDr-Mammo. Following the official split, we use 4,000 mammography images for evaluation.
BRSET. Since no official split is provided, we randomly split 20% of data for test, which cludes 3254 fundus images.
RFMiD 1.0. Following the official split, we use 640 fundus images for evaluation.
A.4.4 REGION-CENTRE TASKS
Region-Text Tasks. We split approximately 10% from the Region- Text dataset for evaluation, in- cluding 36,655 samples for Region-to- Text Identification and 32,691 samples for Text-to-Region Detection.
Report-Grounded Tasks. For chest x-ray grounded reports, we follow the split RadFM (Wu et al., 2023) and filtered the chext x-ray scans paired with annotation boxes. Overall, 3,022 samples are acquired for evaluation. For lesion grounded reports and organ grounded reports, we include 438 and 1,264 data items in the test set, respectively.
B TRAINING DETAILS
B. IMPLEMENTATION DETAILS
We employ InternVI. 1.2 (Chen et al., 2024b) as our general-domain foundation to begin training, which is composed of Intern VTT-6B as the vision encoder, and Nous-Hermes-22 Hermes-2-Yi-344 Yi-34B as the lan- guage model. Our training process divided into two steps: alignment training and instruction tuning. During the alignment training phase, we freeze the vision encoder and language model, only fine-tuning the alignment module with medical image captioning and report generation datasets, which contain about 2.4M data in total. In the instruction tuning stage, apply both public datasets and our Region-Centric Centric datasets, MedReglnstruct, to optimize the language model, while keeping the other components unchanged. The amount of instruction tuning data is approximately 2.2M. The
22

---

Published as a conference paper at ICLR 2025
language model loss is applied as the loss function. We follow the official instruction for finetuning InternVL, and leverage LoRA with DeepSpeed ZeRO Stage to optimize model parameters. The model is trained on 16 NVIDIA H800 GPUs for 1 epoch in the alignment stage and 2 epochs in the instruction tuning stage.
B.2 TASK- PROMPTS
We design task-specific instructions to prompt the model to recognize and address different objec- tives for visual question answering, report generation, image classification and region-centric tasks. The instruction prompt for each task can be found in Figure 12.
Chessfestion
Tas ane medical asslatast. 1.MM. 18P 7.:
EN You are helpful medioal. aosistant. Your baok ie medical Lmage given (NCOALITT)] Lmage possibble
Repert Generatiee
helpdul nedical
EN Tou are helpfell medical assiatant You are zequiced to Fied
You aze given a cless x-ray laage, and you are required to genarate melical report coseleting findings and irpressions based on
copresented canging 0. to 199, These valuee corcesjond to the left top left botton xight x, and botton right
Us zeglose
given dbjects. cegion
Winding includes obracvations, Lepesesion outlines the
Zizak diagnosia.
Cevended Report Geneston
medicol image, and you are required to locate the reglon of ongan or lasion
and generate report for the region. The regloon ie regresented in
Rephon-o Hentlicatis
Excm 369. these valcas yt, yal laft laft botson right and bottow zight
zanging
the object in the zegion. zepien rppeesented the dozm of bounding bax, [xt, yt, s2, ye), with intaquce canging teom to 259. valuue concespend to the top a, top lest y, botton zight
yol.
Rittoli F.
Figure 12: Task-Specific Prompt Examples.
C EXPERIMENT DETAILS
C.I PERFORMANCE COMPARISON ON GENERAL MEDICAL VISION-LANGUAGE TASKS
C.I.1 VISUAL QUESTION ANSWERING
Table 6: Performance comparison Visual Question Answering task. ** indicates that the model is fine-tuned on the dataset. 'N/A' means the scores are not reported.
Dataset
Modality
Metrics
Med-F Fam-iFlaningo
Lea
RadFM
MeaDr
BIoMedCPT
Ours
Visual Question Answering
BLEU-1
11.75
70,76
78.01
76.40
N/A
81.55
F1
13.43
70.96*
78.09
77.50
N/A
82.06
SLAKE
X-ray, MRL, CT
BertScore
40.22
84. 84.89*
87.33
78.71
N/A
91.76
CloseAccuracy
33.24
82.25
83.40
89.90=
85.35
OpenRecall
21.06
83,80*
76.23
74.20
84.30*
80.45
BLEU-I
27.4
44.6*
50.7
59.34
N/A
61.89
F1
28.22
44.77
$1.04
60.99
N/A
62.24
VQA-RAD
X-ray, MRI, CT
BertScore
61.93
69.13*
74.13
78.70
N/A
82.23
CloseAccuracy
44.62
27.39*
60.56
78.49
81.3
75.30
OpenRecall
12.68
67.43*
41.57
41.75
60.90
46.03
BLEU-1
24.31
48.39F
24.84
61.40
N/A
59.52
F1
60.69
PathVQA
Pathology
BertScore
57.45
73.18
58.4
76.00
N/A
80.03
CloseAccuracy
47.86
58.18
40.22
90.20
88 80.00=
89.12
OpenRecall
2.35
39.27
10.08
33,50
28 28.00"
33.56
BLEU-I
30.32
39.19
48.71
64,15
N/A
67.19
F1
33.27
41.98
49.47
64.83
67.88
PMC-VQA
X-ray, MRI,
BertScore
62.21
67.2
70.54
84.41
N/A
85.26
CloscAccuracy
35.88
63.82
45.76
66.18
N/A
81.18
OpenRecall
39.13
53.09
$2.19
65.15
68.07
We implement our model on several medical VQA benchmarks for comparison, such as English and Chinese versions of SLAKE (Liu et al., 2021), VQA-RAD (Lau et al., 2018), PathVQA (He et al., 2020), and PMC-VQA (Zhang al., 2023a). For PMC-VQA. versions. We follow the official splits of all the datasets and apply both natural language generation (NLG) and classification metrics to evaluate the model outputs. Table 6 represents the results
combined the data from both
23

---

Published as a conference paper at ICLR 2025
these datasets. Specifically, LLa and BioMedGPT shows slightly strong performance the SLAKE, VQA-RAD, and Path VQQ datasets because they are fine-tuned on each of the dataset. RadFM is primarily trained on radiology data and struggle to understand pathology modalities, leading to poor performance on the PathVQA dataset. Compared with MedDr, our model achieves more competitive performance, indicating that incorporating regional information into the training data allows the MLLM to better perceive spatial knowledge of various organs and lesion areas in medical images. However, our model performs slightly less competitive on Path likely due to the scarcity of regional information for the pathology images. This highlights the urgency of enhancing region-centric capabilities across wide range of modalities, which would also benefit traditional vision-language tasks.
C.1.2 REPORT GENERATION
Table 7: Performance comparison on English Report Generation task. 'N/A' means the scores
not reported.
Dataset
Modality
Metrics
Med-Flamingo
Led
RadFM
MedDr
BioMedGPT Ours
Report Generation
English
BLEU-1
21.67
20.62
30.92
31.27
N/A
35.18
BLEU-4
1.83
0.43
4.55
7,83
9,90
10.36
NLG
METEOR
15.38
14.09
20.37
23.10
14.20
28.36
ROUGE-L
13.1
10.36
14.83
20.92
24.40
24.70
MIMIC-CXR
X-ray
BertScore
46.46
43.86
53.14
53.05
NIA
58.79
CheXBent
16.67
11.63
21.42
24.72
N/A
34.89
CE
RadGraph
8.92
4.85
13.15
20.28
22.50
22.67
RadCliql
1.2
N/A
0.85
BLEU-1
21.33
14.31
27.2
37.70
NIA
45.73
BLEU-4
3.42
0.13
4.04
12.20
N/A
14.84
NLG
METEOR
19.22
18.39
32.30
12,90
35.51
IU-Xray
X-ray
ROUGE-L
14.47
6.72
12.65
28.30
28.50
30.43
BertSeore
53.15
44.81
53.71
57.19
N/A
66.28
CheXBar
30.78
27.94
33.77"
56.30
N/A
61.TT
CE
RadGraph
13.63
2.85
11.69
33.10
N/A
38.00
RadCliqi
0.27
English Report Generation. For English report generation, we evaluate our model on the report generation task for chest X-ray datasets MIMIC-XXR (Johnson et al., 2019) and IU-Xray (Demner- Fushman et al.. 2016). We apply the training-test split of Wu et al. (2023) for MIMIC-CXR and Chen et al. (2020) for IU-Xray. Natural language generation (NLG) and Clinical Efficiency (CE) metrics are utilized for assessment. Specifically, DeBERTa is deployed to calculate BertScore. Among these models, Med- Med-Flamingo and LLa concentrate on question-answeriin skills but are less effective in producing longer sentences to thoroughly describe the observations. In compari- son with RadFM and MedDr, we integrate the recognition and detection of chest X-ray structures into the training data, which inherently encourages the model to analyze fine-grained anatomical details in the medical scan, thereby accomplishing significantly better results. For instance, the MIMIC-CXB dataset, our model outperforms 3.91% in BLEU-1 and 3.78 in ROUGE-L compared with MedIDr. Besides, MedRegA suparsses BioMEdGPT by 1.93% in ROUGE-L on the IU-Xray dataset. Examples in Figure 13 demonstrate that our model can consistently provide detailed and accurate descriptions of the findings in chest x-ray, closely aligning with the ground truth annota- tions. These results highlight highlight the ability of MedRegA to produce coherent and clinically relevant findings.
Evaluation on ReXrank Benchmark. To provide a comparison of our methods with state-of-the- art methods for chest X-ray report generation in English, we evaluate our method with the official test dataset on ReXrank (Lab, 2024). We present the results in Table 9. Our model achieves results that are comparable to specialist models, such as RGRG (Tanida et al., 2023) and RaDialog (Pelle- grini et al., 2023), which are specifically trained on report generation tasks for chest X-ray datasets. However, unlike these specialist models, our model designed as generalist framework capable of handling multiple tasks, such as VQA and diagnosis, and supporting more modalities. Within the generalist model category, our model ranks second. It accomplishes superior scores in certain metrics compared to MedVersa (Zhou et al., 2024), and outperformed other generalists including RadFM (Wu et al., 2023) and BioMedGPT (Zhang et al., 2024a). comparison with Med Versa, model offers broader versatility in report generation for different modalities of various organs,
24

---

Published as a conference paper at ICLR 2025
MedRegA Prediction: Findings: In comparison with the study there little overall change. Monitoring and support devices remain In place, Continued enlargement cardiac silhouette with bilateral pleural effusions and compressive atelectasis at the bases.
study _ monitoring support devices remain place. Continued substantial enlargement of the cardiac silhouette with bilateral pleural effusions, compressive basilar atelectasis, and moderate pulimonar edema.
Ground Truth: Findings: In comparison
MedRegA Prediction: Findings: As compared to the previous radiograph, the monitoring and support devices are in unchanged position. The lung volumes have decreased, There milé-to-noderate pulmonary edema. The size cardiac silhouette unchanged. No pleural effusions. pneumothorax.
Ground Truth: Finclings: As compared to the previous radiograph, there is no relevant change. The monitoring and support devices constant. Low lung volumes, borderline of the cardiac silhouette. Mild pulmonary edema, Moderate retrocardiac atelectasis, No evidence of pneumonia.
MIMIC-CXR
MedRegA Prediction: The heart is normal in size. The mediastinum is unremarkable. The aorta is tortuous. The lungs are hyperinflated. There no pleural effusion, pneumothorax, or focal airspace disease. degenerative changes noted the thoracic spine.
Ground Truth:
The heart is normal In size. The mediastinum is stable. Atherosclerotic caldiiicatinss of the aorta identified. There no focal consolidation, pleural effusion or pneumothorax. Degenerative changes the thoracic spine noted.
The heart normal in The mediastinum unremarkable. The lungs clear. There no pleural effusion pneumothorax.
MedRegA Prediction:
Ground Truth: The lungs are clear. There is no pleural effusion or pneumothorax. The heart and mediastinum are normal. The skeletal structures are normal
(b) IU-Xray
Figure 13: Generated Report Examples on MIMIC-CXR and IU-Xray datasets.
including X-ray, CT and MRI images of the brain, spine, abdomen, and pelvis, in addition to chest X-rays.
Chinese Report Generation. To evaluate the ability of our model in generating medical report in Chinese, we collect Chinese image-report pairs in real clinical scenarios to construct test dataset covering brain, chest, spine, abdomen and pelvis from the hospital. Table demonstrates the model performance comparison on Chinese report generation. Since MedDr reveals the ability compre- hend and generate Chinese language, we directly prompt it with Chinese texts to fetch the outputs. For other monolingual baselines, obtain English reports and deploy an additional LLM to trans- late them into Chinese for evaluation. To quantify the capability of the models in Chinese report generation, we utilize NLG metrics report the averaged result, where Chinese tokenizer is also adopted to segment the generated sentences into word phrases. We apply Multilingual BERT to calculate the BertScore metric. MedRegA significantly outperforms other baselines in generating Chinese reports, revealing impressive bilingual ability. It can be inferred from the examples in Fig- ure 14 that training on bilingual reports enables MedRegA to retain knowledge in both languages, yielding coherent and fluent medical reports in Chinese. For example, in multiple cases, MedRegA can successfully capture observations of lesions in higher level of granularity, including size and locations.
Evaluation on paired bilingual dataset. Moreover, to evaluate the generalizability of our model across English and Chinese languages, We further apply GPT-4o to create paired bilingual datasets, including a English-to-Chine house data. Both datasets contain 100 samples respectively. Table 10 demonstrates that directly prompting MedRegA to generate report a different language results in lower performance met- rics. This be attributed to the differences in writing style between the English and Chinese report
dataset from Xray and a Chinese-to-Engeeee to-English dataset from our in-
25

---

Published as a conference paper at ICLR 2025
MedRagA Predictions
Ground Truth:
A X5 E6. LTD
BAR E: A
BUHNI 49OTEL PS
EM
A
BoRe 2 UIREETTNN RETTIRKESS RES-RRRMMMETTE zuichhBmm
NESTI
KANN AURH MOA Eire 2aP110rm BEUEE StBler 25f0oRe "M. 2. Bubdie
M P9 Jiliin weAre wareeks MINI WANNEENNN 1 CAR A11 20
Wo R. S. MULLH SEERCIIAAA
000 Flll Pi SupeE BE0 SURTEE
E 3 MA EM BOTTHRRR 11t 20 BAEEEE "CARR
FOR MINLIT ROTmDMIIAAT 20 Ro + 11 do 257 1mm Fill 1.A 7798
THE SURII LIBIEE Mamocial Wo a FiSer Ko Smon. 28 So FRR
MedRegA Predictions
FIRIERR (.F. 2 R. SU HE
M/5ttR f6MMA122//ER
Ground
14/5thF
30 ML.
BE BERRAS t1 111
SA AR 1:500
EM S 2 HERSS
SERURERS
WHFE 1AME.. MA #il H AO0
GEEKMIEXEM EMM
SON RUISO W SaIMara: ME saaia, M U. MIN H5TTA RESSRRR JUMR
MUUHHE PiR. 1818. MAT
1E A Litiin 1
MadRagh Prodiction
19:0 Bist Nalmmm glmm samm deaidxittii 199
UNT E. Mn Be Be% 19 por 1996.
1i molo Sop 19 dram Kram 1
38% 215-9 FOR BO
METTSEE THE 7919
Ground Trutts
25 11 TALEAA BR040 A TEAS
2o Miterr X Mormm soom 97 BA Koth
RESAAAA RMM BAPAAI 20 15 Poa ganm cowd OLL
Esto 201
Figure 14: Generated report examples In-house Chinese report dataset.
26

---

Published as a conference paper at ICLR 2025
Table 8: Performance comparison Chinese Report Generation task. cannot generate valid outputs.
indicates the model
Dataset
Modality
Metrics
Med-Flamingo LLaVA-Med
RadFM
MedDr
Ours
Report Generation in Chinese
BLEU-1
5.17
4.89
5.00
12.31
40.89
BLEU-4
0.03
0.72
20.47
Brain
MRI, CT
NLG
METEOR
3,46
3.19
5.08
12.30
39.95
ROUGE-L
3,54
3.03
5.04
8.37
22.78
BertScore
59.12
58.19
63.79
63.05
72.71
BLEU-1
3,40
3.79
2.35
10.40
38.47
BLEU-4
0.01
0.20
18.11
Chest
CT
NLG
METHOR
2.45
3.25
3.50
9.38
35.06
ROUGE-L
2.69
3.35
3.69
7.29
19.41
BertScore
59.19
56.72
62.02
60.71
69.18
BLEU-1
3.49
8.21
10.7
BLEU-4
0.16
22.57
Spine
X-ray, MRI
METEOR
2.75
5.35
5.64
11.23
29.54
ROUGE-L
3.38
4.80
8.85
BertScore
56.61
58.46
64.17
61.07
74.41
BLEU-1
2.87
2.85
1.79
13.51
45.22
BLEU-4
0.30
20.67
Abdomen
CT
NLG
METEOR
1.99
2.56
2.40
9.56
36.07
ROUGE-L
2.37
2.79
2.66
8.84
18,91
BertScore
58.27
55.67
60.60
61.07
69.46
BLEU-1
3.02
4.82
2.91
13.05
35.12
BLEU-4
0.01
0.23
11.86
Pelvis
MRI
NLG
METEOR
2.31
3.66
3.64
11.09
29.54
ROUGE-I.
2.92
3.93
4.12
8.84
16.10
BertScore
57.62
57.08
62.11
61.04
69.78
Table 9: Evaluation on ReXrank Benchmark. Bold values indicate the best results, Underlined values indicate the second-bes results, and *** indicates the best results among generalist models.
Model
Model Type RadCliQ-viL
RadCliQ-vOl
BLEUT
HertScore
SembScore"
RadGraph
MIMIC-CXR Dataset
ModVersa
Generalist
0.692
2,581*
0.195
0.518
0.601
0,244
RGRG
Specialist
0.803
2.818
0.24
0.447
RadFM
Gencralist
0.815
2.8
0.196
0.479
0,556
0.234
RaDialog
Specialist
0.97
3,044
0.175
0.419
0,545
0,234
BioMedGPT
Generalist
1.044
3.175
0.123
0,361
0,512
0.242
Chgengent
Specialist
1.137
3,272
MedRegA (Ours)
Generalist
0.718
2,634
0,205
0,496
0.613*
0,244
IU Xray Donaset
MedVersa
Generalist
L08$
3,337
0.193
0.43*
0.315
0,273
RGRG
Specialist
1,363
3,723
0.125
0,323
0.337
RadI'M
Generalist
1. 1.604
4,093
0.081
0.281
0.245
0.111
RaDialog
Specialist
1.33
3.647
0.112
0.322
0,381
0.168
BioMedGPP
Generalist
1.9
4,554
0.015
0.163
0.205
0.062
CheXagent
Specialist
1.437
0.094
0,304
MedRegA (Ours)
Generalist
1.303
3.644
0.157
0.358
0.326
0.185
samples. Notably, the gap in BertScore much smaller than BLEU and ROUGE scores, indicat- ing that changing the target language affects styles more significantly than semantics. Figure 15 shows several case studies on the paired bilingual dataset, reflecting the impact language transfer. The examples show that although the style may deviate, model can still capture the significant impressions in the medical image.
C.1.3 MEDICAL IMAGE CLASSIFICATION
conduct experiments on both single-label classification and multi-label classification with a wide range of datasets across radiology, ultrasound, ophthalmology, and dermatology. For all the datasets. we adopt the official test set and prompt the model with the predefined label set. Since MLLMs are inclined to overfit to frequent labels in the training set, mitigate the class dlss-imbalanced issue by oversampling minority classes and downsampling majority classes, which critically increases the classification score. F1 scores for each dataset reported in Table 11. For single-label classifica-
27

---

Published as a conference paper at ICLR 2025
Table 10: Evaluation on paired bilingual data for report generation. "Translated" indicates generating and evaluating with the translated language, and "Original" means using the original language of the report.
Method
Language
BLEU-1
ROUGE-L
BertScore
Results on English-to- Ch-hheeee dataset
Translated Original
CN
10.04
7.06
66.75
EN
48.99
34.89
73.98
Results on Chinese-to-Engee English dataset
Translated
EN
19.95
17.39
62.20
Original
CN
41.66
32.59
73.39
English-to-Chinn
Ground Truch: Contimed enlargement the cordiae silhowett with pulmonary vascular congestion and bilateral pleural effasions compressive stelozasis Central catheter remains in place.
EN Generation: There enfergmeen of the canlicc silhoastc with pulmonary odkms and bilatural pleural offusions compreeivve stelectasis bees
Ground (+ Festt X.
Generation: 2
X Fat EE
3% 2 RR
Chinese-to to---gish
CN Ground Truth: E E8/884255. 23 W that. 8- 2t 25.0. Axid917eenxxnnnnotttm AL R
9 PUBLESE 0t:530
Ben0iss di st.1s
#es ROEE PP- P- 68.6 WA 19. RE
CN Generation: M8
fE/088 B6 11:22 1
EN Ground Truch:
and
the right panglis,
observed with undear
boundariss, mearating approinaidy thee 99amn Sénre with paichy aggull areas incde rarticted diffasion, Surmanding paiday cáema roted The right laleai veatride
and manowed, and madine strccture shiked to let, the ocmmin simooole, a parchy ans obeerved with chor borndartee, to scoonninn edemo, to offect. The recaaliing boxn parcechysra calibiia utifee egrnds with absonnal signal findings
EN Gener atter: Thaie i large, punially calancing, mo-criuaiing mas the right cociital lake with olens. The approcmasely Stmm=))mm=) 3) mon and bac eigntfcant effect on the larra veomileee cauding oberuction and formation of hydhocophaliic gradens. There as also mild edema with the right occipial tobe. No some arrik is sdernited.
Figure 15: Examples on paired bilingual dataset.
tion, MedRegA outperforms existing models on most benchmarks by a large margin. In contrast, ulti-label classification appears more challenging for MILMS due to the difficulties in decoupling subtle symptoms and relating them to corresponding diagnoses. Since Med and RadFM are not exposed to diverse modalities training, they cannot manage to generate satisfactory diagnostic results for unseen modalities, and struggle to extract valid labels from the prompts. It is worth noting that LLaVA-Med is able to acknowledge its limitation and refuse to pro- vide uncertain answers, whereas the other models may give random responses when confronted with queries beyond their knowledge scope. Figure 16 provides examples of the generation of disease diagnosis task, demonstrating the diversity of input modalities and the diagnostic categories.
ooaminan, LLaVA-
C.2 GROUNDED REPORT GENERATION
As a supplement to the main text, we present the comparison in the quality of reports for organ grounded report generation in Table 12. NLG metrics are adopted evaluate the report outputs. By integrating the region-centric task into prediction, our model shows further improvements in the result of report generation, highlighting the value of regional information in enhancing the perfor- mance of textual generation. Additionally, we exemplify the output of MedRegA for the grounded report generation task in Figure 17. The examples reveal that even when the previous SOTA method MedDr successfully detect the abnormality in the given medical scan, it still struggles to illustrate the specific region of the diagnosis, hindering further performance improvement and interoperabil- ity. In comparison, our proposed MedRegA is capable of locating the specific area related to each descriptive sentence, accelerating its feasibility for clinicians.
28

---

Published as a conference paper at ICLR 2025
Table 11: Performance comparison on Medical Image Classification task. cannot generate valid outputs.
indicates the model
Dataset
Modality
Med-Flamingo Single-Labe Label Clossification
LLa
RadFM
MedDr
Ours
MURA
X-ray
34.30
36.31
49.50
37.81
69,99
PneumoniaMNIST
X-ray
69.72
57.43
38.02
87.30
85.23
OrganAMNST
CT
4,78
14.90
1.23
20.70
29.37
OrganCMNIST
CT
1.49
7.50
6.62
6.99
24.22
OrganSMNIST
CT
1.32
7.27
6.49
8.68
18.98
ISIC2016
Dermatology
44,51
43.02
37.60
50.73
65.16
ISIC2018
Dermatology
10.73
4.57
7.46
11.84
PAD-UFES-20
Dermatology
2.59
12.90
13.81
20.24
KatherColon
Pathology
10.37
12.40
6.54
16.65
43.23
Messidor-2
Fundus
17.79
29.84
18.31
39.72
54.74
OCT2017
OCT
13.33
20.97
23.24
41.13
88.45
OCTMNIST
OCT
10.00
31.50
19.97
58.30
64.46
BUSI
Ultrasound
22.60
22.94
29.8
30.90
45,20
BreasIMNIST
Ulraaouud
26.79
39.72
27.11
66.10
70.35
Multi-Label Classification
CheXpert
X-ray
4.66
18.99
15.60
6.95
21.43
ChestMNIST
X-ray
4.90
13,40
11.96
CXR14
X-ray
5.65
7.80
9.35
VinDr-CXR
X-ray
1.59
2.36
6.57
7.11
11.89
VinD--PXXR
X-ray
2.30
1.92
6.29
8.20
6.33
VinDr-SpineXR
X-ray
3.72
2.49
14.20
26.
30.16
VinDr-Mammo
Mamunggraphy
0.85
3.25
5.7
8.87
9.99
BRSET
Fundus
2.48
7.93
6.76
REMID
Fundus
1.53
2.65
5.54
1.89
6.55
Radiology
Ultrasound
Instruction: Please dingnose discases chest x-ray image. The possible diagnoses are [Diseave List].
Instruction:
Please diagnose the disease in the ultrasound image. The possible diagnoses are benign, maligmant, normal.
Pncumonia
Answer: Benign
Dermatology
Ophthalmology
Instruction: Pleasse disgnose the diseases in the dermoscupi image. The possible diagnoses are [D] [Disesse L.st].
Pleaso diagnoss the ciscase in fundus image. The possible diagnoses are [Di Ebleesse List).
Instruction:
Answer:
Melatocytic nevus
Answer: disbetie retinopathy, dinbetic macaler edema
Figure 16: Disease diagnosis examples, [Disease List] represents all the diseases included in the corresponding dataset.
Table 12: Performance comparison on the quality of reports for organ grounded report gener- ation.
Model
BLEU-1
BLEU-4
METEOR
ROUGE-1
ROUGE-L
BertScore
InternVI.
10.00
2.54
7,89
8.90
7.32
59.20
MedDr
13.42
4.15
8.10
11.45
10.32
60.50
MedRegA
28.20
9.55
27.01
22.86
16.72
66.46
29

---

Published as a conference paper at ICLR 2025
SEroIn mnth
MedRigg Predicted
uprn colppe aupntay
MadRagA Predicte Predicted Cround Truth
RIRREESA
An mn was
Tue de
Mad MistBaak Dadilie
Meziond Droficio
Cardican
Tianis danik There oe palceaury alows.
Cardia
Bergeimen part Minglagge Wilt
plenzi speci
Scoand Irghh
18%
Laft
aftens a acrosing smer caleruaa constattioe panans
St R MiC B.
(b)
MadDe Prediction
CC 2O Te
tHts TR, COMMENI
Stagle 20 maans cEraraaa sare rtabl IuD laags
MadRayA
Predictin
UN kdbk SAEEEGGG RE0 SATAAARRRAERR
IN Led Leay, Graad glas acdi in auouprance regasic hupar Acall B
ARR 8M de EAAA
2:14 Seace Tom.
Fagh Lag Qnall plen sut on desaroat on
truth
Ch: fdll:
ES Laf Dieatiates Acsali
C.W:
C/0 70 1u Bod (M S.E. TINT
SaadaM sodiac chtbe densi segartar the Lett lorw libe
S 4....-1-7...... A 18 101.1/801/1111111000
ris ligg Auacoa romr moprii 43 cac, mitf coods is checved en aestric
(c)
400. 6.0 A R85S5RAAA
Mer of have tain en
Beat
CS 1 stis BRL. H. WISHEENE REEKEER
1% aewreeeeeealll
ielenn veveee/aaturae www.aaatttttttttttttt..
FEW. www. whw.eamaaaaaaaaa.oo..... A. Fat. JaMM,, 74 I.R
Tha chanaa soatis (8150 www.hst 00 drecca The Latia chamtal
Prediction
Ae Su lives, MESIFII Can. Eiit. 21 St.l
Adaiici ste orpaa a with Thaass vut nut comkerment
Declica cormin conk kell
1::: 2 Aub. d0
Migh Baler Ao
Catre Tooie Modestor rnder pe on Merr with meseo ret thee snodhecs
Esace ..A
Ground Truth
(d)
Micka kas, (N8 1112 (TBAI the 1p 7.51
Jel. SPICOERNEETT
Laer
A
FISUR POLRRRCRREER A IE: Alattiem E 2. Si
al bil RiE autriig Bild Roa dlautic 3o Buat cuale ortinaapec ecsaagiii dica
SS SWES.
Might Eirrnio
BEURgO BIT
70th
0004 YeaA El IUT
Aosl cranders
Figure 17: Generated Grounded Report Examples. (a) shows the example on the MIMIC-CXR- sourced grounded report dataset. (b) demonstrates the example for the grounded reports focusing the lesion part. (c) and (d) present the examples of organ grounded reports.
30

---

Published as a conference paper at ICLR 2025
C.3 COMPARISON WITH FINE-TUNED BASELINES
We have evaluated the fine-tuning results with two baselines, Med-Flamingo (Moor et al., 2023) and LLLVA-Med (Li et al., 2024) on the region-centrie tasks to prove the effectiveness and potential of our proposed dataset. We apply subset of 5K samples from the proposed MedRegInstruct dataset to fine-tune the baseline models. This subset covers Region-to- Text Identification and Text- to-Region Detection tasks. We evaluated the models using the test split of the same subset. For Med-Flamingo (Moor et al., 2023), a few-shot learner adapted to the medical domain, we perform 5-shot learning with dataset. For LLaVA-Med (Li al., 2024), the sub-dataset for 10 epochs. Table 13 shows the results. It can be observed that both Med- Flamingo (Moor al., 2023) and VA-Meed Li et al. (2024) can improve the regional tasks after finetuning with our proposed dataset, demonstrating the effectiveness of our dataset in extending regional knowledge.
finetuned the model on
Table 13:
Comparision with fine-tuned baselines on Region-Centric tasks. 'N/A' indicates the model cannot generate valid outputs
Med-F Fad-FFlamnngo
LLaVA-Med
Metric
MedRegA
Zero Shot
Few Shot
Base
Fine-tuned
Region-to-
Text Identification Task
BLBU
2.19
32.02
0.05
21.49
59.06
F1
3.79
33.19
0.14
25.08
59.21
Recall
11.70
33.40
1.99
32.23
59.19
Accuracy
6.62
16.87
0.91
31.93
51.72
BertScore
50.36
71.53
32.18
58.21
82.49
Text-1o Region Detection Task
Text-to-Region
Region-Level Fl Alignment Fl loU
18.36
19.68
56.08
N/A
20.65
N/A
19.53
52.52
19.52
25.27
47.43
D REGION-ALIGNEI EVALUATION
To quantitatively evaluate the medical MLLMs' regional perception and comprehension capabili- evaluation framework to measure the
ties on Region-Centric tasks, we introduce a model performance these tasks.
Region-to-Test Identification task. Since the output for region-to-text identification task in the form of pure texts, adopt Natural Language Generation (NLG) metrics for performance measurement, including BLEU-1, F1 score, Recall, Accuracy and BertScore.
Text-to-Region Detection task. We classify thetext-to-region detection task into four categories according to the number of detected objects and the number of regions per object: single-object single-region, single-object multi-region, multi-object single-region, and multi-object multi-region, as illustrated in 1..1.1. We evaluate the performance across three dimensions: (1) Object-Level. assessing the correctly identified objects; (2) Region-Level: assessing accurately detected regions; (3) Object-Region Alignment: assessing whether the detected boxes are correctly aligned with the corresponding object.
For single-object detection, all the output boxes are aligned with the given object, where evaluating the detection performance from the region-level sufficient. The region-level evaluation metrics are represented in Figure 18. If the object is related to only one region, we report the accuracy of the detected boxes. However, if the object corresponds to multiple regions, we calculate the precision, recall and Fl score.
For multi-object detection, the alignment of text and box must also be considered. As illustrated in Algorithm I, we first apply Hungarian Matching get an optimal loU-based assignment for object- region pairs set of each input, where each predicted region is matched with the ground truth with the maximum IoU score. Subsequently, for each matched predicted and ground truth object-region pair, we identify whether the object and region are correctly detected andproperly ligned. We calculate the three-dimensional metries for each sample and utilize the averaged score for overall evaluation.
31

---

Published as a conference paper at ICLR 2025
Singe-object Singe-region
Singe-object Multi-region
Reglon-Level Prediction= Detected Regions
Regions =1/3
Region-Level Accuracy= Detected Regions Regtons
Reglon-Level Recall= Predicten Regions
Regions 1/2
Figure 18: Region-level Evaluation for Single-Object Detection.
Algorithm Region-Aligned Evaluation
Input:
Predicted Object-Region Pairs Sets P for all the inputs
Ground Truth
Pairs Sets 9 for all the inputs
Output:
AeigneeAllinedd Evaluation Metrics
for each P. (pteet, -p giin ) E E, G. (gghert Io
) ) E G do Optimal loU Assignment (p (preooon Im oreeiion) m
Mantgarian_Machingg( Gt) for each region region do b> iterate over each matched object-region pair for an input
Detected Objects +- 0
Detected Regions + 0
Aligned Pairs
Calculate IoU between (pregion and regiion)
is featt = 9/e then
the object correctly detected
Detected Objects +- Detected Objects +
end if
if IoU 2 0.5 then Detected Regions +- Detected Regions + end if prant = glamt & loU 2 0.5 then
>> the region is correctly detected
current object-region pair is correctly aligned
Aligned Pairs Aligned Pairs end if end for Object-level Precision +
Detected Objects
Object-level Recall + Detected Objects
the Objects
Predicted Objects
Region-level Precision <
Dewected Regions
Region-level Recall
Detected Regions
M
Object Region-level Alignment Precision Aligned Pairs
A
Object Region-level Alignment Recall + end for Calculate the averaged metrics over all the inputs
Aligned & calculate the metrics for each sample N
Grounded Report Generation task. The evaluation Grounded Report Generation performance composed of textual evaluation and regional evaluation. For the textual evaluation, apply NLG metries to assess the quality report part, while regional evaluation follows the same discipline text-to togioioon detection task.
E EFFECTIVENESS OF REGIONAL COT
Since our model has the ability to perceive regions, we also explore the impact of guiding the model with abnormal regions to enhance generation. For instance, we test the model on the visual ques- tion answering dataset SLAKE and multi-label classification VinDr Series datasets, by enforcing the model to first identify abnormal areas and then prompt the model with the bounding box. Ta- ble 14 indicates that providing guidance with abnormal region greatly improves the performance, essentially the challenging multi-labe classification task (with 61.75% and 19.74% F1 score BinDr-SpinXXt and VinCXR dataset, respectively), proving the immense value of understanding regional knowledge.
32

---

Published as a conference paper at ICLR 2025
Table 14; Effectiveness of our proposed Regional CoT.
Dataset
Metrics
MedDr
w/o Regional CoT
w/ Regional CoT
IXD
FI
7.11
11.89
19.74
VinDr-PCXR
FI
8.20
6,33
16.49
VinDr-SpineXR
F1
26.80
30.16
61.75
VinDr-Mammo
F1
8.87
9.99
12.91
BLEU-1
76.40
81.55
84.61
F1
77.50
82.06
85.32
SLAKE
CloseAccuracy
83.40
85.35
87.89
OpenRecall
74.20
80.45
83.61
Regional CoT for Diagmosis
Regional CoT for VQA
Ground Truth: Osteophytes, Dise space narrowing
Question: Where is/are the abnormality located? Answer: lower left
(1) Regional
Rogional
Foraminal steroosis
lower right
(ii) w/ Regional CoT
(ii) w/ Regional CoT
Ostcophytes, Disc space narrowing
Stage
Stage
Slage
Stage
(a) Discase Diagnosis
(b) Visual Question Answering
Figure 19: Ablation Study of Regional CoT.
Additionally, present Regional CoT examples tiveness in Figure 19. As shown Figure 19 (a) for disease diagnosis task, the model struggles to directly predict the correct disease but successfully identifies all relevant diseases with Regional COT. This can be attributed to the detection stage in Regional CoT, which encourages the model to first detect the abnormal regions. thereby mimicking the diagnostic process of clinicians. Figure 19 (b) also highlights the impact of Regional CoT on VQA tasks, especially in enhancing the model's awareness of specific locations.
ablation study to further prove its effec-
33


Reviews:
[{"review_id": "WbO1PFImxW", "reviewer": "Reviewer_QLEK", "summary": "This paper proposes a new large-scale, multi-site, medical visual-question answering (VQA) dataset, called MedRegInstruct, with fine-grained region-specific annotation and multiple corresponding region-centric tasks. The proposed dataset contains multiple region-centric evaluation tasks including, region-to-text generation, text-to-region detection, and grounded medical report generation. The paper further proposed a new region-aware multi-modal large language model (MLLM), called, MedRegA, pre-trained with the proposed dataset and regional chain of thought (CoT). The proposed method outperforms multiple existing SOTA medical VQA models on the regional-centric evaluation.", "questions": "Overall, this is a good paper with solid contributions and it has a significant meaning to the field. Still, the reviewer has a few questions here.\n\n1. How is the English version of the text report/caption generated for the in-house Chinese data? What kind of translation was used here?\n2. According to the “Structure Detection” section on page 5, the structure bounding box in the dataset was generated with a fine-tuned MLLM on the existing Region-Text dataset given the corresponding organ prompt. The reviewer wonders if any kind of evaluation was done on this fine-tuned MLLM, how accurate is the extract bounding box? And why not train a more straightforward detection model for this purpose? Considering that the specific detection model usually performs better than a general MLLM.\n3. Also, as mentioned above, the reviewer wonders how much will the other baseline medical MLLMs improve if they were trained with the proposed dataset. The comparison between InternVL and MedRegA in the paper is not very fair as the InternVL was only trained with natural image-text pairs. The reviewer understands it may take tons of extra time/money to conduct such an evaluation, but it would still be an interesting question to explore.\n4. According to the abstract, the code and model will be released to the public, but the reviewer wonders if the data will be released as well. This is pretty critical as the dataset is the major contribution of the paper. The answer to this question will influence the reviewer's final score for the paper.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The proposed new region-centric medical dataset MedRegInstruct addressed the need for fine-grained, region-level, training and evaluation datasets in the medical domain. It is novel to the field and will help the development of the field as provides a more fine-grained training and evaluation annotation. It is proven that the method pre-trained with the new dataset can demonstrate more robust performance in the targeted region-specific tasks and also other related medical tasks. \n2. The paper has provided enough details about the data curation and evaluation, including the process of question-answering pair generation from medical data with region annotation, a prompt for each downstream evaluation, and detailed settings for each task. The author also seems to claim they will release the code and pre-trained model later.\n3. The paper is overall well written with nice figures and a clean presentation. It is easy to follow even if there are many details in it.", "weaknesses": "1. While the proposed dataset is novel and important to the field, the corresponding MedRegA model is not that novel, though I hate to say that, in comparison. The overall model design is basically the same as the base InternVL model and the major change is the training scheme and prompt formulation with the new dataset, which can be extended to other medical VQL models as well. While the author further proposed single-step regional CoT in the paper, it is still relatively straightforward. **Yet**, this is not a major drawback as the performance of this model demonstrates the superiority of training with the proposed dataset.\n2. Similarly, it seems that the baselines used in the evaluation are not all fine-tuned with the proposed dataset, which enables the model to understand regional information. Comparing the model trained on the proposed dataset with baselines that are not trained on this dataset is not very fair, to some extent. It would be interesting to see how well each baseline would improve if they were fine-tuned on the proposed dataset, even if just in a few-shot manner.\n3. Another concern about this paper is the quality of the data. Most of the regional bounding box-text pairs in the dataset were taken from the existing dataset and generated from an automated rule-based system and an LLM. The evaluation of the data quality is missing in this paper, which is very critical as the main purpose is **fine-grained regional evaluation,** where the quality of the bounding box and the correctness of the corresponding text description are very important. It would be better if some sort of data quality check could be done before releasing it to the public.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VTKrqaqg6d", "reviewer": "Reviewer_5dSL", "summary": "In this work, the authors tackle the current problem in MLLMs (multimodal large language models) of how they don't necessarily \"focus\" on particular regions, but rather take the entire image as context and then have to solve a variety of downstream tasks. To address this problem they offer two main sets of contributions. The first, is the formulation of several region specific tasks, namely region-to-text identification, text-to-region detection, and grounded report generation, which they combine into a dataset called MedRegInstruct. The second is the training strategy and modeling paradigm for a MLLM they propose, which they refer to as MedRegA. MedRegA is a bilingual, as it was trained on both English and Chinese paired datapoints, model that they apply to this larger dataset they prepare in comparison to currently available medical MLLMs.", "questions": "• What was the filtering criterion for reducing SA-Med2D-20M?\n\n• Is there a noticeable performance difference between English and Chinese tasks? Is there paired data that this can be evaluated on? • • • \n\n• Overall, it is unclear from the paper why exactly these two particular languages are used, other than being able to source data for both, rather than just arbitrary languages with medical report data.\n\n• Is the MedRegInstruct dataset planning to be release?\nFor the Region-Text dataset, why is the case that there needs to be this split into half for region-to-text and half for text-to-region? Can't every datapoint server the purpose of being either during training?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "• The paper really introduces another method of interaction, in the bounding boxes, that can be quite useful as MedRegA is able to reason about specific regions. This is an immensely useful tool for narrowing of problems as opposed to current trained MLLMs have to do.\n\n• The proposed training strategy for learning how to incorporate the bounding boxes is simple and scalable, seems like it could be generally applied to other strategies that want to apply bounding boxes.\n\n• Regional CoT makes a lot of sense intuitively and provides a nice bias for solving region specific problems.\n\n• MedRegA has very strong performance on the chosen set of benchmarks and metrics in comparison to existing state of the art methods.", "weaknesses": "• The section of the report grounding dataset is quite confusing as to how its constructed, and how much it relies on the models it uses for automation for providing correct labels/groundtruth. For example, is InternLM perfect at sub-select different parts of the reports? \n\n• The paper claims to work across a broad set of modalities, but when considering the grounding dataset it seems that it only considers, as other MLLMs do, Chest Xrays?\n\n• It's unclear if the comparison to existing baselines are exactly fair as the trained MedRegA has access to more data. In fact, compared to the baseline of InternLM, it should be strictly better because MedRegA is trained with InternLM as an initialization. While it's true that the baselines don't provide a mechanism for highlighting particular regions, they still should be adapted to the new data that the authors are evaluating on.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sxnXsOMK8C", "reviewer": "Reviewer_ToYk", "summary": "This work introduces a novel region-centric task and presents a large-scale Chinese medical dataset named MedRegInstruct. To address this new task, the authors developed a bilingual multimodal large language model, MedRegA, which outperforms baseline methods across most evaluation metrics.", "questions": "1.\tPlease explain why certain models were fine-tuned in the comparison while others were not.\n2.\tThe authors state that adding CoT improves model performance; however, no comprehensive ablation experiments were conducted to confirm this.\n3.\tHave the authors considered comparisons using other large models as the backbone?\n4.\tDo the authors plan to open-source this dataset?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper is well-organized.\n2.\tThis work introduces a novel task alongside a large-scale medical dataset.\n3.\tThe authors propose a new model that significantly outperforms other models across the majority of datasets and evaluation metrics, particularly for the newly proposed task.", "weaknesses": "1.\tThe article’s description of the model structure in the main text is incomplete, relying heavily on Figure 4 to convey the model’s architecture. For example, the structure of the encoder and tokenizer remains ambiguous. The authors should clarify which components utilize existing models, which are fine-tuned, and which are trained from scratch.\n2.\tThe fairness of the model comparison is questionable. In Table 1, only a subset of models has been fine-tuned based on the test dataset, leaving others unmodified. Is this due to an inability to fine-tune certain models, or for another reason? The authors should provide detailed criteria for choosing to fine-tune specific models and address the fairness of this experimental design.\n3.\tOn the newly proposed task, the authors’ model demonstrates a substantial performance advantage. They mention that existing open-source models cannot accurately capture coordinates, but it would be insightful to know if fine-tuning on the authors’ proposed dataset could improve this. If fine-tuning other models would yield better results, the authors should do so for a more persuasive comparison. Conversely, if fine-tuning has no impact, the authors should explain the model’s unique advancements to provide deeper insights.\n4.\tThe role of the CoT component is somewhat unclear, with insufficient ablation experiments to support its effectiveness. The authors should conduct a more specific ablation study to illustrate the impact of CoT rather than merely reporting improved results with its inclusion.\n5.\tWhile the authors claim that the proposed model can perform diagnostic tasks, there is no clear evaluation or user study provided. A brief evaluation or explanation of the diagnostic results would add clarity.\n6.\tSeveral prior studies focus on medical image segmentation, which aligns well with the proposed region-centric task. Have the authors considered using such methods as feature extractors, backbones, or even as comparative models?\n7.\tThe experimental results on report generation are insufficient, as the authors rely solely on descriptive analysis without quantitative evaluation or user studies. Similar to point 5, if the authors wish to validate effectiveness without explicit evaluation metrics, a user study would provide valuable support.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vKxEVJoGOz", "reviewer": "Reviewer_WceT", "summary": "This paper presents MedRegA, a region-aware medical multimodal language model capable of handling both image-level and region-level vision-language tasks across diverse medical modalities. The authors propose three Region-Centric tasks and create a large-scale dataset called MedRegInstruct to enhance the model's focus on specific anatomical regions. They introduce a Regional Chain-of-Thought approach to further improve performance. Experiments show that MedRegA outperforms existing models on tasks like visual question answering, report generation, and medical image classification, demonstrating significant versatility and interpretability.", "questions": "Please address the weaknesses mentioned above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The comprehensive MedRegInstruct dataset is valuable and likely to have a significant impact on the community.\n- Introducing Regional Chain-of-Thought is insightful and significantly improves the model's performance.\n- The model achieves state-of-the-art performance without fine-tuning on specific benchmark datasets, indicating strong generalization across different domains.", "weaknesses": "- Human validation of the automatically generated annotations is needed to prove the quality of the dataset.\n- The paper lacks comprehensive comparisons in report generation with state-of-the-art methods, such as those on the ReXrank[1] leaderboard.\n- Detection task of multiple regions still remain challenging for purposed method.\n\n[1] Rajpurkar Lab. ReXrank Leaderboard. 22 July 2024, https://rajpurkarlab.github.io/rexrank/.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)