Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks

## Abstract

Abstract Several medical Multimodal Large Languange Models (MLLMs) have been developed to address tasks involving visual images with textual instructions across various medical modalities, achieving impressive results.
Most current medical generalist models are region-agnostic, treating the entire image as a holistic representation. However, they struggle to identify which specific regions they are focusing on when generating a sentence.
To mimic the behavior of doctors, who typically begin by reviewing the entire image before concentrating on specific regions for a thorough evaluation, we aim to enhance the capability of medical MLLMs in understanding anatomical regions within entire medical scans.
To achieve it, we first formulate Region-Centric tasks and construct a large-scale dataset, MedRegInstruct, to incorporate regional information into training. Combining our collected dataset with other medical multimodal corpora for training, we propose a Region-Aware medical MLLM, MedRegA , which is the first bilingual generalist medical AI system to simultaneously handle image-level and region-level medical vision-language tasks across a broad range of modalities. Our MedRegA not only enables three region-centric tasks, but also achieves the best performance for visual question answering, report generation and medical image classification over 8 modalities, showcasing significant versatility. Experiments demonstrate that our model can not only accomplish powerful performance across various medical vision-language tasks in bilingual settings, but also recognize and detect structures in multimodal medical scans, boosting the interpretability and user interactivity of medical MLLMs. Our project page is https://medrega.github.io .

## 1 Introduction

Figure: Figure 2: The significance of Region-Centric ability. (a) Comparison between the region-agnostic model (MedDr) and the region-centric MedRegA in analyzing lesion area within the medical scan. (b) Performance comparison of prompting the model with and without regional information on five benchmarks of Visual Question Answering (VQA) and classification tasks.
Refer to caption: extracted/5953015/imgs/intro2-6.png

In recent years, Multimodal Large Language Models have witnessed rapid advancements (OpenAI, ; Liu et al., ; Reid et al., ; Bai et al., ; Chen et al., ),which also appear promising for adaption into the healthcare field. By integrating visual and textual modalities, MLLMs can address diverse medical needs, such as patient consultations, medical report generation, and disease diagnosis.
Several medical Artificial Intelligence (AI) models have been proposed to tackle different tasks involving visual images with textual instructions across multiple medical modalities and achieved impressive results (Moor et al., ; Li et al., ; Tu et al., ; Wu et al., ; He et al., ; Zhang et al., ). However, most current medical generalist models are region-agnostic, tailored to treat the entire image as a holistic representation, but struggling to detect or reason about specific regions within a medical scan. This lack of spatial awareness often leads to inaccurate descriptions that negatively affect the quality of the generated reports.
As illustrated in Figure 2-a, region-agnostic models, such as MedDr (He et al., ), mistakenly identified the lesion area in the left part of the brain, while the ground truth is located in the right part. This incorrect localization led MedDr to classify the lesion as surrounding edema instead of glioma, as radiologists noted. Consequently, this makes it difficult for clinicians to verify, trust, or revise the generated report, thereby hindering the model’s interpretability and clinical usability.

The primary challenge arises because previous methods were not trained on region-centric tasks. Specifically, medical reports often consist of descriptions of multiple regions, making it unclear which regions the model is addressing for each generated sentence. This ambiguity reduces reliability and does not align with the clinical workflow. Radiologists typically begin by reviewing the entire image and then concentrating on specific regions to ensure a thorough evaluation. Inspired by this process, we aim to integrate region-centric information into medical MLLMs, mirroring the radiologist’s workflow. This approach is crucial as it ensures the model focuses on the correct regions being described, whether normal, abnormal, uncertain, or in any other state.

To address the above limitations, we first introduce three Region-Centric tasks:
(1) Region-to-Text Identification aims to identify structures, organs, or any abnormalities within a given bounding box region as input; (2) Text-to-Region Detection aims to accurately locate the positions of structures or abnormalities described in the instruction by providing bounding boxes; (3) Grounded Report Generation aims to generate detailed reports with corresponding bounding boxes for associated anatomies in the medical image.
To enable MLLMs to perform these three tasks, we constructed a large-scale dataset, MedRegInstruct, encompassing all the above tasks.
We collect real clinical data from a local hospital, including approximately 25K Chinese scan-report pairs of X-ray, CT, and MRI modalities from 15K patients. Then, we propose an automatic labeling system to curate grounded reports, lowering the expense of manually annotating fine-grained organs within each medical scan.

Combining our collected dataset MedRegInstruct with other medical multimodal corpora for training, we propose a Region-Aware medical MLLM, MedRegA, which is the first bilingual generalist medical AI system to simultaneously handle image-level and region-level vision-language tasks across a broad range of modalities, such as radiology, pathology, dermatology, and ophthalmology; see Figure 1.
Experiments presented in Figure 2-a show that our MedRegA can precisely output the bounding box of the specific region being focused on, while other existing best-performing medical MLLMs cannot. This significantly helps the model to correctly describe the condition of certain areas (Figure 2-a) and further enhances other downstream tasks such as visual question answering and diagnosis (Figure 2-b).
In summary, our MedRegA outperforms MedDr by 3.91% and 8.03% (BLEU-1) on MIMIC-CXR and IU-Xray in English report generation, with an additional improvement of 27.34% for generating Chinese reports. Besides, our MedRegA not only enables three region-centric tasks but also achieves the best performance for visual question answering, report generation, and medical image classification over 8 modalities, showcasing significant versatility.

To summarize, our contributions can be concluded as follows:

(1) We establish Region-Centric tasks with a large-scale dataset, MedRegInstruct, where each sample is paired with coordinates of body structures or lesions. This would expand the model’s functionality to perceive regions within the medical scan, encouraging the model to focus on critical areas and improving interpretability.
To benchmark the regional ability of medical MLLMs in performing those tasks, we propose a Region-Aligned evaluation framework to assess the quality and alignment of the output text and regions.

(2) Based on the proposed dataset, we develop a Region-Aware medical MLLM, MedRegA, as a bilingual generalist medical AI system to perform both image-level and region-level medical vision-language tasks, demonstrating impressive versatility.
In the inference stage, we introduce Regional CoT (Chain-of-Thought) to further enhance model performance with its spatial knowledge.

(3) We evaluate our model on comprehensive medical tasks, including visual question answering, report generation, image classification, region identification, region detection, and grounded report generation. MedRegA outperforms SOTA methods by a large margin on traditional visual-language and region-centric tasks.

## 2 Related Work

Vision-Language Datasets for Medicine.
In the advancement of Multimodal Large Language Models, there is a strong demand for image-text datasets that serve as pretraining data sources of MLLMs, as these datasets play a crucial role in impacting model performance. The most universally applied vision-language datasets in medicine including those for visual question answering (VQA) tasks (Lau et al., ; Liu et al., ; He et al., ; Zhang et al., ; Chen et al., ) and medical report generation (MRG) tasks (Johnson et al., ; Hamamci et al., ). However, these datasets lack region annotations relevant to the texts, impeding MLLMs from further understanding structural information within medical scans. Huang et al. (); Xie et al. () has attempted to close the gap by incorporating region grounding into multimodal conversations, but had limited access to manually written descriptions from real clinical scenarios. Although works of Zhang et al. (); Lei et al. () constructed region-wise reports paired with anomaly masks from clinical data, they concentrated on chest CT and brain MRI respectively, still leaving a blank space for other body structures such as colon, liver, and pancreas. As a consequence, existing datasets have not met the requirements in developing a generalist MLLM in healthcare, especially to comprehend regional information and accommodate diverse multimodal inputs from a wide range of medical systems. This poses significant challenges in creating a model that can effectively assist clinicians across various specialties.

Medical Multimodal Large Language Models.
The rapid progression in Multimodal Large Language Models has driven the integration of MLLMs into the medical field to enhance diagnostic processes, medical research, and patient care.
The development of large-scale medical models follows the trend of progressively scaling up the application scope.
Several AI assistants are specifically designed for a single modality. Given the abundant public datasets for Chest X-ray (CXR) images, Thawkar et al. (); Pellegrini et al. (); Chen et al. (); Hyland et al. () have facilitated MLLMs’ ability to understand CXR scans. Specifically for pathology, Lu et al. () curated a large pathological image dataset and presented a pathological AI assistant.
Since the large-scale parameters of MLLMs enable them to store knowledge across various modalities, extensive works enlarge the coverage of medical imaging to deploy more general and applicable medical MLLMs (Wu et al., ; Chaves et al., ; Bai et al., ; Zhang et al., ; Moor et al., ; Li et al., ; Liu et al., ; Tu et al., ; He et al., ; Zhang et al., ).
However, without the capability of recognizing and detecting anatomies within medical scans, previous models are still confronted with challenges in providing expert-like detailed descriptions of specific structures or lesions, which notably limits their reliability and interpertability.

Multimodal Large Language Models with Regional Ability.
Enhancing the regional ability to perceive particular objects and ground responses with related areas is crucial for facilitating visual understanding, making it a prevailing target in training MLLMs for natural images (Wang et al., ; Peng et al., ; Guo et al., ). However, enabling MLLMs to detect and ground fine-grained structures in medical scans remains largely underexplored. Current research primarily focused on a limited set of region-related tasks involving only a few modalities. For instance, models proposed by Huang et al. (); Alkhaldi et al. (); Zhou et al. () are capable of detecting single structures based on the given prompts, but failing to delineate fine-grained descriptions for each region, as a clinician could. Bannur et al. () introduced a grounded report generation task and developed an MLLM to generate descriptions of all findings in a medical image accompanied with corresponding location, but the model was restricted to receiving merely chest X-ray input. Based on these limitations, we aimed to fully integrate the function of recognizing, detecting and describing regions for a wide range of modalities into our proposed model.

## 3 Region-Centric Dataset Construction

In this section, we introduce the MedRegInstruct dataset, which is composed of two subsets: (1) the Region-Text Dataset containing 550K triplets of image, question-answer pair, and corresponding regions, (2) the Region-Grounded Dataset including 240K instances where each image combines a report annotated with regions and their fine-grained descriptions.
Data distribution details can be found in Appendix A.1.
Compared with the previous vision-language datasets for medicine (Liu et al., ; Johnson et al., ; Hamamci et al., ), our dataset not only places an extra emphasis on cultivating the regional abilities of medical MLLMs, but also substantially enhances diversity by integrating a wider range of anatomical structures and a bilingual setting for report generation.

### 3.1 Region-Text Dataset

The creation of our Region-Text dataset stems from SA-Med2D-20M (Ye et al., ), which is a large-scale segmentation dataset containing 2D medical images of almost all the body parts.
We filter approximately 285K images from the original dataset to maintain the diversity of anatomical structures and anomaly lesions. The original labeled segmentation masks are converted into bounding boxes to represent the highlight region.
The image-region pairs were formulated into two forms: (1) Region-to-Text: the model outputs the information of a highlighted region given the bounding box; (2) Text-to-Region: the model locates the specific regions by generating bounding boxes.
To obtain question-answer pairs, we fill the predefined templates with region names and coordinates.
In statistics, the Region-Text dataset contains 550K data items, with approximately a half designated as Region-to-Text and the other half as Text-to-Region respectively.

### 3.2 Report-Grounded Dataset

The report-grounded dataset is sourced from two databases, MIMIC-CXR dataset (Johnson et al., ), and our in-house clinical data containing 25K X-ray, MRI and CT scans from 15K patients. The in-house data covers multiple regions, including the brain, abdomen, chest, spine, and pelvis. In this section, we will illustrate the automatic data processing procedure to curate grounded reports following three steps: Image-Report Pair Construction, Report Refinement and Structure Detection.

Image-Report Pair Construction. For the MIMIC-CXR dataset, we follow previous works (Wu et al., ) to utilize both frontal and lateral images, and include findings and impressions in the report. However, the MRI and CT scans in our in-house dataset are originally 3D volumes. From these volumes, we extract central slices from each scan to formulate the 2D inputs, which typically provide the most representative views of the anatomical structures. In total, we construct 95K image-report pairs from the collected clinical data, which can also be employed to stimulate the bilingual abilities of MLLM by enhancing it to generate structured reports in Chinese.

Report Refinement. Our objective is to decompose raw report data into fine-grained descriptions for each organ mentioned in medical scans.
We observe that the open-source InternLM model (Cai et al., ) demonstrates impressive textual comprehension ability for Chinese medical reports. To better leverage the LLM’s information extraction ability, we first implement a rule-based strategy to summarize a coarse overview of all the organs referred in the report. Subsequently, we employed InternLM to segment each report into detailed descriptions given the organ list as the prompt.

Structure Detection.
The existing chest x-ray medical report dataset, MIMIC-CXR, has been labeled with bounding box coordinates of 29 anatomical locations by Chest ImaGenome (Wu et al., ), from which we select 12 standardized ones in chest.
We combine the selected bounding boxes with the corresponding description in the segmented reports to form the region-grounded reports for chest X-ray scans. Overall, 220K grounded reports are constructed from the MIMIC-CXR dataset.
In our in-house dataset, certain scans are manually labeled with the lesion area, such as brain tumor, lung cancer, and other abnormalities. For these scans, 6,881
grounded reports specifically for abnormalities are acquired based on the manual annotations.
Nevertheless, the region labels of anatomical structures are still in demand. To automatically locate the observed organs within the medical scan, we first finetune an MLLM based on the Region-Text dataset, which contains scans of the same modalites and organs as our collected data. Then, we input the collected images into the finetuned MLLM, prompted with the organs covered in the reports, in order to acquire the corresponding bounding boxes. Subsequently, we match the annotated abnormality coordinates with descriptions for the lesion part and the detected organ regions with corresponding parts in the segmented report.
By combining the generated region with the relevant fine-grained description, we formulated approximately 11K data items for organ-level grounded reports.

Figure: Figure 3: Data Construction Pipeline for Report-Grounded Dataset. The automatic data processing procedure is composed of three steps: Image-Report Pair Construction, Report Refinement, and Structure Detection.
Refer to caption: x2.png

## 4 Region-Aware Medical MLLM

The Region-Aware Medical MLLM MedRegA served as an interpretable bilingual medical AI system to tackle a variety of vision-language tasks, including visual question answering, report generation and classification, with a specialized regional ability for Region-Centric tasks.
In this section, we first introduce the formulation of Region-Centric tasks in our model. Secondly, we illustrate the training process of the model. Finally, we propose a Region-Guided generation strategy aimed at further improving the inference quality.

Region-Centric Task Formulation.
We define the Region-Centric tasks from three perspectives: (1) Region-to-Text Identification, where the model outputs the name of the specified region; (2) Text-to-Region Detection, where the model detects the area of the given organ or anomaly; (3) Grounded Report Generation, where the model generates a report for the medical scan, aligning each description with the corresponding region. To enable the model to encode the regional information, we represented regions in the format of bounding boxes $[x_{1},y_{1},x_{2},y_{2}]$, where $(x_{1},y_{1})$ indicate the top left point of the box and $(x_{2},y_{2})$ denotes the bottom right one. All the values are normalized into integers within the range of $[0,1000)$. These bounding boxes were inserted into the sentences surrounded by a pair of special tokens, like <box>$[x_{1},y_{1},x_{2},y_{2}]$</box>, while the correlated object was marked with <ref></ref>, as illustrated in Figure 4.

Figure: Figure 4: An illustrative example of performing region-centric tasks with MedRegA.
Refer to caption: x3.png

Figure: Figure 5: Comparison of traditional inference and generation pipeline with Regional CoT.
Refer to caption: x4.png

Model Training.
To train a general medical MLLM not limited to understanding regions, we process and integrate various public datasets targeted at different tasks, including visual question answering, report generation, organ classification, and disease diagnosis. We design task-specific instructions to prompt the model to recognize and address different objectives. Considering the competitive performance of the open-source model InternVL 1.2 (Chen et al., ), we utilize it as our general-domain foundation to begin training.Our training process is divided into two steps: alignment training and instruction tuning. During the alignment training phase, we freeze the vision encoder and language model, only fine-tuning the alignment module with medical image captioning datasets. In the instruction tuning stage, we apply both public datasets and our Region-Centric datasets, MedRegInstruct, to optimize the language model, while keeping the other components unchanged. The language model loss is applied as the loss function. More training details can be found in Appendix B.

Regional CoT.
After training on Region-Centric tasks, the model has been equipped with region identification and detection abilities, which have the potential to further enhance model performance on other general tasks. As proved in Zhang et al. (); Yang et al. (), textualizing additional guidance such as spatial coordinates into the prompt would unleash the language model’s ability to process multimodal information. Following these insights, we incorporate regional CoT into the generation pipeline to better leverage these regional skills, as illustrated in Figure 5. The model is required to initially detect critical regions in the input image, followed by generation prompted by the detected regions.
This approach could encourage the model to attend more to internal structures within the medical scan when answering patient consultations or diagnosing diseases.

Table: Table 1: Performance comparison on general medical tasks and region-centric tasks. ‘N/A’ means the scores are not reported. ‘-’ indicates the model cannot generate valid outputs. ‘✗’ denotes the model cannot tackle corresponding tasks. ‘*’ means that the model is fine-tuned on the dataset.

## 5 Experiment

### 5.1 Performance on General Medical Tasks

To evaluate the ability of MedRegA on general medical tasks, namely visual question answering, report generation and classification, we comprehensively compare our model with the base model InternVL (Chen et al., ) and open source medical MLLMs in general domain, including Med-Flamingo (Moor et al., ), LLaVA-Med (Li et al., ), RadFM (Wu et al., ), MedDr (He et al., ), and BiomedGPT (Zhang et al., )
by reproducing their released checkpoints with official prompting instructions.
An overall performance comparison is shown in Table 1 with averaged results on each task. It should be notified that the existing baselines cannot manage to locate highlighted areas within the medical scan and provide valid regional outputs, exposing a huge limitation in differentiating body structures and detecting fine-grained abnormalities.

Performance on Visual Question Answering. In Visual Quesion Answering task, the model needs to answer questions involving the modality, visible structures, and possible diseases of the given medical scan. We perform comparison on several medical VQA benchmarks, such as English and Chinese versions of SLAKE (Liu et al., ), VQA-RAD (Lau et al., ), and PathVQA (He et al., ).
The averaged results on these datasets revealing that MedRegA outperforms all the baselines in the overall metrics by approximately 2% to 40% in English and over 10% in Chinese.

Performance on Medical Report Generation. Medical Report Generation task requires the model to generate a detailed report based on the provided medical scan.
For English report generation, we evaluate our model on the report generation task for chest X-ray datasets MIMIC-CXR (Johnson et al., ) and IU-Xray (Demner-Fushman et al., ).
Moreover, to evaluate the bilingual ability of our model in report generation, we collect a Chinese report generation test dataset covering brain, chest, spine, abdomen and pelvis from the hospital.
Table 1 shows that MedRegA excels all other baselines in generating reports with both English and Chinese, achieving 5.97% and 28.77% higher averaged BLEU-1 score compared with MedDr, respectively.

Performance on Medical Image Classification.
In Medical Image Classification task, the model is expected to either classify the medical
structure shown in the image or diagnose the specific diseases indicated by the scan.
We conduct experiments on both single-label classification and multi-label classification with a wide range of datasets across radiology, ultrasound, ophthalmology, and dermatology.
We report the averaged F1 score in Table 1.
For single-label classification, MedRegA outperforms existing models by a large margin from 15.32% to 30.98%. Multi-label classification appears more challenging for MLLMs due to the difficulties in decoupling subtle symptoms and relating them to corresponding diagnoses.
To enhance the model’s focus on disease lesions, we employ Regional CoT into multi-label classification, and present the results in Figure 2 (b). Specifically, with Regional CoT, the model achieved an F1 score of 61.75% on VinDr-SpineXR dataset, surpassing MedDr and MedRegA without Regional CoT by and 34.95% and 31.59%, respecitvely.
The improved results indicate that incorporating regional information assists medical MLLMs in establishing explicit relationships between local regions and each class label, rather than considering the entire global image with all labels (as illustrated in Figure 5), which further enhances image perception and handles multi-label classification.

### 5.2 Region-Aligned Evaluation on Region-Centric Tasks

To evaluate the regional ability of MedRegA, we implement our model on the proposed Region-Centric Tasks. To quantitatively evaluate the regional perception and comprehension capabilities of medical MLLMs, we introduce a Region-Aligned evaluation framework to measure the model performance on these tasks.
In this section, we first present the assessment metrics for each task and demonstrate the experimental results. Due to the inability of existing open-souce medical MLLMs to generate precise coordinates, we apply an MLLM with an impressive regional ability on natural images, InternVL (Chen et al., ), as the compared baseline.

Table: Table 2: Performance on the region-to-text identification task.

Figure: Figure 6: Definition of Region-Level, Object-Level, and Object-Region Alignment evaluation. Boxes labeled with roman numerals denote groundtruths and those with capital letters denote predictions.
Refer to caption: x5.png

Table: Table 3: Performance on the text-to-region detection task. ‘N/A’ means that the task does not correspond to the evaluation dimension.

Table: Table 4: Performance on Chest X-ray Grounded Report Generation. ‘✗’ denotes the model cannot tackle corresponding tasks.

#### 5.2.1 Evaluation on Region-to-Text Identification

Settings. In the Region-to-Text Identification task, the model is required to identify the name of the specified region based on the corresponding bounding box.
Since the output for this task is in the form of pure texts, we adopt Natural Language Generation (NLG) metrics for performance measurement, including BLEU-1, F1 score, Recall, Accuracy and BertScore.

Results. We categorize the task into (1) structure identification for identifying anatomies such as structures within the brain, heart, lung, abdomen or spine, and (2) lesion identification focusing on abnormalities like tumors and cancers. Results of these two subtasks are reported in Table 2. It can be observed that MedRegA is able to accurately recognize the regions delineated by bounding boxes and associate them with the corresponding areas with 39.85% higher BertScore than InternVL, while previous methods fail to comprehend region encodings in medical scans.
Notably, the model demonstrates a stronger capability of identifying body structures compared to lesions, possibly because anatomies tend to be larger, whereas lesions are subtler and exhibit more variation in shape.

#### 5.2.2 Evaluation on Text-to-Region Detection

Settings. In the Text-to-Region Detection task, the model should detect the region corresponding to specific organs or anomalies. The input textual instruction contains the name of single or multiple objects, and the model is expected to output bounding boxes for all the relevant regions. We classify the task into four categories according to the number of detected objects and the number of regions per object: single-object single-region, single-object multi-region, multi-object single-region, and multi-object multi-region, as illustrated in Appendix A.1.1.
We evaluate the performance across three dimensions: (1) Object-Level: assessing the correctly identified objects; (2) Region-Level: assessing accurately detected regions; (3) Object-Region Alignment: assessing whether the detected boxes are correctly aligned with the corresponding object.
For single-object detection, all the output boxes are aligned with the given object, where evaluating the detection performance from the region-level is sufficient. The region is correctly detected if its Intersection over Union (IoU) score exceeds the threshold of 0.5.
For multi-object detection, the alignment of text and box must also be considered. Figure 6 illustrates the three-dimensional evaluation in the multi-object setting.

Results. Table 3 demonstrate the model performance for Text-to-Region Detection. When detecting objects corresponding to a single region, our model achieves relatively high scores, with detection accuracy approaching 50%. However, when multiple regions are associated with the given objects,

Metrics
MedDr
InternVL
MedRegA
BLEU-1
11.79
5.76
20.04
ROUGE-L
8.38
4.52
14.01
BertScore
66.59
63.22
74.11
Region Acc
✗
-
25.80
IoU
✗
2.06
35.63

Table: Table 5: Performance on Lesion Grounded Report Generation. ‘✗’ denotes the model cannot tackle corresponding tasks. ‘-’ indicates the model cannot generate valid outputs.

Figure: Figure 7: Examples on Organ Grounded Report Generation. The predicted bounding boxes are labeled with the same color as the detected organ in generated texts.

the detection task becomes more complex for MLLMs. In such cases, recall is higher than precision,
inferring that the model suffers from difficulty in detecting all the required regions thoroughly and tends to conservatively generate fewer bounding boxes.

#### 5.2.3 Performance on Grounded Report Generation.

Chest X-ray Grounded Report Generation.
For the grounded reports constructed from MIMIC-CXR, we follow the same training-test split as the report generation task and evaluate the model with those accompanied by organ bounding boxes, consisting of 3,022 samples in total. The model performance on region detection is assessed as the multi-region single-object sub-task defined in §5.2.2.
Furthermore, we integrate NLG and CE metrics to estimate the quality of the descriptions for each region.
As shown in Table 4, our model can detect the chest structures and generate descriptions simultaneously with impressive scores on both report generation and region detection.

Lesion Grounded Report Generation.
To evaluate our model performance on generating grounded reports that concentrate on the lesion area, we sample 438 pairs of tumor coordinates and corresponding Chinese descriptions annotated by doctors from our collected data. The evaluation metrics can be referred to as the single-region single-object sub-task defined in §5.2.2 since each medical scan is associated with a single area of abnormality.
NLG metrics are applied to evaluate the descriptions.
Table 5 demonstrates that our model outperforms established baselines, especially with 33.57% higher IoU than InternVL.

Organ Grounded Report Generation.
As illustrated in §3.2, we enlarge the collected Chinese image-report data with automatically-annotated organs to enrich the report-grounded data. Through self-training with these data, MedRegA has the capability to yield grounded reports with detailed descriptions on organs, as the examples in Figure 5. Without training on region-centric tasks, MedDr can only generate a broad description for each organ and fail to relate fine-grained descriptions with organ coordinates. In contrast, MedRegA managed to ground generated sentences with corresponding organs, and generated more detailed description on the internal condition of the organ. Though there are subtle deviation in the nodule position, MedRegA still provided a thorough insight on the medical scan, exhibiting a remarkable ability to leverage regional information.

## 6 Conclusion

In this paper, we present an interpretable bilingual generalist medical AI system with an additional intention to enhance the ability of medical MLLMs to investigate critical regions within the given medical scan. From data aspect, we first formulate Region-Centric tasks tailored to region identification, detection, and fine-grained grounded report generation for each recognized regions. Specifically based on these tasks, we construct a large-scale dataset, MedRegInstruct, with a semi-automatic labeling system. In terms of model, we propose a Region-Aware medical MLLM, MedRegA, by leveraging our constructed dataset and multimodal medical corpora across diverse tasks for training. Experiments demonstrate that MedRegA achieves promising results on both image and region-level tasks. We believe that region-centric capability is essential in medical MLLM development, since establishing relations between specific regions and generated texts would not only encourage the model to focus on critical areas, but also facilitate interpretability and clinical interactivity.

## Ethics Statement

The in-house dataset used in our research consists of existing retrospective data collected from a hospital, and is intended to advance research in healthcare. All data have been anonymized, which means that personal identifiers such as names, addresses, and contact details have been removed to protect the privacy and confidentiality of individuals. The data is ensured to be analyzed and interpreted responsibly to avoid any potential negative impact on patients or individuals.

## Appendix of “Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks”

## Appendix A Dataset Details

### A.1 MedRegInstruct Dataset

The motivation of creating the MedRegInstruct Dataset lies in the lack of region-centric abilities and interoperability in existing medical MLLMs. In other words, existing medical MLLMs are global-centric, designed to treat the entire image as a holistic representation and struggling to detect or reason about specific regions within a medical scan. As a result, current medical generalist models cannot manage to concentrate on a critical region within the given image, thus leading to inaccurate descriptions specifically in the lesion location. Besides, merely generating diagnostics or reports without fine-grained locations is less interpretable, making it hard for clinicians to trace the source of the generated text.

Thus, we first establish Region-Centric tasks, aimed at improving medical MLLMs’ capability to encode regional information. Then, we develop a large-scale dataset, MedRegInstruct, covering all those tasks in order to train an interpretable generalist medical MLLM with region-centric abilities.

#### A.1.1 Region-Centric Task

Figure: Figure 8: Region-Centric Task Fomulation. In the Region-to-Text Identification task, each question-answer pair contains one region to recognize. The detected region is composed of two types, body structure and disease lesion. In the Text-to-Region detection task, the model would be asked to simultaneously locate multiple regions for an object or multiple objects. Based on the number of detected objects and the number of regions per object, the detection task can be categorized into four subtasks, namely single-object single-region, single-object multi-region, multi-object single-region, and multi-object multi-region. In the Groubded Report Generation task, each region should be matched with a detailed description, discussing the condition of the located organ or lesion.
Refer to caption: x6.png

As illustrated in 8, we propose three types of Region-Centric tasks, (1) Region-to-Text identification: recognizing the name or condition of a given region; (2) Text-to-Region Detection: locating the position of structures mentioned in the instruction; (3) Grounded Report Generation: providing detailed reports for all highlighted anatomies along with
their corresponding regions in the medical scan.

Based on the specific tasks in mind, we construct the MedRegInstruct dataset, consisting of the Region-Text Dataset for Region-to-Text Identification task and Text-to-Region detection task, and the Report-Grounded Dataset for Grounded Report Generation task.

Figure: Figure 9: MedRegInstruct dataset overview. (a) and (b) demonstrate the distribution over modalities and datasets for Region-to-Text and Text-to-Region data, respectively. (c) and (d) presents the subtask distribution and training-test splits for Region-Text Dataset. (e) shows the body structure distribution of the collected report data. (f) exhibits the statistics of Report-Grounded dataset.
Refer to caption: x7.png

#### A.1.2 Region-Text Dataset

The Region-Text dataset is sourced from SA-Med2D-20M (Ye et al., ), from which approximately 285K images are filtered to construct into image-text-region triplets. To formulate the data into region-to-text identification and text-to-region detection task, we first employed GPT-4 to pre-define 50 templates for each. Template examples are shown in Figure 10.

Figure: Figure 10: Template examples to formulate triplets of (image, text, region) into region-to-text and text-to-region tasks.
Refer to caption: extracted/5953015/imgs/template.png

In statistics, the region-text dataset is composed of 278,923 samples for region-to-text task and 262,031 samples for text-to-region task, respectively.

#### A.1.3 Report-Grounded Dataset

The Report-Grounded dataset sources from MIMIC-CXR and a clinical in-house dataset collected from the hospital, containing approximately 240K samples in total. Examples for the grounded reports can be found in Figure 8.

In-house dataset for Chinese report generation. Here we will provide a detailed overview of the collected clinical data. Our in-house clinical data containing approximately 25K X-ray, MRI and CT scans from 15K patients, each with a piece of report written in Chinese. The in-house data covers multiple regions, including the brain, chest, abdomen, spine, and pelvis, as shown in Figure 9. Also, various lesion types are considered, such as brain tumor, ischemic
stroke, cerebral hemorrhage, nasopharyngeal carcinoma, vestibular schwannoma, lung cancer, pul-
monary embolism, liver cancer, colon cancer, kidney cancer, pancreas cancer, prostate cancer, lum-
bar degenerative disease, disc space narrowing, spondylolisthesis, etc. For scans in 3D volume, we
select the central 2D slice that provides the most representative views of the anatomical structures.
In total, our in-house dataset is composed of 27,357 image-report pairs.
Examples are shown in Figure 11.

Figure: Figure 11: Data examples of the Chinese image-report pairs in our collected dataset.
Refer to caption: extracted/5953015/imgs/report-example.png

Grounded report construction from MIMIC-CXR dataset. We apply Chest-ImaGenome (Wu et al., ), which includes bounding box annotations for 29 structures in the frontal chest x-ray images from MIMIC-CXR dataset. From these atanomies, we select 12 standardized ones in chest to create the report-grounded dataset for chest x-ray examination, including left lung, right lung, mediastinum, cardiac silhouette, left hilar structures, right hilar structures, left clavicle, right clavicle, left hemidiaphragm, right hemidiaphragm, right atrium, and abdomen. We filtered the chext x-ray scans paired with annotation boxes, and obtained 225,610 data samples in total.

Grounded report construction In-house dataset. The constructed report-grounded dataset from our in-house dataset contains two parts: (1) Lesion Grounded Report and (2) Organ Grounded Report.
In our in-house dataset, certain scans are manually labeled with the lesion area by experts. This lesions include 12 types in total, namely, brain tumor, ischemic stroke, cerebral hemorrhage, nasopharyngeal carcinoma, vestibular schwannoma, lung cancer, pulmonary embolism, liver cancer, colon cancer, kidney cancer, pancreas cancer, prostate cancer. For these scans, we match the annotated abnormal region coordinates with specific descriptions for the lesion part, acquiring 6,881 grounded reports specifically for abnormalities.
For the rest cases, we follow the semi-automatical pipeline illustrated in §3.2 to construct organ grounded reports. The organ grounded report data covers body parts including heart, lung, liver, colon, kidney, spleen, and pancreas. We selected 5 slices from each original 3D scan to obtain more diverse views, and obtain 10,907 data items for organ grounded reports.

### A.2 Training Datasets on Traditional Medical Tasks

#### A.2.1 Image Captioning

PMC-OA (Lin et al., ) is a biomedical dataset with 1.6M image-caption pairs collected from PubMedCentral’s OpenAccess subset which covers diverse modalities or diseases. We leveraged the dataset for alignment training.

QUILT (Ikezogwo et al., ) is a large-scale vision-language dataset consisting of 802,144 image and text pairs, which was curated from video frames and corresponding subtitles on YouTube. We leveraged the dataset for alignment training.

#### A.2.2 Visual Question Answering

SLAKE (Liu et al., ) is a bilingual radiology VQA dataset consists of 642 radiology images and over 7000 diverse QA pairs annotated by experienced physicians. Following the official split, we used both English and Chinese versions for training, which contains 4919 and 4916 question-answer pairs repectively.

VQA-RAD  (Lau et al., ) consists of 3.5K question-answering pairs on 314 radiology images, where clinicians asked naturally occurring questions about radiology images and provided reference answers. Following the official split, we use 3,064 question-answer pairs for training.

PathVQA (He et al., ) consists of 32,799 open-ended questions from 4,998 pathology images where each question is manually checked to ensure correctness. Every image is paired with several questions related to multiple aspects such as shape, color and location. Following the official split, we use 19,755 question-answer pairs for training.

PMC-VQA (Zhang et al., ) consists of 1.6 million question-answer pairs, which is a large-scale medical visual question-answering dataset generated from PMC-OA. We combined two versions of PMC-VQA and use 329,551 question-answer pairs for training.

MedPix is collected from MedPix website(^1^11https://medpix.nlm.nih.gov), which is a free open-access online database for medical usage. RadFM (Wu et al., ) separate the dataset into MPx-single and MPx-multi. We apply the MPx-single part and use 92,282 question-answer pairs for training.

#### A.2.3 Report Generation

MIMIC-CXR (Johnson et al., ) is a large-scale chest image-report dataset that consists of 371,920 chest X-rays associated with 227,943 reports from 65,079 patients. Following RadFM (Wu et al., ), we use 354,569 cases for training.

IU-Xray (Demner-Fushman et al., ) consists of 7,470 images and 3,955 reports collected from the Indiana Network. Following R2Gen (Chen et al., ), we use 4,720 cases for training.

In-house Dataset. We split the collected reports at the patient level, ensuring that the training and
test sets do not contain overlapping patients. we use approximately 90% of all the data for training,
which contains 24,608 image-report pairs.

#### A.2.4 Medical Image Classification

VinDr-CXR (Nguyen et al., ) consists of 18,000 images that were manually annotated by a total of 17 experienced radiologists with 22 local labels of rectangles surrounding abnormalities and 6 global labels of suspected diseases. The training set contains 15,000 scans, and 3 radiologists independently label each image. Following the official split, we use 45,000 samples for training.

VinDr-PCXR (Pham et al., ) is a pediatric CXR dataset of 9,125 studies that were retrospectively collected from a major pediatric hospital in Vietnam between 2020-2021. Each scan was manually annotated by an experienced radiologist for the presence of 36 critical findings and 15 diseases. Following the official split, we use 4,585 samples for training.

VinDr-SpineXR (Nguyen et al., ) is a large-scale annotated medical image dataset for spinal lesion detection and classification from radiographs. The dataset contains 10,466 spine X-ray images from 5,000 studies, each of which is manually annotated with 13 types of abnormalities by an experienced radiologist with bounding boxes around abnormal findings. Following RadFM (Wu et al., ), we use 8,389 samples for training.

VinDr-Mammo (Nguyen et al., ) is a large-scale full-field digital mammography dataset of 5,000 four-view exams. Following the official split, we use 16,391 samples for training.

CheXpert (Irvin et al., ) is a large public dataset for chest radiograph interpretation, which retrospectively collected the chest from Stanford Hospital, performed between October 2002 and July 2017. The dataset contains 224,316 chest radiographs of 65,240 patients. Following the official split, we use 223,414 samples for training.

MURA (Rajpurkar et al., ) is a large-scale dataset of musculoskeletal radiographs containing 40,561 images from 14,863 studies, where each study is manually labeled by radiologists as either normal or abnormal. Following the official split, we use 36,808 samples for training.

ISIC2018 (Codella et al., ) is a skin lesion dataset acquired with 7 dermatoscope types. Following the official split, we use 10,015 samples for training.

ISIC2019 (Combalia et al., ) is a skin lesion dataset labeled with 8 different categories. Following the official split, we use 25,331 samples for trianing.

PAD-UFES (Pacheco et al., ) is a skin lesion dataset composed of clinical images collected from smartphone devices and a set of patient clinical data containing up to 22 features. The dataset consists of 1,373 patients, 1,641 skin lesions, and 2,298 images for six different diagnostics. We randomly sample 80% of the dataset for training, which includes 1,838 samples.

Kather colon dataset (Kather et al., ) is a dataset of 100,000 non-overlapping image patches from hematoxylin & eosin (H&E) stained histological images of human colorectal cancer (CRC) and normal tissue, covering 9 tissue classes in total.

BRSET (Nakayama et al., ) is a multi-labeled ophthalmological dataset onsisting of 16,266 images from 8,524 Brazilian patients. Multi-labels are included alongside color fundus retinal photos. We randomly sample 80% of the dataset for training, containing 13,012 samples.

ODIR-5K (Li et al., ) is a structured ophthalmic database of 5,000 patients with age, color fundus photographs from left and right eyes and doctors’ diagnostic keywords from doctors. Following the official split, we use 6,392 samples for training.

OCT2017 (Kermany et al., ) includes 83,484 OCT images of 4,686 patients, consisting of 4 categories, normal, drusen, choroidal neoVascularisation (CNV), and Diabetic Macular Edema (DME). Following the official split, we use 82,484 samples for training.

Butterfly Network ultrasound dataset (ButterflyNetworkInc., ) is a large dataset containing 9 different classes of ultrasound images acquired with the Butterfly IQ on 31 individuals. Following Chen et al. (), 34,325 images are applied for training.

BUSI (Al-Dhabyani et al., ) includes breast ultrasound images among women between 25 and 75 years old. The number of patients is 600 females, patients. The dataset consists of 780 images that are categorized into three classes, namely, standard, benign, and malignant. We randomly sample 80% of the dataset for training, which includes 630 images.

### A.3 Training Datasets on Region-Centric Tasks (MedRegInstruct)

#### A.3.1 Region-Text Tasks

SA-Med2D-20M (Ye et al., ) is a large-scale segmentation dataset of 2D medical images built upon numerous public and private datasets. The dataset consists of 4.6 million 2D medical images and 19.7 million corresponding masks, covering almost the whole body and showing significant diversity. We filter approximately 285K images from the original dataset, and construct 242,268 and 229,340 training samples for Region-to-Text Identification and Text-to-Region Detection respectively.

VinDr Series Dataset is a large-scale classification composed of VinDr-CXR, VinDr-PCXR, VinDr-SpineXR, VinDr-Mammo. The datasets provide radiologist’s bounding-box annotation for abnormal areas. We follow the official split and apply the samples with bounding boxes.

ISIC Challenge Dataset contains lesion segmentation data where the original image is paired with manually annotated lesion boundaries. We follow the official split and convert the segmentation map into bounding boxes.

PanNuke (Gamper et al., ) is a semi-automatically generated nuclei instance segmentation dataset. We follow the official split and convert the segmentation map into bounding boxes to formulate region-text pairs.

#### A.3.2 Report-Grounded Tasks

Chest-ImaGenome (Wu et al., ) applied a CXR bounding box detection pipeline to automatically label frontal chest x-ray images from MIMIC-CXR dataset with 29 annotations, from which we selected 12 standardized structures in the chest. Following the split of MIMIC-CXR in RadFM Wu et al. (), we filtered the chext x-ray scans paired with annotation boxes, and obtained 222,588 samples for training.

In-house dataset. The constructed report-grounded dataset from our in-house dataset contains two parts, Lesion Grounded Report and Organ Grounded Report.
We applied 6,443 lesion grounded reports and 10,907 organ grounded reports for training.

### A.4 Test Datatsets

#### A.4.1 Visual Question Answering

SLAKE. Following the official split, we use 1,061 and 1,033 quesion-answer pairs for test on the English and Chinese version, respectively.

VQA-RAD. Following the official split, we use 451 quesion-answer pairs for evaluation.

PathVQA. Following the official split, we use 6,761 quesion-answer pairs for evaluation.

PMC-VQA. Following the official split, the two versions of PMC-VQA test set contain 50,000 and 33,430 quesion-answer pairs, respectively. We test the model on both versions and report the averaged result.

#### A.4.2 Report Generation

MIMIC-CXR. Following RadFM (Wu et al., ), we use 4,710 cases for test.

IU-Xray. Following R2Gen (Chen et al., ), we use 1,180 cases for training.

In-house Dataset. We split the collected reports at the patient level, ensuring that the training and test sets do not contain overlapping patients. we use approximately 10% of all the data for test, which includes 2,749 image-report pairs.

#### A.4.3 Medical Image Classification

MURA. Following the official split, we use 3,193 X-ray images for test.

PneumoniaMNIST. Following the official split, we use 624 X-ray samples for test.

OrganCMNIST. Following the official split, we use 8,216 CT samples for test.

OrganAMNIST. Following the official split, we use 8,827 CT samples for test.

ISIC2016. Following the official split, we use 379 skin lesion images for test.

ISIC2018. Following the official split, we use 1,512 skin lesion images for test.

PAD-UFES-20. Since no official split is provided, we randomly split 20% of the data for evaluation, which contains 460 images.

Kather Colon Dataset. The original dataset is split into “NCT-CRC-HE-100K” and “CRC-VAL-HE-7K” subsets, which share no overlap with each other. Since the “NCT-CRC-HE-100K” subset is applied for training, we use “CRC-VAL-HE-7K” subset for test, which contains 7,180 pathology images.

Messidor-2. The Messidor-2 dataset is a collection of Diabetic Retinopathy (DR) examinations. We treat this task a binary classification to detect DR disease. Since no official split is provided, we randomly split 20% of the data for evaluation, which includes 378 fundus images.

OCT2017. Following the official split, we use 968 OCT scans for test.

OCTMNIST. Following the official split, we use 1,000 OCT scans for test.

BUSI. Since no official split is provided, we randomly split 20% of the data for test, which includes 150 ultrasound images.

BreastMNIST. Following the official split, we use 156 ultrasound scans for test.

CheXpert. Following the official split, we use 234 chest x-ray images for test.

ChestMNIST. Following the official split, we use 22,433 chest x-ray images for test.

CXR14. We follow the official split in Holste et al. () and apply 21,081 chest x-ray images for test.

VinDr-CXR. Following the official split, we use 3,000 chest x-ray images for evaluation.

VinDr-PCXR. Following the official split, we use 1,397 chest x-ray images for evaluation.

VinDr-SpineXR. Following the official split, we use 2,077 spine x-ray images for evaluation.

VinDr-Mammo. Following the official split, we use 4,000 mammography images for evaluation.

BRSET. Since no official split is provided, we randomly split 20% of the data for test, which includes 3254 fundus images.

RFMiD 1.0. Following the official split, we use 640 fundus images for evaluation.

#### A.4.4 Region-Centric Tasks

Region-Text Tasks. We split approximately 10% from the Region-Text dataset for evaluation, including 36,655 samples for Region-to-Text Identification and 32,691 samples for Text-to-Region Detection.

Report-Grounded Tasks. For chest x-ray grounded reports, we follow the split in RadFM (Wu et al., ) and filtered the chext x-ray scans paired with annotation boxes. Overall, 3,022 samples are acquired for evaluation. For lesion grounded reports and organ grounded reports, we include 438 and 1,264 data items in the test set, respectively.

## Appendix B Training Details

### B.1 Implementation Details

We employ InternVL 1.2 (Chen et al., ) as our general-domain foundation to begin training, which is composed of InternViT-6B as the vision encoder, and Nous-Hermes-2-Yi-34B as the language model.
Our training process is divided into two steps: alignment training and instruction tuning. During the alignment training phase, we freeze the vision encoder and language model, only fine-tuning the alignment module with medical image captioning and report generation datasets, which contain about 2.4M data in total. In the instruction tuning stage, we apply both public datasets and our Region-Centric datasets, MedRegInstruct, to optimize the language model, while keeping the other components unchanged. The amount of instruction tuning data is approximately 2.2M. The language model loss is applied as the loss function. We follow the official instruction for finetuning InternVL, and leverage LoRA with DeepSpeed ZeRO Stage 3 to optimize model parameters. The model is trained on 16 NVIDIA H800 GPUs for 1 epoch in the alignment stage and 2 epochs in the instruction tuning stage.

### B.2 Task-Specific Prompts

We design task-specific instructions to prompt the model to recognize and address different objectives for visual question answering, report generation, image classification and region-centric tasks. The instruction prompt for each task can be found in Figure 12.

Figure: Figure 12: Task-Specific Prompt Examples.
Refer to caption: extracted/5953015/imgs/prompt.png

## Appendix C Experiment Details

### C.1 Performance Comparison on General Medical Vision-Language Tasks

#### C.1.1 Visual Question Answering

Table: Table 6: Performance comparison on Visual Question Answering task. ‘*’ indicates that the model is fine-tuned on the dataset. ‘N/A’ means the scores are not reported.

We implement our model on several medical VQA benchmarks for comparison, such as English and Chinese versions of SLAKE (Liu et al., ), VQA-RAD (Lau et al., ), PathVQA (He et al., ), and PMC-VQA (Zhang et al., ).
For PMC-VQA, we combined the data from both versions.
We follow the official splits of all the datasets and apply both natural language generation (NLG) and classification metrics to evaluate the model outputs.
Table 6 represents the results on these datasets.
Specifically, LLaVa-Med and BioMedGPT shows slightly strong performance on the SLAKE, VQA-RAD, and PathVQA datasets because they are fine-tuned on each of the dataset. RadFM is primarily trained on radiology data and struggle to understand pathology modalities, leading to poor performance on the PathVQA dataset. Compared with MedDr, our model achieves more competitive performance, indicating that incorporating regional information into the training data allows the MLLM to better perceive spatial knowledge of various organs and lesion areas in medical images. However, our model performs slightly less competitive on PathVQA, likely due to the scarcity of regional information for the pathology images. This highlights the urgency of enhancing region-centric capabilities across a wide range of modalities, which would also benefit traditional vision-language tasks.

#### C.1.2 Report Generation

Table: Table 7: Performance comparison on English Report Generation task. ‘N/A’ means the scores are not reported.

Figure: Figure 13: Generated Report Examples on MIMIC-CXR and IU-Xray datasets.
Refer to caption: extracted/5953015/imgs/cxr_example.png

For English report generation, we evaluate our model on the report generation task for chest X-ray datasets MIMIC-CXR (Johnson et al., ) and IU-Xray (Demner-Fushman et al., ). We apply the training-test split of Wu et al. () for MIMIC-CXR and Chen et al. () for IU-Xray.
We apply the training-test split of Wu et al. () for MIMIC-CXR and Chen et al. () for IU-Xray.
Natural language generation (NLG) and Clinical Efficiency (CE) metrics are utilized for assessment. Specifically, DeBERTa is deployed to calculate BertScore.
Among these models, Med-Flamingo and LLaVA-Med concentrate on question-answering skills but are less effective in producing longer sentences to thoroughly describe the observations. In comparison with RadFM and MedDr, we integrate the recognition and detection of chest X-ray structures into the training data, which inherently encourages the model to analyze fine-grained anatomical details in the medical scan, thereby accomplishing significantly better results. For instance, on the MIMIC-CXR dataset, our model outperforms 3.91% in BLEU-1 and 3.78 in ROUGE-L compared with MedDr. Besides, MedRegA suparsses BioMEdGPT by 1.93% in ROUGE-L on the IU-Xray dataset.

To evaluate the ability of our model in generating medical report in Chinese, we collect Chinese image-report pairs in real clinical scenarios to construct a test dataset covering brain, chest, spine, abdomen and pelvis from the hospital. Table 8 demonstrates the model performance comparison on Chinese report generation. Since MedDr reveals the ability to comprehend and generate Chinese language, we directly prompt it with Chinese texts to fetch the outputs. For other monolingual baselines, we obtain English reports and deploy an additional LLM to translate them into Chinese for evaluation.
To quantify the capability of the models in Chinese report generation, we utilize NLG metrics to report the averaged result, where a Chinese tokenizer is also adopted to segment the generated sentences into word phrases. We apply Multilingual BERT to calculate the BertScore metric.
MedRegA significantly outperforms other baselines in generating Chinese reports, revealing impressive bilingual ability.
It can be inferred from the examples in Figure 14 that training on bilingual reports enables MedRegA to retain knowledge in both languages, yielding coherent and fluent medical reports in Chinese.

Table: Table 8: Performance comparison on Chinese Report Generation task. ‘-’ indicates the model cannot generate valid outputs.

Figure: Figure 14: Generated Report Examples on our in-house Chinese report dataset.
Refer to caption: extracted/5953015/imgs/cn_report_generate.png

#### C.1.3 Medical Image Classification

We conduct experiments on both single-label classification and multi-label classification with a wide range of datasets across radiology, ultrasound, ophthalmology, and dermatology.
For all the datasets, we adopt the official test set and prompt the model with the predefined label set.
Since MLLMs are inclined to overfit to frequent labels in the training set, we mitigate the class-imbalanced issue by oversampling minority classes and downsampling majority classes, which critically increases the classification score.
F1 scores for each dataset are reported in Table 9.
For single-label classification, MedRegA outperforms existing models on most benchmarks by a large margin.
In contrast, ulti-label classification appears more challenging for MLLMs due to the difficulties in decoupling subtle symptoms and relating them to corresponding diagnoses.
Since Med-Flamingo, LLaVA-Med and RadFM are not exposed to diverse modalities in training, they cannot manage to generate satisfactory diagnostic results for unseen modalities, and struggle to extract valid labels from the prompts.
It is worth noting that LLaVA-Med is able to acknowledge its limitation and refuse to provide uncertain answers, whereas the other models may give random responses when confronted with queries beyond their knowledge scope.

Table: Table 9: Performance comparison on Medical Image Classification task. ‘-’ indicates the model cannot generate valid outputs.

### C.2 Grounded Report Generation

As a supplement to the main text, we present the comparison in the quality of reports for organ grounded report generation in Table 10. NLG metrics are adopted to evaluate the report outputs. By integrating the region-centric task into prediction, our model shows further improvements in the result of report generation, highlighting the value of regional information in enhancing the performance of textual generation.

Table: Table 10: Performance comparison on the quality of reports for organ grounded report generation.

Figure: Figure 15: Generated Grounded Report Examples. (a) shows the example on the MIMIC-CXR-sourced grounded report dataset. (b) demonstrates the example for the grounded reports focusing on the lesion part. (c) and (d) present the examples of organ grounded reports.
Refer to caption: extracted/5953015/imgs/example5.png

Additionally, we exemplify the output of MedRegA for the grounded report generation task in Figure 15. The examples reveal that even when the previous SOTA method MedDr successfully detect the abnormality in the given medical scan, it still struggles to illustrate the specific region of the diagnosis, hindering further performance improvement and interoperability. In comparison, our proposed MedRegA is capable of locating the specific area related to
each descriptive sentence, accelerating its feasibility for clinicians.

## Appendix D Region-Aligned Evaluation

To quantitatively evaluate the medical MLLMs’ regional perception and comprehension capabilities on Region-Centric tasks, we introduce a Region-Aligned evaluation framework to measure the model performance on these tasks.

Region-to-Text Identification task.
Since the output for region-to-text identification task is in the form of pure texts, we adopt Natural Language Generation (NLG) metrics for performance measurement, including BLEU-1, F1 score, Recall, Accuracy and BertScore.

Text-to-Region Detection task.
We classify thetext-to-region detection task into four categories according to the number of detected objects and the number of regions per object: single-object single-region, single-object multi-region, multi-object single-region, and multi-object multi-region, as illustrated in §A.1.1. We evaluate the performance across three dimensions: (1) Object-Level: assessing the correctly identified objects; (2) Region-Level: assessing accurately detected regions; (3) Object-Region Alignment: assessing whether the detected boxes are correctly aligned with the corresponding object.

For single-object detection, all the output boxes are aligned with the given object, where evaluating the detection performance from the region-level is sufficient. The region-level evaluation metrics are represented in Figure 16.
If the object is related to only one region, we report the accuracy of the detected boxes. However, if the object corresponds to multiple regions, we calculate the precision, recall and F1 score.

Figure: Figure 16: Region-level Evaluation for Single-Object Detection.
Refer to caption: extracted/5953015/imgs/eval-single.png

For multi-object detection, the alignment of text and box must also be considered. As illustrated in Algorithm 1, we first apply Hungarian Matching to get an optimal IoU-based assignment for object-region pairs set of each input, where each predicted region is matched with the ground truth with the maximum IoU score. Subsequently, for each matched predicted and ground truth object-region pair, we identify whether the object and region are correctly detected andproperly ligned. We calculate the three-dimensional metrics for each sample and utilize the averaged score for overall evaluation.

Figure: Algorithm 1 Region-Aligned Evaluation

Grounded Report Generation task. The evaluation of Grounded Report Generation performance is composed of textual evaluation and regional evaluation. For the textual evaluation, we apply NLG metrics to assess the quality of the report part, while regional evaluation follows the same discipline as text-to-region detection task.

## Appendix E Effectiveness of Regional CoT

Table: Table 11: Effectiveness of our proposed Regional CoT.

Since our model has the ability to perceive regions, we also explore the impact of guiding the model with abnormal regions to enhance generation. For instance, we test the model on the visual question answering dataset SLAKE and multi-label classification VinDr Series datasets, by enforcing the model to first identify abnormal areas and then prompt the model with the bounding box. Table 11 indicates that providing guidance with abnormal region greatly improves the performance, essentially on the challenging multi-label classification task (with 61.75% and 19.74% F1 score on VinDr-SpineXR and VinCXR dataset, respectively), proving the immense value of understanding regional knowledge.

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: Overall, this is a good paper with solid contributions and it has a significant meaning to the field. Still, the reviewer has a few questions here.

1. How is the English version of the text report/caption generated for the in-house Chinese data? What kind of translation was used here?
2. According to the “Structure Detection” section on page 5, the structure bounding box in the dataset was generated with a fine-tuned MLLM on the existing Region-Text dataset given the corresponding organ prompt. The reviewer wonders if any kind of evaluation was done on this fine-tuned MLLM, how accurate is the extract bounding box? And why not train a more straightforward detection model for this purpose? Considering that the specific detection model usually performs better than a general MLLM.
3. Also, as mentioned above, the reviewer wonders how much will the other baseline medical MLLMs improve if they were trained with the proposed dataset. The comparison between InternVL and MedRegA in the paper is not very fair as the InternVL was only trained with natural image-text pairs. The reviewer understands it may take tons of extra time/money to conduct such an evaluation, but it would still be an interesting question to explore.
4. According to the abstract, the code and model will be released to the public, but the reviewer wonders if the data will be released as well. This is pretty critical as the dataset is the major contribution of the paper. The answer to this question will influence the reviewer's final score for the paper.
soundness: 3
strengths: 1. The proposed new region-centric medical dataset MedRegInstruct addressed the need for fine-grained, region-level, training and evaluation datasets in the medical domain. It is novel to the field and will help the development of the field as provides a more fine-grained training and evaluation annotation. It is proven that the method pre-trained with the new dataset can demonstrate more robust performance in the targeted region-specific tasks and also other related medical tasks. 
2. The paper has provided enough details about the data curation and evaluation, including the process of question-answering pair generation from medical data with region annotation, a prompt for each downstream evaluation, and detailed settings for each task. The author also seems to claim they will release the code and pre-trained model later.
3. The paper is overall well written with nice figures and a clean presentation. It is easy to follow even if there are many details in it.
summary: This paper proposes a new large-scale, multi-site, medical visual-question answering (VQA) dataset, called MedRegInstruct, with fine-grained region-specific annotation and multiple corresponding region-centric tasks. The proposed dataset contains multiple region-centric evaluation tasks including, region-to-text generation, text-to-region detection, and grounded medical report generation. The paper further proposed a new region-aware multi-modal large language model (MLLM), called, MedRegA, pre-trained with the proposed dataset and regional chain of thought (CoT). The proposed method outperforms multiple existing SOTA medical VQA models on the regional-centric evaluation.
weaknesses: 1. While the proposed dataset is novel and important to the field, the corresponding MedRegA model is not that novel, though I hate to say that, in comparison. The overall model design is basically the same as the base InternVL model and the major change is the training scheme and prompt formulation with the new dataset, which can be extended to other medical VQL models as well. While the author further proposed single-step regional CoT in the paper, it is still relatively straightforward. **Yet**, this is not a major drawback as the performance of this model demonstrates the superiority of training with the proposed dataset.
2. Similarly, it seems that the baselines used in the evaluation are not all fine-tuned with the proposed dataset, which enables the model to understand regional information. Comparing the model trained on the proposed dataset with baselines that are not trained on this dataset is not very fair, to some extent. It would be interesting to see how well each baseline would improve if they were fine-tuned on the proposed dataset, even if just in a few-shot manner.
3. Another concern about this paper is the quality of the data. Most of the regional bounding box-text pairs in the dataset were taken from the existing dataset and generated from an automated rule-based system and an LLM. The evaluation of the data quality is missing in this paper, which is very critical as the main purpose is **fine-grained regional evaluation,** where the quality of the bounding box and the correctness of the corresponding text description are very important. It would be better if some sort of data quality check could be done before releasing it to the public.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: • What was the filtering criterion for reducing SA-Med2D-20M?

• Is there a noticeable performance difference between English and Chinese tasks? Is there paired data that this can be evaluated on? • • • 

• Overall, it is unclear from the paper why exactly these two particular languages are used, other than being able to source data for both, rather than just arbitrary languages with medical report data.

• Is the MedRegInstruct dataset planning to be release?
For the Region-Text dataset, why is the case that there needs to be this split into half for region-to-text and half for text-to-region? Can't every datapoint server the purpose of being either during training?
soundness: 3
strengths: • The paper really introduces another method of interaction, in the bounding boxes, that can be quite useful as MedRegA is able to reason about specific regions. This is an immensely useful tool for narrowing of problems as opposed to current trained MLLMs have to do.

• The proposed training strategy for learning how to incorporate the bounding boxes is simple and scalable, seems like it could be generally applied to other strategies that want to apply bounding boxes.

• Regional CoT makes a lot of sense intuitively and provides a nice bias for solving region specific problems.

• MedRegA has very strong performance on the chosen set of benchmarks and metrics in comparison to existing state of the art methods.
summary: In this work, the authors tackle the current problem in MLLMs (multimodal large language models) of how they don't necessarily "focus" on particular regions, but rather take the entire image as context and then have to solve a variety of downstream tasks. To address this problem they offer two main sets of contributions. The first, is the formulation of several region specific tasks, namely region-to-text identification, text-to-region detection, and grounded report generation, which they combine into a dataset called MedRegInstruct. The second is the training strategy and modeling paradigm for a MLLM they propose, which they refer to as MedRegA. MedRegA is a bilingual, as it was trained on both English and Chinese paired datapoints, model that they apply to this larger dataset they prepare in comparison to currently available medical MLLMs.
weaknesses: • The section of the report grounding dataset is quite confusing as to how its constructed, and how much it relies on the models it uses for automation for providing correct labels/groundtruth. For example, is InternLM perfect at sub-select different parts of the reports? 

• The paper claims to work across a broad set of modalities, but when considering the grounding dataset it seems that it only considers, as other MLLMs do, Chest Xrays?

• It's unclear if the comparison to existing baselines are exactly fair as the trained MedRegA has access to more data. In fact, compared to the baseline of InternLM, it should be strictly better because MedRegA is trained with InternLM as an initialization. While it's true that the baselines don't provide a mechanism for highlighting particular regions, they still should be adapted to the new data that the authors are evaluating on.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: 1.	Please explain why certain models were fine-tuned in the comparison while others were not.
2.	The authors state that adding CoT improves model performance; however, no comprehensive ablation experiments were conducted to confirm this.
3.	Have the authors considered comparisons using other large models as the backbone?
4.	Do the authors plan to open-source this dataset?
soundness: 3
strengths: 1.	The paper is well-organized.
2.	This work introduces a novel task alongside a large-scale medical dataset.
3.	The authors propose a new model that significantly outperforms other models across the majority of datasets and evaluation metrics, particularly for the newly proposed task.
summary: This work introduces a novel region-centric task and presents a large-scale Chinese medical dataset named MedRegInstruct. To address this new task, the authors developed a bilingual multimodal large language model, MedRegA, which outperforms baseline methods across most evaluation metrics.
weaknesses: 1.	The article’s description of the model structure in the main text is incomplete, relying heavily on Figure 4 to convey the model’s architecture. For example, the structure of the encoder and tokenizer remains ambiguous. The authors should clarify which components utilize existing models, which are fine-tuned, and which are trained from scratch.
2.	The fairness of the model comparison is questionable. In Table 1, only a subset of models has been fine-tuned based on the test dataset, leaving others unmodified. Is this due to an inability to fine-tune certain models, or for another reason? The authors should provide detailed criteria for choosing to fine-tune specific models and address the fairness of this experimental design.
3.	On the newly proposed task, the authors’ model demonstrates a substantial performance advantage. They mention that existing open-source models cannot accurately capture coordinates, but it would be insightful to know if fine-tuning on the authors’ proposed dataset could improve this. If fine-tuning other models would yield better results, the authors should do so for a more persuasive comparison. Conversely, if fine-tuning has no impact, the authors should explain the model’s unique advancements to provide deeper insights.
4.	The role of the CoT component is somewhat unclear, with insufficient ablation experiments to support its effectiveness. The authors should conduct a more specific ablation study to illustrate the impact of CoT rather than merely reporting improved results with its inclusion.
5.	While the authors claim that the proposed model can perform diagnostic tasks, there is no clear evaluation or user study provided. A brief evaluation or explanation of the diagnostic results would add clarity.
6.	Several prior studies focus on medical image segmentation, which aligns well with the proposed region-centric task. Have the authors considered using such methods as feature extractors, backbones, or even as comparative models?
7.	The experimental results on report generation are insufficient, as the authors rely solely on descriptive analysis without quantitative evaluation or user studies. Similar to point 5, if the authors wish to validate effectiveness without explicit evaluation metrics, a user study would provide valuable support.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 4
questions: Please address the weaknesses mentioned above.
soundness: 4
strengths: - The comprehensive MedRegInstruct dataset is valuable and likely to have a significant impact on the community.
- Introducing Regional Chain-of-Thought is insightful and significantly improves the model's performance.
- The model achieves state-of-the-art performance without fine-tuning on specific benchmark datasets, indicating strong generalization across different domains.
summary: This paper presents MedRegA, a region-aware medical multimodal language model capable of handling both image-level and region-level vision-language tasks across diverse medical modalities. The authors propose three Region-Centric tasks and create a large-scale dataset called MedRegInstruct to enhance the model's focus on specific anatomical regions. They introduce a Regional Chain-of-Thought approach to further improve performance. Experiments show that MedRegA outperforms existing models on tasks like visual question answering, report generation, and medical image classification, demonstrating significant versatility and interpretability.
weaknesses: - Human validation of the automatically generated annotations is needed to prove the quality of the dataset.
- The paper lacks comprehensive comparisons in report generation with state-of-the-art methods, such as those on the ReXrank[1] leaderboard.
- Detection task of multiple regions still remain challenging for purposed method.

[1] Rajpurkar Lab. ReXrank Leaderboard. 22 July 2024, https://rajpurkarlab.github.io/rexrank/.

Meta-review: This paper developed MedRegA, a Region-Aware medical Multimodal Large Language Model (MLLM) designed to enhance interpretability and interactivity by incorporating anatomical region awareness into medical vision-language tasks across multiple modalities. By leveraging a newly constructed dataset, MedRegInstruct, and enabling both image-level and region-level tasks, MedRegA achieves state-of-the-art performance in bilingual medical applications, including visual question answering, report generation, and image classification.

During the initial review, reviewers praised that this paper introduces the novel MedRegInstruct dataset and the region-aware MedRegA model with an innovative Regional Chain-of-Thought approach, achieving state-of-the-art performance and addressing critical gaps in fine-grained region-specific vision-language tasks.  On the other hand, they also found that the main weaknesses of this paper are the limited novelty of the MedRegA model, as it heavily relies on existing architectures like InternVL with minor modifications in training schemes and prompts; concerns about fairness in baseline comparisons, as many baselines were not fine-tuned on the proposed dataset; and the lack of human validation to ensure the quality of the dataset’s automated annotations. 

During the discussion, the authors provided additional results of finetuning two baselines, Med-Flamingo and LLaVA-Med, and some clarifications about human evaluation. They also provided new results comparing report generation models on IU-Xray and MIMIC-CXR datasets. The reviewers collectively appreciated the authors' detailed rebuttal, clarifications, and additional results, particularly regarding the quality and impact of the MedRegInstruct dataset and the MedRegA model. The dataset's potential to benefit the research community was widely recognized, leading to improved ratings from 3 reviewers, with overall support for acceptance.

The reviewers have reached a consensus, leaning toward accepting the paper. This AC agrees with their assessment of the proposed dataset's significant potential contribution and impact on the research community, which supports the paper's acceptance. While the MedRegA model is not entirely novel, it establishes a solid performance baseline for the dataset. However, this AC strongly encourages the authors to enhance and clarify their discussion of human validation and dataset quality in the final version of the paper. For instance, statements such as “We also conducted a visual evaluation and found that although most bounding boxes are slightly larger, they can still encompass the target region” remain vague and could benefit from more specific details and justification.

Decision: Accept (Poster)