Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks

Authors: Lehan Wang, Haonan Wang, Honglong Yang, Jiaji Mao, Zehong Yang, Jun Shen, Xiaomeng Li
Primary area: foundation or frontier models, including LLMs
Keywords: Multimodal Large Language Model, Biomedicine, Region-Text

Abstract:
Several medical Multimodal Large Languange Models (MLLMs) have been developed to address tasks involving visual images with textual instructions across various medical modalities, achieving impressive results. 
Most current medical generalist models are region-agnostic, treating the entire image as a holistic representation. However, they struggle to identify which specific regions they are focusing on when generating a sentence.
To mimic the behavior of doctors, who typically begin by reviewing the entire image before concentrating on specific regions for a thorough evaluation, we aim to enhance the capability of medical MLLMs in understanding anatomical regions within entire medical scans.
To achieve it, we first formulate \textbf{Region-Centric tasks} and construct a \textbf{large-scale dataset, MedRegInstruct,} to incorporate regional information into training. Combining our collected dataset with other medical multimodal corpora for training, we propose a \textbf{Region-Aware medical MLLM, MedRegA}, which is the first bilingual generalist medical AI system to simultaneously handle image-level and region-level medical vision-language tasks across a broad range of modalities. Our MedRegA not only enables three region-centric tasks, but also achieves the best performance for visual question answering, report generation and medical image classification over 8 modalities, showcasing significant versatility. Experiments demonstrate that our model can not only accomplish powerful performance across various medical vision-language tasks in bilingual settings, but also recognize and detect structures in multimodal medical scans, boosting the interpretability and user interactivity of medical MLLMs. The codes and model will be made publicly available.

Decision: accept (poster)
Reviewer scores: 8, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730412771446

thread_type:
official_review

<thread_object>
creator: reviewer_wcet

<content>
summary:
This paper presents MedRegA, a region-aware medical multimodal language model capable of handling both image-level and region-level vision-language tasks across diverse medical modalities. The authors propose three Region-Centric tasks and create a large-scale dataset called MedRegInstruct to enhance the model's focus on specific anatomical regions. They introduce a Regional Chain-of-Thought approach to further improve performance. Experiments show that MedRegA outperforms existing models on tasks like visual question answering, report generation, and medical image classification, demonstrating significant versatility and interpretability.

strengths:
- The comprehensive MedRegInstruct dataset is valuable and likely to have a significant impact on the community.
- Introducing Regional Chain-of-Thought is insightful and significantly improves the model's performance.
- The model achieves state-of-the-art performance without fine-tuning on specific benchmark datasets, indicating strong generalization across different domains.

weaknesses:
- Human validation of the automatically generated annotations is needed to prove the quality of the dataset.
- The paper lacks comprehensive comparisons in report generation with state-of-the-art methods, such as those on the ReXrank[1] leaderboard.
- Detection task of multiple regions still remain challenging for purposed method.

[1] Rajpurkar Lab. ReXrank Leaderboard. 22 July 2024, https://rajpurkarlab.github.io/rexrank/.

questions:
Please address the weaknesses mentioned above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730481612153

thread_type:
official_review

<thread_object>
creator: reviewer_toyk

<content>
summary:
This work introduces a novel region-centric task and presents a large-scale Chinese medical dataset named MedRegInstruct. To address this new task, the authors developed a bilingual multimodal large language model, MedRegA, which outperforms baseline methods across most evaluation metrics.

strengths:
1.	The paper is well-organized.
2.	This work introduces a novel task alongside a large-scale medical dataset.
3.	The authors propose a new model that significantly outperforms other models across the majority of datasets and evaluation metrics, particularly for the newly proposed task.

weaknesses:
1.	The article’s description of the model structure in the main text is incomplete, relying heavily on Figure 4 to convey the model’s architecture. For example, the structure of the encoder and tokenizer remains ambiguous. The authors should clarify which components utilize existing models, which are fine-tuned, and which are trained from scratch.
2.	The fairness of the model comparison is questionable. In Table 1, only a subset of models has been fine-tuned based on the test dataset, leaving others unmodified. Is this due to an inability to fine-tune certain models, or for another reason? The authors should provide detailed criteria for choosing to fine-tune specific models and address the fairness of this experimental design.
3.	On the newly proposed task, the authors’ model demonstrates a substantial performance advantage. They mention that existing open-source models cannot accurately capture coordinates, but it would be insightful to know if fine-tuning on the authors’ proposed dataset could improve this. If fine-tuning other models would yield better results, the authors should do so for a more persuasive comparison. Conversely, if fine-tuning has no impact, the authors should explain the model’s unique advancements to provide deeper insights.
4.	The role of the CoT component is somewhat unclear, with insufficient ablation experiments to support its effectiveness. The authors should conduct a more specific ablation study to illustrate the impact of CoT rather than merely reporting improved results with its inclusion.
5.	While the authors claim that the proposed model can perform diagnostic tasks, there is no clear evaluation or user study provided. A brief evaluation or explanation of the diagnostic results would add clarity.
6.	Several prior studies focus on medical image segmentation, which aligns well with the proposed region-centric task. Have the authors considered using such methods as feature extractors, backbones, or even as comparative models?
7.	The experimental results on report generation are insufficient, as the authors rely solely on descriptive analysis without quantitative evaluation or user studies. Similar to point 5, if the authors wish to validate effectiveness without explicit evaluation metrics, a user study would provide valuable support.

questions:
1.	Please explain why certain models were fine-tuned in the comparison while others were not.
2.	The authors state that adding CoT improves model performance; however, no comprehensive ablation experiments were conducted to confirm this.
3.	Have the authors considered comparisons using other large models as the backbone?
4.	Do the authors plan to open-source this dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730490084143

thread_type:
official_review

<thread_object>
creator: reviewer_5dsl

<content>
summary:
In this work, the authors tackle the current problem in MLLMs (multimodal large language models) of how they don't necessarily "focus" on particular regions, but rather take the entire image as context and then have to solve a variety of downstream tasks. To address this problem they offer two main sets of contributions. The first, is the formulation of several region specific tasks, namely region-to-text identification, text-to-region detection, and grounded report generation, which they combine into a dataset called MedRegInstruct. The second is the training strategy and modeling paradigm for a MLLM they propose, which they refer to as MedRegA. MedRegA is a bilingual, as it was trained on both English and Chinese paired datapoints, model that they apply to this larger dataset they prepare in comparison to currently available medical MLLMs.

strengths:
• The paper really introduces another method of interaction, in the bounding boxes, that can be quite useful as MedRegA is able to reason about specific regions. This is an immensely useful tool for narrowing of problems as opposed to current trained MLLMs have to do.

• The proposed training strategy for learning how to incorporate the bounding boxes is simple and scalable, seems like it could be generally applied to other strategies that want to apply bounding boxes.

• Regional CoT makes a lot of sense intuitively and provides a nice bias for solving region specific problems.

• MedRegA has very strong performance on the chosen set of benchmarks and metrics in comparison to existing state of the art methods.

weaknesses:
• The section of the report grounding dataset is quite confusing as to how its constructed, and how much it relies on the models it uses for automation for providing correct labels/groundtruth. For example, is InternLM perfect at sub-select different parts of the reports? 

• The paper claims to work across a broad set of modalities, but when considering the grounding dataset it seems that it only considers, as other MLLMs do, Chest Xrays?

• It's unclear if the comparison to existing baselines are exactly fair as the trained MedRegA has access to more data. In fact, compared to the baseline of InternLM, it should be strictly better because MedRegA is trained with InternLM as an initialization. While it's true that the baselines don't provide a mechanism for highlighting particular regions, they still should be adapted to the new data that the authors are evaluating on.

questions:
• What was the filtering criterion for reducing SA-Med2D-20M?

• Is there a noticeable performance difference between English and Chinese tasks? Is there paired data that this can be evaluated on? • • • 

• Overall, it is unclear from the paper why exactly these two particular languages are used, other than being able to source data for both, rather than just arbitrary languages with medical report data.

• Is the MedRegInstruct dataset planning to be release?
For the Region-Text dataset, why is the case that there needs to be this split into half for region-to-text and half for text-to-region? Can't every datapoint server the purpose of being either during training?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730519324040

thread_type:
official_review

<thread_object>
creator: reviewer_qlek

<content>
summary:
This paper proposes a new large-scale, multi-site, medical visual-question answering (VQA) dataset, called MedRegInstruct, with fine-grained region-specific annotation and multiple corresponding region-centric tasks. The proposed dataset contains multiple region-centric evaluation tasks including, region-to-text generation, text-to-region detection, and grounded medical report generation. The paper further proposed a new region-aware multi-modal large language model (MLLM), called, MedRegA, pre-trained with the proposed dataset and regional chain of thought (CoT). The proposed method outperforms multiple existing SOTA medical VQA models on the regional-centric evaluation.

strengths:
1. The proposed new region-centric medical dataset MedRegInstruct addressed the need for fine-grained, region-level, training and evaluation datasets in the medical domain. It is novel to the field and will help the development of the field as provides a more fine-grained training and evaluation annotation. It is proven that the method pre-trained with the new dataset can demonstrate more robust performance in the targeted region-specific tasks and also other related medical tasks. 
2. The paper has provided enough details about the data curation and evaluation, including the process of question-answering pair generation from medical data with region annotation, a prompt for each downstream evaluation, and detailed settings for each task. The author also seems to claim they will release the code and pre-trained model later.
3. The paper is overall well written with nice figures and a clean presentation. It is easy to follow even if there are many details in it.

weaknesses:
1. While the proposed dataset is novel and important to the field, the corresponding MedRegA model is not that novel, though I hate to say that, in comparison. The overall model design is basically the same as the base InternVL model and the major change is the training scheme and prompt formulation with the new dataset, which can be extended to other medical VQL models as well. While the author further proposed single-step regional CoT in the paper, it is still relatively straightforward. **Yet**, this is not a major drawback as the performance of this model demonstrates the superiority of training with the proposed dataset.
2. Similarly, it seems that the baselines used in the evaluation are not all fine-tuned with the proposed dataset, which enables the model to understand regional information. Comparing the model trained on the proposed dataset with baselines that are not trained on this dataset is not very fair, to some extent. It would be interesting to see how well each baseline would improve if they were fine-tuned on the proposed dataset, even if just in a few-shot manner.
3. Another concern about this paper is the quality of the data. Most of the regional bounding box-text pairs in the dataset were taken from the existing dataset and generated from an automated rule-based system and an LLM. The evaluation of the data quality is missing in this paper, which is very critical as the main purpose is **fine-grained regional evaluation,** where the quality of the bounding box and the correctness of the corresponding text description are very important. It would be better if some sort of data quality check could be done before releasing it to the public.

questions:
Overall, this is a good paper with solid contributions and it has a significant meaning to the field. Still, the reviewer has a few questions here.

1. How is the English version of the text report/caption generated for the in-house Chinese data? What kind of translation was used here?
2. According to the “Structure Detection” section on page 5, the structure bounding box in the dataset was generated with a fine-tuned MLLM on the existing Region-Text dataset given the corresponding organ prompt. The reviewer wonders if any kind of evaluation was done on this fine-tuned MLLM, how accurate is the extract bounding box? And why not train a more straightforward detection model for this purpose? Considering that the specific detection model usually performs better than a general MLLM.
3. Also, as mentioned above, the reviewer wonders how much will the other baseline medical MLLMs improve if they were trained with the proposed dataset. The comparison between InternVL and MedRegA in the paper is not very fair as the InternVL was only trained with natural image-text pairs. The reviewer understands it may take tons of extra time/money to conduct such an evaluation, but it would still be an interesting question to explore.
4. According to the abstract, the code and model will be released to the public, but the reviewer wonders if the data will be released as well. This is pretty critical as the dataset is the major contribution of the paper. The answer to this question will influence the reviewer's final score for the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732176887971

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response Part I

comment:
We sincerely appreciate the reviewers for their time and effort in the review. We are glad that they found our proposed tasks, dataset, and model `"novel"`/`"help the development of the field"`/`"immensely useful"`/`"likely to have a significant impact on the community"` (QLEK, 5dSL, WceT) and our paper `"well written"`/`"well-organized"` (QLEK, ToYk), and agreed that our MedRegA model `"demonstrate strong performance"` and `"strong generalization"` (5dSL, WceT).

We would like to address common questions here, followed by detailed responses to each reviewer separately. We have also updated the paper with the revised parts marked in blue.

## Common Questions

**Q1: Fine-tune other baselines with our proposed dataset.** 

We would like to emphasize that our core contribution is to **enhance the ability of medical MLLM to perceive and interpret regions** within the medical scan by proposing the novel MedRegInstruct dataset and the versatile MedRegA model, which remains unexplored in previous works. Specifically, we applied an automatic labeling system to curate the grounded medical reports in our dataset. Our dataset covers comprehensive region-centric tasks, aimed at encouraging the model to focus on critical areas. This significantly improves the interpretability and reliability of medical MLLM in clinical applications.

We have evaluated the finetuning results with two baselines, Med-Flamingo [1] and LLaVA-Med [2]. To save time and cost for evaluation, we use a subset of 5K samples from the proposed MedRegInstruct dataset to finetune the baseline models. This subset covers Region-to-Text Identification and Text-to-Region Detection tasks. We evaluated the models using the test split of the same subset.

For Med-Flamingo [1], a few-shot learner adapted to the medical domain, we perform a 5-shot learning with our dataset. For LLaVA-Med [2], we finetuned the model on the sub-dataset for 10 epochs. The table below shows the results.

**Results on Region-to-Text Identification Task**

| **Metric**    | **Med-Flamingo (Zero Shot)** | **Med-Flamingo (Few Shot)** | **LLaVA-Med (Base)** | **LLaVA-Med (Finetuned)** | **MedRegA** |
| ------------- | :--------------------------: | :-------------------------: | :------------------: | :-----------------------: | :---------: |
| **BLEU**      |             2.19             |            32.02            |         0.05         |           21.49           |  **59.06**  |
| **F1**        |             3.79             |            33.19            |         0.14         |           25.08           |  **59.21**  |
| **Recall**    |            11.70             |            33.40            |         1.99         |           32.23           |  **59.19**  |
| **Accuracy**  |             6.62             |            16.87            |         0.91         |           31.93           |  **51.72**  |
| **BertScore** |            50.36             |            71.53            |        32.18         |           58.21           |  **82.49**  |

**Results on Text-to-Region Detection Task**

| **Metric**                     | **Med-Flamingo (Zero Shot)** | **Med-Flamingo (Few Shot)** | **LLaVA-Med (Base)** | **LLaVA-Med (Finetuned)** | **MedRegA** |
| ------------------------------ | :--------------------------: | :-------------------------: | :------------------: | :-----------------------: | :---------: |
| **Region-Level F1**            |             N/A              |            18.36            |         N/A          |           19.68           |  **56.08**  |
| **Alignment F1** |             N/A              |            20.65            |         N/A          |           19.53           |  **52.52**  |
| **IoU**                        |             N/A              |            19.52            |         N/A          |           25.27           |  **47.43**  |

In the table, 'N/A' indicates that the model cannot generate valid outputs

From the results, we can observe that both Med-Flamingo [1] and LLaVA-Med [2] can improve on the regional tasks even in a few-shot setting or fine-tuning with a small subset in our proposed dataset. For example, they achieve an increase of 29.4% and 24.94% respectively in the F1 score on Region-to-Text Identification task. This proves the effectiveness of our dataset in extending regional knowledge.

We have updated the results and analysis above in **Appendix C.3**.

[1] Moor, Michael, et al. "Med-flamingo: a multimodal medical few-shot learner." Machine Learning for Health (ML4H). PMLR, 2023.

[2] Li, Chunyuan, et al. "Llava-med: Training a large language-and-vision assistant for biomedicine in one day." Advances in Neural Information Processing Systems 36 (2024).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper developed MedRegA, a Region-Aware medical Multimodal Large Language Model (MLLM) designed to enhance interpretability and interactivity by incorporating anatomical region awareness into medical vision-language tasks across multiple modalities. By leveraging a newly constructed dataset, MedRegInstruct, and enabling both image-level and region-level tasks, MedRegA achieves state-of-the-art performance in bilingual medical applications, including visual question answering, report generation, and image classification.

During the initial review, reviewers praised that this paper introduces the novel MedRegInstruct dataset and the region-aware MedRegA model with an innovative Regional Chain-of-Thought approach, achieving state-of-the-art performance and addressing critical gaps in fine-grained region-specific vision-language tasks.  On the other hand, they also found that the main weaknesses of this paper are the limited novelty of the MedRegA model, as it heavily relies on existing architectures like InternVL with minor modifications in training schemes and prompts; concerns about fairness in baseline comparisons, as many baselines were not fine-tuned on the proposed dataset; and the lack of human validation to ensure the quality of the dataset’s automated annotations. 

During the discussion, the authors provided additional results of finetuning two baselines, Med-Flamingo and LLaVA-Med, and some clarifications about human evaluation. They also provided new results comparing report generation models on IU-Xray and MIMIC-CXR datasets. The reviewers collectively appreciated the authors' detailed rebuttal, clarifications, and additional results, particularly regarding the quality and impact of the MedRegInstruct dataset and the MedRegA model. The dataset's potential to benefit the research community was widely recognized, leading to improved ratings from 3 reviewers, with overall support for acceptance.

The reviewers have reached a consensus, leaning toward accepting the paper. This AC agrees with their assessment of the proposed dataset's significant potential contribution and impact on the research community, which supports the paper's acceptance. While the MedRegA model is not entirely novel, it establishes a solid performance baseline for the dataset. However, this AC strongly encourages the authors to enhance and clarify their discussion of human validation and dataset quality in the final version of the paper. For instance, statements such as “We also conducted a visual evaluation and found that although most bounding boxes are slightly larger, they can still encompass the target region” remain vague and could benefit from more specific details and justification.