Improving Medical Large Vision-Language Models with Abnormal-Aware Feedback
Authors: Yucheng Zhou, Lingran Song, Jianbing Shen
Accepted at: acl25

Improving Medical Large Vision-Language Models with Abnormal-Aware Feedback

Yucheng Zhou, Lingran Song, Jianbing Shen*
SKL-IOTSC, CIS, University of Macau
yucheng.zhou@connect.um.edu.mo, jianbingshen@um.edu.mo

Abstract

Existing Medical Large Vision-Language Models (Med-LVLMs), encapsulating extensive medical knowledge, demonstrate excellent capabilities in understanding medical images. However, there remain challenges in visual localization in medical images, which is crucial for abnormality detection and interpretation. To address these issues, we propose a novel UMed-LVLM designed to unveil medical abnormalities. Specifically, we collect a Medical Abnormalities Unveiling (MAU) dataset and propose a two-stage training method for UMed-LVLM training. To collect MAU dataset, we propose a prompt method utilizing the GPT-4V to generate diagnoses based on identified abnormal areas in medical images. Moreover, the two-stage training method includes Abnormal-Aware Instruction Tuning and Abnormal-Aware Rewarding, comprising Relevance Reward, Abnormal Localization Reward and Vision Relevance Reward. Experimental results demonstrate that our UMed-LVLM significantly outperforms existing Med-LVLMs in identifying and understanding medical abnormalities, achieving a 58% improvement over the baseline. In addition, this work shows that enhancing the abnormality detection capabilities of Med-LVLMs significantly improves their understanding of medical images and generalization capability.

1 Introduction

Large Vision-Language Models (LVLMs) demonstrate remarkable capability in various vision-language tasks (OpenAI, 2023a; Liu et al., 2023b; Li et al., 2023a). Medical image analysis poses a significant challenge for LVLMs due to their intricate patterns and structures, thereby demanding an in-depth grasp of nuanced variations to ensure precise diagnoses (Wu et al., 2023). To enhance LVLMs for medical images, some works (Qin et al., 2023) encapsulate a substantial medical corpus into these models, i.e., Medical Large Vision-Language Models (Med-LVLMs). These Med-LVLMs exhibit proficiency in understanding medical images and human queries.

Despite their successes, existing Med-LVLMs exhibit limitations in visual localization capability within medical images, as shown in Table 1. Advanced models like GPT-4V (OpenAI, 2023a), one of the leading LVLMs, exhibit notable shortcomings in accurately interpreting and visual localization in medical images (i.e., abnormality localization) (Wu et al., 2023). The capability for visual localization is critical for two primary reasons: Firstly, bias in visual localization can lead to unreliable responses in diagnosis, undermining the credibility and interpretability of Med-LVLMs. Secondly, some LVLMs (Yin et al., 2023; Guo et al., 2024; Ranasinghe et al., 2024) improve the visual understanding in LVLM by enhancing its visual localization capability. In these works, visual localization in natural scenes can benefit from general detectors. However, abnormality localization in medical images lacks a large amount of data to train specialized detectors, especially for some

<table>
  <tr>
    <th>Method</th>
    <th>Medical Diagnosis</th>
    <th>Multi-modal Medical Image</th>
    <th>Region-Awareness</th>
    <th>Detector & Segmenter-Free</th>
  </tr>
  <tr>
    <td>RegionGPT</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>XrayGPT</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Med-Flamingo</td>
    <td>✓</td>
    <td>✓</td>
    <td>x</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>MedVint</td>
    <td>✓</td>
    <td>✓</td>
    <td>x</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Our UMed-LVLM</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
</table>

Table 1: Comparison of different Med-LVLMs: RegionGPT (Guo et al., 2024), XrayGPT (Thawakar et al., 2023), Med-Flamingo (Moor et al., 2023), MedVint (Zhang et al., 2023a) and Ours. “Medical Diagnosis” denotes the model’s applicability for medical diagnosis; “Multi-modal Medical Image” indicates training on multi-modal medical images; “Region-Awareness” reflects the model’s capability for region recognition; “Detector & Segmenter-Free” specifies independence from external detectors or segmenters.
rare diseases. This limitation emphasizes the necessity for enhancing the inherent visual localization capabilities of medical LVLMs without specific detectors to improve their understanding of medical images and the reliability of responses.

To address these challenges, we propose UMed-LVLM, designed with Unveiling Medical abnormalities for LVLM. To encapsulate the abnormality unveiling capability into UMed-LVLM, we first collect a dataset comprising medical images with abnormality regions. Then, we design a prompt method to generate a diagnosis dataset, i.e., Medical Abnormalities Unveiling (MAU), through the GPT-4V model (OpenAI, 2023b). The MAU dataset encompasses 5,817 medical images, user queries, and diagnosis responses with abnormal areas. This dataset is used to train UMed-LVLM via a two-stage training method, i.e., Abnormal-Aware Instruction Tuning and Abnormal-Aware Rewarding, enabling it to understand medical abnormalities. UMed-LVLM is continually trained on MedViInt (Zhang et al., 2023a). To prevent knowledge catastrophic forgetting of large language models during continual training on large-scale datasets (Luo et al., 2023), we train the model on limited-scale datasets, following that employed in previous studies (Zhou et al., 2023; Yu et al., 2024). Abnormal-Aware Rewarding, comprising Relevance Reward, Abnormal Localization Reward and Vision Relevance Reward, aims to improve the model’s capability to capture abnormal areas.

In the experiments, we evaluate UMed-LVLM and other methods on MAU dataset, and UMed-LVLM outperforms other competitors in understanding medical images and identifying abnormalities. Moreover, an in-depth analysis revealed that while Med-LVLMs are not yet adept at abnormality detection in medical images, enhancing their abnormality detection capabilities can improve their understanding of medical images. Furthermore, our findings suggest that large models possess out-of-distribution (OOD) generalization capabilities, indicating the potential for more robust disease recognition through iterative improvements in large models, even if they are only exposed to diverse medical images of limited disease types (Zhang et al., 2021). In addition, we analyze the cross-modal capabilities and generalization potential of Med-LVLMs, and results show that augmenting models in varied modalities enhances their performance in modality with limited dataset availability.

The main contributions of this work are below:

• We propose a novel UMed-LVLM for medical diagnosis with unveiling abnormality by enhancing its visual localization capability.

• We introduce Abnormal-Aware Instruction Tuning and Abnormal-Aware Rewarding strategy to train the Med-LVLM, aiming to enhance the model’s focus on abnormal areas when generating responses.

• We design a prompt method to create the MAU dataset for UMed-LVLM training. The dataset comprises medical images and diagnoses with abnormality annotations.

• Experimental results show that UMed-LVLM outperforms existing Med-LVLMs in identifying and understanding medical abnormalities. We conduct an in-depth analysis of model training and generalization capabilities, underscoring the potential of incorporating medical abnormalities to enhance Med-LVLM.

2 Related Work

Recent advancements in LVLMs have significantly improved visual comprehension and contextual language understanding (Alayrac et al., 2022; Chen et al., 2023; Zhou et al., 2024a). Notable models like CLIP (Radford et al., 2021) and BLIP-2 (Li et al., 2023b) have achieved impressive results in vision-language tasks by leveraging pre-trained image-text pairs. The introduction of GPT-4 (OpenAI, 2023a) has further propelled this field, with models like LLaVA (Liu et al., 2023b) and its improved version (Liu et al., 2023a) demonstrating exceptional capabilities in multimodal tasks. In the medical domain, models such as LLaVA-Med (Li et al., 2023a), Visual Med-Alpaca (Shu et al., 2023), OphGLM (Gao et al., 2023), and XrayGPT (Thawakar et al., 2023) have specialized in interpreting medical images and providing comprehensive assistance. Observations in (Wu et al., 2023) indicated that although GPT-4V performs well in differentiating medical image modalities and anatomy, it still has difficulties in disease diagnosis and generating comprehensive reports. Additionally, Reinforcement Learning (RL) has been applied to LLMs to enhance their performance and flexibility (Sutton and Barto, 1998; Mnih et al., 2015; Schulman et al., 2015; Mnih et al., 2016; Schulman et al., 2017). The full version can be found in Appendix A.
Figure 1: The two-stage training for our UMed-LVLM. “Rel”, “ALR” and “VRR” denote “Relevance Reward”, “Abnormal Localization Reward” and “Vision Relevance Reward”, respectively, as detailed in Sec. 3.2.

3 Methodology

UMed-LVLM is trained via a two-stage training method, i.e., Abnormal-Aware Instruction Tuning and Abnormal-Aware Rewarding. UMed-LVLM is continually trained on MedViT (Zhang et al., 2023a). Abnormal-Aware Rewarding comprises LLM relevance rewarding, Abnormal Localization Rewarding, and Vision Relevance Rewarding.

3.1 Abnormal-Aware Instruction Tuning

We introduce an abnormal-aware instruction tuning approach to improve the model’s ability to understand medical abnormal regions and generate corresponding diagnoses, building upon previous work in instruction tuning (Wei et al., 2022; Liu et al., 2023b). Given a medical image \( x \) and a user query \( q \), model generates a response \( a \) token-by-token. The response includes the diagnosis and a description of the abnormal region, i.e.,

\[
p(a|x, q; \theta) = \prod_{t=1}^T p(a_t|a_{<t}, x, q; \theta),
\]

where \( \theta \) represents the model parameters, and \( T \) is the length of the response.

During training, we provide the model with medical images, user queries, and responses including diagnoses and descriptions of abnormal regions. Model are optimized by cross-entropy loss, i.e.,

\[
\mathcal{L}_{it} = - \sum_{i=1}^T \log p_i.
\]

where \( p_i \) is the probability of the \( i \)-th token in response \( a \). This approach improves the model’s understanding of abnormal regions in medical images and its ability to provide diagnosis outputs with identified abnormal regions. However, it does not directly guide the model to focus on abnormal regions in medical images.

3.2 Abnormal-Aware Rewarding

To address the challenge of effectively identifying and describing abnormalities within medical images, we propose an Abnormal-Aware Rewarding (AAR) strategy for UMed-LVLM training. This reinforcement learning (RL) training strategy comprises three rewarding strategies, i.e., Relevance Rewarding, Abnormal Localization Rewarding and Vision Relevance Rewarding, designed to optimize the Med-LVLMs based on abnormalities. In contrast to Ouyang et al. (2022) optimize LLMs following user instructions by the Proximal Policy Optimization (PPO (Schulman et al., 2017)), AAR optimizes the Med-LVLMs by a more directed learning towards the accurate medical diagnosis with abnormality recognition.

Relevance Rewarding. The relevance rewarding framework is fundamentally structured around three pivotal components: the policy network, the value network and the LLM reward model. Both networks play a crucial role in guiding the training process, with the policy network (\( \pi \)) generating actions (responses) based on the given state \( s_t \), which encapsulates the medical image \( x \) and user query \( q \). The policy network’s output is mathematically expressed as:

\[
\pi(a_t|s_t; \theta) = \prod_{t=1}^T \pi(a_t|s_t, a_{<t}; \theta),
\]

where \( \theta \) denotes the parameters of policy network.

Simultaneously, the value network (\( V \)) is tasked with estimating the expected return from state \( s_t \), offering a benchmark for calculating advantage function vital for optimizing the policy network:

\[
V(s_t; \phi) = \mathbb{E}\left[ r_t \mid s_t, \pi \right],
\]
where \( \phi \) represents the parameters of the value network, and \( r_t \) denotes the immediate reward associated with the current state \( s_t \).

The LLM relevance rewarding framework incorporates rewards from the policy network, the value network, and the LLM relevance reward model. The total reward, \( r_t^{\pi,V,LLM} \), measures the improvement of the chosen action over the baseline provided by the value network and includes the relevance reward from the LLM model:

\[
r_t^{\pi,V,LLM} = A(s_t, a_t; \theta, \phi) + r_t^{LLM},
\]

where \( A(s_t, a_t; \theta, \phi) = Q(s_t, a_t; \theta) - V(s_t; \phi) \) is the advantage function, and \( r_t^{LLM} \) represents the relevance reward provided by the LLM model. The Q-function \( Q(s_t, a_t; \theta) \) is updated using the Bellman equation:

\[
\Delta Q(s_t, a_t) =
\alpha \left[ r_t + \gamma \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t) \right],
\]
\[
Q(s_t, a_t) \leftarrow Q(s_t, a_t) + \Delta Q(s_t, a_t),
\]

where \( \alpha \) is the learning rate, \( r_t \) is the immediate reward, \( \gamma \) is the discount factor, and \( \max_{a'} Q(s_{t+1}, a') \) is the maximum expected future reward for the next state \( s_{t+1} \).

Abnormal Localization Reward. The Abnormal Localization Reward (ALR), denoted as \( r_t^{loc} \), quantifies the accuracy of predicted bounding boxes by calculating the Intersection over Union (IoU) between the predicted bounding box (Pred-BBox) and the ground truth bounding box (GT-BBox). Specifically, \( r_t^{loc} \) is defined as:

\[
r_t^{loc} = \frac{\text{Overlap(Pred-BBox, GT-BBox)}}{\text{Union(Pred-BBox, GT-BBox)}}.
\]

This reward directly encourages the model to accurately localize the regions of abnormality within the medical image. A higher IoU score indicates a greater overlap between the predicted and ground truth bounding boxes, signifying a more precise localization of the abnormal area.

Vision Relevance Reward. The Vision Relevance Reward (VRR), \( r_t^{att} \), is computed by aggregating attention weights between abnormal category tokens and image patches identified as abnormal regions. Specifically, within a transformer framework, the VRR quantifies the model’s focus on crucial visual areas by analyzing the attention scores allocated to tokens corresponding to abnormal categories and their association with image patches indicative of abnormalities. This mechanism enhances diagnostic accuracy by reinforcing the model’s concentration on critical visual information. The VRR is calculated as:

\[
r_t^{att} = \sum_{i \in N} \sum_{j \in \bar{N}} \frac{\exp \left( \frac{Q_i K_j^\top}{\sqrt{d_k}} \right)}{\sum_{k \in \bar{N}} \exp \left( \frac{Q_i K_k^\top}{\sqrt{d_k}} \right)},
\]

where \( N \) denotes the set of tokens associated with identified textual abnormalities, and \( \bar{N} \) represents the set of image patches characterized as abnormal regions. \( Q_i \) and \( K_j \) are the query vector for token \( i \) and the key vector for image patch \( j \), respectively; \( d_k \) is the dimensionality of the key vectors. This approach leverages the transformer’s attention mechanism to quantify the model’s focus on critical areas. By emphasizing the importance of abnormality-related tokens and their corresponding attention weights over abnormal image patches, the model is incentivized to attend to visual information indicative of medical abnormalities, ultimately improving diagnostic performance.

Reward Normalization and Aggregation. To achieve equilibrium between the ALR (\( r_t^{loc} \)) and the VRR (\( r_t^{att} \)), we individually normalize these rewards for responses generated by the same query. This normalization ensures that each reward type contributes equally to the final reward calculation. The combined reward for each response is computed as follows:

\[
r_t = r_t^{\pi,V,LLM} + \frac{r_t^{loc}}{\max(r_t^{loc})} + \frac{r_t^{att}}{\max(r_t^{att})},
\]

where \( \max(r_t^{loc}) \) and \( \max(r_t^{att}) \) are the maximum values of the abnormal-aware localization and attention reward, respectively, for all responses to a particular query.

Optimization Process. As an improved version of the PPO, our policy network is refined by maximizing an objective function that incorporates the combined reward \( r_t \) into the PPO. Specifically, we aim to maximize an objective function that combines the clipped surrogate objective with an entropy bonus to encourage exploration:

\[
\mathcal{L}^{\text{CLIP+ENT}}(\theta) = \mathbb{E} \Big[ \mathcal{L}^{\text{CLIP}}(\theta) + c_1 r_t - c_2 \mathcal{L}^{VF}(\phi) + c_3 S[\pi(\cdot|s_t)] \Big],
\]
where \( \mathcal{L}^{CLIP}(\theta) \) is the clipped part of the PPO objective, \( r_t \) is the combined normalized reward as defined previously, \( \mathcal{L}^{VF}(\phi) \) is the value function loss, \( S[\pi(\cdot|s_t)] \) denotes the entropy of the policy for state \( s_t \), and \( c_1, c_2, c_3 \) are coefficients balancing the contribution of each term.

4 MAU Dataset

To enhance the abnormality unveiling capabilities of the Med-LVLM, we construct the MAU dataset for UMed-LVLM training. Firstly, we collect medical image datasets with abnormal annotations. Then, we design a Prompt Method to construct the MAU dataset, a medical diagnosis dataset with abnormal annotations, by GPT-4V (OpenAI, 2023b).

Collecting Medical Image Datasets with Abnormal Areas. We collect a medical image dataset annotated with abnormal areas for training the Med-LVLM. This dataset encompasses five distinct sub-datasets, namely DeepLesion (Yan et al., 2017), KidneyStone (TEZ, 2023), NIH (Wang et al., 2017), TBX11K (Liu et al., 2020), and KVASIR (Pogorelov et al., 2017), each originating from diverse sources. DeepLesion includes 32,120 axial CT slices with eight types of abnormalities. The KidneyStone dataset contains 1,300 renal CT scans of various kidney stones in different sizes, shapes, and locations. The NIH dataset has 112,120 chest X-ray images across 14 pathological categories. The TBX11K dataset consists of 11,200 chest X-ray images for tuberculosis detection. The KVASIR dataset provides 8,000 endoscopic images of the gastrointestinal tract, with eight types of abnormalities. These sub-datasets span various medical imaging modalities, including X-rays, CT scans, and gross pathology. We utilized a portion of these datasets, and Appendix B summarizes the specific details of the segments used from each sub-dataset. Each image is annotated with the type of abnormality present and includes bounding box information for identified abnormal areas.

Medical Abnormal Unveiling Dataset Construction. To construct the Medical Abnormal Unveiling (MAU) Dataset, we design a Prompt Method that utilizes the GPT-4V model to generate diagnosis annotations with medical abnormal areas. The pipeline, as shown in Figure 2, comprises two stages: diagnosing abnormalities in medical images and reflecting on the previous diagnosis. Firstly, we integrate collected medical images, corresponding abnormality categories, and the locations of abnormal areas into our designed prompt. This prompt is then passed into GPT-4V to generate diagnosis responses based on the given abnormality categories and areas. To build a diagnosis with step-by-step thoughts, a reflection prompt is designed to reorganize these diagnosis responses, starting from abnormality detection, identifying bounding box regions, and finally recognizing abnormality categories. The processed responses, the medical images, and queries form medical image diagnosis samples. Through this Prompt Method, we use GPT-4V to generate an MAU dataset (examples in Appendix B). Our plug-and-play prompt method
easily integrates with other medical datasets and is disease-agnostic.

Review by Human Experts. To ensure the reliability of the MAU dataset, all generated data have been reviewed and filtered by human expert reviewers (i.e., three doctoral students specializing in medicine). During review, only 13 samples were found to have errors, which were corrected manually by reviewers. As shown in Table 2, using GPT-4V for direct diagnosis without additional information yields limited performance. However, providing the model with the location of the abnormal area leads to a significant performance improvement. Crucially, in our dataset construction, we provide GPT-4V with both the abnormal area and the ground-truth diagnosis. GPT-4V’s role is not to perform the diagnosis itself, but rather to generate an explanation (the “diagnosis” text) of the reasoning process, given the known diagnosis and the location of the abnormality. This approach leverages the model’s strong language generation capabilities while mitigating its limitations in direct medical image interpretation, resulting in a dataset of high quality and reliability.

5 Experiments

5.1 Experimental Setups

We evaluate our method on five benchmarks: test set of our MAU, VQA-RAD (Lau et al., 2018) and SLAKE (Liu et al., 2021b), PMC-VQA-test (Zhang et al., 2023a) and MedMNIST (Yang et al., 2021). Our MAU comprises five sub-datasets: DeepLesion, KidneyStone, NIH, TBX11K, and KVASIR. We utilized MedVInt (Zhang et al., 2023b) as the initialization for our model. UMed-LVLM is trained on the training set of 4,653 examples and then evaluated on 1,164 examples on MAU. Crucially, we ensure no case overlap between the training and test sets, based on the original data annotations, to prevent data leakage. We employ Accuracy (ACC) as the evaluation metric. ACC assesses the correctness of diagnosis results, i.e., whether the response text includes the correct category of abnormalities, following (Wu et al., 2024) to evaluate the correctness of outputs from LVLMs. The implementation details and details of the compared LVLMs methods are in Appendix C and Appendix D.

<table>
  <tr>
    <th>Method</th>
    <th>DL</th>
    <th>KS</th>
    <th>KV</th>
    <th>NIH</th>
    <th>TBX</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>MimiGPT-4</td>
    <td>0.02</td>
    <td>0.00</td>
    <td>0.02</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td>mPLUG-Owl</td>
    <td>0.05</td>
    <td>0.00</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td>LLaVA</td>
    <td>0.20</td>
    <td>0.00</td>
    <td>0.04</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.05</td>
  </tr>
  <tr>
    <td>Qwen-VL</td>
    <td>0.13</td>
    <td>0.00</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.03</td>
  </tr>
  <tr>
    <td>XrayGPT</td>
    <td>0.18</td>
    <td>0.12</td>
    <td>0.02</td>
    <td>0.07</td>
    <td>0.06</td>
    <td>0.09</td>
  </tr>
  <tr>
    <td>LLaVA-Med</td>
    <td>0.22</td>
    <td>0.04</td>
    <td>0.12</td>
    <td>0.03</td>
    <td>0.01</td>
    <td>0.08</td>
  </tr>
  <tr>
    <td>Med-Flamingo</td>
    <td>0.27</td>
    <td>0.15</td>
    <td>0.15</td>
    <td>0.09</td>
    <td>0.02</td>
    <td>0.14</td>
  </tr>
  <tr>
    <td>MedVInt</td>
    <td>0.29</td>
    <td>0.11</td>
    <td>0.27</td>
    <td>0.08</td>
    <td>0.09</td>
    <td>0.17</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="7">Trained on MAU Dataset</th>
  </tr>
  <tr>
    <td>MedVInt (SFT)</td>
    <td>0.42</td>
    <td>0.93</td>
    <td>0.93</td>
    <td>0.28</td>
    <td>0.78</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>MedVInt (SFT&PPO)</td>
    <td>0.44</td>
    <td>0.94</td>
    <td>0.95</td>
    <td>0.30</td>
    <td>0.80</td>
    <td>0.69</td>
  </tr>
  <tr>
    <td>UMed-LVLM</td>
    <td><b>0.53</b></td>
    <td><b>0.99</b></td>
    <td><b>0.98</b></td>
    <td><b>0.37</b></td>
    <td><b>0.86</b></td>
    <td><b>0.75</b></td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="7">Closed-Source Model</th>
  </tr>
  <tr>
    <td>GPT-4V</td>
    <td>0.27</td>
    <td>0.36</td>
    <td>0.53</td>
    <td>0.18</td>
    <td>0.19</td>
    <td>0.31</td>
  </tr>
  <tr>
    <td>GPT-4V w/ bbox</td>
    <td>0.50</td>
    <td>0.95</td>
    <td>0.95</td>
    <td>0.32</td>
    <td>0.81</td>
    <td>0.72</td>
  </tr>
</table>

Table 2: Comparison on test set.“DL”, “KS” and “KV” denote “Deep Lesion”, “Kidney Stone” and “KVASIR”, respectively. The comparison models include LVLMs (i.e., MiniGPT-4 (Zhu et al., 2024a), mPLUG-Owl (Ye et al., 2023), LLaVA (Liu et al., 2023b), Qwen-VL (Bai et al., 2023), GPT-4V (OpenAI, 2023b)) and Med-LVLMs (i.e., XrayGPT (Thawakar et al., 2023), LLaVA-Med (Li et al., 2023a), Med-Flamingo (Moor et al., 2023), MedVInt (Zhang et al., 2023a)). MedVInt (SFT) denotes MedVInt trained with SFT on the MAU dataset, while MedVInt (SFT&PPO) indicates MedVInt trained with SFT and PPO on the MAU dataset. GPT-4V w/ bbox denotes that GPT-4V was provided with abnormal region location information.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">VQA-RAD</th>
    <th colspan="2">SLAKE</th>
  </tr>
  <tr>
    <th>Open</th>
    <th>Close</th>
    <th>Open</th>
    <th>Close</th>
  </tr>
  <tr>
    <td>MEVF-BAN (Nguyen et al., 2019)</td>
    <td>49.2</td>
    <td>77.2</td>
    <td>77.8</td>
    <td>79.8</td>
  </tr>
  <tr>
    <td>CPRD-BAN (Liu et al., 2021a)</td>
    <td>52.5</td>
    <td>77.9</td>
    <td>79.5</td>
    <td>83.4</td>
  </tr>
  <tr>
    <td>M3AE (Chen et al., 2022)</td>
    <td>67.2</td>
    <td>83.5</td>
    <td>80.3</td>
    <td>87.8</td>
  </tr>
  <tr>
    <td>PMC-CLIP (Lin et al., 2023)</td>
    <td>67.0</td>
    <td>84.0</td>
    <td>81.9</td>
    <td>88.0</td>
  </tr>
  <tr>
    <td>MedVInt (Zhang et al., 2023a)</td>
    <td>69.3</td>
    <td>84.2</td>
    <td>88.2</td>
    <td>87.7</td>
  </tr>
  <tr>
    <td>UMed-LVLM</td>
    <td><b>74.9</b></td>
    <td><b>87.6</b></td>
    <td><b>90.4</b></td>
    <td><b>89.5</b></td>
  </tr>
</table>

Table 3: Results on VQA-RAD (Lau et al., 2018) and SLAKE (Liu et al., 2021b).

5.2 Results and Discussion

As shown in Table 2, our method outperforms other methods on the MAU. By comparing UMed-LVLM with MedVInt (SFT) and MedVInt (SFT&PPO), the performance gain can be attributed to the MAU dataset and our Abnormal-Aware Rewarding approach. From the results, UMed-LVLM significantly outperforms GPT-4V. Furthermore, providing ground-truth abnormal region locations to GPT-4V (GPT-4V w/ bbox) leads to a significant improvement, emphasizing the importance of abnormal region localization for diagnosis. Table 3 and Table 4 present results on VQA-RAD, SLAKE, and PMC-VQA-test datasets. UMed-LVLM outperforms other methods, showcasing
<table>
  <tr>
    <th>Method</th>
    <th>Choice</th>
    <th>Blanking</th>
  </tr>
  <tr>
    <td>PMC-CLIP (Lin et al., 2023)</td>
    <td>24.7</td>
    <td>-</td>
  </tr>
  <tr>
    <td>BLIP-2 (Li et al., 2023b)</td>
    <td>24.3</td>
    <td>21.8</td>
  </tr>
  <tr>
    <td>Open-Flamingo (OpenAI, 2023a)</td>
    <td>26.4</td>
    <td>26.5</td>
  </tr>
  <tr>
    <td>LLAVA-Med (Li et al., 2023a)</td>
    <td>34.8</td>
    <td>29.4</td>
  </tr>
  <tr>
    <td>MedVInT (Zhang et al., 2023a)</td>
    <td>39.2</td>
    <td>35.3</td>
  </tr>
  <tr>
    <th>UMed-LVLM</th>
    <th>42.6</th>
    <th>38.1</th>
  </tr>
</table>

Table 4: Comparison of different methods on PMC-VQA-test (Zhang et al., 2023a) (metric is accuracy).

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Pneumonia</th>
    <th colspan="2">Breast</th>
    <th colspan="2">Derma</th>
  </tr>
  <tr>
    <th>AUC↑</th>
    <th>ACC↑</th>
    <th>AUC↑</th>
    <th>ACC↑</th>
    <th>AUC↑</th>
    <th>ACC↑</th>
  </tr>
  <tr>
    <td>ResNet50</td>
    <td>96.2</td>
    <td>88.4</td>
    <td>86.6</td>
    <td>84.2</td>
    <td>91.2</td>
    <td>73.1</td>
  </tr>
  <tr>
    <td>DWT-CV</td>
    <td>95.7</td>
    <td>88.7</td>
    <td>89.8</td>
    <td>85.7</td>
    <td>91.7</td>
    <td>74.8</td>
  </tr>
  <tr>
    <td>SADAE</td>
    <td>98.3</td>
    <td>91.8</td>
    <td>91.5</td>
    <td>87.8</td>
    <td>92.7</td>
    <td>75.9</td>
  </tr>
  <tr>
    <td>PMC-CLIP</td>
    <td>99.0</td>
    <td>95.4</td>
    <td>94.6</td>
    <td>91.4</td>
    <td>93.4</td>
    <td>79.8</td>
  </tr>
  <tr>
    <td>MedVInT</td>
    <td>98.5</td>
    <td>94.9</td>
    <td>93.4</td>
    <td>90.4</td>
    <td>93.7</td>
    <td>80.0</td>
  </tr>
  <tr>
    <th>UMed-LVLM</th>
    <th><b>99.1</b></th>
    <th><b>95.8</b></th>
    <th><b>95.2</b></th>
    <th><b>92.8</b></th>
    <th><b>94.2</b></th>
    <th><b>84.1</b></th>
  </tr>
</table>

Table 5: Performance comparison on MedMNIST (Yang et al., 2021): Pneumonia (chest X-ray), Breast (ultrasound), and Derma (dermoscopy). The comparison models include ResNet50 (He et al., 2016), DWT-CV (Cheng et al., 2022), SADAE (Ge et al., 2022), PMC-CLIP, and MedVInT.

its ability to understand and reason about medical images and related questions. In Table 5, UMed-LVLM outperforms other methods on MedMNIST (Pneumonia, Breast, Derma). Remarkably, despite continual training without Ultrasound and Dermatoscopy data, UMed-LVLM generalizes well to these modalities, demonstrating its effectiveness and generalization. The case study can be found in Appendix F.

5.3 Ablation Study

To verify the efficacy of components in AAR, we conduct experiments by progressively removing each component from UMed-LVLM to observe the impact on performance. As shown in Table 6, we compare the performance of UMed-LVLM and its three variants, i.e., “w/o ALR”, “w/o VRR”, and “w/o AAR”. The results show a performance decline as components are removed, demonstrating that each component plays a critical role in enhancing diagnosis accuracy for medical images. The VRR, by aligning abnormal identification and attention regions, ensures that the model does not overlook subtle but critical abnormalities in the images. The performance degradation observed when either component is removed substantiates the hypothesis that a dual-reward system, which addresses abnormality recognition and localization, is beneficial in diagnosis.

<table>
  <tr>
    <th>Method</th>
    <th>DL</th>
    <th>KS</th>
    <th>KV</th>
    <th>NIH</th>
    <th>TBX</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <th>UMed-LVLM</th>
    <th>0.53</th>
    <th>0.99</th>
    <th>0.98</th>
    <th>0.37</th>
    <th>0.86</th>
    <th>0.75</th>
  </tr>
  <tr>
    <td>w/o VRR</td>
    <td>0.49</td>
    <td>0.97</td>
    <td>0.95</td>
    <td>0.30</td>
    <td>0.82</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>w/o ALR</td>
    <td>0.48</td>
    <td>0.96</td>
    <td>0.96</td>
    <td>0.35</td>
    <td>0.83</td>
    <td>0.72</td>
  </tr>
  <tr>
    <td>w/o AAR</td>
    <td>0.42</td>
    <td>0.93</td>
    <td>0.93</td>
    <td>0.28</td>
    <td>0.78</td>
    <td>0.67</td>
  </tr>
</table>

Table 6: Ablation study verifies the effectiveness of components in the Abnormal-Aware Rewarding (AAR), including two components: Abnormal Localization Reward (ALR) and Abnormal-Vision Relevance Reward (VRR). “w/o AAR” denotes UMed-LVLM only trained on Abnormal-Aware Instruction Tuning.

![Performance of UMed-LVLM with different IoUs.](page_1012_1042_324_246.png)

Figure 3: Performance of UMed-LVLM with different IoUs.

![Performance of Abnormal-Aware Instruction Tuning across different epochs (Left) and data scales(Right).](page_1012_1322_324_246.png)

Figure 4: Performance of Abnormal-Aware Instruction Tuning across different epochs (Left) and data scales(Right).

5.4 Analysis

Impact of Abnormal Localization. To verify the impact of localization ability during the reasoning process on Med-LVLM, we injected bounding boxes (bbox) corresponding to different IoU scores during model inference for observation. The experimental results, as shown in Figure 3, indicate that enhancing localization ability can influence the diagnostic results. As anomaly localization becomes more accurate, the model’s diagnostic performance improves. However, when IoU exceeds 0.6, the improvement in the model’s diagnostic performance begins to plateau. This suggests that Med-LVLM does not require extremely high localization accuracy to leverage the reasoning gains from localization; a certain level of localization ability is sufficient to enhance diagnostic reasoning. The red dot in the figure represents the performance achieved by our UMed-LVLM. From the trend, it is evident that the localization ability of our method effectively enhances model’s diagnostic capability.
<table>
  <tr>
    <th>Method</th>
    <th>Abdomen</th>
    <th>Lung</th>
    <th>Pelvis</th>
  </tr>
  <tr>
    <td>UMed-LVLM</td>
    <td><b>0.35</b></td>
    <td><b>0.39</b></td>
    <td><b>0.27</b></td>
  </tr>
  <tr>
    <td>MedVInt (SFT)</td>
    <td>0.16</td>
    <td>0.15</td>
    <td>0.16</td>
  </tr>
  <tr>
    <td>MedVInt (SFT&PPO)</td>
    <td>0.15</td>
    <td>0.12</td>
    <td>0.15</td>
  </tr>
  <tr>
    <td>MedVInt</td>
    <td>0.05</td>
    <td>0.04</td>
    <td>0.07</td>
  </tr>
</table>

Table 7: Model generalization to categories excluded from training set. MedVInt (SFT) and MedVInt (SFT&PPO) can be found in Table 2.

<table>
  <tr>
    <th>Method</th>
    <th>TBX11K</th>
    <th>Deep Lesion</th>
  </tr>
  <tr>
    <td>UMed-LVLM</td>
    <td><b>0.57</b></td>
    <td><b>0.42</b></td>
  </tr>
  <tr>
    <td>MedVInt (SFT)</td>
    <td>0.29</td>
    <td>0.28</td>
  </tr>
  <tr>
    <td>MedVInt (SFT&PPO)</td>
    <td>0.30</td>
    <td>0.22</td>
  </tr>
  <tr>
    <td>MedVInt</td>
    <td>0.10</td>
    <td>0.08</td>
  </tr>
</table>

Table 8: Performance of UMed-LVLM w/o training on TBX11K and DeepLesion datasets, respectively. Performance is evaluated on the excluded datasets.

Impact of Instruction Tuning Epoch. As shown in Figure 4(Left), we investigate the effects of employing Instruction Tuning over different epochs on the performance of our model. The performance exhibits an upward trajectory until epoch 4, achieving a peak of approximately 0.75, which suggests that the model becomes progressively more adept at executing tasks as per the user queries and medical image inputs.

Impact of Training Set Scale for Instruction Tuning. As shown in Figure 4(Right), the performance of our model exhibits a positive correlation with the scale of the training data. We can observe a substantial increment as the data scale progresses from 20% to 100%. This trend illustrates that the model benefits from a larger volume of data, refining its ability to understand abnormalities and execute instructions more effectively. This improvement underscores the model’s evolving proficiency in identifying the exact spatial extents of abnormalities, which is crucial for subsequent diagnosis tasks. The incremental nature of this trend indicates that the model’s spatial comprehension capabilities can be significantly honed with more extensive training data. These results collectively highlight the critical role that data scale plays in the efficacy of instruction tuning for Med-LVLM.

Generalization Capability for Untrained Category. This setting aims to evaluate the generalization capability of the UMed-LVLM on medical categories not present in the training data. Specifically, we removed data in three categories (i.e., “Abdomen”, “Lung”, and “Pelvis” on “DeepLesion”) on the training set. Then we evaluated the trained model on these categories to observe its ability to handle untrained categories. The results, shown in Table 7, present the performance in each category. The performance demonstrates that our method exhibits a degree of generalization capability on untrained categories of medical images. Although the performance on untrained categories is somewhat reduced compared to training on these categories, these findings indicate that the model can generalize knowledge learned during training to new categories not present in the training data. In contrast, MedVInt shows a slight ability to generalize to these untrained categories with scores close to zero across all categories. It demonstrates the effectiveness of abnormal-aware learning.

Generalization Capability of Abnormality Unveiling. To assess the robustness and flexibility of UMed-LVLM and MedVInt, we respectively removed the TBX11K and DeepLesion datasets from the training data to verify the models’ generalization capabilities. We evaluated the models on the excluded datasets to determine their ability to generalize across different medical datasets. As shown in Table 8, UMed-LVLM demonstrated significantly better generalization than other methods. The performance disparity shows our model’s effectiveness in adapting to varied medical scenarios.

Generalization Capability for Cross-Modality. To verify the generalization capabilities of UMed-LVLM, we train it on single-modality medical images and evaluate it across different modalities. As shown in Table 9, our approach demonstrates better performance in cross-modal generalization compared to other methods. The performance of our model is attributed to our abnormal-aware training method that enhances the model’s capability to adapt to various medical images. The Abnormal-Aware Instruction Tuning and AAR boost the model’s capability to localize abnormalities and improve diagnostic accuracy in medical images. This is particularly important in scenarios where the modalities differ substantially (e.g., CT vs. Gross Pathology).

6 Conclusion

This study introduces UMed-LVLM, a novel Med-LVLM designed to enhance medical diagnosis by the visual localization of abnormalities in medi-
<table>
  <tr>
    <th>Method</th>
    <th>C -> GX</th>
    <th>G -> CX</th>
    <th>X -> CG</th>
  </tr>
  <tr>
    <td>UMed-LVLM</td>
    <td><b>0.27</b></td>
    <td><b>0.31</b></td>
    <td><b>0.22</b></td>
  </tr>
  <tr>
    <td>MedVInt (SFT)</td>
    <td>0.12</td>
    <td>0.08</td>
    <td>0.13</td>
  </tr>
  <tr>
    <td>MedVInt (SFT&PPO)</td>
    <td>0.09</td>
    <td>0.08</td>
    <td>0.11</td>
  </tr>
  <tr>
    <td>MedVInt</td>
    <td>0.05</td>
    <td>0.04</td>
    <td>0.07</td>
  </tr>
</table>

Table 9: Cross-modal generalization capability models trained on single modality medical images. “C”, “G”, and “X” represent CT, Gross Pathology, and X-ray, respectively.

cal images. Through a specialized training process involving the collection of a Medical Abnormalities Unveiling dataset and the implementation of Abnormal-Aware Instruction Tuning and Abnormal-Aware Rewarding. Results show UMed-LVLM surpasses existing Med-LVLMs in accurately detecting and interpreting medical anomalies. The AAR incorporates innovative reward mechanisms that sharpen the model’s focus on abnormal areas, thereby improving diagnostic reliability and interpretability. Furthermore, the in-depth analysis demonstrates the generalization capability of UMed-LVLM.

Limitations

Our study’s limitation is the limited computational resources. Recent advancements have led to the development of increasingly large LVLMs, which require substantial computational power to train and deploy effectively. As a result of these computational limitations, we were unable to apply our methodologies to the largest and most complex open-source models currently available. This limits our ability to expand and validate our approach on larger-scale LVLMs.

References

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. 2022. Flamingo: a visual language model for few-shot learning. In NeurIPS.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. CoRR, abs/2308.12966.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In NeurIPS 2020.

Yuanjiang Cao, Quan Z. Sheng, Julian J. McAuley, and Lina Yao. 2023. Reinforcement learning for generative AI: A survey. CoRR, abs/2308.14328.

Xi Chen, Xiao Wang, Soravit Changpinyo, A. J. Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V. Thapliyal, James Bradbury, and Weicheng Kuo. 2023. Pali: A jointly-scaled multilingual language-image model. In ICLR 2023. OpenReview.net.

Zhihong Chen, Yuhao Du, Jinpeng Hu, Yang Liu, Guanbin Li, Xiang Wan, and Tsung-Hui Chang. 2022. Multi-modal masked autoencoders for medical vision-and-language pre-training. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2022 - 25th International Conference, Singapore, September 18-22, 2022, Proceedings, Part V, volume 13435 of Lecture Notes in Computer Science, pages 679–689. Springer.

Jianhong Cheng, Hulin Kuang, Qichang Zhao, Yahui Wang, Lei Xu, Jin Liu, and Jianxin Wang. 2022. DWT-CV: dense weight transfer-based cross validation strategy for model selection in biomedical data analysis. Future Gener. Comput. Syst., 135:20–29.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna.lmsys.org (accessed 14 April 2023), 2(3):6.

Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4299–4307.

Hejie Cui, Lingjun Mao, Xin Liang, Jieyu Zhang, Hui Ren, Quanzheng Li, Xiang Li, and Carl Yang. 2024. Biomedical visual instruction tuning with clinician preference alignment. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR 2021. OpenReview.net.

Weihao Gao, Zhuo Deng, Zhiyuan Niu, Fuju Rong, Chucheng Chen, Zheng Gong, Wenze Zhang, Daimin Xiao, Fang Li, Zhenjie Cao, Zhaoyi Ma, Wenbin Wei, and Lan Ma. 2023. Ophglm: Training an ophthalmology large language-and-vision assistant based on instructions and dialogue. CoRR, abs/2306.12174.

Xiaolong Ge, Yanpeng Qu, Changjing Shang, Longzhi Yang, and Qiang Shen. 2022. A self-adaptive discriminative autoencoder for medical applications. IEEE Trans. Circuits Syst. Video Technol., 32(12):8875–8886.

Qiushan Guo, Shalini De Mello, Hongxu Yin, Wonmin Byeon, Ka Chun Cheung, Yizhou Yu, Ping Luo, and Sifei Liu. 2024. Regionopt: Towards region understanding vision language model. CoRR, abs/2403.02330.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770–778. IEEE Computer Society.

Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara L. Berg. 2014. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP 2014, pages 787–798. ACL.

Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR 2015.

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vis., 123(1):32–73.

Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. 2018. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1–10.

Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Nau mann, Hoifung Poon, and Jianfeng Gao. 2023a. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. In NeurIPS 2023.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023b. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML 2023, volume 202 of Proceedings of Machine Learning Research, pages 19730–19742. PMLR.

Weixiong Lin, Ziheng Zhao, Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023. PMC-CLIP: contrastive language-image pre-training using biomedical documents. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2023 - 26th International Conference, Vancouver, BC, Canada, October 8-12, 2023, Proceedings, Part VIII, volume 14227 of Lecture Notes in Computer Science, pages 525–536. Springer.

Bo Liu, Li-Ming Zhan, and Xiao-Ming Wu. 2021a. Contrastive pre-training and representation distillation for medical visual question answering based on radiology images. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2021 - 24th International Conference, Strasbourg, France, September 27 - October 1, 2021, Proceedings, Part II, volume 12902 of Lecture Notes in Computer Science, pages 210–220. Springer.

Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. 2021b. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 18th IEEE International Symposium on Biomedical Imaging, ISBI 2021, Nice, France, April 13-16, 2021, pages 1650–1654. IEEE.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a. Improved baselines with visual instruction tuning. CoRR, abs/2310.03744.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. Visual instruction tuning. In NeurIPS 2023.

Junling Liu, Ziming Wang, Qichen Ye, Dading Chong, Peilin Zhou, and Yining Hua. 2023c. Qilin-med-v1: Towards chinese large vision-language model for general healthcare. CoRR, abs/2310.17956.

Yun Liu, Yu-Huan Wu, Yunfeng Ban, Huifang Wang, and Ming-Ming Cheng. 2020. Rethinking computer-aided tuberculosis diagnosis. In CVPR 2020, pages 2643–2652. Computer Vision Foundation / IEEE.

Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. CoRR, abs/2308.08747.

Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. OK-VQA: A visual question answering benchmark requiring external knowledge. In CVPR 2019, pages 3195–3204. Computer Vision Foundation / IEEE.

Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. 2019. OCR-VQA: visual question answering by reading text in images. In 2019 International Conference on Document Analysis and Recognition, ICDAR 2019, Sydney, Australia, September 20-25, 2019, pages 947–952. IEEE.

Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley,
David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In ICML 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1928–1937. JMLR.org.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. 2015. Human-level control through deep reinforcement learning. Nat., 518(7540):529–533.

Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. 2023. Med-flamingo: a multimodal medical few-shot learner. In ML4H@NeurIPS 2023, volume 225 of Proceedings of Machine Learning Research, pages 353–367. PMLR.

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021. Webgpt: Browser-assisted question-answering with human feedback. CoRR, abs/2112.09332.

Binh D. Nguyen, Thanh-Toan Do, Binh X. Nguyen, Tuong Do, Erman Tjiputra, and Quang D. Tran. 2019. Overcoming data limitation in medical visual question answering. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2019 - 22nd International Conference, Shenzhen, China, October 13-17, 2019, Proceedings, Part IV, volume 11767 of Lecture Notes in Computer Science, pages 522–530. Springer.

OpenAI. 2023a. GPT-4 technical report. CoRR, abs/2303.08774.

OpenAI. 2023b. Gpt-4v(ision) system card.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In NeurIPS 2022.

Konstantin Pogorelov, Kristin Ranheim Randel, Carsten Griwodz, Sigrun Losada Eskeland, Thomas de Lange, Dag Johansen, Concetto Spampinato, Duc-Tien Dang-Nguyen, Mathias Lux, Peter Thelin Schmidt, Michael Riegler, and Pål Halvorsen. 2017. KVASIR: A multi-class image dataset for computer aided gastrointestinal disease detection. In Proceedings of the 8th ACM on Multimedia Systems Conference, MM-Sys 2017, Taipei, Taiwan, June 20-23, 2017, pages 164–169. ACM.

Ziyuan Qin, Huahui Yi, Qicheng Lao, and Kang Li. 2023. MEDICAL IMAGE UNDERSTANDING WITH PRETRAINED VISION LANGUAGE MODELS: A COMPREHENSIVE STUDY. In ICLR 2023. OpenReview.net.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In ICML 2021, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR.

Kanchana Ranasinghe, Satya Narayan Shukla, Omid Poursaeed, Michael S. Ryoo, and Tsung-Yu Lin. 2024. Learning to localize objects improves spatial reasoning in visual-llms. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 12977–12987. IEEE.

John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. 2015. Trust region policy optimization. In ICML 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 1889–1897. JMLR.org.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. CoRR, abs/1707.06347.

Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-OKVQA: A benchmark for visual question answering using world knowledge. In ECCV 2022, volume 13668 of Lecture Notes in Computer Science, pages 146–162. Springer.

Chang Shu, Baian Chen, Fangyu Liu, Zihao Fu, Ehsan Shareghi, and Nigel Collier. 2023. Visual med-alpaca: A parameter-efficient biomedical llm with visual capabilities.

Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. 2020. Textcaps: A dataset for image captioning with reading comprehension. In ECCV 2020, volume 12347 of Lecture Notes in Computer Science, pages 742–758. Springer.

Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. 2020. Learning to summarize from human feedback. CoRR, abs/2009.01325.

Richard S. Sutton and Andrew G. Barto. 1998. Reinforcement learning - an introduction. Adaptive computation and machine learning. MIT Press.
TEZ. 2023. Tez_roi_aug dataset. https://universe.roboflow.com/tez-nwkf5/tez_roi_aug. Visited on 2024-05-21.

Omkar Thawakar, Abdelrahman Shaker, Sahal Shahj Mullappilly, Hisham Cholakkal, Rao Muhammad Anwer, Salman H. Khan, Jorma Laaksonen, and Farhad Shahbaz Khan. 2023. Xraygpt: Chest radiographs summarization using medical vision-language models. CoRR, abs/2306.07971.

Guangyu Wang, Guoxing Yang, Zongxin Du, Longjun Fan, and Xiaohu Li. 2023. Clinicalgpt: Large language models finetuned with diverse medical data and comprehensive evaluation. CoRR, abs/2306.09968.

Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M. Summers. 2017. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In CVPR 2017, pages 3462–3471. IEEE Computer Society.

Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. 2022. Medclip: Contrastive learning from unpaired medical images and text. In EMNLP 2022, pages 3876–3887. Association for Computational Linguistics.

Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. Finetuned language models are zero-shot learners. In ICLR 2022. OpenReview.net.

Chaoyi Wu, Jiayu Lei, Qiaoyu Zheng, Weike Zhao, Weixiong Lin, Xiaoman Zhang, Xiao Zhou, Ziheng Zhao, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023. Can gpt-4v(ision) serve medical applications? case studies on GPT-4V for multimodal medical diagnosis. CoRR, abs/2310.09909.

Shao-Hong Wu, Wen-Juan Tong, Ming-De Li, Hang-Tong Hu, Xiao-Zhou Lu, Ze-Rong Huang, Xin-Xin Lin, Rui-Fang Lu, Ming-De Lu, Li-Da Chen, and Wei Wang. 2024. Collaborative enhancement of consistency and accuracy in us diagnosis of thyroid nodules using large language models. Radiology, 310(3):e232255. PMID: 38470237.

Peng Xia, Ze Chen, Juanxi Tian, Yangrui Gong, Ruibo Hou, Yue Xu, Zhenbang Wu, Zhizhuan Fan, Yiyang Zhou, Kangyu Zhu, Wenhao Zheng, Zhaoyang Wang, Xiao Wang, Xuchao Zhang, Chetan Bansal, Marc Niethammer, Junzhou Huang, Hongtu Zhu, Yun Li, Jimeng Sun, Zongyuan Ge, Gang Li, James Zou, and Huaxiu Yao. 2024a. CARES: A comprehensive benchmark of trustworthiness in medical vision language models. CoRR, abs/2406.06007.

Peng Xia, Kangyu Zhu, Haoran Li, Hongtu Zhu, Yun Li, Gang Li, Linjun Zhang, and Huaxiu Yao. 2024b. RULE: reliable multimodal RAG for factuality in medical vision language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 1081–1093. Association for Computational Linguistics.

Yunfei Xie, Ce Zhou, Lang Gao, Juncheng Wu, Xianhang Li, Hong-Yu Zhou, Sheng Liu, Lei Xing, James Zou, Cihang Xie, and Yuvin Zhou. 2024. Medtrinity-25m: A large-scale multimodal dataset with multigranular annotations for medicine. CoRR, abs/2408.02900.

Ke Yan, Xiaosong Wang, Le Lu, and Ronald M. Summers. 2017. Deeplesion: Automated deep mining, categorization and detection of significant radiology image findings using large-scale clinical lesion annotations. CoRR, abs/1710.01766.

Jiancheng Yang, Rui Shi, and Bingbing Ni. 2021. Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis. In 18th IEEE International Symposium on Biomedical Imaging, ISBI 2021, Nice, France, April 13-16, 2021, pages 191–195. IEEE.

Songhua Yang, Hanjie Zhao, Senbin Zhu, Guangyu Zhou, Hongfei Xu, Yuxiang Jia, and Hongying Zan. 2024. Zhongjing: Enhancing the chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue. In AAAI 2024, pages 19368–19376. AAAI Press.

Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. 2023. mplug-owl: Modularization empowers large language models with multimodality. CoRR, abs/2304.14178.

Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianno Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. 2023. Woodpecker: Hallucination correction for multimodal large language models. CoRR, abs/2310.16045.

Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, and Maosong Sun. 2024. RLHF-V: towards trustworthy mlms via behavior alignment from fine-grained correctional human feedback. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 13807–13816. IEEE.

Oliver Zhang, Jean-Benoit Delbrouck, and Daniel L. Rubin. 2021. Out of distribution detection for medical images. In PIPPI 2021, volume 12959 of Lecture Notes in Computer Science, pages 102–111. Springer.

Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023a. PMC-VQA: visual instruction tuning for medical visual question answering. CoRR, abs/2305.10415.
Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023b. Pmc-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305.10415.

Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. LIMA: less is more for alignment. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Yucheng Zhou, Xiang Li, Qianning Wang, and Jianbing Shen. 2024a. Visual in-context learning for large vision-language models. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 15890–15902. Association for Computational Linguistics.

Yucheng Zhou, Zhi Rao, Jun Wan, and Jianbing Shen. 2024b. Rethinking visual dependency in long-context reasoning for large vision-language models. CoRR, abs/2410.19732.

Yucheng Zhou, Jiha Zhang, Guanjie Chen, Jianbing Shen, and Yu Cheng. 2024c. Less is more: Vision representation compression for efficient video generation with large language models.

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2024a. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In ICLR 2024. OpenReview.net.

Kangyu Zhu, Peng Xia, Yun Li, Hongtu Zhu, Sheng Wang, and Huaxiu Yao. 2024b. Mmedpo: Aligning medical vision-language models with clinical-aware multimodal preference optimization. CoRR, abs/2412.06141.
A  Related Work

A.1  Large Vision-Language Models

Recently, there have been remarkable advancements within the domain of LVLMs (Alayrac et al., 2022; Chen et al., 2023; Zhou et al., 2024b,c). These models effectively bridge the gap between visual comprehension and contextual language understanding, presenting a robust solution for reconciling disparities between visual and textual data, and thereby, enhancing their capacity to address various vision-language tasks. By pre-training on image-text pairs, CLIP (Radford et al., 2021) achieved zero-shot transfer to diverse computer vision tasks without requiring task-specific training. In contrast, BLIP-2 (Li et al., 2023b) employed frozen pre-trained image encoders and LLMs to bridge the modality gap, achieving effective representation and generative learning. The introduction of GPT-4 (OpenAI, 2023a) ushered in a new era of LVLMs (Zhu et al., 2024a; Ye et al., 2023; Bai et al., 2023). LLaVA (Liu et al., 2023b) leveraged a language-only GPT-4 to generate high-quality multimodal instruction data effectively. Notably, LLaVA demonstrated exceptional chat capabilities, even when provided with novel images and instructions. The improved version LLaVA (Liu et al., 2023a) elevated its performance by incorporating academic-task-oriented VQA datasets (Marino et al., 2019; Mishra et al., 2019; Sidorov et al., 2020; Schwenk et al., 2022) and simple response formatting prompts, thus establishing stronger baselines. It is worth mentioning that the inclusion of region-level VQA datasets (Kazemzadeh et al., 2014; Krishna et al., 2017) could significantly enhance the model’s ability to localize fine-grained visual details precisely.

A.2  Medical Large Vision-Language Models

With the rapid development of LVLMs, their applications in the medical domain have also increased significantly. LLaVA-Med (Li et al., 2023a) extended the capabilities of LLaVA (Liu et al., 2023b) to the medical domain with excellent multimodal dialogue capabilities. Unlike Visual Med-Alpaca (Shu et al., 2023), which connected image captioning models with an LLM and employed a classifier for model assignment, LLaVA-Med is an end-to-end model specifically designed for the medical field. In specific medical domains, LVLMs also play a significant role. In ophthalmology, OphGLM (Gao et al., 2023) combined visual and language capabilities to provide comprehensive ophthalmic assistance. To solve open-ended questions about chest radiographs, XrayGPT (Thawakar et al., 2023) effectively aligned the medical visual encoder (Wang et al., 2022) with a fine-tuned large language model (Chiang et al., 2023) by employing a straightforward linear transformation. The scarcity of non-English language models prompted the introduction of Qinlin-Med-VL (Liu et al., 2023c) as the Chinese large vision-language model that combines a pre-trained Vision Transformer (Dosovitskiy et al., 2021) with a foundational Large Language Model. Observations in (Wu et al., 2023) indicated that although GPT-4V performs well in differentiating medical image modalities and anatomy, it still has difficulties in disease diagnosis and generating comprehensive reports. More specifically, GPT-4V demonstrates inadequate performance in accurately identifying the structures or abnormalities in medical images. In addition, several studies have focused on aligning models with clinical preferences, as highlighted by Cui et al. (2024) who emphasize the importance of clinician preference alignment in VLM fine-tuning. MMedPO (Zhu et al., 2024b) proposes a clinical-aware multimodal preference optimization method to improve the model’s understanding of clinical data. Addressing the critical issue of factual accuracy, Xia et al. (2024b) introduce a reliable multimodal retrieval-augmented generation (RAG) approach. To facilitate the development of medical VLMs, Xie et al. (2024) present MedTrinity-25M, a large-scale multimodal dataset with multi-granular annotations. Furthermore, CARES (Xia et al., 2024a) offers a comprehensive benchmark for evaluating the trustworthiness of medical VLMs. In contrast to these works, our approach focuses on leveraging abnormal regions to guide model optimization.

A.3  Reinforcement Learning for Large Language Models

Reinforcement Learning (RL) can be defined as a training paradigm that could learn from interactions with environments (Sutton and Barto, 1998). Following the inception of RL, a multitude of approaches have been proposed, some of the more widely used are Q-learning (Mnih et al., 2015), Trust Region Policy Optimization (TRPO) (Schulman et al., 2015), Asynchronous Advantage Actorcritic (A3C) (Mnih et al., 2016) and Proximal Policy Optimization (PPO) (Schulman et al., 2017).
<table>
  <tr>
    <th>Dataset</th>
    <th>Type</th>
    <th>Class</th>
    <th>Train</th>
    <th>Test</th>
    <th>Total</th>
  </tr>
  <tr>
    <td>DeepLesion</td>
    <td>CT</td>
    <td>9</td>
    <td>1,584</td>
    <td>396</td>
    <td>1,980</td>
  </tr>
  <tr>
    <td>KidneyStone</td>
    <td>CT</td>
    <td>1</td>
    <td>843</td>
    <td>211</td>
    <td>1,054</td>
  </tr>
  <tr>
    <td>NIH</td>
    <td>X-ray</td>
    <td>14</td>
    <td>787</td>
    <td>197</td>
    <td>984</td>
  </tr>
  <tr>
    <td>TBX11K</td>
    <td>X-ray</td>
    <td>1</td>
    <td>639</td>
    <td>160</td>
    <td>799</td>
  </tr>
  <tr>
    <td>KVASIR</td>
    <td>Pathology</td>
    <td>1</td>
    <td>800</td>
    <td>200</td>
    <td>1,000</td>
  </tr>
  <tr>
    <th>Total</th>
    <th></th>
    <th>3</th>
    <th>4,653</th>
    <th>1,164</th>
    <th>5,817</th>
  </tr>
</table>

Table 10: Specific details of the segments used from datasets.

Different from the distribution modeling objective of supervised and unsupervised learning, RL is more flexible in terms of reward functions, which facilitates the application of RL methods to generative AI models (Cao et al., 2023). In the domain of LLMs, RL can be a vital technique for their development to unlock significant potential. As a successful application of RL combined with LLM, WebGPT (Nakano et al., 2021) optimized the reward function through RL and rejection sampling with the aim of improving its behavior. Another illustrative example of employing RL to LLM is InstructGPT (Ouyang et al., 2022), which used methodology of both reinforcement learning with human feedback (RLHF (Stiennon et al., 2020; Christiano et al., 2017)) and PPO to proficiently fine-tune GPT-3 (Brown et al., 2020), thereby empowering it to follow a diverse range of written instructions. It is worth noting that, in addition to the realm of general LLMs, there are already some examples of medical LLMs that make use of RL. Some brilliant works about medical LLMs like ClinicalGPT (Wang et al., 2023) and Zhongjing (Yang et al., 2024) also utilized RL to reduce the bias of LLM and improve performance. While RL has been extensively harnessed in LLMs and medical LLMs, its potential for enhancing capacity of identifying and describing abnormalities within medical images remains to be explored.

B MAU Dataset

The details of dataset are shown in Table 10. This dataset encompasses five distinct sub-datasets, namely DeepLesion (Yan et al., 2017), Kidney-Stone (TEZ, 2023), NIH (Wang et al., 2017), TBX11K (Liu et al., 2020), and KVASIR (Pogorelov et al., 2017), each originating from diverse sources. DeepLesion consists of 32,120 axial CT slices featuring eight different types of abnormalities such as lesions in the lungs, abdomen, mediastinum, liver, pelvis, soft tissue, kidneys, and bones. The KidneyStone dataset contains 1,300 renal CT scans depicting various kidney stones, covering a range of sizes, shapes, and locations within the urinary system. The NIH dataset includes 112,120 chest X-ray images, covering 14 different pathological categories including, but not limited to, pneumonia, tuberculosis, lung cancer, pulmonary edema, and pulmonary fibrosis. The TBX11K dataset comprises 11,200 chest X-ray images focused on the classification and detection of tuberculosis (TB), categorizing images into healthy, sick but non-TB, active TB, latent TB, and indeterminate TB. The KVASIR dataset contains 8,000 endoscopic images of the gastrointestinal (GI) tract, focusing on GI diseases with eight different types of abnormalities, such as polyps, tumors, or other abnormal tissues. As shown in Figure 7, we present some examples from the MAU dataset.

C Implementation Details

In the experiments, we employ MedViNT (Zhang et al., 2023b) to initialize our model. During the instruction tuning phase, we employed an Adam optimizer (Kingma and Ba, 2015) with a learning rate of \( 1 \times 10^{-5} \), weight decay of 0.01 and a batch size of 128. We train the model on four epochs, utilizing a linear decay strategy for the learning rate. For the AAR method, we set \( \gamma = 0.99, c_1 = 0.5, c_2 = 0.5, \) and \( c_3 = 0.01 \). We used an Adam optimizer with a learning rate of \( 1 \times 10^{-6} \) and a batch size of 16, training the model for one epoch. To enhance the diversity of sampling responses, we set the number of sampling candidates to eight and decoded the model with a temperature of 0.9 and a probability threshold (p) of 0.9. Our training was conducted on an NVIDIA H800 GPU.

D Large Vision-Language Models

In this study, we evaluate a variety of large vision-language models (LVLMs) and medical LVLMs (Med-LVLMs), including general-purpose models such as MiniGPT-4 (Zhu et al., 2024a), mPLUG-Owl (Ye et al., 2023), LLaVA (Liu et al., 2023b), Qwen-VL (Bai et al., 2023), and specialized models like Med-LVLMs, including XrayGPT (Thawakar et al., 2023), LLaVA-Med (Li et al., 2023a), Med-Flamingo (Moor et al., 2023), and MedViNT (Zhang et al., 2023a). While MiniGPT-4, mPLUG-Owl, LLaVA, and Qwen-VL are designed for general visual tasks, LLaVA-Med, XrayGPT, Med-Flamingo, and MedViNT are specif-
<table>
  <tr>
    <th></th>
    <th>UMed-LVLM</th>
    <th>GPT-4V</th>
  </tr>
  <tr>
    <td><img src="page_120_120_320_240.png" alt="Chest X-ray image"></td>
    <td>Based on the provided information, the lesion identified within the bounding box coordinates [bbox_start] 0.364, 0.218, 0.184, 0.144 [bbox_end] is located in the upper lobe of the lung. The chest radiograph reveals signs consistent with pulmonary tuberculosis.</td>
    <td>The analysis of the chest X-ray has identified a large area which has been highlighted with a bounding box. However, the bounding box covers the entire image, which indicates that the thresholding method used may not have been appropriate for identifying specific abnormalities in this X-ray. It’s worth note that accurately localizing abnormalities in medical images typically requires deep medical knowledge and often involves advanced techniques beyond simple thresholding. For a definitive analysis, a radiologist or a trained medical professional would need to review the image. With the method used, I was unable to localize a specific abnormality.</td>
  </tr>
  <tr>
    <th>IoU</th>
    <td>0.46</td>
    <td>0</td>
  </tr>
  <tr>
    <td><img src="page_120_370_320_240.png" alt="Endoscopic image"></td>
    <td>The lesion is enclosed within a bounding box with coordinates [bbox_start] 0.52, 0.374, 0.388, 0.464 [bbox_end]. The image depicts a polyp within the gastrointestinal tract, indicating an abnormal growth. This polyp requires further evaluation by a medical professional to assess its characteristics and determine the appropriate course of treatment.</td>
    <td>The image appears to be a clinical photograph from an endoscopy. There is a prominent growth in the image with the following coordinates: [0.2, 0.3, 0.6, 0.1], which seems to be the focal point of concern. However, as an AI, I’m not able to provide medical advice or diagnoses.<br>For a definitive assessment, the image should be examined by a medical professional who can provide an accurate diagnosis and advice. If this is a personal medical image, please consult with your healthcare provider. If this image is for educational or research purposes, you may want to have it reviewed by a medical expert in the field of gastroenterology.</td>
  </tr>
  <tr>
    <th>IoU</th>
    <td>0.44</td>
    <td>0.20</td>
  </tr>
</table>

Figure 5: The comparison of diagnosis results between UMed-LVLM and GPT-4V on randomly sampled samples.

Q1: What disease can be finding on this medical image?

A1: Based on the provided information, the chest radiograph shows evidence consistent with pulmonary tuberculosis, specifically within the bounding box [bbox_start] 0.578, 0.19, 0.22, 0.257 [bbox_end]. The lesion area is located in the upper lobe of the lung.

Q2: What are the typical symptoms of pulmonary tuberculosis?

A2: Pulmonary tuberculosis typically presents with a persistent cough (lasting over three weeks, possibly producing thick, cloudy, or bloody sputum), chest pain (worsening with deep breathing or coughing), unexplained weight loss, fatigue, low-grade fever (often with chills), night sweats, loss of appetite, and swollen lymph nodes (especially in the neck).

Q3: How is pulmonary tuberculosis diagnosed?

A3: Pulmonary tuberculosis diagnosis involves a combination of reviewing the patient's medical history and performing a physical exam, a chest X-ray to visualize the lungs, sputum tests (microscopic examination and culture) to detect TB bacteria, and a tuberculin skin test (TST) or blood test to determine TB infection.

Q4: Is pulmonary tuberculosis contagious?

A4: Yes, pulmonary tuberculosis is contagious. It spreads through the air when an individual with active TB coughs, sneezes, or speaks, releasing bacteria-containing droplets. Prolonged exposure in poorly ventilated spaces increases transmission risk.

Figure 6: Multi-turn conversation from UMed-LVLM.

<table>
  <tr>
    <th>(c1, c2)</th>
    <th>(0.3, 0.7)</th>
    <th>(0.4, 0.6)</th>
    <th>(0.5, 0.5)</th>
    <th>(0.6, 0.4)</th>
    <th>(0.7, 0.3)</th>
  </tr>
  <tr>
    <th>MAU (Avg.)</th>
    <td>0.69</td>
    <td>0.72</td>
    <td><b>0.75</b></td>
    <td>0.71</td>
    <td>0.70</td>
  </tr>
</table>

Table 11: Hyperparameter search for Equ. 11.

E Hyperparameter Choices for Equ. 11

For hyperparameter in Equ. 11, our reference was the source code of PPO\(^1\). Furthermore, we conducted hyperparameter search experiments to fine-tune these parameters for our specific environment

\(^1\)https://spinningup.openai.com/en/latest/algorithms/ppo.html
Q: What can you deduce from the provided medical image and describe the anomaly’s location?

A: The bounding box of the lesion, marked to delineate the enlarged cardiac silhouette, is defined as follows: [bbox_start] 0.214, 0.381, 0.525, 0.406 [bbox_end]. This area highlights the region within the chest cavity where the heart appears enlarged. Cardiomegaly can have several potential implications, including symptoms such as shortness of breath, fatigue, and swelling of the legs. It may be indicative of underlying conditions such as hypertension, heart valve disease, or cardiomyopathy, necessitating further diagnostic evaluation and appropriate management.

Q: Based on the given information, can you discuss the presence of an anomaly in the medical image and where it is situated?

A: As indicated by the coordinates [bbox_start] 0.493, 0.402, 0.449, 0.553 [bbox_end], the polyp is located within the intestinal lumen. This places the lesion approximately at the center-right of the image. Polyps in the colon can be asymptomatic but may cause symptoms like rectal bleeding, changes in bowel habits, and abdominal pain. The primary concern is that some polyps can develop into colorectal cancer if left untreated. A biopsy is recommended to determine if the polyp is benign or malignant, and regular follow-up colonoscopies may be necessary to monitor for the development of additional polyps.

Q: What can you discern from the provided information about the anomaly within this medical image?

A: Upon reviewing the abdominal CT scan, an abnormal area was identified within the abdomen. The specific region of interest is marked by the following bounding box coordinates:
- **Lesion Area** [bbox_start] 0.469, 0.422, 0.033, 0.052 [bbox_end]
This indicates that the lesion starts at approximately 46.9% from the left edge and 42.2% from the top edge of the image. The lesion spans 3.3% of the image width and 5.2% of the image height.
The presence of a lesion in the abdominal region, as identified in the CT scan, may indicate a variety of conditions ranging from benign cysts to malignant tumors. Further evaluation by a specialist is necessary to determine the exact nature of the anomaly and to assess any related symptoms or health implications. A biopsy or additional imaging may be required for a definitive diagnosis.

Q: Based on the given details, can you discuss the presence of an abnormality in the medical image and provide insights into where it is located?

A: The pleural thickening is located within the region marked as [bbox_start] 0.623, 0.418, 0.127, 0.151 [bbox_end]. This bounding box indicates the specific area within the right hemithorax where the anomaly is present. Pleural thickening can be indicative of chronic inflammation, fibrosis, or pleural disease. The presence of this thickening suggests the possibility of underlying conditions such as chronic infection, exposure to asbestos, or pleural malignancy. Further evaluation with a CT scan is recommended for better characterization and to rule out other potential underlying pathologies like pleural effusion or malignancy.

Q: Can you deduce any information about the anomaly’s location and features within the medical image?

A: The anomaly detected in the CT scan is located in the abdominal region. The specific area of the lesion is identified within the coordinates:
- [bbox_start] 0.41, 0.409, 0.109, 0.125 [bbox_end]
- This bounding box indicates the top-left corner at (0.41, 0.409) and spans a width of 0.109 and a height of 0.125 in the normalized coordinate system of the image.
- The abnormality’s location in the abdomen could potentially relate to various symptoms such as abdominal pain, bloating, or changes in bowel habits. Further evaluation is necessary to determine the exact nature and implications of this anomaly.

Q: What can you infer from the medical image provided and what needs to indicate the anomaly’s location?

A: The bounding box coordinates for the polyp’s location are [bbox_start] 0.201, 0.105, 0.486, 0.665 [bbox_end]. Polyps are abnormal growths of tissue projecting from a mucous membrane, which can sometimes lead to further complications if not addressed.
The presence of a polyp could potentially indicate a risk of it being malignant, which might necessitate a biopsy to determine its nature (benign or malignant). Follow-up endoscopic examinations and consultations with a gastroenterologist are recommended to monitor and possibly remove the polyp to prevent any further complications.

Figure 7: Examples in MAU datasets.
F  Case Study

As shown in Figure 5, we randomly sample examples to compare our method (i.e., UMed-LVLM) and GPT-4V. For instance, in the first medical image, a chest X-ray, the UMed-LVLM identified a suspicious lesion in the upper lobe of the lung. Despite the IoU score being moderate, the system was able to localize the abnormality, which corresponds to clinical findings consistent with pulmonary tuberculosis. In contrast, GPT-4V failed to localize the abnormality, as it deemed the method it used not suitable for pinpointing specific abnormalities. Similarly, in the gastrointestinal tract image, our UMed-LVLM delineated a polyp with an IoU score of 0.44, suggesting the presence of abnormal growth, whereas GPT-4V with IoU of 0.20. These show our model’s proficiency in localizing medical abnormalities. We also showcase examples of our UMed-LVLM’s capability for multi-turn dialogues in Figure 6. It demonstrates its potential for interactive medical consultations and diagnostic support.

Reviewer paper summaries: The paper presents UMed-LVLM, a novel Medical LVLM designed to improve abnormality detection and interpretation in medical imaging. They introduce the MAU dataset and a two-stage training approach.

This submission aims to improve the performance and visual understanding of medical LVLMs by enhancing alignment with visual details and abnormalities. Motivated by the need for better visual grounding, the authors construct a new instruction-tuning dataset from existing datasets with bounding boxes. They further propose a training pipeline designed to guide the model to focus on abnormal regions. Experimental results demonstrate the effectiveness of the proposed approach.
Summary of strengths: - The MAU dataset is a significant contribution, integrating medical images with abnormality annotations generated through a structured GPT-4V pipeline.
- The introduction of Abnormal-Aware Rewarding and Instruction Tuning is compelling, effectively improving the model's attention to abnormal regions and its diagnostic accuracy. I like this RL-based idea.
- The language is clearly presented. The authors use precise and concise language so that the reader can easily understand the background, methodology, and results of the study.

- The submission introduces a reward-based training design that encourages collaborative visual grounding for abnormality diagnosis.
- A visual-grounded training pipeline is proposed and validated through experiments on the MedVInt model, showing its practical value.
- The authors repurpose existing datasets with bounding boxes into a new instruction-tuning format, which can benefit other medical LVLM research efforts.
Summary of weaknesses: - I have only one question on the diversity of MAU dataset. I do not see any analysis of the diversity within the MAU dataset.
- Other questions in the review of last round are addressed.

- Lack of grounding evaluation: Although the model is trained to generate bounding boxes (ALR) and align attention to visual regions during diagnosis (as in Eq. 9, VRR design), there is no explicit evaluation of its visual grounding ability: (1) I only notice 2 examples in appendix and there are no quantitative results evaluating the generated bounding boxes, even though some datasets (e.g., your proposed MAU, SLAKE) might include ground truth annotations that could be used for this. (2) No qualitative or interpretability analysis is provided (e.g., visualizations of attention maps) to demonstrate the model’s alignment with abnormalities during the generation process.
- Inconsistent baselines across datasets: The choice of baselines varies from one dataset to another, making it harder to draw consistent conclusions. For example: LVLMs compared in Tables 2 and 4 are not included in experiments for datasets like VQA-RAD and SLAKE (Tables 3 and 5). In addition, stronger baselines such as HuatuoGPT-Vision and CheXagent are missing and can be considered for a more comprehensive evaluation.
- Some concerns with experimental settings:
  1. In Section 5.4, under *“Generalization Capability for Untrained Category,”* the authors remove the "lung" category from DeepLesion but keep datasets like NIH that contain chest X-ray images, including lungs. This weakens the claim of evaluating generalization to unseen categories (although they are different radiology image types).
  2. Similarly, under *“Generalization Capability of Abnormality Unveiling,”* although TBX11K is removed to simulate an unseen abnormality, the NIH dataset remains, which includes many chest X-rays and overlaps in modality and pathology. This raises concerns about the validity of the generalization claim.
Comments, suggestions and typos: N/A

(1) Typo at line 443.

(2) Some questions that need clarification:
- On datasets like VQA-RAD and SLAKE, did you fine-tune the model on each dataset before evaluation, or were the results based on zero-shot inference?
- For tasks such as those in table 2, how is "accuracy" computed? Is it based on rule-based or semantic matching? Those diagnosis tasks are likely in an open-ended style, are here any better metrics?
- Some medical LVLMs like LLaVA-Med perform poorly on NIH and TBX despite being trained on chest X-rays. Could you explain the cause of this performance gap? (maybe the problem with metric)
- It would be valuable to show how your proposed training pipeline transfers to other existing LVLMs, such as LLaVA-Med, to demonstrate broader applicability.
Overall assessment scores: 3.5

3
Metareview score: 3.5
Metareview: This paper proposes UMed-LVLM, a novel medical large vision-language model that improves abnormality localization and diagnosis through an Abnormal-Aware Feedback training pipeline. The authors contribute (1) a new benchmark dataset MAU, created by combining and reformatting several medical imaging datasets using GPT-4V, and (2) a two-stage training method involving Instruction Tuning and Reward-based Optimization with novel reward signals (Relevance Reward, Abnormal Localization Reward, and Vision Relevance Reward). The paper targets the challenge of aligning medical LVLMs with clinically relevant visual cues. The authors demonstrate that their model significantly improves performance on multiple datasets across different medical imaging modalities and settings, including generalization to unseen abnormalities and categories.
Summary of reasons to publish: * The MAU dataset is a valuable contribution, covering diverse modalities (CT, X-ray, endoscopy) and pathology types across five datasets. The diversity is well-analyzed and relevant to practical deployment scenarios.

* The reward-based training integrates visual and diagnostic alignment, representing a meaningful advance over existing SFT-only approaches. The proposed rewards address both language-vision relevance and visual localization.

* The model is evaluated across seven datasets and compared against strong baselines, including HuatuoGPT-Vision and CheXagent, with added results for LLaVA-Med-1.5 during rebuttal.

* The authors addressed most reviewer concerns with clarity and additional experiments (e.g., new baseline results, fine-tuning settings, visual grounding clarification, performance across training epochs).