Focus on What Matters: Enhancing Medical Vision-Language Models with Automatic Attention Alignment Tuning
Authors: Aofei Chang, Le Huang, Alex James Boyd, Parminder Bhatia, Taha Kass-Hout, Cao Xiao, Fenglong Ma
Accepted at: acl25

Focus on What Matters: Enhancing Medical Vision-Language Models with Automatic Attention Alignment Tuning

Aofei Chang1, Le Huang2, Alex James Boyd2, Parminder Bhatia2, Taha Kass-Hout2, Cao Xiao2*, Fenglong Ma1*
1Pennsylvania State University, 2GE Healthcare,
1{aofei, fenglong}@psu.edu,
2{Lena.Huang, Alex.Boyd, Parminder.Bhatia, Taha.Kass-Hout, Cao.Xiao}@gehealthcare.com

Abstract
Medical Large Vision-Language Models (Med-LVLMs) often exhibit suboptimal attention distribution on visual inputs, leading to hallucinated or inaccurate outputs. Existing mitigation methods primarily rely on inference-time interventions, which are limited in attention adaptation or require additional supervision. To address this, we propose A3TUNE, a novel fine-tuning framework for Automatic Attention Alignment Tuning. A3TUNE leverages zero-shot weak labels from SAM, refines them into prompt-aware labels using BiomedCLIP, and then selectively modifies visually-critical attention heads to improve alignment while minimizing interference. Additionally, we introduce a A3MOE module, enabling adaptive parameter selection for attention tuning across diverse prompts and images. Extensive experiments on medical VQA and report generation benchmarks show that A3TUNE outperforms state-of-the-art baselines, achieving enhanced attention distributions and performance.1

1 Introduction
While medical Large Vision-Language Models (Med-LVLMs) have shown significant progress in the medical domain (Li et al., 2024; Chen et al., 2024d; Thawkar et al., 2023; Moor et al., 2023), they often produce inaccurate or hallucinated outputs that deviate from the provided visual medical information, as revealed by recent benchmarks (Xia et al., 2024; Gu et al., 2024; Chen et al., 2024a; Chang et al., 2025). An example of medical visual question answering (VQA) extracted from the SLAKE (Liu et al., 2021) dataset is shown in Figure 1(A), where we visualize the average attention map on the image inputs during generating answers. We can observe that LLaVA-Med in Figure 1(A.2) generates the hallucinated response “Alzheimer’s disease”, neglecting the tumor region and over-focusing on irrelevant background areas, as shown in the corresponding attention map. This reveals a significant bias in attention distribution on visual inputs that limits the model’s effectiveness, which have been identified in general LVLMs (Gong et al., 2024; Woo et al., 2024; Liu et al., 2025).

Unfortunately, none of the specified bias mitigation strategies have been proposed in the medical domain. In the general domain, research primarily focuses on inference-time interventions to reduce attention biases, employing two main approaches. The first approach, contrastive decoding (Leng et al., 2024; Favero et al., 2024; Liu et al., 2025; Woo et al., 2024; Gong et al., 2024), introduces a contrastive adjustment to the decoding logits. However, this method does not directly modify the attention distribution, meaning it cannot guarantee that the model attends to diagnostically critical regions. The second approach directly modifies attention maps during inference, as seen in ControlMLLM (Wu et al., 2024), which enforces the model to focus on pre-annotated regions of interest (RoIs). However, as shown in Figure1(A.3), this method still generates partially hallucinated content. Besides, it requires additional tuning and ground truth RoIs for each inference process, making it impractical for real-world applications.

To overcome the limitations of inference-time intervention, an ideal solution is to automatically adjust attention maps towards RoIs during fine-tuning for downstream medical tasks. This approach enables the model to place more attention on critical regions during inference, eliminating the need for additional labels or interventions. However, implementing such a method poses several challenges:

(1) Limited availability of medical segmentation labels. As shown in Figure 1(A.3), using segmentation labels (RoIs) as guidance, as done in ControlMLLM, can enhance the learning of accurate attention maps and answer correctness. How-
<table>
  <tr>
    <th>Prompt</th>
    <th>Ground Truth Segments</th>
  </tr>
  <tr>
    <td>(1) Does the picture contain liver?</td>
    <td></td>
  </tr>
  <tr>
    <td>(2) Where is the spleen located?</td>
    <td></td>
  </tr>
</table>

Figure 1: (A) Examples of medical VQA and attention maps on medical images. In this example of Brain MRI from the SLAKE dataset, red box denotes the RoI of the brain tumor that LLaVA-Med should focus on. Red texts and green texts indicate wrong answers and correct answers, respectively. (B) Example of ground truth RoIs for different prompts on an Abdomen CT from SLAKE.

ever, such labels are often unavailable in medical datasets.

(2) Trade-off between attention alignment and model stability. Assuming that RoI labels are available, directly modifying the attention maps of all attention heads towards the labels without any strategy is still risky. This may lead to the over-alignment issue that potentially impacts the output stability and overall performance. Therefore, achieving the right balance between attention alignment and model stability is essential.

(3) Adapting attention to diverse prompts and images. Even if attention alignment and model stability are balanced, the parameter-sharing strategy in fine-tuning remains a limitation for adaptive attention alignment. For example, as shown in Figure 1, the optimal RoIs can vary significantly based on the input prompt and image, requiring dynamic alignment. While the computation of attention maps adjusts to input representations, the shared parameters in fine-tuning limit the model’s ability to flexibly learn attention distribution across diverse inputs.

To address these challenges simultaneously, we propose A³TUNE, a novel fine-tuning framework designed for Automatic Attention Alignment Tuning. As shown in Figure 2, A³TUNE integrates a set of cooperative strategies to ensure that attention in Med-LVLMs is well aligned, minimally disruptive, and highly adaptable across diverse medical tasks. Firstly, to overcome the lack of segmentation labels (Challenge 1), A³TUNE utilizes zero-shot segmentation labels generated by SAM (Roy et al., 2023) and further refines them into prompt-aware weak labels using BioMedCLIP (Zhang et al., 2023). These weak labels serve as guidance for attention alignment, eliminating the need for manual annotations. However, weak labels alone are not sufficient — uncontrolled modifications on all attention heads can disrupt model stability (Challenge 2). To further balance attention alignment with model stability, A³TUNE selectively modifies only the most “visually-critical” attention heads, minimizing the risk of over-alignment and instability. Furthermore, the parameter-sharing strategy in alignment tuning remains a limitation, as RoIs vary significantly based on the input prompt and image (Challenge 3). To address this, we incorporate a custom-designed Mixture-of-Experts A³MoE into A³TUNE on attention modules, allowing the model to dynamically select parameters and adjust attention maps for different images and prompts.

In summary, this work makes the following contributions: (1) We propose A³TUNE, a novel visual attention tuning approach that utilizes zero-shot weak labels to refine the visual focus of Med-LVLMs and enhance their performance. (2) We develop a set of designs for cooperative attention alignment tuning: (i) weak label supervision, (ii) selective tuning of visually-critical attention heads, and (iii) a A³MoE module for adaptive attention adjustment. (3) We conduct extensive experiments on five medical VQA and two report generation benchmarks against ten baselines, demonstrating that A³TUNE outperforms state-of-the-art methods in both effectiveness and interpretability, improving visual grounding and overall model performance.

2 Related Work

While some efforts, such as CoMT (Jiang et al., 2024), have attempted to reduce hallucinations in Med-LVLMs for report generation by training on hierarchical QA pairs derived from real clinical
image reports, mitigation strategies specifically designed for Med-LVLMs remain largely underexplored. Since Med-LVLMs share the same structure and training process as general LVLMs, hallucination issues are a common challenge across both. As a result, many inference-time mitigation strategies developed for LVLMs are also applicable to Med-LVLMs, including: (1) Enhancing visual information and mitigating text over-reliance: VCD (Leng et al., 2024), M3ID (Favero et al., 2024), PAI (Liu et al., 2025), HELPD (Yuan et al., 2024), RBD (Liang et al., 2024). (2) Mitigating visual token attention bias, with approaches such as AVISC (Woo et al., 2024) and DAMRO (Gong et al., 2024). (3) Refining and modifying LLM generation patterns during decoding, OPERA (Huang et al., 2024), DoLa (Chuang et al., 2023). In addition, other mitigation methods, such as Truth-Flow (Wang et al., 2025), leverage trained steering vectors to guide the model toward more truthful outputs in LLMs. Despite these advancements, significant challenges remain in mitigation attention biases in Med-LVLMs.

3 Preliminaries

3.1 Background of Med-LVLMs

Med-LVLMs share the same fundamental architecture as general LVLMs (Liu et al., 2024; Chen et al., 2023), consisting of three primary components: the Visual Encoder, the Visual Alignment Layer, and the Large Language Model (LLM). The input image is firstly divided into \( N \) patches and processed by the Visual Encoder, which converts it into a sequence of visual tokens \( \mathbf{T}_v \) with embeddings represented as \( \mathbf{X}_v \in \mathbb{R}^{N \times d_v} \), where \( d_v \) denotes the dimension of the visual hidden representation.

The visual tokens are then projected through the Visual Alignment Layer into \( \mathbf{X}'_v \in \mathbb{R}^{N \times d_p} \), where \( d_p \) denotes the dimension of the LLM token space. These aligned visual tokens are subsequently forwarded to the LLM along with the embeddings \( \mathbf{X}_p \in \mathbb{R}^{V \times d_p} \) of the tokenized textual prompts \( \mathcal{T}_p \), where \( V \) is the number of text tokens.

The final objective of the LLM in Med-LVLMs is to predict the next token \( y_t \) based on the current visual input \( \mathbf{X}'_v \), prompt input \( \mathbf{X}_p \), and previously generated tokens \( y_{<t} \), formulated as:
\[
p_t = p(y_t \mid \mathbf{X}'_v, \mathbf{X}_p, y_{<t}; \Theta^*),
\]
where \( \Theta^* \) denotes the parameters of the LLM.

3.2 Visual Attention Map

Most Med-LVLMs adopt a Transformer (Vaswani et al., 2017) decoder-based LLM, which processes inputs through \( L \) decoder layers, each equipped with a multi-head attention module. In layer \( l \), the module includes \( H \) attention heads, where each head \( h \) (with \( 1 \leq h \leq H \)) computes attention separately using their corresponding attention map \( \mathbf{M}_{lh} \). This attention mechanism models relationships between visual and textual tokens.

To analyze how visual tokens contribute to text generation, we focus on the attention scores between visual tokens and subsequent text tokens, referred to as visual attention map \( \mathbf{M}_{lh}^v \), a submatrix of the overall attention map \( \mathbf{M}_{lh} \). To gain a global understanding of attention distribution on visual tokens, the averaged visual attention map \( \mathbf{M}^v \) can be obtained by aggregating attention across all heads and layers (Wu et al., 2024):
\[
\mathbf{M}^v = \frac{1}{LH} \sum_{l=1}^L \sum_{h=1}^H \mathbf{M}_{lh}^v.
\]

![Overview of the Proposed A^3Tune and details of the designed visual attention alignment tuning](page_98_120_1292_627.png)

Figure 2: (A) The overview of A^3Tune and (B) the details of the designed visual attention alignment tuning.
4 The Proposed A³TUNE

A³TUNE aims to automatically align attention to enhance visual grounding and improve the performance of Med-LVLMs. The fine-tuning pipeline is illustrated in Figure 2. Given an input image \( I \) and prompt \( P \), we firstly generate a set of prompt-aware weak labels \( S \) using a zero-shot method (Section 4.1). To achieve effective attention alignment (Section 4.2) with the guidance of \( S \), we first identify visually-critical attention heads, then integrate an A³MOE design into each decoder layer to enable flexible attention distribution learning.

4.1 Prompt-Aware Weak Labels Generation

Given a medical image \( I \), we first use SAM (Kirillov et al., 2023) to generate a set of candidate segments \( S^* \) following (Yang et al., 2024). However, using all segments in \( S^* \) for attention tuning introduces noise, as only a subset is relevant to each prompt. Therefore, it is necessary to select prompt-aware segments as weak labels to adaptively guide attention tuning. To filter prompt-aware weak labels, we embed each segment \( s \in S^* \) into a feature representation \( \mathbf{E}_s \) using BioMedCLIP’s vision encoder, while the text prompt \( P \) is embedded into \( \mathbf{E}_P \) using its text encoder. We then select an adaptive threshold \( \tau_K \) to select \( K \) segments that are most similar to the text prompt based on cosine similarity (Sim) in the embedding space:

\[
S = \{ s \in S^* \mid \operatorname{Sim}(\mathbf{E}_s, \mathbf{E}_P) \geq \tau_K \}.
\]

(3)

4.2 Visual Attention Alignment Tuning

The goal of visual attention alignment tuning is to align averaged visual attention map \( M^e \) with fixed weak labels \( S \) (i.e., masking regions) via Eq. (3) during fine-tuning. To achieve this goal, we first design a new parameter-efficient fine-tuning strategy and select “visually-critical” attention heads to the alignment of attention maps with \( S \).

4.2.1 A³MOE Design

We build A³TUNE based on a parameter-efficient fine-tuning technique, LoRA (Hu et al., 2021), and apply it to all linear modules of the LLM in the Med-LVLM, including the Query and Key matrices in attention modules. In standard LoRA fine-tuning, the trainable LoRA parameters \( \Delta \mathbf{W}_q \) and \( \Delta \mathbf{W}_k \) for Query and Key matrices are shared across all training instances. However, this static parameter-sharing strategy in attention is insufficient for A³TUNE. As shown in the abdominal organ analysis tasks (Figure 3), weak labels \( S \) vary with the prompt, yet attention maps generated with shared \( \Delta \mathbf{W}_q \) and \( \Delta \mathbf{W}_k \) lack the flexibility to adapt effectively to different prompts and images.

To address this limitation, we introduce A³MOE, a Mixture-of-Experts mechanism specifically designed for A³TUNE, with two sub-modules: Q-MoE and K-MoE, applied to the LoRA parameters \( \Delta \mathbf{W}_q^{(l)} \) and \( \Delta \mathbf{W}_k^{(l)} \) in each LLM decoder layer \( l \). For clarity, we omit the explicit annotation of \( l \) in subsequent equations, as A³MOE is applied consistently across all decoder layers. The following sections describe these sub-modules in detail.

Q-MoE: Prompt-Level MoE on Query Matrix.
As shown in Figure 3, the target RoIs often depend on the text prompt, even for the same image. To handle this, we introduce Q-MoE, a prompt-level MoE on \( \Delta \mathbf{W}_q \), consisting of \( O^q \) experts. For the \( o \)-th expert (\( 1 \leq o \leq O^q \)), we define \( \mathcal{E}_o^q = \mathbf{B}_o \mathbf{A}_o \), where \( \mathbf{B}_o \in \mathbb{R}^{d_p \times r} \) and \( \mathbf{A}_o \in \mathbb{R}^{r \times d_p} \) are matrices with a low rank \( r \).

To dynamically route experts based on the prompt, we apply a prompt-level gating mechanism that generates router weights \( \alpha \) based on the hidden states of the prompt \( \mathbf{H}_p \), which is obtained at each layer \( l \) after processing \( \mathbf{X}_p \) from the previous decoder layers. To capture the prompt’s overall context, \( \mathbf{H}_p \) is first averaged using a pooling operation. The routing and parameter computation are formulated as:

\[
\alpha = \operatorname{softmax}(\operatorname{MLP}(\operatorname{Pooling}(\mathbf{H}_p))),
\]
\[
\Delta \mathbf{W}_q = \sum_{o=1}^{O^q} \alpha_o \mathcal{E}_o^q,
\]

where \( \alpha \in \mathbb{R}^{O^q} \) represents the router weights, with each \( \alpha_o \) determining the contribution of the

<table>
  <tr>
    <th>Prompt</th>
    <th>Prompt-aware weak labels</th>
    <th>Attention map learned using shared parameters</th>
  </tr>
  <tr>
    <td>Does the picture contain liver?</td>
    <td>![Prompt-aware weak labels: red bounding boxes and green inner segments](page_682_153_277_181.png)</td>
    <td>![Attention map learned using shared parameters: blue heatmaps](page_962_153_277_181.png)</td>
  </tr>
  <tr>
    <td>Where is the spleen located?</td>
    <td>![Prompt-aware weak labels: red bounding boxes and green inner segments](page_682_340_277_181.png)</td>
    <td>![Attention map learned using shared parameters: blue heatmaps](page_962_340_277_181.png)</td>
  </tr>
</table>

Figure 3: Motivation for using A³MOE. The second column shows prompt-aware weak labels, with red bounding boxes and green inner segments. The third column shows the attention maps generated using shared parameters for the Query and Key matrices.
corresponding expert \( \mathcal{E}_o^k \). MLP denotes the multi-layer perceptron used in the gating mechanism.

**K-MoE: Visual Token-Level Sparse MoE in Key Matrix.** Unlike text prompts, which can be summarized with pooling, visual inputs contain fine-grained information at the token level, where subtle differences can indicate abnormalities that require varied attention. To capture these nuances, we introduce K-MoE, a visual token-level MoE applied on \( \Delta \mathbf{W}_k \). K-MoE includes \( O^k \) experts, each implemented using LoRA.

For each visual token \( c \), the gating mechanism dynamically select experts based on its hidden states \( \mathbf{H}_c^v \), and the LoRA parameters \( \Delta \mathbf{W}_k^c \) for visual token \( c \) are computed as:

\[
\beta^c = \operatorname{softmax}(\operatorname{MLP}(\mathbf{H}_c^v)), \tag{6}
\]
\[
\Delta \mathbf{W}_k^c = \sum_{o=1}^{O_k} \mathbb{1}_o \beta_o^c \mathcal{E}_o^k, \tag{7}
\]

where \( \beta^c \in \mathbb{R}^{O^k} \) and \( \mathbb{1}_o \) is a binary indicator enforcing sparsity by retaining only the top-\( B \) gating weights. It is set to 1 if expert \( \mathcal{E}_o^k \) is among the top-\( B \) most relevant experts, otherwise it is set to 0. This mechanism ensures that for each visual token, only the most relevant experts contribute, minimizing interference among visual tokens while improving efficiency in alignment tuning.

**4.2.2 “Visually-Critical” Heads Selection**
Each layer \( l \) in the LLM decoder fine-tuned with A\( ^3 \)MOE learns \( H \) attention heads and the total number of attention heads is \( L \times H \). Among them, the attention heads that assign higher weights to visual information are more crucial in processing visual features. Thus in layer \( l \), the importance of each head \( h \) can be quantified using the visual attention ratio \( r_{lh} \), which measures the proportion of attention allocated to visual tokens relative to all tokens as follows:

\[
r_{lh} = \frac{\sum_{c \in \mathcal{T}_v} \mathbf{M}_{lh}^v[c]}{\sum_{c' \in \mathcal{T}_v \cup \mathcal{T}_p} \mathbf{M}_{lh}^v[c']}, \tag{8}
\]

where \( \mathbf{M}_{lh}^v[c] \) represents the attention score assigned by attention head \( h \) in layer \( l \) to token \( c \).

To refine the visual attention matrix \( \widetilde{\mathbf{M}}^v \), we select top-\( R \) “visually-critical” heads based on the visual attention ratios \( r_{lh} \). To achieve this, we use a binary indicator \( \mathbb{1}_{lh} \) to identify and retain only the most influential attention heads. Specifically, it is set to 1 if attention head \( (l, h) \) is among the \( R \) heads with the highest visual attention ratios and 0 otherwise. The updated visual attention matrix is then computed as:

\[
\widetilde{\mathbf{M}}^v = \frac{1}{R} \sum_{l=1}^L \sum_{h=1}^H \mathbb{1}_{lh} \mathbf{M}_{lh}^v. \tag{9}
\]

**4.2.3 Attention Alignment Function**
After obtaining refined \( \widetilde{\mathbf{M}}^v \), we define how it is adjusted to align with the weak labels \( \mathcal{S} \). Intuitively, in the averaged attention map \( \bar{\mathbf{M}}^v \), tokens within the masking regions \( \mathcal{S} \) should receive higher attention values. Motivated by the cross-attention control design for image generation in (Chen et al., 2024c), we use a mask-based energy function to guide the attention alignment tuning:

\[
\mathcal{L}_{\text{align}} = \sum_{s \in \mathcal{S}} \left( 1 - \frac{\sum_{c \in s} \widetilde{\mathbf{M}}_c^v}{\sum_{c'=1}^N \widetilde{\mathbf{M}}_{c'}^v} \right)^2, \tag{10}
\]

where \( s \) represents each referring segment in \( \mathcal{S} \), \( c \) and \( c' \) is the visual token index, and \( N \) is the number of visual tokens. Among all \( N \) visual tokens, this function encourages higher attention on visual tokens within each \( s \), minimizing the loss and guiding \( \widetilde{\mathbf{M}}^v \) to effectively focus on the prompt-aware regions \( \mathcal{S} \).

**4.3 Final Objective**
In the fine-tuning process, we incorporate the loss of proposed visual attention alignment tuning \( \mathcal{L}_{\text{align}} \) (Eq. 10) as a regularization term for downstream medical tasks. This term is combined with the language modeling objective \( \mathcal{L}_{LLM} \):

\[
\mathcal{L}_{LLM} = - \sum_{t=1}^T \log p_t(y_t \mid \mathbf{X}_v', \mathbf{X}_p, y_{<t}; \Theta, \Theta^*), \tag{11}
\]

where \( \Theta^* \) denotes the frozen parameters of LLM, \( \Theta \) denotes the trainable parameters in LoRA and A\( ^3 \)MOE, and \( y_{<t} \) denotes the generated tokens before time step \( t \). The final objective loss is:

\[
\mathcal{L} = \mathcal{L}_{LLM} + \lambda \mathcal{L}_{\text{align}}, \tag{12}
\]

where \( \lambda > 0 \) is a hyperparameter that controls the strength of attention tuning.

**5 Experiments**
We evaluate our method on representative Med-LVLMs, including LLaVA-Med (Li et al., 2024) and LLaVA-Med-1.5. The experimental results for LLaVA-Med-1.5 are provided in Appendix G.
Table 1: Performance comparison on medical VQA benchmarks using LLaVA-Med. The best results are highlighted in bold, and the second-best results are underlined. OmniVQA corresponds to the OmniMedVQA dataset.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Method</th>
    <th colspan="2">SLAKE</th>
    <th colspan="2">VQA-RAD</th>
    <th colspan="2">PathVQA</th>
    <th colspan="2">IU-Xray</th>
    <th colspan="2">OmniVQA</th>
  </tr>
  <tr>
    <th>Open</th>
    <th>Closed</th>
    <th>Open</th>
    <th>Closed</th>
    <th>Open</th>
    <th>Closed</th>
    <th>Open</th>
    <th>Closed</th>
    <th>Open</th>
    <th>Closed</th>
  </tr>
  <tr>
    <td rowspan="8">LLaVA-Med</td>
    <td>LLaVA-Med</td>
    <td>41.28</td>
    <td>57.75</td>
    <td>32.48</td>
    <td>68.90</td>
    <td>10.34</td>
    <td>52.49</td>
    <td>72.83</td>
    <td>31.79</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Greedy</td>
    <td>42.81</td>
    <td>60.00</td>
    <td>35.68</td>
    <td>68.11</td>
    <td>11.35</td>
    <td>52.34</td>
    <td>75.00</td>
    <td>31.11</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Beam (Sutskever et al., 2014)</td>
    <td>42.31</td>
    <td>61.69</td>
    <td>33.57</td>
    <td>66.93</td>
    <td>9.86</td>
    <td>53.79</td>
    <td>73.59</td>
    <td>30.89</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Nucleus (Holtzman et al., 2020)</td>
    <td>41.06</td>
    <td>61.41</td>
    <td>32.64</td>
    <td>68.11</td>
    <td>9.93</td>
    <td>53.17</td>
    <td>73.09</td>
    <td>30.96</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>VCD (Leng et al., 2024)</td>
    <td>39.76</td>
    <td>59.44</td>
    <td>33.93</td>
    <td>66.54</td>
    <td>11.17</td>
    <td>53.52</td>
    <td>74.36</td>
    <td>32.89</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>DoLa (Chuang et al., 2023)</td>
    <td>42.37</td>
    <td>59.72</td>
    <td>35.68</td>
    <td>68.50</td>
    <td>11.38</td>
    <td>52.55</td>
    <td>74.87</td>
    <td>31.04</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>OPERA (Huang et al., 2024)</td>
    <td>40.44</td>
    <td>60.00</td>
    <td>35.54</td>
    <td>68.50</td>
    <td>9.77</td>
    <td>53.17</td>
    <td>75.00</td>
    <td>31.72</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>AVISC (Woo et al., 2024)</td>
    <td>41.46</td>
    <td>59.72</td>
    <td>35.43</td>
    <td>64.17</td>
    <td>11.03</td>
    <td>52.20</td>
    <td>73.85</td>
    <td>32.48</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>M3ID (Favero et al., 2024)</td>
    <td>38.85</td>
    <td>60.28</td>
    <td>35.31</td>
    <td>62.60</td>
    <td>9.80</td>
    <td>52.93</td>
    <td>72.32</td>
    <td>31.11</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>DAMRO (Gong et al., 2024)</td>
    <td>41.33</td>
    <td>59.72</td>
    <td>32.73</td>
    <td>66.54</td>
    <td>10.80</td>
    <td>51.84</td>
    <td>72.19</td>
    <td>31.45</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>PAI (Liu et al., 2025)</td>
    <td>43.18</td>
    <td>60.28</td>
    <td>35.10</td>
    <td>68.50</td>
    <td>11.09</td>
    <td>52.46</td>
    <td>74.87</td>
    <td>32.13</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="8">LLaVA-Med + LoRA</td>
    <td>LLaVA-Med + LoRA</td>
    <td>80.65</td>
    <td>82.82</td>
    <td>33.37</td>
    <td>66.54</td>
    <td>31.92</td>
    <td>90.95</td>
    <td>83.29</td>
    <td>90.65</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Greedy</td>
    <td>81.12</td>
    <td>85.07</td>
    <td>31.88</td>
    <td>68.50</td>
    <td>33.90</td>
    <td>91.86</td>
    <td>83.93</td>
    <td>90.73</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Beam (Sutskever et al., 2014)</td>
    <td>81.32</td>
    <td><b>86.76</b></td>
    <td>32.55</td>
    <td>68.90</td>
    <td>33.60</td>
    <td>91.92</td>
    <td>84.06</td>
    <td>90.58</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Nucleus (Holtzman et al., 2020)</td>
    <td>80.18</td>
    <td>83.35</td>
    <td>31.34</td>
    <td>68.50</td>
    <td>30.14</td>
    <td>90.92</td>
    <td>83.41</td>
    <td>90.05</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>VCD (Leng et al., 2024)</td>
    <td>79.58</td>
    <td>84.23</td>
    <td>32.96</td>
    <td>67.72</td>
    <td>30.20</td>
    <td>90.86</td>
    <td>83.93</td>
    <td>90.73</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>DoLa (Chuang et al., 2023)</td>
    <td>81.84</td>
    <td>86.48</td>
    <td>31.94</td>
    <td>68.50</td>
    <td>34.00</td>
    <td>91.86</td>
    <td>84.06</td>
    <td>90.69</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>OPERA (Huang et al., 2024)</td>
    <td>81.25</td>
    <td>86.48</td>
    <td>33.18</td>
    <td>68.90</td>
    <td>33.64</td>
    <td>91.80</td>
    <td>84.06</td>
    <td>90.42</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>AVISC (Woo et al., 2024)</td>
    <td>80.15</td>
    <td>85.63</td>
    <td>33.66</td>
    <td>69.29</td>
    <td>32.37</td>
    <td>90.62</td>
    <td>84.06</td>
    <td>90.54</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>M3ID (Favero et al., 2024)</td>
    <td>79.83</td>
    <td>84.79</td>
    <td>31.40</td>
    <td>68.90</td>
    <td>32.47</td>
    <td>91.15</td>
    <td>84.95</td>
    <td>90.73</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>DAMRO (Gong et al., 2024)</td>
    <td>82.19</td>
    <td>83.66</td>
    <td>32.41</td>
    <td>66.14</td>
    <td>32.27</td>
    <td>90.12</td>
    <td>84.69</td>
    <td>90.08</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>PAI (Liu et al., 2025)</td>
    <td>81.02</td>
    <td><b>86.76</b></td>
    <td>32.14</td>
    <td>68.11</td>
    <td>33.46</td>
    <td>91.77</td>
    <td>84.31</td>
    <td>90.95</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>A³TUNE (ours)</td>
    <td><b>82.36</b></td>
    <td><b>86.76</b></td>
    <td><b>36.97</b></td>
    <td><b>70.87</b></td>
    <td><b>34.61</b></td>
    <td><b>92.19</b></td>
    <td><b>85.97</b></td>
    <td><b>91.98</b></td>
    <td></td>
    <td></td>
  </tr>
</table>

5.1 Settings

Datasets. We evaluate our method on two key tasks in medical application of Med-LVLMs: medical VQA and medical report generation. For medical VQA, we use diverse datasets including SLAKE (Liu et al., 2021), VQA-RAD (Lau et al., 2018), PathVQA (He et al., 2020), IU-Xray (Demner-Fushman et al., 2016) and OmniMedVQA (Hu et al., 2024). For medical report generation, we use MIMIC-CXR (Johnson et al., 2019) and IU-Xray. Details of dataset processing and settings are provided in Appendix A.

Baselines. We compare our approach with widely used hallucination mitigation methods with or without fine-tuning, including decoding strategies and contrastive decoding techniques2. The decoding baselines include Greedy decoding, Nucleus sampling (Holtzman et al., 2020), Beam search (Sutskever et al., 2014). For contrastive decoding techniques, we specifically compare with: VCD (Leng et al., 2024), OPERA (Huang et al., 2024), DoLa (Chuang et al., 2023), AVISC (Woo et al., 2024), M3ID (Favero et al., 2024), DAMRO (Gong et al., 2024) and PAI (Liu et al., 2025). Additionally, we include ControlMLLM (Wu et al., 2024) as a baseline only when the ground truth RoIs are available. Detailed settings of baselines and implementation are in Appendix B and Appendix C, respectively.

Metrics. (1) Metrics for Performance Evaluation. For medical VQA, we report Accuracy for close-ended questions and Recall for open-ended questions, following LLaVA-Med (Li et al., 2024). For medical report generation, we use standard metrics for generation tasks, including BLEU (Papineni et al., 2002), ROUGE-L (Lin, 2004), METEOR (Banerjee and Lavie, 2005), and BERTScore (Zhang et al., 2019). Additionally, we report domain-specific metrics designed for medical report generation: CheXbert (Smit et al., 2020), RadGraph (Jain et al., 2021) and RaTEScore (Zhao et al., 2024). More details of these metrics are provided in Appendix D.1. (2) Metrics for Attention Maps Evaluation. For datasets with ground truth RoIs (e.g., SLAKE), we design two metrics to evaluate the attention distribution on images: (1) Coverage Score (Coverage), which measures spatial alignment, and (2) Intensity Alignment (Intensity), which assesses the degree of focus. Details of these two metrics are in Appendix D.2.

Table 2: Comparison of Visual Attention Distribution.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Metrics</th>
  </tr>
  <tr>
    <th>Coverage ↑</th>
    <th>Intensity ↑</th>
  </tr>
  <tr>
    <td>LLaVA-Med</td>
    <td>0.122</td>
    <td>0.076</td>
  </tr>
  <tr>
    <td>LLaVA-Med + LoRA</td>
    <td>0.132</td>
    <td>0.076</td>
  </tr>
  <tr>
    <td>A³TUNE</td>
    <td><b>0.275</b></td>
    <td><b>0.147</b></td>
  </tr>
</table>

2We do not compare with the methods requiring tailored de-hallucination training pipelines such as CoMT (Jiang et al., 2024) and HELPD (Yuan et al., 2024).
Table 3: Performance on report generation benchmarks using LLaVA-Med fine-tuned with LoRA.

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Metric</th>
    <th colspan="12">Method</th>
  </tr>
  <tr>
    <th>LLaVA-Med + LoRA</th>
    <th>Greedy</th>
    <th>Beam</th>
    <th>Nucleus</th>
    <th>VCD</th>
    <th>DoLa</th>
    <th>OPERA</th>
    <th>AVISC</th>
    <th>M3ID</th>
    <th>DAMRO</th>
    <th>PAI</th>
    <th>A³TUNE</th>
  </tr>
  <tr>
    <td rowspan="7">IU-Xray</td>
    <td>BLEU</td>
    <td>7.70</td>
    <td>8.86</td>
    <td>9.34</td>
    <td>7.80</td>
    <td>8.83</td>
    <td>8.93</td>
    <td>8.27</td>
    <td>8.52</td>
    <td>8.63</td>
    <td>7.29</td>
    <td>8.87</td>
    <td>11.05</td>
  </tr>
  <tr>
    <td>ROUGE-L</td>
    <td>26.15</td>
    <td>27.09</td>
    <td>27.56</td>
    <td>26.72</td>
    <td>27.36</td>
    <td>26.94</td>
    <td>27.14</td>
    <td>26.97</td>
    <td>27.79</td>
    <td>25.61</td>
    <td>26.74</td>
    <td>30.00</td>
  </tr>
  <tr>
    <td>METEOR</td>
    <td>29.50</td>
    <td>26.01</td>
    <td>26.44</td>
    <td>30.33</td>
    <td>31.77</td>
    <td>25.74</td>
    <td>29.66</td>
    <td>31.14</td>
    <td>31.65</td>
    <td>30.03</td>
    <td>25.99</td>
    <td>34.26</td>
  </tr>
  <tr>
    <td>BERTScore</td>
    <td>88.36</td>
    <td>88.50</td>
    <td>88.52</td>
    <td>88.28</td>
    <td>88.30</td>
    <td>88.42</td>
    <td>88.41</td>
    <td>88.47</td>
    <td>88.52</td>
    <td>88.14</td>
    <td>88.39</td>
    <td>89.05</td>
  </tr>
  <tr>
    <td>CheXbert</td>
    <td>53.81</td>
    <td>52.55</td>
    <td>52.88</td>
    <td>52.73</td>
    <td>51.86</td>
    <td>52.27</td>
    <td>56.26</td>
    <td>53.33</td>
    <td>54.45</td>
    <td>51.50</td>
    <td>52.45</td>
    <td>57.19</td>
  </tr>
  <tr>
    <td>RadGraph</td>
    <td>20.44</td>
    <td>20.76</td>
    <td>21.29</td>
    <td>20.85</td>
    <td>22.02</td>
    <td>20.63</td>
    <td>21.87</td>
    <td>22.27</td>
    <td>22.22</td>
    <td>20.75</td>
    <td>20.48</td>
    <td>24.24</td>
  </tr>
  <tr>
    <td>RaTEScore</td>
    <td>58.37</td>
    <td>58.24</td>
    <td>58.77</td>
    <td>57.84</td>
    <td>58.93</td>
    <td>58.10</td>
    <td>58.73</td>
    <td>59.21</td>
    <td>59.37</td>
    <td>59.19</td>
    <td>57.81</td>
    <td>63.03</td>
  </tr>
  <tr>
    <td rowspan="7">MIMIC-CXR</td>
    <td>BLEU</td>
    <td>3.28</td>
    <td>4.07</td>
    <td>3.39</td>
    <td>3.29</td>
    <td>3.53</td>
    <td>3.99</td>
    <td>3.14</td>
    <td>3.34</td>
    <td>3.62</td>
    <td>3.37</td>
    <td>4.32</td>
    <td>4.56</td>
  </tr>
  <tr>
    <td>ROUGE-L</td>
    <td>16.54</td>
    <td>18.75</td>
    <td>17.25</td>
    <td>16.14</td>
    <td>16.58</td>
    <td>18.62</td>
    <td>15.52</td>
    <td>16.79</td>
    <td>16.67</td>
    <td>15.75</td>
    <td>18.64</td>
    <td>19.03</td>
  </tr>
  <tr>
    <td>METEOR</td>
    <td>17.90</td>
    <td>18.81</td>
    <td>17.36</td>
    <td>17.98</td>
    <td>18.68</td>
    <td>18.68</td>
    <td>14.70</td>
    <td>18.29</td>
    <td>18.22</td>
    <td>17.03</td>
    <td>19.78</td>
    <td>20.23</td>
  </tr>
  <tr>
    <td>BERTScore</td>
    <td>85.57</td>
    <td>86.14</td>
    <td>85.78</td>
    <td>85.42</td>
    <td>85.57</td>
    <td>86.14</td>
    <td>84.75</td>
    <td>85.59</td>
    <td>85.58</td>
    <td>85.27</td>
    <td>86.10</td>
    <td>86.17</td>
  </tr>
  <tr>
    <td>CheXbert</td>
    <td>22.14</td>
    <td>24.24</td>
    <td>23.05</td>
    <td>21.76</td>
    <td>23.38</td>
    <td>25.14</td>
    <td>20.07</td>
    <td>22.74</td>
    <td>22.95</td>
    <td>22.76</td>
    <td>25.78</td>
    <td>24.93</td>
  </tr>
  <tr>
    <td>RadGraph</td>
    <td>9.43</td>
    <td>10.73</td>
    <td>9.78</td>
    <td>9.20</td>
    <td>9.93</td>
    <td>10.75</td>
    <td>7.74</td>
    <td>9.52</td>
    <td>9.93</td>
    <td>9.40</td>
    <td>11.17</td>
    <td>11.55</td>
  </tr>
  <tr>
    <td>RaTEScore</td>
    <td>40.00</td>
    <td>41.06</td>
    <td>38.59</td>
    <td>39.26</td>
    <td>40.95</td>
    <td>40.73</td>
    <td>35.72</td>
    <td>40.08</td>
    <td>39.87</td>
    <td>39.62</td>
    <td>41.03</td>
    <td>42.73</td>
  </tr>
</table>

5.2 Medical VQA Results

The performance of A³TUNE on diverse medical VQA benchmarks is presented in Table 1, indicating A³TUNE maintains its superiority across all datasets and its effectiveness across diverse medical images and VQA tasks. In addition, we also analyze visual attention distribution on the test set of SLAKE with annotated RoIs (Table 2). Since the baselines in Table 1 improve the model only on the decoding side without modifying attentions, their attention distributions remain similar to the base model, LLaVA-Med and we compare only against LLaVA-Med and LLaVA-Med + LoRA. We observe that A³TUNE achieves the highest scores in both coverage (0.275) and intensity (0.147), outperforming baselines. These results highlight A³TUNE’s ability to focus more effectively on RoIs in medical images, explaining its better VQA performance and improved interpretability.

5.3 Medical Report Generation Results

Table 3 presents the evaluation of A³TUNE on medical report generation tasks using traditional metrics (e.g., BLEU, ROUGE-L) and domain-specific metrics such as RaTEScore. The baselines are applied on fine-tuned LLaVA-Med using LoRA, as the original LLaVA-Med performs significantly worse on this task, as shown in the full results in Appendix E. Similar to the results in Table 1, A³TUNE outperforms all baselines across both datasets and almost all metrics.

5.4 Module Effectiveness Analysis

5.4.1 Weak Labels Generation

(1) Quality of Weak Labels. This experiment evaluates the upper bound of A³TUNE’s performance by upgrading weak labels to high-quality ground truth labels (GT). We use a subset of SLAKE with RoIs annotations, including 992 training samples and 206 test samples and take ControlMLLM (Wu et al., 2024) as the baseline.

Figure 4 (B.2) shows that replacing weak labels (Weak) with high-quality labels (GT in A³TUNE further improves performance, validating the reasonableness of using weak labels. Additionally, while ControlMLLM (Control) improves performance in LLaVA-Med (Figure 4 (A)), it negatively impacts fine-tuned LLaVA-Med+LoRA in Figure 4 (B.1), whereas A³TUNE achieve improved results. These results highlight both the effectiveness of A³TUNE and its potential for further improvement when ground truth RoI labels are available.

(2) Selection of Weak Labels. To obtain prompt-aware weak labels, we select \( K \) segments that are most similar to the text prompt. In this experiment,

![Effectiveness analysis of RoIs labels. Base is the base model, Control means adding ControlMLLM to align attention maps with ground truth labels, Weak uses weak labels, and GT uses ground truth labels.](page_1012_120_388_246.png)

Figure 4: Effectiveness analysis of RoIs labels. Base is the base model, Control means adding ControlMLLM to align attention maps with ground truth labels, Weak uses weak labels, and GT uses ground truth labels.

![Analysis of (a) the number of selected attention heads R in A³TUNE and (b) the number of weak labels K, evaluated on the SLAKE dataset.](page_1012_420_388_246.png)

Figure 5: Analysis of (a) the number of selected attention heads \( R \) in A³TUNE and (b) the number of weak labels \( K \), evaluated on the SLAKE dataset.
<table>
  <tr>
    <th>Image Modality</th>
    <th>Image and Labels</th>
    <th>Weak Labels</th>
    <th>LLaVA-Med + LoRA Attention Map</th>
    <th>A³Tune Attention Map</th>
    <th>Input Prompt & Output Text</th>
  </tr>
  <tr>
    <td>Chest X-ray</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>Prompt: What diseases are included in the picture?<br>LLaVA-Med + LoRA: Cardiomegaly ✗<br>A³Tune: Pneumothorax ✔</td>
  </tr>
  <tr>
    <td>Brain MRI</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>Prompt: Where is/are the abnormality located?<br>LLaVA-Med + LoRA: Lower Left ✗<br>A³Tune: Lower Right ✔</td>
  </tr>
  <tr>
    <td>Abdomen CT</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>Prompt: What is the organ on the lower right corner of the body in this image?<br>LLaVA-Med + LoRA: Liver ✗<br>A³Tune: Spleen ✔</td>
  </tr>
</table>

Figure 6: Case study for fine-grained effectiveness analysis. The red box in the first column (not provided as input to the model) highlights the RoI that LLaVA-Med should focus on. The second column shows weak segmentation labels, with red bounding boxes and green inner segments generated using the method described in Section 4.1.

![Ablation settings and number of selected experts plots](page_328_579_484_246.png)

Figure 7: Analysis of (a) the contribution of different modules in A³MOE, and (b) the number of experts in K-MoE within A³MOE. In (a), each module’s impact is evaluated by removing it from A³TUNE, denoted "w/o".

we analyze the impact of different values of \( K \) on model performance using the SLAKE dataset. As shown in Figure 5 (a) increasing \( K \) initially improves performance by adding more relevant labels. However, performance peaks at \( K = 4 \), after which additional segments introduce noise, leading to a performance decline.

5.4.2 Selection of Attention Heads
In A³TUNE, we select \( R \) most “visually-critical” attention heads to balance model stability and effectiveness of A³TUNE. As shown in Figure 5 (b), disabling attention tuning (\( R = 0 \)) results in noticeably lower performance. Performance peaks at \( R = 128 \), which strikes a balance between the strength of attention tuning and stability. Thus, we set \( R = 128 \) in our experiments.

5.4.3 A³MOE Design
(1) Ablation Study of A³MOE. Figure 7 (a) shows the impact of removing key components of A³MOE from A³TUNE: Q-MoE, K-MoE and A³MOE as a whole. The removal of each module leads to a noticeable performance drop, particularly for K-MoE. This highlights the importance of K-MoE, which is designed for fine-grained attention tuning at the visual token level and is critical for datasets with diverse image modalities like SLAKE. Additionally, the removal of the entire A³MOE leads to a further decline in performance. However, even with these components removed, the performance of A³TUNE remains above the baseline LoRA tuning on LLaVA-Med, demonstrating the effectiveness of the overall framework.

(2) The Number of Selected Experts in K-MoE.
As shown in Figure 7(b), performance peaks at \( B = 3 \). Beyond this point, additional experts lead to diminishing and unstable returns, likely due to saturation and increased interference among experts handling different visual tokens.

5.5 Case Study
Figure 6 presents a case study for three image modalities. The attention map visualizations demonstrate that our method effectively redirects the model’s focus to relevant regions, mitigating hallucination issues compared to baselines. In addition, we provide fine-grained effectiveness evaluation across Chest X-ray, Abdomen CT, and Brain MRI images from SLAKE in Appendix F Table 10.

5.6 Generalization to Other Medical LVLMs
We further evaluate our approach on medical VQA (Table 4) and report generation (Table 5) using a more recent and stronger Med-LVLM, HuatouGPT-Vision-7B (Chen et al., 2024b). As shown in the results, transferring A³TUNE to this new backbone consistently achieves the best performance across all metrics on IU-Xray for report generation and SLAKE for VQA. These results demonstrate the flexibility and strong generalization ability of A³TUNE, along with its capacity to further enhance
Table 4: Performance on SLAKE VQA benchmark using HuatuoGPT-Vision-7B (denoted as HuatuoGPT-V). We report both open-ended and close-ended performance.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Metric</th>
    <th colspan="10">Method</th>
  </tr>
  <tr>
    <th>Base</th>
    <th>Greedy</th>
    <th>Beam</th>
    <th>Nucleus</th>
    <th>VCD</th>
    <th>DoLa</th>
    <th>OPERA</th>
    <th>AVISC</th>
    <th>M3ID</th>
    <th>DAMRO</th>
    <th>PAI</th>
    <th>A^3Tune</th>
  </tr>
  <tr>
    <td>HuatuoGPT-V + LoRA</td>
    <td>Open-ended</td>
    <td>85.46</td>
    <td>84.89</td>
    <td>85.62</td>
    <td>85.57</td>
    <td>85.57</td>
    <td>85.19</td>
    <td>85.07</td>
    <td>84.42</td>
    <td>85.41</td>
    <td>85.57</td>
    <td>83.85</td>
    <td>86.77</td>
  </tr>
  <tr>
    <td></td>
    <td>Close-ended</td>
    <td>91.27</td>
    <td>89.86</td>
    <td>89.86</td>
    <td>90.99</td>
    <td>90.99</td>
    <td>90.42</td>
    <td>89.86</td>
    <td>90.70</td>
    <td>91.27</td>
    <td>90.99</td>
    <td>90.99</td>
    <td>91.55</td>
  </tr>
</table>

Table 5: Performance on report generation benchmarks using HuatuoGPT-Vision-7B fine-tuned with LoRA.

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Metric</th>
    <th colspan="2">HuatuoGPT-V + LoRA</th>
    <th colspan="10">Method</th>
  </tr>
  <tr>
    <th>Greedy</th>
    <th>Beam</th>
    <th>Nucleus</th>
    <th>VCD</th>
    <th>DoLa</th>
    <th>OPERA</th>
    <th>AVISC</th>
    <th>M3ID</th>
    <th>DAMRO</th>
    <th>PAI</th>
    <th>A^3TUNE</th>
  </tr>
  <tr>
    <td rowspan="7">IU-Xray</td>
    <td>BLEU</td>
    <td>8.65</td>
    <td>9.34</td>
    <td>10.21</td>
    <td>8.19</td>
    <td>9.10</td>
    <td>9.03</td>
    <td>10.01</td>
    <td>7.47</td>
    <td>8.35</td>
    <td>9.10</td>
    <td>6.92</td>
    <td>10.52</td>
  </tr>
  <tr>
    <td>ROUGE-L</td>
    <td>27.34</td>
    <td>28.17</td>
    <td>28.64</td>
    <td>26.28</td>
    <td>27.80</td>
    <td>27.50</td>
    <td>28.57</td>
    <td>24.78</td>
    <td>27.05</td>
    <td>27.80</td>
    <td>24.36</td>
    <td>28.85</td>
  </tr>
  <tr>
    <td>METEOR</td>
    <td>31.48</td>
    <td>31.76</td>
    <td>34.23</td>
    <td>30.72</td>
    <td>32.13</td>
    <td>31.24</td>
    <td>34.10</td>
    <td>30.88</td>
    <td>32.17</td>
    <td>32.13</td>
    <td>30.84</td>
    <td>36.30</td>
  </tr>
  <tr>
    <td>BERTScore</td>
    <td>88.35</td>
    <td>88.53</td>
    <td>88.60</td>
    <td>88.19</td>
    <td>88.36</td>
    <td>88.39</td>
    <td>88.51</td>
    <td>87.77</td>
    <td>88.19</td>
    <td>88.36</td>
    <td>87.44</td>
    <td>88.67</td>
  </tr>
  <tr>
    <td>CheXbert</td>
    <td>54.21</td>
    <td>55.16</td>
    <td>55.84</td>
    <td>53.86</td>
    <td>54.34</td>
    <td>53.89</td>
    <td>55.01</td>
    <td>52.11</td>
    <td>53.78</td>
    <td>54.34</td>
    <td>50.09</td>
    <td>56.27</td>
  </tr>
  <tr>
    <td>RadGraph</td>
    <td>21.35</td>
    <td>21.86</td>
    <td>22.47</td>
    <td>20.17</td>
    <td>21.65</td>
    <td>21.04</td>
    <td>22.59</td>
    <td>19.71</td>
    <td>21.19</td>
    <td>21.65</td>
    <td>18.26</td>
    <td>23.51</td>
  </tr>
  <tr>
    <td>RaTEScore</td>
    <td>58.21</td>
    <td>58.66</td>
    <td>59.78</td>
    <td>58.29</td>
    <td>58.29</td>
    <td>57.84</td>
    <td>59.33</td>
    <td>56.49</td>
    <td>57.86</td>
    <td>58.29</td>
    <td>55.16</td>
    <td>60.51</td>
  </tr>
</table>

performance when built upon more capable backbone models.

6 Conclusion

In this work, we present A^3TUNE, a novel fine-tuning framework designed to enhance the visual grounding capabilities of Med-LVLMs. By leveraging prompt-aware weak labels and integrating a A^3MOE design, A^3TUNE dynamically aligns attention distributions to RoIs across diverse medical tasks and datasets, without requiring inference-time adjustments. Extensive experiments highlights A^3TUNE as a promising direction for enhancing Med-LVLMs in downstream applications.

Limitations

While the use of weak labels in A^3TUNE demonstrates its effectiveness, it also introduces noise that can limit performance. In some cases, the model can only focus on generally correct regions, lacking accuracy but providing directions for future research. As shown in Section 5.4.1, using high-quality labels leads to further performance improvements. Additionally, our framework is currently restricted to fine-tuning for downstream tasks, limiting its broader applicability. Furthermore, the metrics used to evaluate visual attention distribution are constrained to patch-level granularity due to the inherent design of Med-LVLMs, rather than achieving pixel-level precision.

Acknowledgments

The authors thank the anonymous referees for their valuable comments and helpful suggestions. This work is partially supported by the National Science Foundation under Grant No. 2238275 and the National Institutes of Health under Grant No. R01AG077016.

References

Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72.

Aofei Chang, Le Huang, Parminder Bhatia, Taha Kass-Hout, Fenglong Ma, and Cao Xiao. 2025. Medheval: Benchmarking hallucinations and mitigation strategies in medical large vision-language models. arXiv preprint arXiv:2503.02157.

Jiawei Chen, Dingkang Yang, Tong Wu, Yue Jiang, Xiaolou Hou, Mingcheng Li, Shunli Wang, Dongling Xiao, Ke Li, and Lihua Zhang. 2024a. Detecting and evaluating medical hallucinations in large vision language models. arXiv preprint arXiv:2406.10185.

Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghu Ramakrishnan, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. 2023. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478.

Junying Chen, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, Guangjun Yu, Xiang Wan, and Benyou Wang. 2024b. HuatuoGPT-vision, towards injecting medical visual knowledge into multimodal llms at scale.

Minghao Chen, Iro Laina, and Andrea Vedaldi. 2024c. Training-free layout control with cross-attention guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5343–5353.

Zhihong Chen, Maya Varma, Jean-Benoit Delbrouck, Magdalini Paschali, Louis Blankemeier, Dave Van
Veen, Jeya Maria Jose Valanarasu, Alaa Youssef, Joseph Paul Cohen, Eduardo Pontes Reis, Emily B. Tsai, Andrew Johnston, Cameron Olsen, Tanishq Mathew Abraham, Sergios Gatidis, Akshay S Chaudhari, and Curtis Langlotz. 2024d. Chexagent: Towards a foundation model for chest x-ray interpretation. arXiv preprint arXiv:2401.12208.

Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2023. Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint arXiv:2309.03883.

Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. 2016. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2):304–310.

Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. 2024. Multi-modal hallucination control by visual information grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14303–14312.

Xuan Gong, Tianshi Ming, Xinpeng Wang, and Zhihua Wei. 2024. Damro: Dive into the attention mechanism of lvlm to reduce object hallucination. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7696–7712.

Zishan Gu, Changchang Yin, Fenglin Liu, and Ping Zhang. 2024. Medvh: Towards systematic evaluation of hallucination for large vision language models in the medical context. arXiv preprint arXiv:2407.02730.

Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. 2020. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286.

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.

Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. 2024. Omnimed-vqa: A new large-scale comprehensive evaluation benchmark for medical lvlm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22170–22183.

Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13418–13427.

Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven QH Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P Lungren, Andrew Y Ng, et al. 2021. Radgraph: Extracting clinical entities and relations from radiology reports. arXiv preprint arXiv:2106.14463.

Yue Jiang, Jiawei Chen, Dingkang Yang, Mingcheng Li, Shunli Wang, Tong Wu, Ke Li, and Lihua Zhang. 2024. Comt: Chain-of-medical-thought reduces hallucination in medical report generation.

Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng. 2019. Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. 2023. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026.

Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. 2018. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1–10.

Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13872–13882.

Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Nau mann, Hoifung Poon, and Jianfeng Gao. 2024. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36.

Xiaoyu Liang, Jiayuan Yu, Lianrui Mu, Jiedong Zhuang, Jiaqi Hu, Yuchen Yang, Jiangnan Ye, Lu Lu, Jian Chen, and Haoji Hu. 2024. Mitigating hallucination in visual-language models via re-balancing contrastive decoding. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pages 482–496. Springer.

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. 2021. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1650–1654. IEEE.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruction tuning. Advances in neural information processing systems, 36.

Shi Liu, Kecheng Zheng, and Wei Chen. 2025. Paying more attention to image: A training-free method for alleviating hallucination in lvms. In European Conference on Computer Vision, pages 125–140. Springer.

Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. 2023. Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (MLAH), pages 353–367. PMLR.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.

Saikat Roy, Tassilo Wald, Gregor Koehler, Maximilian R Rokuss, Nico Disch, Julius Holzschuh, David Zimmerer, and Klaus H Maier-Hein. 2023. Sammd: Zero-shot medical image segmentation capabilities of the segment anything model. arXiv preprint arXiv:2304.05396.

Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Y Ng, and Matthew Lungren. 2020. Combining automatic labelers and expert annotations for accurate radiology report labeling using bert. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1500–1519.

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc.

Omkar Thawkar, Abdelrahman Shaker, Sahal Shaji Mullapilly, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Jorma Laaksonen, and Fahad Shahbaz Khan. 2023. Xraygpt: Chest radiographs summarization using medical vision-language models. arXiv preprint arXiv:2306.07971.

Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Neural Information Processing Systems.

Hanyu Wang, Bochuan Cao, Yuanpu Cao, and Jinghui Chen. 2025. Truthflow: Truthful llm generation via representation flow correction. arXiv preprint arXiv:2502.04556.

Sangmin Woo, Donguk Kim, Jaehyuk Jang, Yubin Choi, and Changick Kim. 2024. Don’t miss the forest for the trees: Attentional vision calibration for large vision language models. arXiv preprint arXiv:2405.17820.

Mingrui Wu, Xinyue Cai, Jiayi Ji, Jiale Li, Oucheng Huang, Gen Luo, Hao Fei, GUANNAN JIANG, Xiaoshuai Sun, and Rongrong Ji. 2024. ControlMLLM: Training-free visual prompt learning for multimodal large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Peng Xia, Ze Chen, Juanxi Tian, Yangrui Gong, Ruibo Hou, Yue Xu, Zhenbang Wu, Zhiyuan Fan, Yiyang Zhou, Kangyu Zhu, et al. 2024. Cares: A comprehensive benchmark of trustworthiness in medical vision language models. arXiv preprint arXiv:2406.06007.

Lingfeng Yang, Yueze Wang, Xiang Li, Xinlong Wang, and Jian Yang. 2024. Fine-grained visual prompting. Advances in Neural Information Processing Systems, 36.

Feiyang Yu, Mark Endo, Rayan Krishnan, Ian Pan, Andy Tsai, Eduardo Pontes Reis, Eduardo Kaiser Ururahy Nunes Fonseca, Henrique Min Ho Lee, Zahra Shaker Hossein Abad, Andrew Y Ng, et al. 2023. Evaluating progress in automatic chest x-ray radiology report generation. Patterns, 4(9).

Fan Yuan, Chi Qin, Xiaogang Xu, and Piji Li. 2024. Helpd: Mitigating hallucination of lvms by hierarchical feedback learning with vision-enhanced penalty decoding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1768–1785.

Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. 2023. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.00915.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.

Weike Zhao, Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2024. Ratescore: A metric for radiology report generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15004–15019.

A Dataset Processing and Statistics

For IU-Xray and OmniVQA in medical VQA task, we utilize the preprocessed datasets provided by the CARES benchmark (Xia et al., 2024), splitting each dataset into training and test sets with a 7:3 ratio. For the MIMIC-CXR dataset used in the report
<table>
  <tr>
    <th>Task</th>
    <th>Dataset</th>
    <th>Training</th>
    <th>Test</th>
  </tr>
  <tr>
    <td rowspan="5">Medical VQA</td>
    <td>SLAKE</td>
    <td>4,919</td>
    <td>1,061</td>
  </tr>
  <tr>
    <td>VQA-RAD</td>
    <td>1,797</td>
    <td>451</td>
  </tr>
  <tr>
    <td>PathVQA</td>
    <td>19,755</td>
    <td>6,761</td>
  </tr>
  <tr>
    <td>IU-Xray</td>
    <td>1,789</td>
    <td>784</td>
  </tr>
  <tr>
    <td>OmniMedVQA</td>
    <td>6,155</td>
    <td>2,642</td>
  </tr>
  <tr>
    <td rowspan="2">Report Generation</td>
    <td>IU-Xray</td>
    <td>2,069</td>
    <td>590</td>
  </tr>
  <tr>
    <td>MIMIC-CXR</td>
    <td>1,902</td>
    <td>441</td>
  </tr>
</table>

Table 6: Dataset statistics for the Medical VQA and Report Generation tasks.

Table 7: Fine-tuning epochs and \( \lambda \) values for different datasets.

<table>
  <tr>
    <th>Dataset</th>
    <th>Epochs</th>
    <th>\( \lambda \)</th>
  </tr>
  <tr>
    <th colspan="3">Medical VQA</th>
  </tr>
  <tr>
    <td>SLAKE</td>
    <td>6</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>VQA-RAD</td>
    <td>9</td>
    <td>0.06</td>
  </tr>
  <tr>
    <td>PathVQA</td>
    <td>3</td>
    <td>0.02</td>
  </tr>
  <tr>
    <td>IU-Xray</td>
    <td>6</td>
    <td>0.12</td>
  </tr>
  <tr>
    <td>OmniMedVQA</td>
    <td>3</td>
    <td>0.03</td>
  </tr>
  <tr>
    <th colspan="3">Medical Report Generation</th>
  </tr>
  <tr>
    <td>IU-Xray</td>
    <td>12</td>
    <td>0.08</td>
  </tr>
  <tr>
    <td>MIMIC-CXR</td>
    <td>12</td>
    <td>0.05</td>
  </tr>
</table>

generation task, we randomly sample 2,000 image-report pairs from the preprocessed MIMIC-CXR-JPG dataset (Johnson et al., 2019) for the training set and 500 pairs for the test set. We extract the “Findings” and “Impression” sections from each report in sampled MIMIC-CXR reports, filtering out those with an extremely low word count. The statistics of those datasets are shown in Table 6.

B Implementation Details of Baselines

Generally, we follow the recommended settings for all baselines while making necessary adjustments to adapt them to Med-LVLMs. The detailed settings are listed as follows:

• Beam Search (Sutskever et al., 2014): The number of beams is set to 5.

• Nucleus Sampling (Holtzman et al., 2020): The top-\( p \) value for sampling is 0.9.

• VCD (Leng et al., 2024): The contrastive decoding parameters are set to \( \alpha = 1 \) and \( \beta = 0.1 \). Diffusion noise is added to images using 500 steps.

• DoLa (Chuang et al., 2023): The mature layer is set to 32, while the early candidate mature layers are [0, 2, 4, 6, 8, 10, 12, 14].

• OPERA (Huang et al., 2024): The number of beams is set to 5, with a scale factor of 50, threshold of 15, and num-attn-candidates = 5. The penalty weight is set to 1. Notably, for LLaVA-Med-1.5 in the report generation task, the scale factor is set to 25 and the threshold is adjusted to 25, as the default values result in nonsensical decoded content.

• AVISC (Woo et al., 2024): We select the top-10 outlier image tokens to construct the negative decoding object. The contrastive decoding parameters are set to \( \alpha = 1 \) and \( \beta = 0.1 \).

• M3ID (Favero et al., 2024): The contrastive decoding parameters are set as follows: \( \lambda = 0.02 \) and \( \gamma_t = \exp(-\lambda \cdot t) \), where \( t \) denotes the current decoding step.

• DAMRO (Gong et al., 2024): We select the top-10 tokens with the highest attention to the [CLS] token in the visual encoder as outlier tokens. The contrastive decoding parameters are set to \( \alpha = 0.5 \) and \( \beta = 0.1 \).

• PAI (Liu et al., 2025): In the inference intervention, the start layer and end layer are set to 2 and 32, respectively. \( \gamma = 1.1 \) and \( \alpha = 0.2 \).

• ControlMLLM (Wu et al., 2024): In inference-time tuning, it is configured with \( \beta = 0.5, \alpha = 400 \), and a learning rate of 4. For LLaVA-Med-1.5, the same parameters are applied but with a reduced learning rate of 1.

C Implementation Details

C.1 Hyperparameter setting

All fine-tuning tasks are performed using the same seed for LoRA initialization. The LoRA rank is set to 64 and the rank of each expert in A^3MoE is set to 16, with a learning rate of 2e-4. For A^3MoE, the default number of experts in K-MoE \( O^k \) is 8, while the default number of experts in Q-MoE \( O^q \) is 4. For example, we use these settings in all report generation tasks for Chest X-rays. However, for large datasets with diverse image modalities such as SLAKE, PathVQA and OmniMedVQA, the number of experts is increased to 16 and 8, respectively. Some other key hyperparameters are: \( K = 4, R = 128, B = 3 \) when \( O^k = 16 \) and \( B = 2 \) when \( O^k = 8 \).
Table 8: Full results of report generation on IU-Xray, based on LLaVA-Med and the LoRA fine-tuned LLaVA-Med.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Method</th>
    <th colspan="7">IU-Xray</th>
  </tr>
  <tr>
    <th>BLEU</th>
    <th>ROUGE-L</th>
    <th>METEOR</th>
    <th>BERTScore</th>
    <th>CheXbert</th>
    <th>RadGraph</th>
    <th>RaTEScore</th>
  </tr>
  <tr>
    <td rowspan="10">LLaVA-Med</td>
    <td>LLaVA-Med</td>
    <td>1.30</td>
    <td>11.55</td>
    <td>16.09</td>
    <td>84.12</td>
    <td>35.21</td>
    <td>6.46</td>
    <td>42.32</td>
  </tr>
  <tr>
    <td>Greedy</td>
    <td>1.21</td>
    <td>13.97</td>
    <td>19.12</td>
    <td>84.61</td>
    <td>35.82</td>
    <td>6.74</td>
    <td>42.32</td>
  </tr>
  <tr>
    <td>Beam</td>
    <td>1.19</td>
    <td>13.42</td>
    <td>19.49</td>
    <td>84.24</td>
    <td>39.41</td>
    <td>8.69</td>
    <td>44.56</td>
  </tr>
  <tr>
    <td>Nucleus</td>
    <td>1.33</td>
    <td>11.15</td>
    <td>16.28</td>
    <td>83.89</td>
    <td>32.39</td>
    <td>6.39</td>
    <td>41.39</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>1.19</td>
    <td>10.26</td>
    <td>15.46</td>
    <td>83.49</td>
    <td>33.25</td>
    <td>5.67</td>
    <td>41.20</td>
  </tr>
  <tr>
    <td>DoLa</td>
    <td>1.20</td>
    <td>13.93</td>
    <td>19.14</td>
    <td>84.60</td>
    <td>35.61</td>
    <td>6.74</td>
    <td>42.31</td>
  </tr>
  <tr>
    <td>OPERA</td>
    <td>0.89</td>
    <td>8.46</td>
    <td>13.90</td>
    <td>82.52</td>
    <td>33.19</td>
    <td>3.48</td>
    <td>39.19</td>
  </tr>
  <tr>
    <td>AVISC</td>
    <td>1.16</td>
    <td>11.12</td>
    <td>17.34</td>
    <td>83.86</td>
    <td>30.99</td>
    <td>5.08</td>
    <td>43.30</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>1.40</td>
    <td>12.01</td>
    <td>16.48</td>
    <td>84.07</td>
    <td>34.32</td>
    <td>6.71</td>
    <td>42.02</td>
  </tr>
  <tr>
    <td>DAMRO</td>
    <td>1.23</td>
    <td>10.69</td>
    <td>16.68</td>
    <td>83.72</td>
    <td>30.21</td>
    <td>5.63</td>
    <td>43.62</td>
  </tr>
  <tr>
    <td>PAI</td>
    <td>1.14</td>
    <td>12.90</td>
    <td>18.46</td>
    <td>84.33</td>
    <td>38.64</td>
    <td>6.96</td>
    <td>42.92</td>
  </tr>
  <tr>
    <td rowspan="10">LLaVA-Med + LoRA</td>
    <td>LLaVA-Med + LoRA</td>
    <td>7.70</td>
    <td>26.15</td>
    <td>29.50</td>
    <td>88.36</td>
    <td>53.81</td>
    <td>20.44</td>
    <td>58.37</td>
  </tr>
  <tr>
    <td>Greedy</td>
    <td>8.86</td>
    <td>27.09</td>
    <td>26.01</td>
    <td>88.50</td>
    <td>52.55</td>
    <td>20.76</td>
    <td>58.24</td>
  </tr>
  <tr>
    <td>Beam</td>
    <td>9.34</td>
    <td>27.56</td>
    <td>26.44</td>
    <td>88.52</td>
    <td>52.88</td>
    <td>21.29</td>
    <td>58.77</td>
  </tr>
  <tr>
    <td>Nucleus</td>
    <td>7.80</td>
    <td>26.72</td>
    <td>30.33</td>
    <td>88.28</td>
    <td>52.73</td>
    <td>20.85</td>
    <td>57.84</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>8.83</td>
    <td>27.56</td>
    <td>31.77</td>
    <td>88.30</td>
    <td>51.86</td>
    <td>22.02</td>
    <td>58.93</td>
  </tr>
  <tr>
    <td>DoLa</td>
    <td>8.93</td>
    <td>26.94</td>
    <td>25.74</td>
    <td>88.42</td>
    <td>52.27</td>
    <td>20.63</td>
    <td>58.10</td>
  </tr>
  <tr>
    <td>OPERA</td>
    <td>8.27</td>
    <td>27.14</td>
    <td>29.66</td>
    <td>88.41</td>
    <td>56.26</td>
    <td>21.87</td>
    <td>58.73</td>
  </tr>
  <tr>
    <td>AVISC</td>
    <td>8.52</td>
    <td>26.97</td>
    <td>31.14</td>
    <td>88.47</td>
    <td>53.33</td>
    <td>22.27</td>
    <td>59.21</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>8.63</td>
    <td>27.79</td>
    <td>31.65</td>
    <td>88.52</td>
    <td>54.45</td>
    <td>22.72</td>
    <td>59.37</td>
  </tr>
  <tr>
    <td>DAMRO</td>
    <td>7.29</td>
    <td>25.61</td>
    <td>30.03</td>
    <td>88.14</td>
    <td>51.50</td>
    <td>20.75</td>
    <td>59.19</td>
  </tr>
  <tr>
    <td>PAI</td>
    <td>8.87</td>
    <td>26.74</td>
    <td>25.99</td>
    <td>88.39</td>
    <td>52.45</td>
    <td>20.48</td>
    <td>57.81</td>
  </tr>
  <tr>
    <td>A³TUNE (ours)</td>
    <td>11.05</td>
    <td>30.00</td>
    <td>34.26</td>
    <td>89.05</td>
    <td>57.19</td>
    <td>24.24</td>
    <td>63.03</td>
  </tr>
</table>

![Line plot showing performance vs λ for A³TUNE on SLAKE dataset](page_1012_1342_388_246.png)

Figure 8: Analysis of the value of \( \lambda \) in A³TUNE on the SLAKE dataset.

C.2 \( \lambda \) Selection

In our final loss in Eq. (12), we use a key hyperparameter \( \lambda \) to balance the two loss terms. Here, we conduct an analysis to select the optimal value of \( \lambda \). Figure 8, illustrate the impact of \( \lambda \), which controls the strength of attention alignment tuning. We can observe that on SLAKE, performance peaks at \( \lambda = 0.1 \). Beyond this point, the performance declines due to over-alignment and extreme attention distributions. Notably, \( \lambda \) varies with the scale of datasets and the training epochs, and the values of \( \lambda \) and fine-tuning epochs for all the datasets are as shown in Table 7. All the experiments are conducted using four A6000 GPUs.

D Metrics

D.1 Metrics for Report Generation

We evaluate model performance using commonly used metrics for generation tasks. These include BERTScore (Zhang et al., 2019), which measures the similarity between the embeddings of predicted and reference texts, and METEOR (Banerjee and Lavie, 2005), which evaluates alignment between generated answers and reference texts, accounting for synonyms and stemming. Additionally, we employ ROUGE-L (Lin, 2004), which measures n-gram overlap and the longest common subsequence, and BLEU (Papineni et al., 2002), which calculates n-gram precision in the predicted text relative to the reference, focusing on exact matches. In addition, we include the following domain-specific metrics designed for medical report generation:

• CheXbert (Smit et al., 2020) is an automatic labeler that extracts pathology indicators from radiology reports. We follow (Yu et al., 2023) to calculate the CheXbert vector similarity that measures the cosine similarity between pathology indicator vectors derived from ground truth and model-generated reports.

• RadGraph (Jain et al., 2021) is a tool that extracts entity and relation from radiology reports. We use RadGraph to specifically indicate RadGraph F1, which measures the overlap of clinical entities and their relations extracted from ground truth and model-generated reports.

• RaTEScore (Zhao et al., 2024) is a recently
Table 9: Full results of report generation on MIMIC-CXR, based on LLaVA-Med and the LoRA fine-tuned LLaVA-Med.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Method</th>
    <th colspan="6">MIMIC-CXR</th>
  </tr>
  <tr>
    <th>BLEU</th>
    <th>ROUGE-L</th>
    <th>METEOR</th>
    <th>BERTScore</th>
    <th>CheXbert</th>
    <th>RadGraph</th>
    <th>RaTEScore</th>
  </tr>
  <tr>
    <td rowspan="8">LLaVA-Med</td>
    <td>LLaVA-Med</td>
    <td>1.38</td>
    <td>12.28</td>
    <td>13.20</td>
    <td>84.24</td>
    <td>15.45</td>
    <td>3.14</td>
    <td>32.91</td>
  </tr>
  <tr>
    <td>Greedy</td>
    <td>0.42</td>
    <td>9.23</td>
    <td>6.77</td>
    <td>83.94</td>
    <td>14.23</td>
    <td>1.42</td>
    <td>29.81</td>
  </tr>
  <tr>
    <td>Beam</td>
    <td>1.20</td>
    <td>13.06</td>
    <td>13.00</td>
    <td>83.25</td>
    <td>10.90</td>
    <td>3.16</td>
    <td>34.30</td>
  </tr>
  <tr>
    <td>Nucleus</td>
    <td>1.22</td>
    <td>12.49</td>
    <td>11.99</td>
    <td>83.65</td>
    <td>15.79</td>
    <td>3.31</td>
    <td>34.17</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>1.12</td>
    <td>12.21</td>
    <td>11.53</td>
    <td>83.53</td>
    <td>14.25</td>
    <td>2.83</td>
    <td>34.06</td>
  </tr>
  <tr>
    <td>DoLa</td>
    <td>0.39</td>
    <td>9.13</td>
    <td>6.54</td>
    <td>83.96</td>
    <td>14.26</td>
    <td>1.38</td>
    <td>29.59</td>
  </tr>
  <tr>
    <td>OPERA</td>
    <td>1.06</td>
    <td>12.90</td>
    <td>11.93</td>
    <td>83.11</td>
    <td>12.90</td>
    <td>2.80</td>
    <td>36.12</td>
  </tr>
  <tr>
    <td>AVISC</td>
    <td>1.20</td>
    <td>11.90</td>
    <td>13.08</td>
    <td>83.20</td>
    <td>13.09</td>
    <td>2.62</td>
    <td>35.22</td>
  </tr>
  <tr>
    <td rowspan="5">LLaVA-Med + LoRA</td>
    <td>M3ID</td>
    <td>1.10</td>
    <td>12.13</td>
    <td>11.47</td>
    <td>83.69</td>
    <td>15.08</td>
    <td>2.77</td>
    <td>33.68</td>
  </tr>
  <tr>
    <td>DAMRO</td>
    <td>1.27</td>
    <td>12.40</td>
    <td>12.84</td>
    <td>83.40</td>
    <td>13.76</td>
    <td>3.36</td>
    <td>35.71</td>
  </tr>
  <tr>
    <td>PAI</td>
    <td>0.48</td>
    <td>9.74</td>
    <td>7.22</td>
    <td>83.81</td>
    <td>14.39</td>
    <td>1.45</td>
    <td>30.57</td>
  </tr>
  <tr>
    <td>LLaVA-Med + LoRA</td>
    <td>LLaVA-Med + LoRA</td>
    <td>3.28</td>
    <td>16.54</td>
    <td>17.90</td>
    <td>85.57</td>
    <td>22.14</td>
    <td>9.43</td>
    <td>40.00</td>
  </tr>
  <tr>
    <td>Greedy</td>
    <td>4.07</td>
    <td>18.75</td>
    <td>18.81</td>
    <td>86.14</td>
    <td>24.24</td>
    <td>10.73</td>
    <td>41.06</td>
  </tr>
  <tr>
    <td>Beam</td>
    <td>3.39</td>
    <td>17.25</td>
    <td>17.36</td>
    <td>85.78</td>
    <td>23.05</td>
    <td>9.78</td>
    <td>38.59</td>
  </tr>
  <tr>
    <td>Nucleus</td>
    <td>3.29</td>
    <td>16.14</td>
    <td>17.98</td>
    <td>85.42</td>
    <td>21.76</td>
    <td>9.20</td>
    <td>39.26</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>3.53</td>
    <td>16.58</td>
    <td>18.68</td>
    <td>85.57</td>
    <td>23.38</td>
    <td>9.93</td>
    <td>40.95</td>
  </tr>
  <tr>
    <td>DoLa</td>
    <td>3.99</td>
    <td>18.62</td>
    <td>18.68</td>
    <td>86.14</td>
    <td>25.14</td>
    <td>10.75</td>
    <td>40.73</td>
  </tr>
  <tr>
    <td>OPERA</td>
    <td>3.14</td>
    <td>15.52</td>
    <td>14.70</td>
    <td>84.75</td>
    <td>20.07</td>
    <td>7.74</td>
    <td>35.72</td>
  </tr>
  <tr>
    <td>AVISC</td>
    <td>3.34</td>
    <td>16.79</td>
    <td>18.29</td>
    <td>85.59</td>
    <td>22.74</td>
    <td>9.52</td>
    <td>40.08</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>3.62</td>
    <td>16.67</td>
    <td>18.22</td>
    <td>85.58</td>
    <td>22.95</td>
    <td>9.93</td>
    <td>39.87</td>
  </tr>
  <tr>
    <td>DAMRO</td>
    <td>3.37</td>
    <td>15.75</td>
    <td>17.03</td>
    <td>85.27</td>
    <td>22.76</td>
    <td>9.40</td>
    <td>39.62</td>
  </tr>
  <tr>
    <td>PAI</td>
    <td>4.32</td>
    <td>18.64</td>
    <td>19.78</td>
    <td>86.10</td>
    <td>25.78</td>
    <td>11.17</td>
    <td>41.03</td>
  </tr>
  <tr>
    <td>A²TUNE (ours)</td>
    <td><b>4.56</b></td>
    <td><b>19.03</b></td>
    <td><b>20.23</b></td>
    <td><b>86.17</b></td>
    <td><b>24.93</b></td>
    <td><b>11.55</b></td>
    <td><b>42.73</b></td>
  </tr>
</table>

proposed metric that prioritizes crucial medical entities, including diagnostic outcomes and anatomical details. This metric is robust to complex medical synonyms and sensitive to negation expressions, aligning more closely with human judgment compared to existing metrics.

D.2 Metrics for Attention Distribution

(a) Coverage Score. The Coverage Score measures the proportion of the ground truth region that is covered by the attention map. Let B denote the binary mask of the ground truth segment (where \( \mathbf{B}(i,j) = 1 \) for pixels belonging to the ground truth and \( \mathbf{B}(i,j) = 0 \) otherwise) and M denote the attention map output by the model. The score is defined as:

\[
\text{Coverage} = \frac{\sum_{i,j} \mathbf{B}(i,j) \cdot \mathbf{M}_\tau(i,j)}{\sum_{i,j} \mathbf{B}(i,j)},
\]

where \( \mathbf{M}_\tau \) is the thresholded attention map, i.e., \( \mathbf{M}_\tau(i,j) = 1 \) if \( \mathbf{M}(i,j) \geq \tau \), and \( \mathbf{M}_\tau(i,j) = 0 \) otherwise. This metric quantifies how well the attention aligns spatially with the ground truth. In our experiments, we set \( \tau \) as 0.15.

(b) Intensity Alignment. The Intensity Alignment metric evaluates the average attention intensity within the ground truth region. It is computed as:

\[
\text{Intensity Alignment} = \frac{\sum_{i,j} \mathbf{B}(i,j) \cdot \mathbf{M}(i,j)}{\sum_{i,j} \mathbf{B}(i,j)}
\]

This score reflects the degree to which the model focuses its attention on the ground truth segment, considering the intensity values of the attention map.

Table 10: Results on different medical image modalities.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Chest X-ray</th>
    <th colspan="2">Abdomen CT</th>
    <th colspan="2">Brain MRI</th>
  </tr>
  <tr>
    <th>Open</th>
    <th>Closed</th>
    <th>Open</th>
    <th>Closed</th>
    <th>Open</th>
    <th>Closed</th>
  </tr>
  <tr>
    <td>LLaVA-Med + LoRA</td>
    <td>76.96</td>
    <td>89.08</td>
    <td>79.67</td>
    <td>81.15</td>
    <td>79.15</td>
    <td>79.37</td>
  </tr>
  <tr>
    <td>Greedy</td>
    <td>78.53</td>
    <td>89.92</td>
    <td>81.50</td>
    <td>83.61</td>
    <td>80.57</td>
    <td>82.54</td>
  </tr>
  <tr>
    <td>Beam</td>
    <td>79.83</td>
    <td>89.92</td>
    <td>81.76</td>
    <td>84.43</td>
    <td>78.90</td>
    <td>84.12</td>
  </tr>
  <tr>
    <td>Nucleus</td>
    <td>80.95</td>
    <td>87.40</td>
    <td>79.49</td>
    <td>79.51</td>
    <td>79.72</td>
    <td>80.95</td>
  </tr>
  <tr>
    <td>DoLa</td>
    <td>78.53</td>
    <td>89.92</td>
    <td>81.50</td>
    <td>83.61</td>
    <td>79.89</td>
    <td>80.95</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>81.39</td>
    <td>86.55</td>
    <td>80.54</td>
    <td>80.33</td>
    <td>79.62</td>
    <td>69.84</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>79.54</td>
    <td>85.71</td>
    <td>76.70</td>
    <td>80.33</td>
    <td>80.57</td>
    <td>73.01</td>
  </tr>
  <tr>
    <td>AVISC</td>
    <td>80.52</td>
    <td>87.40</td>
    <td>81.41</td>
    <td>78.69</td>
    <td>80.49</td>
    <td>85.71</td>
  </tr>
  <tr>
    <td>OPERA</td>
    <td>80.19</td>
    <td>89.92</td>
    <td>81.15</td>
    <td>83.61</td>
    <td>77.55</td>
    <td>82.54</td>
  </tr>
  <tr>
    <td>DAMRO</td>
    <td>79.23</td>
    <td>88.24</td>
    <td>80.19</td>
    <td>83.61</td>
    <td>75.60</td>
    <td>80.95</td>
  </tr>
  <tr>
    <td>PAI</td>
    <td>79.78</td>
    <td>89.92</td>
    <td>80.45</td>
    <td>81.15</td>
    <td>81.95</td>
    <td>80.95</td>
  </tr>
  <tr>
    <td>A²TUNE (ours)</td>
    <td><b>82.21</b></td>
    <td><b>89.92</b></td>
    <td><b>83.33</b></td>
    <td><b>84.43</b></td>
    <td><b>82.41</b></td>
    <td><b>87.30</b></td>
  </tr>
</table>

![Bar chart showing VQA Types where A²TUNE outperforms LLaVA-Med + LoRA across three image modalities.](page_1012_1342_340_246.png)

Figure 9: VQA Types where A²TUNE outperforms LLaVA-Med + LoRA across three image modalities.
Case(A) Prompt: What disease is included in the medical image?

<table>
  <tr>
    <th>A.1 Input Image</th>
    <th>A.2 LLaVA-Med-1.5</th>
    <th>A.3 LLaVA-Med-1.5 + ControlMLLM</th>
  </tr>
  <tr>
    <td>Ground truth: “A brain tumor in the upper left region.”</td>
    <td>Generated answer: “The picture includes a case of Creutzfeldt-Jakob disease (CJD), which is a rare and rapidly progressive neurodegenerative disorder.”</td>
    <td>Generated answer: “The picture includes a case of glioblastoma, which is a type of brain tumor.”</td>
  </tr>
</table>

Case(B) Prompt: Is there any abnormality on the liver?

<table>
  <tr>
    <th>B.1 Input Image</th>
    <th>B.2 LLaVA-Med-1.5</th>
    <th>B.3 LLaVA-Med-1.5 + ControlMLLM</th>
  </tr>
  <tr>
    <td>Ground truth: “A liver tumor in the upper left region.”</td>
    <td>Generated answer: “According to the image, the liver appears to be normal.”</td>
    <td>Generated answer: “Yes, there appears to be a liver abscess in the image.”</td>
  </tr>
</table>

Figure 10: Case study on LLaVA-Med-1.5. The red box in the Input Image (not provided as input to the model) highlights the RoI that model should focus on. Red texts and green texts indicate wrong answers and correct answers, respectively.

Table 11: Results on report generation benchmarks, based on LLaVA-Med-1.5.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Method</th>
    <th colspan="7">IU-Xray</th>
  </tr>
  <tr>
    <th>BLEU</th>
    <th>ROUGE-L</th>
    <th>METEOR</th>
    <th>BERTScore</th>
    <th>CheXbert</th>
    <th>RadGraph</th>
    <th>RaTEScore</th>
  </tr>
  <tr>
    <td rowspan="9">LLaVA-Med-1.5</td>
    <td>LLaVA-Med-1.5</td>
    <td>1.40</td>
    <td>12.41</td>
    <td>16.30</td>
    <td>84.55</td>
    <td>38.20</td>
    <td>7.77</td>
    <td>41.15</td>
  </tr>
  <tr>
    <td>Greedy</td>
    <td>1.04</td>
    <td>12.15</td>
    <td>9.87</td>
    <td>85.43</td>
    <td>38.04</td>
    <td>5.43</td>
    <td>34.97</td>
  </tr>
  <tr>
    <td>Beam</td>
    <td>1.09</td>
    <td>11.17</td>
    <td>19.59</td>
    <td>83.43</td>
    <td>40.13</td>
    <td>9.42</td>
    <td>48.64</td>
  </tr>
  <tr>
    <td>Nucleus</td>
    <td>1.44</td>
    <td>12.10</td>
    <td>15.60</td>
    <td>81.45</td>
    <td>38.04</td>
    <td>6.51</td>
    <td>40.41</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>1.42</td>
    <td>12.26</td>
    <td>15.72</td>
    <td>84.54</td>
    <td>36.57</td>
    <td>6.49</td>
    <td>39.93</td>
  </tr>
  <tr>
    <td>DoLa</td>
    <td>0.99</td>
    <td>12.15</td>
    <td>9.36</td>
    <td>85.64</td>
    <td>38.22</td>
    <td>5.40</td>
    <td>34.93</td>
  </tr>
  <tr>
    <td>OPERA</td>
    <td>1.13</td>
    <td>11.49</td>
    <td>14.63</td>
    <td>83.76</td>
    <td>37.38</td>
    <td>1.41</td>
    <td>35.96</td>
  </tr>
  <tr>
    <td>AVISC</td>
    <td>1.18</td>
    <td>11.32</td>
    <td>16.66</td>
    <td>83.76</td>
    <td>35.83</td>
    <td>6.63</td>
    <td>40.36</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>1.33</td>
    <td>12.31</td>
    <td>16.31</td>
    <td>84.45</td>
    <td>37.54</td>
    <td>6.42</td>
    <td>40.41</td>
  </tr>
  <tr>
    <td>DAMRO</td>
    <td>1.27</td>
    <td>11.56</td>
    <td>16.42</td>
    <td>84.08</td>
    <td>35.60</td>
    <td>6.80</td>
    <td>40.08</td>
  </tr>
  <tr>
    <td>PAI</td>
    <td>1.11</td>
    <td>12.05</td>
    <td>10.99</td>
    <td>85.03</td>
    <td>37.56</td>
    <td>5.21</td>
    <td>34.83</td>
  </tr>
  <tr>
    <td rowspan="9">LLaVA-Med-1.5 + LoRA</td>
    <td>LLaVA-Med-1.5 + LoRA</td>
    <td>8.04</td>
    <td>26.52</td>
    <td>30.37</td>
    <td>88.24</td>
    <td>51.32</td>
    <td>20.35</td>
    <td>56.97</td>
  </tr>
  <tr>
    <td>Greedy</td>
    <td>9.36</td>
    <td>27.57</td>
    <td>27.91</td>
    <td><b>88.55</b></td>
    <td>52.44</td>
    <td>21.28</td>
    <td>58.61</td>
  </tr>
  <tr>
    <td>Beam</td>
    <td>9.54</td>
    <td>28.41</td>
    <td>35.40</td>
    <td>88.45</td>
    <td>53.70</td>
    <td>22.43</td>
    <td>59.65</td>
  </tr>
  <tr>
    <td>Nucleus</td>
    <td>7.80</td>
    <td>26.72</td>
    <td>30.33</td>
    <td>88.28</td>
    <td>52.73</td>
    <td>20.85</td>
    <td>57.84</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>8.83</td>
    <td>27.36</td>
    <td>31.77</td>
    <td>88.30</td>
    <td>51.86</td>
    <td>22.02</td>
    <td>58.93</td>
  </tr>
  <tr>
    <td>DoLa</td>
    <td>8.93</td>
    <td>26.94</td>
    <td>25.74</td>
    <td>88.42</td>
    <td>52.27</td>
    <td>20.63</td>
    <td>58.10</td>
  </tr>
  <tr>
    <td>OPERA</td>
    <td>9.23</td>
    <td>27.48</td>
    <td>34.17</td>
    <td>88.17</td>
    <td>51.65</td>
    <td>21.37</td>
    <td>57.89</td>
  </tr>
  <tr>
    <td>AVISC</td>
    <td>5.57</td>
    <td>21.71</td>
    <td>26.84</td>
    <td>87.34</td>
    <td>47.32</td>
    <td>16.87</td>
    <td>53.66</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>8.44</td>
    <td>26.21</td>
    <td>30.86</td>
    <td>88.20</td>
    <td>51.13</td>
    <td>20.77</td>
    <td>59.37</td>
  </tr>
  <tr>
    <td>DAMRO</td>
    <td>8.21</td>
    <td>25.77</td>
    <td>30.58</td>
    <td>88.09</td>
    <td>50.10</td>
    <td>22.33</td>
    <td>57.31</td>
  </tr>
  <tr>
    <td>PAI</td>
    <td>8.52</td>
    <td>26.97</td>
    <td>28.63</td>
    <td>88.42</td>
    <td>52.22</td>
    <td>20.99</td>
    <td>58.21</td>
  </tr>
  <tr>
    <td>A³TUNE (ours)</td>
    <td><b>10.51</b></td>
    <td><b>28.76</b></td>
    <td><b>35.74</b></td>
    <td><b>88.51</b></td>
    <td><b>53.88</b></td>
    <td><b>23.10</b></td>
    <td><b>59.66</b></td>
  </tr>
</table>

E Full Results on Report Generation Benchmarks

We present the comparison results of report generation using LLaVA-Med in Table 8 (IU-Xray) and Table 9 (MIMIC-CXR). These tables include the original results of baselines applied to LLaVA-Med without fine-tuning, where the model performs significantly worse on report generation. For example, in Table 8, the best-performing baseline, M3ID, achieves a BLEU score of 1.40, which is much lower than fine-tuned LLaVA-Med baselines. This highlights the challenge of generating professional medical reports without fine-tuning. As discussed in the experiments section, A³TUNE consistently achieves the best performance, outperforming all baselines by a large margin.

F Fine-grained Effectiveness Analysis

In this experiment, we conduct fine-grained analyses to evaluate the effectiveness of our proposed method across different medical image modalities. Specifically, we examine how our approach improves model performance on Chest X-ray, Abdomen CT, and Brain MRI images from SLAKE. As shown in Table 10, our method outperforms all baselines, particularly on Brain MRI. These results demonstrates the effectiveness of A³TUNE across diverse medical images and its generaliza-
Table 12: Results of Report Generation on MIMIC-CXR

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Method</th>
    <th colspan="7">MIMIC-CXR</th>
  </tr>
  <tr>
    <th>BLEU</th>
    <th>ROUGE-L</th>
    <th>METEOR</th>
    <th>BERTScore</th>
    <th>CheXbert</th>
    <th>RadGraph</th>
    <th>RaTEScore</th>
  </tr>
  <tr>
    <td rowspan="9">LLaVA-Med-1.5</td>
    <td>LLaVA-Med-1.5</td>
    <td>0.98</td>
    <td>10.73</td>
    <td>10.95</td>
    <td>82.29</td>
    <td>14.11</td>
    <td>1.83</td>
    <td>31.90</td>
  </tr>
  <tr>
    <td>Greedy</td>
    <td>0.93</td>
    <td>10.90</td>
    <td>9.45</td>
    <td>83.09</td>
    <td>14.11</td>
    <td>1.09</td>
    <td>28.07</td>
  </tr>
  <tr>
    <td>Beam</td>
    <td>1.13</td>
    <td>10.95</td>
    <td>13.01</td>
    <td>82.34</td>
    <td>12.51</td>
    <td>1.87</td>
    <td>32.86</td>
  </tr>
  <tr>
    <td>Nucleus</td>
    <td>0.96</td>
    <td>10.55</td>
    <td>10.70</td>
    <td>78.53</td>
    <td>14.11</td>
    <td>1.64</td>
    <td>31.54</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>1.00</td>
    <td>10.84</td>
    <td>11.31</td>
    <td>83.11</td>
    <td>13.56</td>
    <td>1.97</td>
    <td>31.75</td>
  </tr>
  <tr>
    <td>DoLa</td>
    <td>0.65</td>
    <td>9.88</td>
    <td>7.83</td>
    <td>83.47</td>
    <td>14.07</td>
    <td>1.09</td>
    <td>28.07</td>
  </tr>
  <tr>
    <td>OPERA</td>
    <td>1.09</td>
    <td>11.51</td>
    <td>12.22</td>
    <td>82.69</td>
    <td>13.00</td>
    <td>0.52</td>
    <td>27.77</td>
  </tr>
  <tr>
    <td>AVISC</td>
    <td>1.16</td>
    <td>11.14</td>
    <td>12.30</td>
    <td>82.61</td>
    <td>14.11</td>
    <td>1.97</td>
    <td>32.20</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>1.06</td>
    <td>11.50</td>
    <td>11.54</td>
    <td>83.15</td>
    <td>13.98</td>
    <td>2.24</td>
    <td>32.74</td>
  </tr>
  <tr>
    <td>DAMRO</td>
    <td>1.14</td>
    <td>11.01</td>
    <td>12.35</td>
    <td>82.80</td>
    <td>13.73</td>
    <td>2.26</td>
    <td>32.18</td>
  </tr>
  <tr>
    <td>PAI</td>
    <td>1.12</td>
    <td>11.67</td>
    <td>10.63</td>
    <td>82.80</td>
    <td>14.05</td>
    <td>0.98</td>
    <td>28.07</td>
  </tr>
  <tr>
    <td rowspan="9">LLaVA-Med-1.5 + LoRA</td>
    <td>LLaVA-Med-1.5 + LoRA</td>
    <td>3.50</td>
    <td>16.38</td>
    <td>18.95</td>
    <td>85.56</td>
    <td>21.45</td>
    <td>9.54</td>
    <td>40.45</td>
  </tr>
  <tr>
    <td>Greedy</td>
    <td>3.50</td>
    <td>16.49</td>
    <td>18.71</td>
    <td>85.54</td>
    <td>23.43</td>
    <td>9.63</td>
    <td>40.49</td>
  </tr>
  <tr>
    <td>Beam</td>
    <td>3.66</td>
    <td>16.85</td>
    <td>20.68</td>
    <td>85.51</td>
    <td>25.90</td>
    <td>9.91</td>
    <td>41.46</td>
  </tr>
  <tr>
    <td>Nucleus</td>
    <td>3.48</td>
    <td>16.35</td>
    <td>18.93</td>
    <td>85.50</td>
    <td>22.27</td>
    <td>9.27</td>
    <td>40.08</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>3.74</td>
    <td>16.88</td>
    <td>19.03</td>
    <td>85.56</td>
    <td>22.98</td>
    <td>9.56</td>
    <td>40.93</td>
  </tr>
  <tr>
    <td>DoLa</td>
    <td>3.48</td>
    <td>16.45</td>
    <td>18.66</td>
    <td>85.54</td>
    <td>23.34</td>
    <td>9.52</td>
    <td>40.49</td>
  </tr>
  <tr>
    <td>OPERA</td>
    <td>3.56</td>
    <td>16.77</td>
    <td>20.10</td>
    <td>85.46</td>
    <td>24.31</td>
    <td>9.81</td>
    <td>41.33</td>
  </tr>
  <tr>
    <td>AVISC</td>
    <td>3.31</td>
    <td>16.36</td>
    <td>18.64</td>
    <td>85.48</td>
    <td>23.31</td>
    <td>9.02</td>
    <td>40.36</td>
  </tr>
  <tr>
    <td>M3ID</td>
    <td>3.14</td>
    <td>16.13</td>
    <td>18.52</td>
    <td>85.39</td>
    <td>22.42</td>
    <td>9.15</td>
    <td>39.74</td>
  </tr>
  <tr>
    <td>DAMRO</td>
    <td>3.42</td>
    <td>16.63</td>
    <td>18.87</td>
    <td>85.48</td>
    <td>23.30</td>
    <td>9.46</td>
    <td>40.80</td>
  </tr>
  <tr>
    <td>PAI</td>
    <td>3.63</td>
    <td>16.65</td>
    <td>18.61</td>
    <td>85.60</td>
    <td>24.51</td>
    <td>9.60</td>
    <td>40.49</td>
  </tr>
  <tr>
    <td>A³TUNE (ours)</td>
    <td>4.22</td>
    <td>18.02</td>
    <td>20.69</td>
    <td>85.75</td>
    <td>25.37</td>
    <td>10.52</td>
    <td>42.15</td>
  </tr>
</table>

tion ability in medical applications. To explain this improvement, we also include case studies with attention map visualizations for each modality in Section 5.5.

Furthermore, we analyze the VQA types where our method outperforms fine-tuned LLaVA-Med with LoRA across the three image modalities. As shown in Figure 9, our approach improves the model’s performance in three key areas sensitive to image information:

• General Radiology Knowledge: Understanding medical modality types and their features.

• Anatomical Structures: Recognizing features of key anatomical structures, such as organ count, location.

• Abnormalities: Identifying abnormalities based on features like location and color.

These improvements highlight the effectiveness of attention tuning in handling VQA types relying on accurate visual information and interaction.

G Analysis on LLaVA-Med-1.5

G.1 Attention Biases and Hallucination Issues in LLaVA-Med-1.5

The attention biases we address are not unique to LLaVA-Med but are prevalent across Med-LVLMs. For instance, we include cases from LLaVA-Med-1.5 in this section. Although LLaVA-Med-1.5 is an enhanced version of LLaVA-Med, with improved training data and an increased number of visual tokens (from 256 to 576), the attention biases and hallucination issues persist.

In Figure 10, we visualize the attention biases and corresponding hallucination issues in LLaVA-Med-1.5. As shown, the model often fails to focus on the correct RoIs and generates hallucinated outputs. Even with attention tuning via ControlMLLM, hallucinations persist. For example, in Case (B.3), while the model identifies an abnormality in the liver, it incorrectly classifies it as a liver abscess instead of liver cancer.

These attention biases are common issues in Med-LVLMs, underscoring the need for continued research on emergent Med-LVLMs to mitigate such challenges effectively.

G.2 Experiment Results on LLaVA-Med-1.5

Similar to the LLaVA-Med experiments in Section 5.3, Table 11 and Table 12 show that A³TUNE outperforms all baselines across almost all metrics on both IU-Xray and MIMIC-CXR, excelling in both language quality and clinical accuracy. These results underscore A³TUNE’ effectiveness in medical applications across diverse Med-LVLMs, demonstrating its strong generalization ability.

Reviewer paper summaries: This paper introduces A³TUNE, a framework designed to enhance the performance of medical vision-language models (Med-LVLMs). In particular, A³TUNE employs three key strategies: (1) zero-shot weak label generation using SAM and BioMedCLIP to guide attention toward relevant regions without manual annotations; (2) selective tuning of visually-critical attention heads to balance alignment and model stability; and (3) an A³MoE module enabling adaptive parameter selection for dynamic attention adjustment across diverse prompts and images. Extensive experiments on five medical VQA benchmarks and two report generation tasks demonstrate that A³TUNE outperforms state-of-the-art baselines, achieving higher accuracy and improved attention distribution.
Summary of strengths: Overall Readability: The paper is generally well-written and easy to read, with clear organization.

Sound Rationale: The core idea of using weakly supervised labels to enhance model alignment and understanding, combined with parameter-efficient fine-tuning and Mixture-of-Experts (MoE) for performance optimization, is technical sound. While these techniques are not novel, their integration into medical vision-language models (Med-LVLMs) addresses a critical gap in the field.

Extensive Comparisons: The authors comprehensively compare their method against numerous baselines across diverse medical tasks (e.g., VQA, report generation), demonstrating robustness and superiority.

Thorough Ablation Analysis: Key ablation experiments are conducted to validate the contributions of each component (e.g., weak label selection, MoE design).
Summary of weaknesses: 1. The authors use LLaVA-Med as their baseline method, which is a relatively outdated model. I think their proposed approach could be applied to newer Med-LVLMs. Conducting additional experiments on these models would further validate the generalizability of their method.

2. While the authors improve weak label supervision (e.g., Eq. 10), it would be informative to see ablation results using simpler losses like L1 or L2. This could clarify whether their specific alignment mechanism truly outperforms traditional approaches.

3. For normal images (without diseases), how do the automatically generated weak labels guide attention? Will the incorrect guidance impair model performance? The paper focuses on abnormal cases, but understanding how the model handles normal scenarios is critical for clinical reliability.
Comments, suggestions and typos: No additional comments.
Overall assessment scores: 3
Metareview score: 4
Metareview: This paper introduces A³TUNE, a framework designed to enhance the performance of medical vision-language models (Med-LVLMs). In particular, A³TUNE employs three key strategies:

(1) zero-shot weak label generation using SAM and BioMedCLIP to guide attention toward relevant regions without manual annotations; 

(2) selective tuning of visually-critical attention heads to balance alignment and model stability; 

(3) an A³MoE module enabling adaptive parameter selection for dynamic attention adjustment across diverse prompts and images. 

Extensive experiments on five medical VQA benchmarks and two report generation tasks demonstrate that A³TUNE outperforms state-of-the-art baselines, achieving higher accuracy and improved attention distribution.
Summary of reasons to publish: The core idea of using weakly supervised labels to enhance model alignment and understanding, combined with parameter-efficient fine-tuning and Mixture-of-Experts (MoE) for performance optimization, is technical sound. While these techniques are not novel, their integration into medical vision-language models (Med-LVLMs) addresses a critical gap in the field.

Extensive Comparisons: The authors comprehensively compare their method against numerous baselines across diverse medical tasks (e.g., VQA, report generation), demonstrating robustness and superiority.

Ablation Analysis: Key ablation experiments are conducted to validate the contributions of each component (e.g., weak label selection, MoE design).