CLAIM: Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Intervention
Authors: Zekai Ye, Qiming Li, Xiaocheng Feng, Libo Qin, Yichong Huang, Baohang Li, Kui Jiang, Yang Xiang, Zhirui Zhang, Yunfei Lu, Duyu Tang, Dandan Tu, Bing Qin
Accepted at: acl25

CLAIM: Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Intervention

Zekai Ye1*, Qiming Li1*, Xiaocheng Feng1,2, Libo Qin3, Yichong Huang1, Baohang Li1, Kui Jiang1, Yang Xiang2, Zhirui Zhang4, Yunfei Lu4, Duyu Tang4, Dandan Tu4, Bing Qin1,2
1Harbin Institute of Technology 2Peng Cheng Laboratory 3Central South University 4Huawei Technologies Co., Ltd {zkye,qmli,xcfeng}@ir.hit.edu.cn

Abstract

Large Vision-Language Models (LVLMs) have demonstrated impressive multimodal abilities but remain prone to multilingual object hallucination, with a higher likelihood of generating responses inconsistent with the visual input when utilizing queries in non-English languages compared to English. Most existing approaches to address these rely on pre-training or fine-tuning, which are resource-intensive. In this paper, inspired by observing the disparities in cross-modal attention patterns across languages, we propose Cross-Lingual Attention Intervention for Mitigating Multilingual Object Hallucination (CLAIM) in LVLMs, a novel near training-free method by aligning attention patterns. CLAIM first identifies language-specific cross-modal attention heads, then estimates language shift vectors from English to the target language, and finally intervenes in the attention outputs during inference to facilitate cross-lingual visual perception capability alignment. Extensive experiments demonstrate that CLAIM achieves an average improvement of 13.56% (up to 30% in Spanish) on the POPE and 21.75% on the hallucination subsets of the MME benchmark across various languages. Further analysis reveals that multilingual attention divergence is most prominent in intermediate layers, highlighting their critical role in multilingual scenarios.

1 Introduction

Large Vision-Language Models (LVLMs) have made significant strides in bridging visual and textual content (Bai et al., 2023b; Liu et al., 2024c; Ye et al., 2023), leading to notable developments in numerous downstream tasks (Shah et al., 2023; Zhu et al., 2023; Zhang et al., 2024). However, LVLMs still suffer from serious object hallucination, i.e., generating responses that are inconsistent with the visual input (S. et al., 2023; Z. et al., 2024; Huang et al., 2023), such as misidentifying the presence of objects in an image or providing inaccurate descriptions of their attributes. This issue becomes even more severe when processing non-English queries (Schneider and Sitaram, 2024; Qu et al., 2024; Romero et al., 2024), a challenge referred to as multilingual object hallucination in LVLMs.

Recent research (Qu et al., 2024) focus on mitigating multilingual object hallucination in LVLMs via adopting Supervised Fine-Tuning (SFT) (Liu et al., 2023) and Direct Preference Optimization (DPO) (Zhao et al., 2023). However, these technologies rely on large-scale annotated image-text datasets, which are extremely expensive, time-consuming, and computation-consuming for non-English especially low-resource languages. Therefore, it is urgent to develop a training-free method for mitigating multilingual object hallucination, with further understanding of the behavioral disparities of LVLMs in multilingual scenarios.

Inspired by prior research (Liu et al., 2025; Bi et al., 2024; Chen et al., 2024b; Jiang et al., 2024) highlighting the crucial role of attention in bridging textual and visual information, we discover the significant difference in attention patterns of LVLMs across languages. Specifically, as illustrated in Figure 1, the model pays attention to distinct areas of the image when processing queries in different

![A comparison of attention weights map between Chinese and English query. In English query, LVLM correctly focuses on the key object "bird" in the image, leading to an accurate response. However, in Chinese query, the model exhibits hallucination.](page_1012_384_393_180.png)

Figure 1: A comparison of attention weights map between Chinese and English query. In English query, LVLM correctly focuses on the key object "bird" in the image, leading to an accurate response. However, in Chinese query, the model exhibits hallucination.
languages. Under non-English queries, LVLMs’ intermediate layers can even exhibit approximately a 32% decrease in attention to image regions relevant to queries (§5.1). This finding motivates us to guide the inference process of LVLMs for non-English queries by leveraging the cross-modal attention patterns in English scenarios, as LVLMs are typically well-trained on large English image-text data and perform best in English.

To this end, we propose Cross-Lingual Attention Intervention for Mitigating Multilingual Object Hallucination (CLAIM) in LVLMs, a near training-free, plug-and-play method that is applicable during the inference stage. We first identify language-specific cross-modal attention heads, i.e., the attention heads behaving quite differently for visual tokens in the same meaning queries across various languages. Next, we estimate language shift vectors for caption queries of images from English to the target language. During the inference stage, we apply shift vectors to intervene in attention outputs of these heads to align with English visual perception capabilities, reducing the likelihood of multilingual object hallucination.

Experiments conducted on LLaVA-1.5 (Liu et al., 2024a) and Qwen-VL-Chat (Bai et al., 2023b) demonstrate that CLAIM results in an average improvement of 13.56% on the POPE (Li et al., 2023) benchmark and 21.75% on the hallucination subsets of the MME (Fu et al., 2023).

Our contributions are summarized as follows:
• We reveal significant cross-modal attention divergence across languages in LVLMs.
• We propose CLAIM, a novel inference-time method that aligns non-English attention patterns with English, mitigating multilingual object hallucination with low cost.
• We analyze LVLMs’ attention patterns in multilingual scenarios, highlighting the role of intermediate layers in cross-modal inference.

2 Related Work

Multilingual Large Vision-Language Models
Leveraging the advanced capabilities of Large Language Models (LLMs) (Touvron et al., 2023a; Chiang et al., 2023; Qin et al., 2025), Large Vision-Language Models (LVLMs) (Yin et al., 2023; Wu et al., 2023) integrate visual encoders (Dosovitskiy, 2020; Radford et al., 2021) and feature projectors, allowing them to process and generate content from both visual and textual inputs. Built on the strong multilingual language model LLaMA-2 (Touvron et al., 2023b), which is trained on a diverse multilingual corpus, LLaVA-1.5 (Liu et al., 2024a) is inherently a multilingual LVLM. Qwen-VL-Chat (Bai et al., 2023b) is another multilingual LVLM with an English-centric design, trained on a large corpus of Chinese-language data and built upon Qwen (Bai et al., 2023a). Existing research (Geigle et al., 2023; Andersland, 2024; Maaz et al., 2024) employ training-based approaches to enhance the multilingual capabilities of LVLMs. Despite significant progress, LVLMs still struggle with multilingual object hallucination, limiting their global applicability in diverse countries and languages.

Hallucination in LVLMs
LVLMs often generate text outputs that are inconsistent with the visual input, a issue commonly referred to as the hallucination phenomenon. To mitigate hallucination, some methods focusing on the training phase utilize instruction (Liu et al., 2023), reinforcement learning with human/AI feedback (Yu et al., 2024), or model structure enhancement (Chen et al., 2024a). Another line of methods (Leng et al., 2024; Chen et al., 2024c; Zhong et al., 2024; Huang et al., 2024) reduce the likelihood of hallucination by performing conservative decoding on the original inputs and the inputs with disturbed contents. However, the above approaches proposed for mitigating hallucination only focus on their effectiveness in English. MHR (Qu et al., 2024) first attempts to mitigate multilingual object hallucination by SFT and DPO. In this paper, we propose a novel method for mitigating multilingual object hallucination without datasets construction and training.

3 Methodology

In this section, we first introduce the overall process of CLAIM in Figure 2, followed by the preliminary for the attention mechanism of LVLMs. Then, we describe the three parts of CLAIM in detail.

3.1 Preliminary
Modern LVLMs (Bai et al., 2023b; Liu et al., 2024a) typically comprise three key components: a visual encoder, a feature projector, and a language decoder. Specifically, the visual encoder first transforms the input visual image into visual features. The feature projector then maps these features to the input space of the language decoder, producing the visual embeddings \( P = \{p_i\}_{i=0}^n \), where \( p_i \) represents the visual embedding corresponding to the
Figure 2: Overview of our proposed CLAIM method. A block of MHA in the figure represents a attention head. CLAIM intervene in identified language-specific cross-modal attention heads using estimated language shift vectors. (1) Identification of Language-Specific Cross-Modal Attention Heads §3.2: We train probes to identify the language-specific cross-modal attention heads, which exhibit significantly different behavior across languages associated with visual perception. (2) Estimation of Language Shift Vectors §3.3: We estimate the language shift vectors in attention outputs from English to the target non-English language for identical images queried with captions. (3) Intervention during Inference §3.4: During inference, we apply language shift vectors to intervene in the language-specific cross-modal attention heads for mitigating multilingual object hallucination.

i-th image patch, and \( n \) denotes the total number of patches. Similarly, the textual input is mapped to textual embeddings \( T = \{ t_i \}_{i=0}^m \), where \( t_i \) corresponds to the i-th textual token, and \( m \) represents the number of text tokens. The visual and textual embeddings are then concatenated to form the input embeddings \( X = [P, T] \) for the language decoder.

During the forward pass of the language decoder, the input embeddings \( X \) serve as the hidden states for the first self-attention layer (Vaswani, 2017). The h-th attention head in the l-th self-attention layer applies linear transformations to project the hidden states into queries \( Q_h^l \in \mathbb{R}^{e \times d} \), keys \( K_h^l \in \mathbb{R}^{e \times d} \), and values \( V_h^l \in \mathbb{R}^{e \times d} \). Here, \( e = n + m \) and \( d \) denotes the head-specific hidden dimension. The attention scores \( A_h^l \in \mathbb{R}^{e \times e} \) are then calculated based on \( Q_h^l \) and \( K_h^l \) as follows:

\[
\tilde{A}_h^l = \operatorname{softmax}(A_h^l + M), \quad A_h^l = \frac{Q_h^l K_h^{lT}}{\sqrt{d}}, \tag{1}
\]
\[
M[i, j] = \begin{cases}
0 & \text{if } j \leq i \\
-\infty & \text{if } j > i
\end{cases} \tag{2}
\]

where \( M \) is the causal mask matrix. The attention weights \( \tilde{A}_h^l \) estimate the relevance of each token, which are used to reweight the values \( V_h^l \) from each token, producing the attention outputs \( O_h^l \in \mathbb{R}^{e \times d} \),

\[
O_h^l = \tilde{A}_h^l V_h^l. \tag{3}
\]

At each layer, the hidden states pass through multi-head attention (MHA), which comprises \( H \) independent attention heads, each performing separate linear transformations. Specifically, the MHA mechanism can be formulated as:

\[
X^{l+1} = X^l + \sum_{h=1}^H O_h^l W_h^l,
\]

where \( W_h^l \in \mathbb{R}^{d \times Hd} \) maps d-dimensional attention outputs of heads into hidden state representations, which are then fed into a standard multilayer perceptron (MLP) for further processing. Finally, the hidden state of the last token is decoded into a next-token prediction distribution.

3.2 Identification of Language-Specific Cross-Modal Attention Heads

Since LVLMs generate tokens in an auto-regressive manner, our method focuses on the attention matrices of the last input token, \( A_h^l[e] \), which aggregates the most comprehensive visual and textual information. We mask \( A_h^l[e] \) to exclude attention toward all textual tokens, allowing us to identify the language-specific cross-modal attention heads, which capture the variations in cross-modal attention patterns where semantically equivalent text in different languages attends to the same image. If we do not exclude the attention from the last input token to preceding text tokens, it would introduce
substantial text-language-specific features, leading to the identification of attention heads specialized for the input text’s language, which diverges from the primary motivation and purpose of our work.

\[
\widehat{M}[i, j] = \begin{cases}
0 & \text{if } j \leq i \\
-\infty & \text{if } j > i \text{ or } (i = e \text{ and } j > n)
\end{cases}
\]
(5)

\[
\widehat{O}_h^l = \operatorname{softmax}(\mathcal{A}_h^l + \widehat{M})V_h^l.
\]
(6)

For each image \( P_i \), we construct caption queries in both English and the target non-English language. A caption query refers to a request where the goal is to generate a textual description (a caption) for a given image which designed to stimulate LVLMs’ visual perception capabilities and contributes to mitigating multilingual object hallucination. The English query, \( T_{en} \), is “What is it in the image?”, while \( T_{tgt} \) is its translation into the target language. Then, both queries are fed into the model along with the image for the standard inference process, deriving \( x_i \in \left\{ \widehat{O}_{i,h}^{en,l}[e_i], \widehat{O}_{i,h}^{tgt,l}[e_i] \right\} \).

Probe (Li et al., 2024) \( f_h^* \) is a binary classifier, trained to predict language labels based on \( x_i \). The language labels \( y_i \in \{1, -1\} \) corresponds to English and the target language respectively. We train probes using \( B \) samples for each attention head \( H_h^l \). Finally, we evaluate probes using test samples, identifying language-specific cross-modal attention heads for Top-K classification accuracy. The formulas are summarized as:

\[
f_h^* = \arg\min_{f_h^l} \sum_{i=1}^B \mathcal{L}\left(f_h^l(x_i), y_i\right),
\]
(7)

\[
H_s = \{H_h^l | H_h^l \in TopK(\operatorname{Acc}(f_h^*))\},
\]
(8)

where \( \mathcal{L} \) is the loss function of probes and \( H_s \) is the set of language-specific cross-modal attention heads, \( K \) denoted as the number of selected heads.

3.3 Estimation of Language Shift Vectors

Given the sets \( \{(T_{en}, P_i)\}_{i=1}^B \) and \( \{(T_{tgt}, P_i)\}_{i=1}^B \), we derive \( \{O_{i,h}^{en,l}\}_{i=1}^B \) and \( \{O_{i,h}^{tgt,l}\}_{i=1}^B \) respectively through the standard inference process of the model, estimating the language shift vectors \( S_h^l \):

\[
S_h^l = \frac{1}{B} \sum_{i=1}^B \left(O_{i,h}^{en,l}[e_i] - O_{i,h}^{tgt,l}[e_i]\right).
\]
(9)

The shifts estimate the attention disparities between English and the target language for visual perception alignment. Notably, we do not use \( \widehat{O}_{i,h}^{en,l}, \widehat{O}_{i,h}^{tgt,l} \) to estimate language shift vectors as these matrices are not directly derived from the standard inference process. Instead, in order to preserve the original representation space of the model, we opt to utilize \( O_{i,h}^{en,l} \) and \( O_{i,h}^{tgt,l} \) providing more reliable shifts. As the visual and textual representations in \( O_h^l \) are not orthogonal, the language shift vectors inherently capture multimodal information understanding which aids in mitigating multilingual object hallucination.

3.4 Intervention during Inference

Finally, we apply language shift vectors to intervene in language-specific cross-modal attention heads during inference in non-English queries:

\[
X^{l+1} = X^l + \sum_{h=1}^H (O_h^l + I_h^l \alpha S_h^l)W_h^l.
\]
(10)

\( I_h^l \) is an indicator function, which is 1 if \( H_h^l \in H_s \) and 0 otherwise, and \( \alpha \) denotes the intensity of the intervention. After intervention, LVLMs leverage their strongest English visual perception proficiency even when processing non-English queries. Since the intervention are pre-computed, CLAIM hardly incurs additional latency during the inference stage. We estimate the inference speed and discuss the results in Appendix D.

4 Experiment

4.1 Datasets

POPE POPE (Li et al., 2023) is designed to evaluate object hallucination in the VQA paradigm. It queries LVLMs about the presence of specific objects in a given image while maintaining a balanced 1:1 ratio between existent and non-existent objects. The benchmark employs three distinct sampling strategies for negative samples - random, popular, adversarial - with their difficulty levels increasing in that order. POPE integrates data from three major repositories: MSCOCO (Lin et al., 2014), AOKVQA (Schwenk et al., 2022), and GQA (Hudson and Manning, 2019). Evaluation is conducted using accuracy as the primary metric. As the original POPE benchmark is available only in English, we translating all queries into multiple languages using Google Translate and meticulously refine the translation results to maintain superior benchmark quality. Since the text is fairly simple, we directly use Google Translate and find it feasible to verify the translation quality, as detailed in Appendix B.
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Setup</th>
    <th rowspan="2">Method</th>
    <th colspan="6">LLaVA-1.5</th>
    <th colspan="6">Qwen-VL-Chat</th>
  </tr>
  <tr>
    <th>En</th><th>Zh</th><th>Es</th><th>Ru</th><th>Pt</th><th>Bg</th><th>Avg.</th>
    <th>En</th><th>Zh</th><th>Es</th><th>Ru</th><th>Hi</th><th>De</th><th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="3">Random</td>
    <td></td>
    <td><i>Baseline</i></td>
    <td>88.50</td><td>81.00</td><td>63.03</td><td>72.33</td><td>78.97</td><td>72.23</td><td>73.51</td>
    <td>86.63</td><td>84.57</td><td>68.13</td><td>76.83</td><td>56.97</td><td>77.53</td><td>72.81</td>
  </tr>
  <tr>
    <td></td>
    <td>VCD</td>
    <td>-</td><td>81.47</td><td>67.40</td><td>73.33</td><td>78.07</td><td>72.47</td><td>74.55</td>
    <td>-</td><td>84.70</td><td>72.17</td><td>75.37</td><td>48.37</td><td>78.27</td><td>71.78</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td>-</td><td>86.50</td><td>87.33</td><td>87.50</td><td>85.40</td><td>80.50</td><td>85.45</td>
    <td>-</td><td>88.03</td><td>86.93</td><td>82.60</td><td>67.57</td><td>88.07</td><td>82.64</td>
  </tr>
  <tr>
    <td rowspan="3">COCO</td>
    <td>Popular</td>
    <td><i>Baseline</i></td>
    <td>87.43</td><td>83.07</td><td>62.93</td><td>69.20</td><td>82.03</td><td>71.17</td><td>73.68</td>
    <td>85.67</td><td>83.40</td><td>68.03</td><td>75.73</td><td>62.20</td><td>77.20</td><td>73.31</td>
  </tr>
  <tr>
    <td></td>
    <td>VCD</td>
    <td>-</td><td>83.20</td><td>67.20</td><td>70.17</td><td>80.10</td><td>69.47</td><td>74.03</td>
    <td>-</td><td>82.70</td><td>72.70</td><td>74.20</td><td>57.40</td><td>76.70</td><td>72.74</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td>-</td><td>88.00</td><td>87.53</td><td>84.03</td><td>86.13</td><td>80.07</td><td>85.15</td>
    <td>-</td><td>85.47</td><td>85.90</td><td>80.63</td><td>69.23</td><td>86.43</td><td>81.35</td>
  </tr>
  <tr>
    <td rowspan="3">Adversarial</td>
    <td></td>
    <td><i>Baseline</i></td>
    <td>85.20</td><td>73.40</td><td>62.87</td><td>66.07</td><td>73.70</td><td>65.93</td><td>68.39</td>
    <td>83.90</td><td>80.87</td><td>67.53</td><td>72.80</td><td>63.10</td><td>75.90</td><td>72.04</td>
  </tr>
  <tr>
    <td></td>
    <td>VCD</td>
    <td>-</td><td>74.27</td><td>67.30</td><td>66.47</td><td>73.63</td><td>66.33</td><td>69.60</td>
    <td>-</td><td>79.40</td><td>71.97</td><td>70.83</td><td>54.73</td><td>75.03</td><td>70.39</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td>-</td><td>77.67</td><td>83.27</td><td>79.43</td><td>79.67</td><td>74.57</td><td>78.92</td>
    <td>-</td><td>82.33</td><td>80.70</td><td>77.53</td><td>69.70</td><td>81.50</td><td>78.35</td>
  </tr>
  <tr>
    <td rowspan="3">Random</td>
    <td></td>
    <td><i>Baseline</i></td>
    <td>91.00</td><td>76.13</td><td>63.40</td><td>70.30</td><td>75.03</td><td>69.23</td><td>70.82</td>
    <td>88.47</td><td>85.10</td><td>67.70</td><td>74.63</td><td>59.53</td><td>75.17</td><td>72.43</td>
  </tr>
  <tr>
    <td></td>
    <td>VCD</td>
    <td>-</td><td>77.93</td><td>68.17</td><td>72.17</td><td>75.67</td><td>70.33</td><td>72.85</td>
    <td>-</td><td>84.30</td><td>72.23</td><td>73.67</td><td>49.07</td><td>75.33</td><td>70.92</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td>-</td><td>86.80</td><td>85.97</td><td>85.63</td><td>83.87</td><td>80.63</td><td>84.58</td>
    <td>-</td><td>88.13</td><td>86.03</td><td>83.47</td><td>68.83</td><td>86.13</td><td>82.52</td>
  </tr>
  <tr>
    <td rowspan="3">OKVQA</td>
    <td>Popular</td>
    <td><i>Baseline</i></td>
    <td>86.97</td><td>74.50</td><td>63.27</td><td>69.40</td><td>77.03</td><td>67.73</td><td>70.39</td>
    <td>88.70</td><td>85.60</td><td>67.90</td><td>75.43</td><td>58.87</td><td>75.27</td><td>72.61</td>
  </tr>
  <tr>
    <td></td>
    <td>VCD</td>
    <td>-</td><td>77.57</td><td>67.10</td><td>68.40</td><td>77.03</td><td>68.20</td><td>71.66</td>
    <td>-</td><td>84.17</td><td>71.57</td><td>73.30</td><td>54.20</td><td>75.63</td><td>71.41</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td>-</td><td>85.00</td><td>83.40</td><td>81.87</td><td>84.27</td><td>77.97</td><td>82.50</td>
    <td>-</td><td>88.10</td><td>85.70</td><td>83.90</td><td>68.50</td><td>86.03</td><td>82.45</td>
  </tr>
  <tr>
    <td rowspan="3">Adversarial</td>
    <td></td>
    <td><i>Baseline</i></td>
    <td>79.57</td><td>64.40</td><td>62.37</td><td>63.93</td><td>67.97</td><td>63.03</td><td>64.34</td>
    <td>82.40</td><td>79.97</td><td>66.80</td><td>72.50</td><td>60.17</td><td>72.30</td><td>70.35</td>
  </tr>
  <tr>
    <td></td>
    <td>VCD</td>
    <td>-</td><td>68.97</td><td>65.17</td><td>63.90</td><td>68.67</td><td>64.07</td><td>66.16</td>
    <td>-</td><td>78.77</td><td>69.00</td><td>69.63</td><td>50.80</td><td>71.90</td><td>68.02</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td>-</td><td>77.70</td><td>75.40</td><td>76.17</td><td>75.67</td><td>71.97</td><td>75.38</td>
    <td>-</td><td>81.10</td><td>77.20</td><td>78.73</td><td>67.73</td><td>78.87</td><td>76.73</td>
  </tr>
  <tr>
    <td rowspan="3">Random</td>
    <td></td>
    <td><i>Baseline</i></td>
    <td>89.47</td><td>77.03</td><td>63.83</td><td>70.23</td><td>74.87</td><td>68.07</td><td>70.81</td>
    <td>87.23</td><td>82.53</td><td>73.03</td><td>74.00</td><td>65.10</td><td>74.30</td><td>73.95</td>
  </tr>
  <tr>
    <td></td>
    <td>VCD</td>
    <td>-</td><td>78.00</td><td>68.33</td><td>71.47</td><td>73.63</td><td>70.20</td><td>72.33</td>
    <td>-</td><td>83.63</td><td>76.10</td><td>73.43</td><td>55.50</td><td>80.20</td><td>73.77</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td>-</td><td>84.27</td><td>85.13</td><td>83.57</td><td>82.53</td><td>82.67</td><td>83.63</td>
    <td>-</td><td>85.17</td><td>79.93</td><td>81.90</td><td>62.03</td><td>80.40</td><td>77.89</td>
  </tr>
  <tr>
    <td rowspan="3">GQA</td>
    <td>Popular</td>
    <td><i>Baseline</i></td>
    <td>83.90</td><td>69.60</td><td>64.03</td><td>64.53</td><td>71.53</td><td>63.40</td><td>66.62</td>
    <td>85.80</td><td>81.90</td><td>72.17</td><td>75.07</td><td>59.20</td><td>70.40</td><td>71.93</td>
  </tr>
  <tr>
    <td></td>
    <td>VCD</td>
    <td>-</td><td>73.60</td><td>67.50</td><td>65.23</td><td>71.87</td><td>66.20</td><td>68.88</td>
    <td>-</td><td>82.20</td><td>73.73</td><td>74.17</td><td>54.70</td><td>76.87</td><td>72.33</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td>-</td><td>81.57</td><td>78.50</td><td>77.93</td><td>79.60</td><td>80.57</td><td>79.63</td>
    <td>-</td><td>84.43</td><td>80.60</td><td>82.60</td><td>61.30</td><td>78.07</td><td>77.40</td>
  </tr>
  <tr>
    <td rowspan="3">Adversarial</td>
    <td></td>
    <td><i>Baseline</i></td>
    <td>81.17</td><td>63.17</td><td>63.30</td><td>64.10</td><td>67.90</td><td>62.10</td><td>64.11</td>
    <td>82.63</td><td>78.97</td><td>69.73</td><td>73.07</td><td>61.73</td><td>71.63</td><td>71.52</td>
  </tr>
  <tr>
    <td></td>
    <td>VCD</td>
    <td>-</td><td>67.33</td><td>66.97</td><td>63.87</td><td>67.83</td><td>63.37</td><td>65.87</td>
    <td>-</td><td>79.57</td><td>73.47</td><td>71.83</td><td>54.97</td><td>76.17</td><td>71.20</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td>-</td><td>77.57</td><td>75.93</td><td>76.27</td><td>75.23</td><td>76.47</td><td>76.29</td>
    <td>-</td><td>79.97</td><td>76.77</td><td>79.53</td><td>62.47</td><td>77.63</td><td>75.27</td>
  </tr>
  <tr>
    <td rowspan="3">Existence</td>
    <td></td>
    <td><i>Baseline</i></td>
    <td>190.0</td><td>175.0</td><td>130.0</td><td>145.0</td><td>155.0</td><td>125.0</td><td>146.0</td>
    <td>185.0</td><td>190.0</td><td>135.0</td><td>140.0</td><td>106.7</td><td>195.0</td><td>153.3</td>
  </tr>
  <tr>
    <td></td>
    <td>VCD</td>
    <td>-</td><td>180.0</td><td>130.0</td><td>145.0</td><td>150.0</td><td>128.3</td><td>146.7</td>
    <td>-</td><td>185.0</td><td>155.0</td><td>155.0</td><td>78.30</td><td>195.0</td><td>153.7</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td>-</td><td>195.0</td><td>175.0</td><td>185.0</td><td>175.0</td><td>180.0</td><td>182.0</td>
    <td>-</td><td>190.0</td><td>155.0</td><td>185.0</td><td>128.3</td><td>195.0</td><td>170.7</td>
  </tr>
  <tr>
    <td rowspan="3">Count</td>
    <td></td>
    <td><i>Baseline</i></td>
    <td>155.0</td><td>70.00</td><td>55.00</td><td>80.00</td><td>85.00</td><td>69.67</td>
    <td>150.0</td><td>130.0</td><td>143.3</td><td>113.3</td><td>60.00</td><td>136.7</td><td>116.7</td>
  </tr>
  <tr>
    <td></td>
    <td>VCD</td>
    <td>-</td><td>73.30</td><td>73.30</td><td>53.30</td><td>61.70</td><td>105.0</td><td>73.33</td>
    <td>-</td><td>140.0</td><td>131.7</td><td>100.0</td><td>80.00</td><td>128.3</td><td>116.0</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td>-</td><td>125.0</td><td>130.0</td><td>110.0</td><td>130.0</td><td>135.0</td><td>126.0</td>
    <td>-</td><td>148.3</td><td>153.3</td><td>120.0</td><td>111.7</td><td>137.0</td><td>134.0</td>
  </tr>
  <tr>
    <td rowspan="3">MME</td>
    <td>Color</td>
    <td><i>Baseline</i></td>
    <td>165.0</td><td>80.00</td><td>135.0</td><td>75.00</td><td>110.0</td><td>80.00</td><td>96.00</td>
    <td>180.0</td><td>170.0</td><td>150.0</td><td>153.3</td><td>103.3</td><td>165.0</td><td>148.3</td>
  </tr>
  <tr>
    <td></td>
    <td>VCD</td>
    <td>-</td><td>95.00</td><td>145.0</td><td>85.00</td><td>130.0</td><td>88.30</td><td>108.7</td>
    <td>-</td><td>150.0</td><td>165.0</td><td>146.7</td><td>93.30</td><td>160.0</td><td>143.7</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td>-</td><td>120.0</td><td>155.0</td><td>125.0</td><td>150.0</td><td>108.3</td><td>131.7</td>
    <td>-</td><td>170.0</td><td>165.0</td><td>158.3</td><td>90.00</td><td>165.0</td><td>149.7</td>
  </tr>
  <tr>
    <td rowspan="3">Position</td>
    <td></td>
    <td><i>Baseline</i></td>
    <td>118.3</td><td>53.30</td><td>63.30</td><td>55.00</td><td>50.00</td><td>46.70</td><td>53.67</td>
    <td>131.7</td><td>63.30</td><td>120.0</td><td>93.30</td><td>45.00</td><td>105.0</td><td>85.33</td>
  </tr>
  <tr>
    <td></td>
    <td>VCD</td>
    <td>-</td><td>48.30</td><td>93.00</td><td>60.00</td><td>51.70</td><td>56.70</td><td>61.93</td>
    <td>-</td><td>78.30</td><td>101.7</td><td>101.7</td><td>43.30</td><td>93.30</td><td>83.67</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td>-</td><td>66.70</td><td>78.30</td><td>85.00</td><td>86.70</td><td>58.30</td><td>75.00</td>
    <td>-</td><td>63.30</td><td>116.7</td><td>103.3</td><td>55.00</td><td>106.0</td><td>88.86</td>
  </tr>
  <tr>
    <td rowspan="3">Total Scores</td>
    <td></td>
    <td><i>Baseline</i></td>
    <td>628.3</td><td>378.3</td><td>383.3</td><td>333.3</td><td>395.0</td><td>336.7</td><td>365.3</td>
    <td>646.7</td><td>553.3</td><td>548.3</td><td>500.0</td><td>315.0</td><td>601.7</td><td>503.7</td>
  </tr>
  <tr>
    <td></td>
    <td>VCD</td>
    <td>-</td><td>396.7</td><td>441.3</td><td>343.3</td><td>393.3</td><td>378.3</td><td>390.6</td>
    <td>-</td><td>553.3</td><td>556.7</td><td>503.3</td><td>295.0</td><td>576.7</td><td>497.0</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td>-</td><td>506.7</td><td>538.3</td><td>505.0</td><td>541.7</td><td>481.7</td><td>514.7</td>
    <td>-</td><td>571.7</td><td>590.0</td><td>566.6</td><td>385.0</td><td>603.0</td><td>543.3</td>
  </tr>
</table>

Table 1: Main results on POPE from COCO, OKVQA, GQA and the hallucination subsets of MME.

![Bar chart showing average scores for LLaVA-1.5 across five languages on the MME full dataset.](page_1342_1812_377_246.png)

Figure 3: Average scores for LLaVA-1.5 across five languages on the MME full dataset.
Figure 4: Logit lens observation for interpreting LLaVA-1.5 in multilingual scenarios. The depth of block color for the \( i \)-th token at layer \( l \) indicates the magnitude of its contribution to the logits of the final predicted token. The color represents the corresponding tokens in image or text. The query means "Is there a car in the image?".

MME  The MME dataset (Fu et al., 2023) serves as a comprehensive benchmark for evaluating LVLMs, including 14 subtasks designed to assess both the perceptual and cognitive abilities of LVLMs. Performance is measured based on the sum of accuracy scores across individual questions and images. Following Leng et al. (2024), in addition to adapting the full dataset, we focus specifically on the existence and count subsets for object-level hallucination evaluation, as well as the position and color subsets for attribute-level assessment. Similar to POPE, we translate MME into seven languages using Google Translate and correct the mistakes by human check.

4.2 Models and Implementation Details

Following prior research, we adopt the widely used LLaVA-1.5-7b (Liu et al., 2024a) and Qwen-VL-Chat (Bai et al., 2023b) as our baseline LVLMs. More LVLM results can be found in Appendix C. Since no existing training-free methods tailored to mitigating multilingual object hallucination, we employ the widely used method, VCD (Leng et al., 2024), as a strong baseline for comparison. We sample 1,000 images from the COCO-2017 training dataset to complete identification and estimation, discussing the impact of training set size in Appendix F. To determine optimal hyperparameter values for \( \alpha \) (intervention intensity) and \( K \) (the number of heads involved in the intervention), we employ a sequential optimization approach. Additional details are provided in Appendix A.

4.3 Main Results

Results on POPE. (1) CLAIM effectively mitigates object-level hallucination by aligning the strong visual perception capability used for processing English queries with that used for processing non-English queries. As shown in Table 1, the intervention achieves an average improvement of 17.5% over the baseline on LLaVA-1.5 and 9.8% on Qwen-VL-Chat across various languages and settings. (2) CLAIM enhances performance across both low- and high-resource non-English languages. This improvement can be attributed to its robust ability to facilitate cross-lingual visual perception capability alignment almost regardless of the language’s resource availability. (3) The intervention yields improvements across datasets with different distributions, suggesting that the intervention represents a generalizable direction to mitigating multilingual object hallucination rather than merely tailored to a specific dataset.

Results on MME. This subset extends beyond POPE’s scope, encompassing both object-level and attribute-level hallucinations (1) CLAIM effectively reduces both object-level and attribute-level hallucination. As shown in Table 1, CLAIM achieves an average improvement of 40.9% on LLaVA-1.5 and 7.9% on Qwen-VL-Chat over the baseline across various languages, outperforming VCD. Specifically, CLAIM not only mitigates object-level hallucination, as evidenced by the results on the existence and count subsets, but also mitigates attribute-level hallucination, as demonstrated by the color and position subsets. Detailed results can be found in Appendix E. (2) CLAIM could generally facilitate cross-lingual visual perception capability alignment by intervening the attention patterns, enabling LVLMs to transfer their English proficiency across various tasks in multilingual queries. Illustrated in Figure 3, the intervention significantly enhances perception-based tasks and generalizes well to cognitive reasoning tasks, as strong image perception serves as the foundation for cognitive processing. Meanwhile, our method primarily activates attention heads associ-
ated with perception, which could unintentionally affect the reasoning pathways of LVLMs.

5 Analysis and Discussion

5.1 Multilingual Attention Differences
As shown in Figure 1, when queried in different languages about an object’s presence, the model exhibits distinct attention patterns across image regions. In order to quantitatively analyze multilingual cross-modal attention differences, we conduct a statistical experiment to validate our motivation. Specifically, using bounding box annotations from the POPE-COCO dataset, we localize object regions and compute the sum of attention weights within ground-truth bounding boxes for queries in different languages, obtaining \( A_b(l, \mathcal{B}) \).

\[
A_b(l, \mathcal{B}) = \frac{n}{H|\mathcal{B}|} \cdot \sum_{h=1}^{H} \left( \frac{\sum_{j \in \mathcal{B}} \widetilde{A}_h^l(e, j)}{\sum_{j=1}^{n} \widetilde{A}_h^l(e, j)} \right)
\] (11)

\( \mathcal{B} \) denotes the set of patches included in bounding boxes. The first fraction serves as a normalization factor, eliminating the influence of varying bounding box sizes associated with different queries and averaging \( H \) attention heads in each layer.

![Line graph showing attention per layer of LLaVA-1.5 across four languages, and the per-layer average change rate of non-English languages \( A_b \) relative to English.](page_324_1097_682_246.png)

Figure 5: \( A_b \) per layer of LLaVA-1.5 across four languages, and the per-layer average change rate of non-English languages \( A_b \) relative to English.

As shown in Figure 5, in the initial layers of the model (before layer 10), the model pays little attention to image regions related to queries, and there is almost no difference between multilingual queries, indicating that the model is performing basic global understanding of the image at this stage. In the middle layers (10-25), the attention to key regions in the image for non-English languages decreases significantly compared to English, suggesting that the model exhibits disparities in perceptual capabilities under different language queries. Notably, a peak appears at layer 14, even though the model’s total attention weights for visual tokens are low at this stage (shown in Figure 7 (a)). This may indicate that the model has noticed image regions relevant to queries during its internal reasoning process, performing fine-grained perception.

5.2 Multilingual Inference in LVLMs
In order to better illustrate the mechanism behind CLAIM and improve the interpretability of LVLMs in multilingual scenarios, by leveraging the logit lens (Nostalgebraist, 2020) method, which aims to decode the hidden states of the language decoder at various layers, we investigate the internal inference mechanism of LVLMs and uncover how they process and integrate multimodal information, particularly in non-English queries. The internal inference process of LLaVA-1.5 is illustrated in Figure 4 as a case study.

For multilingual VQA tasks, English-centric LVLMs face the dual challenge of not only bridging the modality gap between visual and textual information but also mapping non-English queries into the English semantic space to ensure accurate responses. **In middle layers, visual tokens are often decoded into their corresponding English concepts.** Similarly, when processing Chinese queries, the model maps them to the English semantic space at these intermediate layers such as the layer 17, consistent with the findings (Wendler et al., 2024) in multilingual LLMs. During pretraining, the alignment of vision-text modality relies heavily on English corpora, which guides LVLMs toward interpreting images through an English-centric pathway. **LVLMs interpret the important entities in the query based on the critical information from the image at intermediate stage.** The original semantic meaning of "car" in the query (such as the 620th token) is enriched to "bus" under the influence of the image information at the layer 21.

5.3 Analysis of Intervention
In order to elucidate the underlying reasons for the efficacy of CLAIM and to substantiate the robustness of our methodology, we conduct a further investigation with two questions.

*Does the intervention truly facilitate the cross-lingual alignment of cross-modal attention distribution?* Illustrated in Figure 6, under standard inference, the projection values of English and non-English vectors are distinctly separated at the zero point, indicating that the cross-modal attention patterns of LVLMs exhibit significant misalignment for identical images depending on the language used in the same meaning query. After intervention, the distributions of non-English languages shift closer to that of English, with more pronounced density peaks. This alignment supports the hypoth-
<table>
  <tr>
    <th rowspan="2">Setup</th>
    <th rowspan="2">Method</th>
    <th colspan="5">LLaVA-1.5</th>
    <th colspan="5">Qwen-VL-Chat</th>
  </tr>
  <tr>
    <th>Zh</th>
    <th>Es</th>
    <th>Ru</th>
    <th>Pt</th>
    <th>Bg</th>
    <th>Avg.</th>
    <th>Zh</th>
    <th>Es</th>
    <th>Ru</th>
    <th>Hi</th>
    <th>De</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="4">Random</td>
    <td><i>Baseline</i></td>
    <td>81.00</td>
    <td>63.03</td>
    <td>72.33</td>
    <td>78.97</td>
    <td>72.23</td>
    <td>73.51</td>
    <td>84.57</td>
    <td>68.13</td>
    <td>76.83</td>
    <td>56.97</td>
    <td>77.53</td>
    <td>72.81</td>
  </tr>
  <tr>
    <td><i>Mono-Shift</i></td>
    <td><span style="color:green;">86.50</span></td>
    <td>86.70</td>
    <td>87.40</td>
    <td>84.50</td>
    <td>76.77</td>
    <td>84.37</td>
    <td><span style="color:green;">88.03</span></td>
    <td>74.87</td>
    <td>80.67</td>
    <td>67.27</td>
    <td>83.03</td>
    <td>78.77</td>
  </tr>
  <tr>
    <td><i>Multi-Shift</i></td>
    <td>85.80</td>
    <td>87.00</td>
    <td>87.87</td>
    <td>83.60</td>
    <td>79.80</td>
    <td>84.81</td>
    <td>86.70</td>
    <td>86.40</td>
    <td>81.37</td>
    <td>63.30</td>
    <td>87.97</td>
    <td>81.15</td>
  </tr>
  <tr>
    <td><i>Specific-Shift</i></td>
    <td><span style="color:green;">86.50</span></td>
    <td>87.33</td>
    <td>87.50</td>
    <td>85.40</td>
    <td>80.50</td>
    <td>85.45</td>
    <td>88.03</td>
    <td>86.93</td>
    <td>82.60</td>
    <td>67.57</td>
    <td>88.07</td>
    <td>82.64</td>
  </tr>
  <tr>
    <td rowspan="4">Popular</td>
    <td><i>Baseline</i></td>
    <td>83.07</td>
    <td>62.93</td>
    <td>69.20</td>
    <td>82.03</td>
    <td>71.17</td>
    <td>73.68</td>
    <td>83.40</td>
    <td>68.03</td>
    <td>75.73</td>
    <td>62.20</td>
    <td>77.20</td>
    <td>73.31</td>
  </tr>
  <tr>
    <td><i>Mono-Shift</i></td>
    <td><span style="color:green;">88.00</span></td>
    <td>86.90</td>
    <td>83.00</td>
    <td>85.30</td>
    <td>79.10</td>
    <td>84.46</td>
    <td><span style="color:green;">85.47</span></td>
    <td>74.77</td>
    <td>78.80</td>
    <td>68.07</td>
    <td>81.73</td>
    <td>77.77</td>
  </tr>
  <tr>
    <td><i>Multi-Shift</i></td>
    <td>87.67</td>
    <td>87.23</td>
    <td>83.33</td>
    <td>86.17</td>
    <td>79.13</td>
    <td>84.71</td>
    <td>84.30</td>
    <td>85.03</td>
    <td>80.37</td>
    <td>68.93</td>
    <td>85.43</td>
    <td>80.81</td>
  </tr>
  <tr>
    <td><i>Specific-Shift</i></td>
    <td><span style="color:green;">88.00</span></td>
    <td>87.53</td>
    <td>84.03</td>
    <td>86.13</td>
    <td>80.07</td>
    <td>85.15</td>
    <td>85.47</td>
    <td>85.90</td>
    <td>80.63</td>
    <td>69.23</td>
    <td>86.43</td>
    <td>81.53</td>
  </tr>
  <tr>
    <td rowspan="4">Adversarial</td>
    <td><i>Baseline</i></td>
    <td>73.40</td>
    <td>62.87</td>
    <td>66.07</td>
    <td>73.70</td>
    <td>65.93</td>
    <td>68.39</td>
    <td>80.87</td>
    <td>67.53</td>
    <td>72.80</td>
    <td>63.10</td>
    <td>75.90</td>
    <td>72.01</td>
  </tr>
  <tr>
    <td><i>Mono-Shift</i></td>
    <td><span style="color:green;">77.67</span></td>
    <td>82.63</td>
    <td>78.13</td>
    <td>78.33</td>
    <td>72.60</td>
    <td>77.87</td>
    <td><span style="color:green;">82.33</span></td>
    <td>73.83</td>
    <td>76.00</td>
    <td>69.30</td>
    <td>79.57</td>
    <td>76.21</td>
  </tr>
  <tr>
    <td><i>Multi-Shift</i></td>
    <td>76.70</td>
    <td>82.13</td>
    <td>79.53</td>
    <td>78.10</td>
    <td>74.83</td>
    <td>78.66</td>
    <td>81.70</td>
    <td>80.50</td>
    <td>76.57</td>
    <td>69.97</td>
    <td>81.17</td>
    <td>77.98</td>
  </tr>
  <tr>
    <td><i>Specific-Shift</i></td>
    <td><span style="color:green;">77.67</span></td>
    <td>83.27</td>
    <td>79.43</td>
    <td>79.67</td>
    <td>74.57</td>
    <td>78.92</td>
    <td>82.33</td>
    <td>80.70</td>
    <td>77.53</td>
    <td>69.70</td>
    <td>81.50</td>
    <td>78.35</td>
  </tr>
</table>

Table 2: Evaluation results of different intervention estimation approaches on POPE-COCO. Green means the best performance while gray means the second-best results.

esis that CLAIM effectively mitigates multilingual object hallucination by reinforcing cross-lingual consistency in visual perception attention.

![Kernel density estimate plot of cross-modal attention outputs across languages of LLaVA-1.5 before and after multilingual intervention. The x-axis represents the inner product between attention outputs and the normal vector of the hyperplane, while the y-axis indicates the density of samples occurring at x value.](page_1012_1342_377_246.png)

Figure 6: Kernel density estimate plot of cross-modal attention outputs across languages of LLaVA-1.5 before and after multilingual intervention. The x-axis represents the inner product between attention outputs and the normal vector of the hyperplane, while the y-axis indicates the density of samples occurring at x value.

Does a unified multilingual intervention (Multi-Shift) work? In our main experiment, we pair English with each non-English language individually, estimating the Specific-Shift for each pair and selecting specialized attention heads to precisely align each non-English languages with English proficiency. To examine the impact of multilingual interactions, we construct a mixed set that includes all non-English languages paired with English, estimating the Multi-Shift between English and the entire non-English group. As shown in Table 2, the Multi-Shift demonstrates moderate improvements, even outperforming CLAIM in some subsets by integrating shared linguistic features. Additionally, to further validate the generalizability of the intervention, we apply the Mono-Shift intervention, derived from the English-Chinese pair, to other languages and observe performance improvements. It indicates that the cross-lingual intervention can generalize well to unseen non-English languages.

5.4 Analysis of Attention Heads

LVLMs extract and interpret cross-modal information within language-specific cross-modal attention heads. As shown in Figure 7 (b), these attention heads are predominantly located in the intermediate layers of the model, particularly at layers 10-17, suggesting that cross-modal integration occurs primarily at this stage. Beyond this, we examine how different layers influence the final prediction. In Figure 4, we observe that around layer 21, visual and textual tokens—such as the 339th and 620th tokens, which correspond to the semantics of "bus"—have a significant impact on the logits of the final prediction. Notably, attention heads in these layers exhibit high classification accuracy, indicating their crucial role in linguistic visual perception and understanding. Conversely, Figure 7 (a) shows that in the early layers (0 and 1), the attention weights of the last input token to visual tokens are strongest. However, classification accuracy remains very low in these heads, suggesting that while LVLMs engage in basic image feature extraction at the initial layers, they contribute minimally to linguistic understanding.

5.5 Impact of Hyperparameters

CLAIM is primarily governed by two key hyperparameters: the intervention intensity \( \alpha \) and the number of heads \( K \) involved in the intervention. Illustrated in Figure 8, the ablation experiments vary one parameter while keeping the other fixed,
yielding several key insights. When \( \alpha \) is too small, the intervention is insufficient, resulting in suboptimal improvements. However, an excessively large \( \alpha \) imposes an overly strong intervention, disrupting the LVLMs’ capabilities. For the hyperparameter \( K \), we observe that a small \( K \) leads to inadequate intervention in language-specific cross-modal attention heads, reducing effectiveness. On the other hand, a large \( K \) introduces unnecessary interference by affecting attention heads that encode irrelevant information, ultimately degrading performance. Overall, CLAIM achieves performance improvements across a wide range of hyperparameter settings, demonstrating strong robustness to hyperparameter selection.

Figure 7: Heatmap of (a) sum attention weights of the last input token toward all visual tokens and (b) the classification accuracy of probes of LLaVA-1.5 across all 32x32 multi-heads, sorted row-wise by value.

Figure 8: Impact of hyperparameters \( \alpha \) and \( K \) on the Accuracy for LLaVA-1.5 and Qwen-VL-Chat on the POPE-COCO popular subset. The "x" symbol indicates a value extraordinarily lower than normal.

6 Conclusion

In this paper, we propose Cross-Lingual Attention Intervention for Mitigating Multilingual Object Hallucination (CLAIM) in LVLMs, a near training-free method that aligns attention patterns across languages. Extensive evaluations on POPE and MME benchmarks demonstrate that CLAIM effectively mitigates multilingual object hallucination and generalizes well across languages and datasets. Further analysis reveals that attention discrepancies primarily occur in intermediate layers and LVLMs extract and interpret cross-modal information within language-specific cross-modal attention heads., providing deeper insights into multilingual LVLMs inference pathways.

7 Limitations

CLAIM requires distinguishing text and vision information within the attention mechanism to identify language-specific cross-modal attention heads, making it applicable only to LVLMs that treat visual and textual tokens equally in the language decoder. Additionally, our method requires access to the internal layers and representations of LVLMs, limiting its applicability to closed-source models. How to mitigate multilingual object hallucination as a plug-and-play tool for all LVLMs, including those with restricted access, requires further investigation. Besides, since CLAIM aligns non-English attention patterns with English patterns, it may inadvertently reinforce English-centric biases rather than fostering truly multicultural comprehension.

8 Acknowledgments

Xiaocheng Feng is the corresponding author of this work. We thank the anonymous reviewers for their insightful comments. This work was supported by the National Natural Science Foundation of China (NSFC) (grant 62276078, U22B2059), the Key R&D Program of Heilongjiang via grant 2022ZX01A32, and the Fundamental Research Funds for the Central Universities (Grant No.HIT.OCEF.2023018).

References

Michael Andersland. 2024. Amharic llama and llava: Multimodal llms for low resource languages. arXiv preprint arXiv:2403.06354.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023a. Qwen technical report. arXiv preprint arXiv:2309.16609.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023b. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966.

Jing Bi, Junjia Guo, Yunlong Tang, Lianggong Bruce Wen, Zhang Liu, and Chenliang Xu. 2024. Unveiling visual perception in language models: An
attention head analysis approach. arXiv preprint arXiv:2412.18108.

Beitao Chen, Xinyu Lyu, Lianli Gao, Jingkuan Song, and Heng Tao Shen. 2024a. Alleviating hallucinations in large vision-language models through hallucination-induced optimization. Preprint, arXiv:2405.15356.

Junzhe Chen, Tianshu Zhang, Shiyu Huang, Yuwei Niu, Linfeng Zhang, Lijie Wen, and Xuming Hu. 2024b. Ict: Image-object cross-level trusted intervention for mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2411.15268.

Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. 2024c. Halc: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.00425.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna.lmsys.org (accessed 14 April 2023), 2(3):6.

Corinna Cortes. 1995. Support-vector networks. Machine Learning.

Alexey Dosovitskiy. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. 2023. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394.

Gregor Geigle, Abhay Jain, Radu Timofte, and Goran Glavas. 2023. mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMs. CoRR, abs/2307.06930. ArXiv: 2307.06930.

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. Preprint, arXiv:2311.05232.

Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13418–13427.

Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709.

Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. 2024. Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens. arXiv preprint arXiv:2411.16724.

Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13872–13882.

Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2024. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36.

Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucination in large vision-language models. In The 2023 Conference on Empirical Methods in Natural Language Processing.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer.

Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. 2023. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024a. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306.

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024b. Llava-next: Improved reasoning, ocr, and world knowledge.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024c. Visual instruction tuning. Advances in neural information processing systems, 36.

Shi Liu, Kecheng Zheng, and Wei Chen. 2025. Paying more attention to image: A training-free method for alleviating hallucination in lvms. In European Conference on Computer Vision, pages 125–140. Springer.

Muhammad Maaz, Hanoona Abdul Rasheed, Abdelrahman M. Shaker, Salman H. Khan, Hisham Cholakkal, Rao Muhammad Anwer, Tim Baldwin, Michael Felsberg, and Fahad Shahbaz Khan. 2024. PALO: A Polyglot Large Multimodal Model for 5B People. CoRR, abs/2402.14818. ArXiv: 2402.14818.
Nostalgebraist. 2020. Interpreting gpt: The logit lens. LessWrong.

OpenAI. 2023. GPT-4. https://openai.com/gpt-4.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830.

Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S Yu. 2025. A survey of multilingual large language models. Patterns, 6(1).

Xiaoye Qu, Mingyang Song, Wei Wei, Jianfeng Dong, and Yu Cheng. 2024. Mitigating multilingual hallucination in large vision-language models. arXiv preprint arXiv:2408.00550.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR.

David Romero, Chenyang Lyu, Haryo Akbarianto Bibowo, Teresa Lynn, Injy Haméd, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atanafu Lambebo Tonja, Bontu Fufa Balcha, et al. 2024. CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark. CoRR, abs/2406.05967. ArXiv: 2406.05967.

Yin S., Fu C., Zhao S., Li K., Sun X., Xu T., and Chen E. 2023. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549.

Florian Schneider and Sunayana Sitaram. 2024. M5 - A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pages 4309–4345. Association for Computational Linguistics.

Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pages 146–162. Springer.

Dhruv Shah, Blažej Osiński, Sergey Levine, et al. 2023. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. In Conference on robot learning, pages 492–504. PMLR.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. arXiv preprint.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems.

Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do llamas work in english? on the latent language of multilingual transformers. arXiv preprint arXiv:2402.10588.

Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and S Yu Philip. 2023. Multimodal large language models: A survey. In 2023 IEEE International Conference on Big Data (BigData), pages 2247–2256. IEEE.

Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2023. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprint arXiv:2311.04257.

Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549.

Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwan He, Zhiyuan Liu, Tat-Seng Chua, et al. 2024. Rlaf-v: Aligning mlms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405.17220.

Bai Z., Wang P., Xiao T., He T., Han Z., Zhang Z., and Shou MZ. 2024. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930.

Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. 2024. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence.

Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. 2023. Beyond hallucinations: Enhancing ivlms through hallucination-aware direct preference optimization. Preprint, arXiv:2311.16839.

Weihong Zhong, Xiaocheng Feng, Liang Zhao, Qiming Li, Lei Huang, Yuxuan Gu, Weitao Ma, Yuan Xu, and Bing Qin. 2024. Investigating and mitigating the multimodal hallucination snowballing in large vision-language models. arXiv preprint arXiv:2407.00569.

Peipei Zhu, Xiao Wang, Lin Zhu, Zhenglong Sun, Wei-Shi Zheng, Yaowei Wang, and Changwen Chen. 2023. Prompt-based learning for unpaired image captioning. IEEE Transactions on Multimedia, 26:379–393.
A Implementation Details

In our experiments, we use greedy search to ensure reproducibility. We employ the experimental settings as default in the VCD code repository. All datasets used in this paper are licensed under a Creative Commons Attribution 4.0 License. We conduct extensive experiments on languages exhibiting varying performance levels across the two models. Probe is implemented as a linear Support Vector Machine (SVM) (Cortes, 1995), using the default LinearSVC API from Scikit-learn (Pedregosa et al., 2011). When calculating metrics for discriminative tasks, we consider "yes" and "no" as right and wrong labels. The output of LVLMs are considered correct if its meaning matches the label. We utilize ChatGPT (OpenAI, 2023) to assist us with coding and polishing the paper.

The hyperparameters under consideration included \( \alpha \) and \( K \). The hyperparameter tuning strategy employed in this study follows a sequential optimization approach. We conduct hyperparameter tuning exclusively on the popular subset of POPE-COCO, with the search space for \( K \) defined as {50, 100, 150, 200, 250, 300} and for \( \alpha \) as {0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5}. Initially, \( K \) was fixed at a value of 100, while \( \alpha \) was systematically adjusted to identify its optimal setting. Through this process, the optimal value for \( \alpha \) was determined. Subsequently, \( \alpha \) was fixed at this optimal value, and \( K \) was iteratively tuned. This led to the identification of the optimal value for \( K \). Consequently, the optimal hyperparameter combination was established as shown in Figure 8. This stepwise optimization strategy ensures a focused and efficient exploration of the hyperparameter space, leading to the identification of the most effective parameter configuration for the model.

B Evaluation of Translation Quality

We sample 100 translated queries from the POPE-COCO random subset for each language and back-translate them into English using Google Translate. The back-translated English queries are then input into LVLMs to test whether their predictions align with those generated from the original English queries. High prediction consistency indicates that the translated data maintains superior benchmark quality. Prediction consistency: Zh-100%, Es-100%, Ru-100%, Pt-100%, Bg-100%, Hi-100%, De-100%. These results demonstrate the reliability of our constructed multilingual dataset.

C Results of LLaVA-NeXT

To comprehensively demonstrate the effectiveness of CLAIM, we also conduct experiments on an advanced LVLM, LLaVA-NeXT-7b (Liu et al., 2024b), with stronger English capability than LLaVA-1.5-7b, as shown in Table 3, also demonstrating strong effectiveness.

D Inference Speed

We evaluate Tokens Per Second (TPS) of LLaVA-1.5 on the popular subset of POPE-COCO using different methods, with the experiments conducted on the H100 GPUs. The results show that VCD leads to a significant decrease in inference speed. We attribute this slowdown to the fact that contrastive-decoding-based methods typically require multiple inference runs or involve substantial additional computations during the inference process. In contrast, CLAIM introduces almost no extra computational overhead during inference, further showing the advantages of our method.

<table>
  <tr>
    <th>Method</th>
    <th>TPS</th>
    <th>Acc (%)</th>
  </tr>
  <tr>
    <td>LLaVA-1.5-7b</td>
    <td>55.46 × 1.0</td>
    <td>73.68</td>
  </tr>
  <tr>
    <td>+VCD</td>
    <td>20.67 × 0.4</td>
    <td>74.03</td>
  </tr>
  <tr>
    <td>+Ours</td>
    <td>54.79 × 1.0</td>
    <td>85.15</td>
  </tr>
</table>

E Detailed Results on MME

In Table 4, we present the performance of LVLM baselines on the 14 tasks of the MME benchmark. The Existence and Count subsets assess object-level hallucination, while the Color and Position subsets focus on attribute-level hallucination. These four subsets form the hallucination evaluation set of MME, which, together with the remaining six categories—Artwork, Celebrity, OCR, Landmark, Scene, and Poster—collectively evaluate the LVLMs’ perception capabilities. Besides, Code Reasoning, Numerical Calculation, Text Translation and Commonsense Reasoning evaluate the LVLMs’ recognition capabilities. The deployment of CLAIM nearly consistently improves their perceptual competencies, indicating the effectiveness of CLAIM for cross-lingual visual perception alignment. Furthermore, The results on recognition-related tasks indicate that the application of CLAIM, while mitigating hallucination issues and augmenting perceptual capabilities, remains effective on some reasoning tests.
<table>
  <tr>
    <th>Dataset</th>
    <th>Setup</th>
    <th>Method</th>
    <th>En</th>
    <th>Zh</th>
    <th>Es</th>
    <th>Ru</th>
    <th>Pt</th>
    <th>Bg</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="3">Random</td>
    <td></td>
    <td><i>Baseline</i></td>
    <td>88.50</td>
    <td>88.33</td>
    <td>80.30</td>
    <td>88.37</td>
    <td>86.77</td>
    <td>81.70</td>
    <td>85.09</td>
  </tr>
  <tr>
    <td></td>
    <td>VCD</td>
    <td>-</td>
    <td>85.83</td>
    <td>69.47</td>
    <td>84.53</td>
    <td>81.97</td>
    <td>75.67</td>
    <td>79.49</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td>-</td>
    <td>88.73</td>
    <td>88.47</td>
    <td>88.13</td>
    <td>86.73</td>
    <td>81.67</td>
    <td>86.75</td>
  </tr>
  <tr>
    <td rowspan="3">COCO Popular</td>
    <td></td>
    <td><i>Baseline</i></td>
    <td>87.37</td>
    <td>87.53</td>
    <td>80.07</td>
    <td>85.60</td>
    <td>85.13</td>
    <td>79.30</td>
    <td>83.53</td>
  </tr>
  <tr>
    <td></td>
    <td>VCD</td>
    <td>-</td>
    <td>85.43</td>
    <td>69.20</td>
    <td>82.07</td>
    <td>82.93</td>
    <td>75.67</td>
    <td>79.06</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td>-</td>
    <td>88.43</td>
    <td>88.13</td>
    <td>85.83</td>
    <td>85.40</td>
    <td>80.33</td>
    <td>85.62</td>
  </tr>
  <tr>
    <td rowspan="3">Adversarial</td>
    <td></td>
    <td><i>Baseline</i></td>
    <td>86.30</td>
    <td>80.80</td>
    <td>79.20</td>
    <td>80.60</td>
    <td>80.37</td>
    <td>76.10</td>
    <td>79.41</td>
  </tr>
  <tr>
    <td></td>
    <td>VCD</td>
    <td>-</td>
    <td>77.23</td>
    <td>68.10</td>
    <td>76.93</td>
    <td>76.27</td>
    <td>70.30</td>
    <td>73.77</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td>-</td>
    <td>80.63</td>
    <td>83.83</td>
    <td>80.20</td>
    <td>80.30</td>
    <td>75.83</td>
    <td>80.16</td>
  </tr>
  <tr>
    <td rowspan="3">Random</td>
    <td></td>
    <td><i>Baseline</i></td>
    <td>91.00</td>
    <td>83.03</td>
    <td>67.50</td>
    <td>85.00</td>
    <td>78.97</td>
    <td>81.20</td>
    <td>79.14</td>
  </tr>
  <tr>
    <td></td>
    <td>VCD</td>
    <td>-</td>
    <td>82.20</td>
    <td>69.27</td>
    <td>82.23</td>
    <td>79.37</td>
    <td>75.70</td>
    <td>77.75</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td>-</td>
    <td>85.47</td>
    <td>87.50</td>
    <td>86.50</td>
    <td>83.33</td>
    <td>80.90</td>
    <td>84.74</td>
  </tr>
  <tr>
    <td rowspan="3">OKVQA Popular</td>
    <td></td>
    <td><i>Baseline</i></td>
    <td>89.00</td>
    <td>81.93</td>
    <td>67.40</td>
    <td>80.73</td>
    <td>80.80</td>
    <td>79.90</td>
    <td>78.15</td>
  </tr>
  <tr>
    <td></td>
    <td>VCD</td>
    <td>-</td>
    <td>81.23</td>
    <td>69.33</td>
    <td>78.33</td>
    <td>79.77</td>
    <td>74.27</td>
    <td>76.59</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td>-</td>
    <td>85.33</td>
    <td>85.27</td>
    <td>82.13</td>
    <td>85.07</td>
    <td>80.13</td>
    <td>83.59</td>
  </tr>
  <tr>
    <td rowspan="3">Adversarial</td>
    <td></td>
    <td><i>Baseline</i></td>
    <td>81.97</td>
    <td>70.80</td>
    <td>66.47</td>
    <td>74.17</td>
    <td>71.27</td>
    <td>73.33</td>
    <td>71.21</td>
  </tr>
  <tr>
    <td></td>
    <td>VCD</td>
    <td>-</td>
    <td>71.57</td>
    <td>66.97</td>
    <td>71.67</td>
    <td>70.83</td>
    <td>69.63</td>
    <td>70.13</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td>-</td>
    <td>75.40</td>
    <td>76.93</td>
    <td>76.73</td>
    <td>75.17</td>
    <td>73.70</td>
    <td>75.59</td>
  </tr>
  <tr>
    <td rowspan="3">Random</td>
    <td></td>
    <td><i>Baseline</i></td>
    <td>89.93</td>
    <td>82.53</td>
    <td>67.73</td>
    <td>84.47</td>
    <td>80.07</td>
    <td>79.87</td>
    <td>78.93</td>
  </tr>
  <tr>
    <td></td>
    <td>VCD</td>
    <td>-</td>
    <td>81.03</td>
    <td>68.60</td>
    <td>81.57</td>
    <td>77.63</td>
    <td>74.53</td>
    <td>76.67</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td>-</td>
    <td>84.23</td>
    <td>87.13</td>
    <td>85.47</td>
    <td>83.30</td>
    <td>79.97</td>
    <td>84.02</td>
  </tr>
  <tr>
    <td rowspan="3">GQA Popular</td>
    <td></td>
    <td><i>Baseline</i></td>
    <td>85.97</td>
    <td>77.17</td>
    <td>67.70</td>
    <td>73.93</td>
    <td>74.90</td>
    <td>79.73</td>
    <td>74.69</td>
  </tr>
  <tr>
    <td></td>
    <td>VCD</td>
    <td>-</td>
    <td>76.80</td>
    <td>69.53</td>
    <td>73.60</td>
    <td>75.23</td>
    <td>74.20</td>
    <td>73.87</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td>-</td>
    <td>80.50</td>
    <td>83.77</td>
    <td>76.50</td>
    <td>81.83</td>
    <td>81.03</td>
    <td>80.73</td>
  </tr>
  <tr>
    <td rowspan="3">Adversarial</td>
    <td></td>
    <td><i>Baseline</i></td>
    <td>82.60</td>
    <td>70.33</td>
    <td>66.93</td>
    <td>73.90</td>
    <td>71.50</td>
    <td>74.03</td>
    <td>71.34</td>
  </tr>
  <tr>
    <td></td>
    <td>VCD</td>
    <td>-</td>
    <td>69.57</td>
    <td>67.50</td>
    <td>72.90</td>
    <td>71.57</td>
    <td>69.33</td>
    <td>70.17</td>
  </tr>
  <tr>
    <td></td>
    <td>Ours</td>
    <td>-</td>
    <td>74.07</td>
    <td>78.10</td>
    <td>76.93</td>
    <td>76.53</td>
    <td>74.87</td>
    <td>76.10</td>
  </tr>
</table>

Table 3: Main results of LLaVA-NeXT on POPE from COCO, OKVQA, GQA.

F Impact of Training Size

In the main experiment, we sample 1,000 images from the COCO-2017 training dataset to identify the language-specific cross-modal attention heads and estimate language shift vectors. Of these, 80% are used as the training set for the probe, while the remaining 20% serve as the test set for the probe. Specifically, CLAIM achieves strong performance even with a minimal training set. We validate this on the popular subset of POPE-COCO (Chinese), as shown in Figure 9. Notably, most improvements are achieved with as few as N = 50 training samples. As the training set size increases, identification becomes more robust; however, an excessively large training set may introduce additional noise when estimating the mean of the language shift vectors.

![Line plot showing the impact of training data size on accuracy for LLaVA-1.5.](page_1012_1342_388_246.png)

Figure 9: Impact of training data size on the Accuracy for LLaVA-1.5.
<table>
  <tr>
    <th rowspan="2">Task</th>
    <th rowspan="2">Method</th>
    <th colspan="6">LLaVA-1.5</th>
    <th colspan="6">Qwen-VL-Chat</th>
  </tr>
  <tr>
    <th>En</th><th>Zh</th><th>Es</th><th>Ru</th><th>Pt</th><th>Bg</th><th>Avg.</th>
    <th>En</th><th>Zh</th><th>Es</th><th>Ru</th><th>Hi</th><th>De</th><th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="3">Existence</td>
    <td><i>Baseline</i></td>
    <td>190.0</td><td>175.0</td><td>130.0</td><td>145.0</td><td>155.0</td><td>125.0</td><td>146.0</td><td>185.0</td>
    <td>190.0</td><td>135.0</td><td>140.0</td><td>106.7</td><td>195.0</td><td>153.3</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>-</td><td>180.0</td><td>130.0</td><td>145.0</td><td>150.0</td><td>128.3</td><td>146.7</td><td>-</td>
    <td>185.0</td><td>155.0</td><td>155.0</td><td>78.30</td><td>195.0</td><td>153.7</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>-</td><td>195.0</td><td>175.0</td><td>185.0</td><td>175.0</td><td>180.0</td><td>182.0</td><td>-</td>
    <td>190.0</td><td>155.0</td><td>185.0</td><td>128.3</td><td>195.0</td><td>170.7</td>
  </tr>
  <tr>
    <td rowspan="3">Count</td>
    <td><i>Baseline</i></td>
    <td>155.0</td><td>70.00</td><td>55.00</td><td>58.30</td><td>80.00</td><td>85.00</td><td>69.67</td><td>150.0</td><td>130.0</td><td>143.3</td><td>113.3</td><td>60.00</td><td>136.7</td><td>116.7</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>-</td><td>73.30</td><td>73.30</td><td>53.30</td><td>61.70</td><td>105.0</td><td>73.33</td><td>-</td>
    <td>140.0</td><td>131.7</td><td>100.0</td><td>80.00</td><td>128.3</td><td>116.0</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>-</td><td>125.0</td><td>130.0</td><td>110.0</td><td>130.0</td><td>135.0</td><td>126.0</td><td>-</td>
    <td>148.3</td><td>153.3</td><td>120.0</td><td>111.7</td><td>137.0</td><td>134.0</td>
  </tr>
  <tr>
    <td rowspan="3">Color</td>
    <td><i>Baseline</i></td>
    <td>165.0</td><td>80.00</td><td>135.0</td><td>75.00</td><td>110.0</td><td>80.00</td><td>96.00</td><td>180.0</td><td>170.0</td><td>150.0</td><td>153.3</td><td>103.3</td><td>165.0</td><td>148.3</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>-</td><td>95.00</td><td>145.0</td><td>85.00</td><td>130.0</td><td>88.30</td><td>108.7</td><td>-</td>
    <td>150.0</td><td>165.0</td><td>146.7</td><td>93.30</td><td>160.0</td><td>143.7</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>-</td><td>120.0</td><td>155.0</td><td>125.0</td><td>150.0</td><td>108.3</td><td>131.7</td><td>-</td>
    <td>170.0</td><td>165.0</td><td>158.3</td><td>90.00</td><td>165.0</td><td>149.7</td>
  </tr>
  <tr>
    <td rowspan="3">Position</td>
    <td><i>Baseline</i></td>
    <td>118.3</td><td>53.30</td><td>63.30</td><td>55.00</td><td>50.00</td><td>46.70</td><td>53.67</td><td>131.7</td><td>63.30</td><td>120.0</td><td>93.30</td><td>45.00</td><td>105.0</td><td>85.33</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>-</td><td>48.30</td><td>93.00</td><td>60.00</td><td>51.70</td><td>56.70</td><td>61.93</td><td>-</td>
    <td>78.30</td><td>101.7</td><td>101.7</td><td>43.30</td><td>93.30</td><td>83.67</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>-</td><td>66.70</td><td>78.30</td><td>85.00</td><td>86.70</td><td>58.30</td><td>75.00</td><td>-</td>
    <td>63.30</td><td>116.7</td><td>103.3</td><td>55.00</td><td>106.0</td><td>88.86</td>
  </tr>
  <tr>
    <td rowspan="3">Artwork</td>
    <td><i>Baseline</i></td>
    <td>121.8</td><td>86.25</td><td>57.75</td><td>73.00</td><td>81.25</td><td>61.25</td><td>71.90</td><td>135.3</td><td>152.8</td><td>105.8</td><td>70.00</td><td>65.75</td><td>130.5</td><td>105.0</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>-</td><td>96.75</td><td>79.50</td><td>81.75</td><td>74.50</td><td>72.25</td><td>80.95</td><td>-</td>
    <td>137.5</td><td>108.5</td><td>71.75</td><td>55.00</td><td>123.3</td><td>99.30</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>-</td><td>117.5</td><td>118.0</td><td>103.5</td><td>107.3</td><td>98.25</td><td>108.9</td><td>-</td>
    <td>146.0</td><td>110.0</td><td>115.0</td><td>83.00</td><td>133.8</td><td>117.6</td>
  </tr>
  <tr>
    <td rowspan="3">Celebrity</td>
    <td><i>Baseline</i></td>
    <td>138.2</td><td>137.4</td><td>55.29</td><td>26.18</td><td>119.7</td><td>13.24</td><td>70.35</td><td>150.0</td><td>162.9</td><td>159.4</td><td>52.65</td><td>57.65</td><td>126.8</td><td>111.9</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>-</td><td>139.7</td><td>79.12</td><td>23.82</td><td>99.41</td><td>10.59</td><td>70.53</td><td>-</td>
    <td>160.9</td><td>158.5</td><td>67.35</td><td>57.35</td><td>121.2</td><td>112.1</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>-</td><td>156.5</td><td>158.2</td><td>61.47</td><td>145.3</td><td>27.65</td><td>109.8</td><td>-</td>
    <td>157.9</td><td>160.3</td><td>110.6</td><td>76.47</td><td>106.2</td><td>122.3</td>
  </tr>
  <tr>
    <td rowspan="3">OCR</td>
    <td><i>Baseline</i></td>
    <td>125.0</td><td>65.00</td><td>50.00</td><td>55.00</td><td>15.00</td><td>55.00</td><td>48.00</td><td>102.5</td><td>80.00</td><td>115.0</td><td>62.50</td><td>32.50</td><td>132.5</td><td>84.50</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>-</td><td>62.50</td><td>57.50</td><td>52.50</td><td>35.00</td><td>57.50</td><td>53.00</td><td>-</td>
    <td>87.50</td><td>100.0</td><td>52.50</td><td>42.50</td><td>115.0</td><td>79.50</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>-</td><td>80.00</td><td>82.50</td><td>57.50</td><td>32.50</td><td>65.00</td><td>63.50</td><td>-</td>
    <td>87.50</td><td>100.0</td><td>77.50</td><td>50.00</td><td>135.0</td><td>90.00</td>
  </tr>
  <tr>
    <td rowspan="3">Landmark</td>
    <td><i>Baseline</i></td>
    <td>165.3</td><td>113.3</td><td>53.75</td><td>81.50</td><td>127.3</td><td>108.5</td><td>96.90</td><td>172.8</td><td>179.0</td><td>123.3</td><td>78.25</td><td>61.50</td><td>163.3</td><td>121.1</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>-</td><td>125.5</td><td>64.25</td><td>97.25</td><td>126.3</td><td>108.0</td><td>102.3</td><td>-</td>
    <td>167.0</td><td>131.3</td><td>78.25</td><td>57.00</td><td>138.0</td><td>114.3</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>-</td><td>147.8</td><td>155.5</td><td>146.5</td><td>159.3</td><td>148.3</td><td>151.5</td><td>-</td>
    <td>178.3</td><td>129.8</td><td>148.0</td><td>88.25</td><td>167.5</td><td>144.9</td>
  </tr>
  <tr>
    <td rowspan="3">Scene</td>
    <td><i>Baseline</i></td>
    <td>159.5</td><td>159.3</td><td>79.50</td><td>120.0</td><td>144.3</td><td>122.5</td><td>125.1</td><td>161.5</td><td>178.8</td><td>124.8</td><td>114.5</td><td>72.75</td><td>131.5</td><td>124.5</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>-</td><td>149.8</td><td>87.30</td><td>119.5</td><td>137.5</td><td>118.0</td><td>122.4</td><td>-</td>
    <td>162.3</td><td>122.8</td><td>108.0</td><td>71.30</td><td>128.8</td><td>118.7</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>-</td><td>149.5</td><td>146.8</td><td>154.3</td><td>144.3</td><td>145.0</td><td>148.0</td><td>-</td>
    <td>162.5</td><td>123.5</td><td>153.0</td><td>92.50</td><td>143.5</td><td>135.0</td>
  </tr>
  <tr>
    <td rowspan="3">Poster</td>
    <td><i>Baseline</i></td>
    <td>143.5</td><td>106.1</td><td>67.35</td><td>57.14</td><td>110.2</td><td>68.71</td><td>81.90</td><td>173.1</td><td>156.8</td><td>142.9</td><td>116.7</td><td>79.16</td><td>134.4</td><td>125.4</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>-</td><td>116.7</td><td>89.80</td><td>65.31</td><td>90.48</td><td>80.95</td><td>88.64</td><td>-</td>
    <td>147.3</td><td>137.4</td><td>101.0</td><td>65.99</td><td>126.9</td><td>115.7</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>-</td><td>133.7</td><td>156.8</td><td>97.62</td><td>129.9</td><td>121.8</td><td>128.0</td><td>-</td>
    <td>161.6</td><td>150.3</td><td>138.8</td><td>109.9</td><td>137.8</td><td>139.7</td>
  </tr>
  <tr>
    <td rowspan="3">Code Reasoning</td>
    <td><i>Baseline</i></td>
    <td>67.50</td><td>65.00</td><td>50.00</td><td>47.50</td><td>55.00</td><td>22.50</td><td>48.00</td><td>55.00</td><td>57.50</td><td>57.50</td><td>20.00</td><td>12.50</td><td>52.50</td><td>40.00</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>-</td><td>67.50</td><td>57.50</td><td>72.50</td><td>62.50</td><td>57.50</td><td>57.50</td><td>-</td>
    <td>42.50</td><td>57.50</td><td>37.50</td><td>35.00</td><td>45.00</td><td>43.50</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>-</td><td>55.00</td><td>60.00</td><td>47.50</td><td>65.00</td><td>25.00</td><td>50.50</td><td>-</td>
    <td>72.50</td><td>50.00</td><td>45.00</td><td>10.00</td><td>52.50</td><td>46.00</td>
  </tr>
  <tr>
    <td rowspan="3">Numerical Calculation</td>
    <td><i>Baseline</i></td>
    <td>70.00</td><td>47.50</td><td>50.00</td><td>45.00</td><td>20.00</td><td>20.00</td><td>36.50</td><td>32.50</td><td>65.00</td><td>45.00</td><td>27.50</td><td>45.00</td><td>37.50</td><td>44.00</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>-</td><td>75.00</td><td>50.00</td><td>62.50</td><td>37.50</td><td>20.00</td><td>49.00</td><td>-</td>
    <td>80.00</td><td>60.00</td><td>37.50</td><td>45.00</td><td>45.00</td><td>53.50</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>-</td><td>67.50</td><td>50.00</td><td>55.00</td><td>20.00</td><td>30.00</td><td>44.50</td><td>-</td>
    <td>45.00</td><td>45.00</td><td>55.00</td><td>45.00</td><td>27.50</td><td>43.50</td>
  </tr>
  <tr>
    <td rowspan="3">Text Translation</td>
    <td><i>Baseline</i></td>
    <td>70.00</td><td>77.50</td><td>50.00</td><td>70.00</td><td>120.0</td><td>52.50</td><td>74.00</td><td>155.0</td><td>60.00</td><td>115.0</td><td>110.0</td><td>17.50</td><td>65.00</td><td>73.50</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>-</td><td>80.00</td><td>50.00</td><td>72.50</td><td>75.00</td><td>72.50</td><td>70.00</td><td>-</td>
    <td>55.00</td><td>87.50</td><td>85.00</td><td>45.00</td><td>72.50</td><td>69.00</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>-</td><td>95.00</td><td>57.50</td><td>50.00</td><td>90.00</td><td>57.50</td><td>70.00</td><td>-</td>
    <td>87.50</td><td>122.5</td><td>110.0</td><td>50.00</td><td>72.50</td><td>88.50</td>
  </tr>
  <tr>
    <td rowspan="3">Commonsense Reasoning</td>
    <td><i>Baseline</i></td>
    <td>124.3</td><td>72.14</td><td>64.29</td><td>61.43</td><td>72.86</td><td>64.29</td><td>67.00</td><td>125.0</td><td>92.14</td><td>100.7</td><td>50.71</td><td>33.57</td><td>74.29</td><td>70.28</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>-</td><td>80.71</td><td>71.43</td><td>70.00</td><td>71.14</td><td>62.86</td><td>72.43</td><td>-</td>
    <td>84.29</td><td>98.57</td><td>48.57</td><td>29.29</td><td>78.57</td><td>67.86</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>-</td><td>90.00</td><td>87.14</td><td>86.43</td><td>87.14</td><td>68.57</td><td>85.86</td><td>-</td>
    <td>91.43</td><td>107.1</td><td>80.00</td><td>39.29</td><td>84.29</td><td>80.43</td>
  </tr>
  <tr>
    <td rowspan="3">Total Scores</td>
    <td><i>Baseline</i></td>
    <td>1813</td><td>1307</td><td>961.3</td><td>970.1</td><td>1261</td><td>925.2</td><td>1085</td><td>1909</td><td>1738</td><td>1638</td><td>1203</td><td>790.0</td><td>1650</td><td>1404</td>
  </tr>
  <tr>
    <td>VCD</td>
    <td>-</td><td>1391</td><td>1128</td><td>1051</td><td>1209</td><td>1009</td><td>1157</td><td>-</td>
    <td>1683</td><td>1619</td><td>1191</td><td>794.1</td><td>1571</td><td>1371</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>-</td><td>1599</td><td>1611</td><td>1375</td><td>1522</td><td>1269</td><td>1475</td><td>-</td>
    <td>1762</td><td>1689</td><td>1612</td><td>1029</td><td>1664</td><td>1551</td>
  </tr>
</table>

Table 4: Detailed results on full subsets of MME.

G  Comparison to PAI

According to the suggestion of the anonymous reviewer, we conduct a comparison experiment with PAI (Liu et al., 2025), which intervenes on attention heads by leveraging their original direction. In Table 5, CLAIM still achieves best performance.

H  Case Study

Illustrated in Figure 10, we present examples demonstrating how CLAIM effectively mitigates multilingual object hallucination. We compare the changes in the model’s attention weight maps for the same image before and after applying CLAIM.
<table>
  <tr>
    <th>Setup</th>
    <th>Method</th>
    <th>Zh</th>
    <th>Es</th>
    <th>Ru</th>
    <th>Pt</th>
    <th>Bg</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="4">Random</td>
    <td><i>Baseline</i></td>
    <td>81.00</td>
    <td>63.03</td>
    <td>72.33</td>
    <td>78.97</td>
    <td>72.23</td>
    <td>73.51</td>
  </tr>
  <tr>
    <td><i>VCD</i></td>
    <td>81.47</td>
    <td>67.40</td>
    <td>73.33</td>
    <td>78.07</td>
    <td>72.47</td>
    <td>74.55</td>
  </tr>
  <tr>
    <td><i>PAI</i></td>
    <td>80.10</td>
    <td>67.97</td>
    <td>83.50</td>
    <td>79.13</td>
    <td>71.10</td>
    <td>76.36</td>
  </tr>
  <tr>
    <td><i>Ours</i></td>
    <td>86.50</td>
    <td>87.33</td>
    <td>87.50</td>
    <td>85.40</td>
    <td>80.50</td>
    <td>85.45</td>
  </tr>
  <tr>
    <td rowspan="4">Popular</td>
    <td><i>Baseline</i></td>
    <td>83.07</td>
    <td>62.93</td>
    <td>69.20</td>
    <td>82.03</td>
    <td>71.17</td>
    <td>73.68</td>
  </tr>
  <tr>
    <td><i>VCD</i></td>
    <td>83.20</td>
    <td>67.20</td>
    <td>70.17</td>
    <td>80.10</td>
    <td>69.47</td>
    <td>74.03</td>
  </tr>
  <tr>
    <td><i>PAI</i></td>
    <td>83.90</td>
    <td>67.73</td>
    <td>79.17</td>
    <td>82.90</td>
    <td>71.90</td>
    <td>77.12</td>
  </tr>
  <tr>
    <td><i>Ours</i></td>
    <td>88.00</td>
    <td>87.53</td>
    <td>84.03</td>
    <td>86.13</td>
    <td>80.07</td>
    <td>85.15</td>
  </tr>
  <tr>
    <td rowspan="4">Adversarial</td>
    <td><i>Baseline</i></td>
    <td>73.40</td>
    <td>62.87</td>
    <td>66.07</td>
    <td>73.70</td>
    <td>65.93</td>
    <td>68.39</td>
  </tr>
  <tr>
    <td><i>VCD</i></td>
    <td>74.27</td>
    <td>67.30</td>
    <td>66.47</td>
    <td>73.63</td>
    <td>66.33</td>
    <td>69.60</td>
  </tr>
  <tr>
    <td><i>PAI</i></td>
    <td>76.60</td>
    <td>67.43</td>
    <td>75.20</td>
    <td>75.53</td>
    <td>67.73</td>
    <td>72.50</td>
  </tr>
  <tr>
    <td><i>Ours</i></td>
    <td>77.67</td>
    <td>83.27</td>
    <td>79.43</td>
    <td>79.67</td>
    <td>74.57</td>
    <td>78.92</td>
  </tr>
</table>

Table 5: Comparison to PAI on POPE of LLaVA-1.5.

English Query: Is there a motorcycle in the image?
English Output: No ✔️
Chinese Query: 图片里有摩托车吗?
Chinese Output: 是 ✗
Chinese Output with Ours: 不 ✔️

![English Attention Map](page_184_624_246_120.png)   ![Chinese Attention Map](page_430_624_246_120.png)   ![Chinese Attention Map with Ours](page_676_624_246_120.png)

English Attention Map        Chinese Attention Map        Chinese Attention Map with Ours

English Query: Is there a computer in the image?
English Output: Yes ✔️
Russian Query: Есть ли на изображении компьютер?
Russian Output: Нет ✗
Russian Output with Ours: Да ✔️

![English Attention Map](page_184_880_246_120.png)   ![Russian Attention Map](page_430_880_246_120.png)   ![Russian Attention Map with Ours](page_676_880_246_120.png)

English Attention Map        Russian Attention Map        Russian Attention Map with Ours

Figure 10: Case study.

Reviewer paper summaries: The paper aims to address multilingual object hallucination with large vision-language models, i.e. when incorrect text is generated about an image. It is found that cross-modal attention distributions differ significantly between English and other languages. The solution proposed is to identify language-specific attention heads, estimate the shift between En and other languages, and apply this shift at inference time. Demonstrable performance improvements are shown on POPE and MME datasets. It is also notably found that the middle attention layers are important in multilingual, cross-modal inference.
Summary of strengths: - Study of this problem is important in the area of multilingual, multimodal models. It makes sense that there are attention distribution gaps between English (the dominant language in pretrained models) and other languages. 

- The method is intuitive. A strong advantage is that the method does not need data/training of the multimodal LLM, making it simple to implement to improve results. 

- The results are compelling across multiple datasets (POPE/MME) and models (LLaVA/Qwen).

- Paper is pretty well-written. Insights are clear. I like how this paper presents the problem, solution, and insights into research questions w.r.t. the problem/solution. Fig. 5 is a good illustration of the differences before and after intervention.
Summary of weaknesses: - The method is limited by English interpretation of text. Past work (e.g. [1]) has noted differences in how different groups perceive content. Therefore, relying on English attention distributions may be limiting. Though I still think this is good starting point to address cross-lingual gap. 

- The method is claimed as training-free, though there is still training needed for identification of language-specific attention probes. It would help to be clearer about terminology. 

- The logit lens evaluation in Fig. 4 is a little unclear. Is this before or after intervention? It would be interesting to see/understand how this distribution changes after intervention. 

- It is not apparent how the amount of multilingual pretraining data for a model affects utility of the method/insights. Is this method needed for non-English centric models? 

[1] Visually Grounded Reasoning across Languages and Cultures, EMNLP21
Comments, suggestions and typos: Some occasional typos - EX: "Vison" Fig 4
Overall assessment scores: 4