Understanding and Mitigating Hallucination in Large Vision-Language Models via Modular Attribution and Intervention
Authors: Tianyun Yang
Ziniu Li
Juan Cao
Chang Xu
Venue: ICLR 2025 Poster (2025)
Keywords: Large Vision-Language Models
Hallucination
Interpretability
Primary area: foundation or frontier models, including LLMs

Abstract: Large Vision-Language Models (LVLMs) exhibit impressive capabilities in complex visual tasks but are prone to hallucination, especially in open-ended generation tasks. This paper explores why LVLMs tend to hallucinate and how to mitigate it. First, we conduct causal mediation analysis through counterfactual edits on specific modules in LVLMs. Our results disclose that Multi-Head Attention (MHA) modules contribute more to the probability of generating hallucination words than multi-layer perceptron modules. We then identify specific heads that are responsible for hallucination, referred to as hallucination heads. Second, we examine the behavior of hallucination heads. We find that they are concentrated in the middle and deeper layers, displaying a strong attention bias toward text tokens. Further, we show that the attention patterns of certain hallucination heads exhibit greater similarity to the base language model and change slowly during the instruction tuning process. Finally, we propose two simple yet effective methods to mitigate hallucination: one is training-free and can be applied directly during decoding, while the other involves fine-tuning. Both methods are targeted for hallucination heads to reduce their reliance on text tokens. Notably, our methods achieve up to 1.7x reduction in hallucination rate for the LLaVA-v1.5-7B model in COCO captioning task, outperforming existing baselines. Overall, our findings suggest that hallucinations in LVLMs are likely to stem from certain modules, and targeted interventions can effectively mitigate these issues.

Published as a conference paper at ICLR 2025
UNDERSTANDING AND MITIGATING HALLUCINATION IN LARGE LAR ATTRIBUTION AND INTERVENTION
MODELS VIA MODU-
Tianyun Yang-.2 Ziniu Li Juan Cao1.2+ Chang Xu
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
2 University of Chinese Academy of Sciences, Beijing, China
3 School of Data Science, The Chinese University of Hong Kong. Shenzhen, China
4 School of Computer Science, Faculty of Engineering, University of Sydney, Australia yangt i anyun 9z ac. cn, ziniulie inkk cwhk. edu.cerccccc ca caojuanéicr ww..cttcccccc................... ac. cn, c. xu@sydrey, edu. au
ABSTRACT
Large Vision-Language Models (LVLMs) exhibit impressive capabilities in com- plex visual tasks but are prone to hallucination, especially in open-ended genera- tion tasks. This paper explores why LVLMs tend to hallucinate and how to mit- igate it. First, we conduct causal mediation analysis through counterfactual edits on specific modules LVLMs. Our results disclose that Multi-Head Attention (MHA) modules contribute more the probability of generating hallucination words than multi-layer perceptron modules. We then identify specific heads that responsible for hallucination, referred as hallucination heads. Second, we examine the behavior of ballucination heads. We find that they are concentrated in the middle and deeper layers, displaying strong attention bias toward text to- kens. Further, we show that the attention patterns of certain hallucination heads exhibit greater similarity the base language model and change slowly during the instruction tuning process. Finally, we propose two simple yet effective methods mitigate hallucination: one is training-fres and can be applied directly during decoding, while the other involves fine-tuning. Both methods are targeted for hallucination heads to reduce their reliance on text tokens. Notably, our methods achieve up to 1.7x reduction in hallucination rate for the LLaVA-v1.5-7B model in COCO captioning task, outperforming existing baselines. Overall, our findings suggest that hallucinations in LVLMs are likely stem from certain modules, and targeted interventions can effectively mitigate these issues.
INTRODUCTION
Large Vision-Language Models (LVLMs) (Liu et al., 2024; Zhu et al., 2023; Tong et al., 2024; Li et al., 2023a; Bai et al., 2023) have demonstrated impressive capabilities in open-world visual tasks, ranging from basic perceptual tasks such as image description to complex reasoning tasks. Despite their strong capabilities, LVLMs often hallucinate, generating content that deviates from the provided image information (Bai et al., 2024), especially in long-form open-ended tasks like captioning. This reveals fundamental weakness in current models and poses a risk of misleading users and undermining trust. Understanding the causes of hallucination LVLMs and developing strategies to mitigate this issue is crucial for improving their reliability.
It is challenging trace the causes of hallucinations. This complexity arises from the intricate training pipeline of LVLMs, which involves visual pre-training and instruction tuning built on top of a a pre-trained language model (Liu et al., 2024; Zhu al., 2023). Additionally, underlying architectures, often based on Transformers (Vaswani et al., 2017), are inherently complex, making interpretation difficult. Several studies have explored different factors contributing hallucinations. Research by Leng et al. (2024) demonstrated that visual uncertainty from distorted input images can
"Corresponding anthor
Code is available https: / thub. coc.b.comm/ii/ http:/.cm/mmannnoonoooooooom luci nat Cn-AAtrr intrrbution

---

Published as a conference paper at ICLR 2025
amplify hallucinations, while Huang et al. (2024) suggested that hallucinations often arise when the model overly relies on certain summary tokens during text generation, neglecting vital image infor- mation. Additionally, Li et al. (2023b) and Zhou et al. (2024) found that models tend to hallucinate by generating objects that frequently appear or co-occur with those in the images, leading to their erroneous inclusion in generated descriptions.
In this paper, we investigate the causes of hallucinations in LVLMs through the lens of attribution and intervention (Bach al., 2015; Voita al., 2019; Covert et al., 2021; Shah et al., 2024). Our goal is to uncover how the internal components of LVLMs contribute to hallucination words. Our study has three stages, which we will elaborate on below. Before diving in, we note that our primary focus is on the LLaVA-v1.5-77 model (LLaVA-7B for short) for open- open-ended generation tasks, with a particular focus on object hallucination. But we also investigate the LVLM model MiniGPT-4 (Zhu et al., 2023) in the Appendix to ensure our findings generalize.
First, we identify components responsible for generating hallucination words using counterfactual edits (zero-ablating specific modules), tool widely used in mechanism interpretability (Nanda et al., 2023). We find that Multi-Head Attention (MHA) modules have a greater impact than multi-layer perceptron modules in producing hallucination words. Further, we localize this effect to specific heads, classifying them as hallucination or non-hallnn haa-allluciaiionn heads. We observe that hallucination heads mainly distribute in the middle and deeper layers of the Transformer.
Second, we examine how the identified hallucination heads contribute to hallucination, with two key insights. For one thing, hallucination heads favor textual inputs over visual inputs by up five times. For another thing, the attention patterns of hallucination heads show a greater similarity to the base language model compared con-haalucination heads. Furthermore, extensive instruction tuning (e.g., 665k samples in LLaVA-7B) results in limited changes to the attention patterns of prominent hallucination heads, suggesting that they are "lazy" when LVLMs are tuned with full parameters.
Finally, we develop methods to mitigate hallucination. Our preliminary results show that downscal- ing attention weights for text tokens significantly reduces the hallucination rate, while increasing attention on image tokens has no effect. Importantly, this intervention is effective specifically for hallucination heads. Based on these findings, we develop two approaches: one training-fre free for use during decoding, while the other involves tine-tuning hallucination heads. Both methods reduce hallucination heads' reliance on text tokens and achieve up to 1.7x reduction hallucination rate for the VL-7 model on the COCO captioning task, outperforming existing baselines.
To summarize, our contributions have three fold:
Attribution of Hallucination Components: We systematically identify and localize the compo- nents most responsible for hallucination generation in LVLMs. Specifically, we show that MHA modules, particularly certain heads in the middle and deeper layers, are key contributors.
We show that hallucination heads strongly favor previously generated text over visual inputs. We also reveal that this pattern is inherited from the base language model and changes slowly during the visual instruction tuning process.
Analysis of Attention Bias:
Hallucination Mitigation Techniques: We develop two targeted strategies: one training-fres for de- coding and one involving fine-tuning. Both methods reduce over-reliance on text tokens, achieving a significant reduction in hallucination rates, outperforming existing baselines.
2 RELATED WORK
Mitigating Hallucinations in LVLMs. LVLMs' hallcunation behaviors are particularly severe in open-ended generation tasks (Huang et al., 2024; Zhou et al., 2024; Zhang et al., 2023). Many approaches have been explored to mitigate hallucinations in LVLMs, most of which focus on bet- ter decoding strategies. For instance, Leng et al. (2024) introduced visual contrastive decoding, which compares output distributions from original and distorted visual inputs to correct the model's ' over-reliance on unimodal priors and statistical bias. Moreover, Huang et al. (2024) observed that LVLMs frequently depend on the so-called summary tokens and proposed a method combining beam-search with octrospection-allocatit
penalizing over-reliance on these tokens. Additionally, Chen al. (2024) highlighted the importance of incorporating both local and global visual con- text, with the HALC method using an external grounding module during decoding. Furthermore, Zhou al. (2024) developed LURE, which rectifies text by revising generated content to mitigate
2

---

Published as a conference paper at ICLR 2025
hallucination issues like co-occurrence errors and object ambiguity. Deng et al. (2024) proposed a -LIP-guided decoding method to mitigate hallucination. Liu et al. (2023) tackled the issue by creating the LRV-Instruction dataset, featuring positive and negative instructions to improve robust- ness in visual instruction tuning. Zhang et al. (2024) introduced REVERIE, a large-scale dataset with reflective rational annotations, enabling models to justify response correctness. In contrast to previous studies, our work identifies a modulc-level cause of hallucination in LVLMs and develops targeted intervention strategies to mitigate hallucination effectively. In addition, a key feature of our method is that it requires only a single generation forward process during decoding, which is faster than existing methods.
Interpretability of Transformers. Understanding and explaining neural networks, particularly Transformers (Vaswani al., 2017), crucial for identifying their behaviors and limitations (Zhao et al., 2024). A widely used approach is causal mediation analysis (Hicks & Tingley, 2011), which attributes the contributions of key components, often employing "knock-out" techniques (Wang et al., 2023) to assess the impact of removing specific model elements on the output. Previous research (Voita et al., 2019; Olsson et al., 2022; Yu et al., 2023; Gandelsman et al., 2023) has demonstrated that individual attention heads in Transformers frequently assume distinct roles, such as induction, copying, and memorization. While prior studies, such as (Zhou et al., 2024), have also explored to understand hallucination in LVLMs, work approaches it through the attribution of key components and intervenes in them specifically.
PRELIMINARY
LVLMs usually process both visual and linguistic data using three components: a vision encoder, a connector, and a Large Language Model (LLM). The vision encoder processes visual input, the connector aligns it with text tokens, and the LLM generates responses from this multimodal input. The LLM, structured as transformer (Vaswani et al., 2017), consists of L layers. Each layer includes a Multi-Head Attention (MHA) module and a Multi-Layer Perceptron (MLP), applying two primary residual transformations the output of the previous layer Zt-1;
2' MHA'
Zd-1, MLP' (2)) +'
In this framework, a layer é employs an MHA module consisting of H attention heads. Each head executes a self-attention operation, where the attention score is computed using query, key, and value matrices derived from the input. Specifically, for the i-th head in layer É, tte operation is given by:
({(K))
head (z--1)- ))= Attention (Q, Kt, V softmax
V
Vdb
the query matrix for the i-th head, and K. = Ze-L the key matrix for the i-th head, and V = = is the value matrix for the i-th head, and de is the dimensionality of the key vectors. The outputs from all H heads are then concatenated and projected using an output projection matrix WO.F;
where = ===
Concat(head (Z*-1), head For common 7B models such as LLaVA (Liu et al., 2024) and MiniGPT4 (Zhu al., 2023), we have = 32 and H = 32 and dg = 128.
MHA (Z
(Z
HALLUCINATION ATTRIBUTION AND INTERVENTION
In this paper, we diagnose hallucination behaviors in LVLMs by examining their internal compo- nents. We first identify the components most responsible for generating hallucination words in Section 4.1. Next, we analyze the patterns of these problematic components in Section 4.2. Finally, we propose strategies to mitigate hallucinations in Section 4.3.
4.1 TRACING HALLUCINATION BEHAVIORS TO MODEL COMPONENTS
The first step in our workflow is to select a neural network with notable hallucination behaviors for analysis. For this purpose, we choose the well-known LLAVA-7B. 2 We then break the neural net- work into smaller units for detailed investigation, focusing specifically on the MHA and MLP. We
"Our analysis and conclusions also hold for other models such MiniGPT4; Appendix A.1.
3

---

Published as a conference paper at ICLR 2025
leverage causal mediation analysis (Hicks & Tingley, 2011). Specifically, we employ a "knockout" technique (Wang et al., 2023), selectively disabling the function of each component while main- taining the rest of the model intact. This is achieved through zero ablation', where the output of targeted component is set to zero. Let M be a vision-language model, and M \ e be the counterpart of knock-outing of component c. For generated response, each component's influence score on hallucination is quantified using the following formula:
T
Ie
(yl E hallucination) [Par (ye u, x. V<t) - (
where m Ei= {yr E hallucination) is the total number of hallucination words, yr is the token time step a generated response /1:7,, and y< refers to the sequence of generated tokens prior to time step t. The variables v and x correspond to the visual context and user query, respectively. high value I. means that component e contribute a to hallucination generation.
For our analysis, we use the COCO dataset (Lin et al., 2014), sampling 1,500 images from the training set. Following stan- dard practice (Huang et al., 2024), we prompt the LVLM with the instruction:"Pleaat describe the image in detail." Objects that match the ground truth labels are marked as non-hallucinated, while mismatches are classified as hallucinated. Subsequently, perform zero-ablation each MLP and MHA indepen- dently, calculate the influence score as in Equation (1), and
0.008
0.007
Intervention Effect
0.006
0.005
3.5
0.004
0.003
present the average results for all MIPs and MHAs in Figure 1.
0.002
0.001
MHAS have greater effects than MLPs for Hallucination. We observe that removing MHAs has significantly larger im- pact compared to removing MLPs, even though MHAs account for only half of the MLP parameters in LLaVA-7B. This find- ing aligns with (Gandelsman et al., 2023), which demonstrated that MHAs have a significant effect on classification accuracy in the transformer model CLIP (Radford et al., 2021). Intuitively, MHAs focus on capturing relationships and dependencies be- tween tokens through attention weights, whereas MLPs primarily process the information output by MHAs. The differing effects of these interventions suggest that hallucinations often stem from the model's attention to specific patterns biases, highlighting the need for more targeted analysis of these components.
0.000
MLP
MHA
Figure 1: Influence scores of MLP and MHAs in LLaVa-7B on generation probability of hal- lucination objects.
Building the above findings, seek further explore the behaviors of attention heads, espe- cially those linked to hallucinations. The first step is identifying such attention heads. This task is not easy, attention heads in Transformers fulfill many roles (Voita et al., 2019; Olsson et al., 2022) and often exhibit polysemantic functionality (Nanda & Sharkey, 2023; Bills et al., 2023), meaning they can influence multiple behaviors simultancously. This makes difficult to precisely isolate and identify the specific functions of each head. However, because our focus is on the generation of hal- lucination words, is crucial minimize the influence of non-hallucination words when attributing attention heads. To achieve this, we introduce new criterion called contrastive influence score, which measures the difference between the intervention effect on hallucination words and its effect on non-hallucination words: In. constrasttive In, hallucciiinooo It, Joo hallaiitalccin
(2) where Ip, Ln.lallucination is the intervention effect of attention head on generating hallucination words (see Equation (1)), while In Znn.mm-allhcination is the counterpart for non-hallucination worlds. Using the same model and setup as before, we calculate the constrastive influence values for a total of 1,024 attention heads across 32 layers, with 32 heads in each layer. The results shown in Figure 2.
Hallucination Heads Distribute in Middle and Last Layers. For clarity, we categorize attention heads into two groups based on their contrastive influence scores: hallucination heads, which exhibit
Thee technique of mean mean-ablating can also be used and similar results are observed; see Appendix A.I. 'It is important to use the constrastive influence score for diagnosing attention heads, relying solely on I hallucimation for attribution and intervention does work well in practice; see Appendix A.
4

---

Published as a conference paper at ICLR 2025
high contrastive influence score, and non houllllucination heads, which show low contrastive influence score. Rather than using a strict threshold to define these categories, we apply a top-k selection, focusing on heads with the highest and lowest contrastive influence scores. Notably, both hallucina- tion and non-hallucination heads, particularly the most prominent ones (e.g., the top 20 highlighted in boxes in Figure 2), are predominantly located in the middle and deeper layers of the model. This finding aligns with previous studies on Transformer models (Voita et al., 2019), which have shown that deeper layers tend capture more abstract and task-specific representations.
1.0
Text Attention
mage Attention
Attention Weights
Head Indix
0
8
4.025
0.4
x4.75
8
0059
1.35
0.0
24
32
Hol ucination Heads Non-Ha lucination Heads (Top 20)
Index
Figure 2: Contrastive influence values of atten- tion heads in the LLaVA-BB model, with blue boxes for the top 20 hallucination heads and red boxes for non-hallucination heads.
Figure 3: Attention weights text and image tokens for hallucination and non-hallucination heads. Hallucination heads strongly favor text tokens over image tokens.
4.2 BEHAVIOR ANALYSIS OF HALLUCINATION HEADS
After attributing attention heads responsible for hallucination in Section 4.1, 1, we analyze their be- havior patterns this section. To understand why they induce hallucinations, attention maps. In particular, we divide the attention weights of an attention head into two parts: text attention and image attention. For each attention head, text attention is calculated by summing the attention weights assigned to tokens corresponding to instructions and responses", while image attention is determined by summing the attention weights assigned to tokens representing image features. See Figure 3 for the results.
examine their
Hallucination Heads Favor Texts Over Image Inputs. We observe that for hallucination heads, text attention is 4.75 times higher than image attent i ion. In contrast, non-hallucination heads allocate attention more evenly between text and images tokens. This suggests that hallucina- tion heads primarily focus on contextual text, causing LVLMs to rely internal knowledge rather than image inputs when generating relevant words. This behavior helps explain the tendency toward hallucination. Our finding aligns with observations from previous research (Leng al., 2024; Huang et al., 2024), which also suggest that LVL.Ms tend to overlook visual information during generation. However, key difference that show this overlooking of visual information primarily occurs in hallucination heads rather than across all attention heads. This insight offers actionable guidance for developing targeted strategies mitigate hallucinations, discussed in Section 4.3.
So far, our analysis has focused on the model after visual instruction tuning. Now, we aim to take step further and to investigate why hallucination heads over rely on text tokens the tuned model. This is challenging question, as factors like the training dataset, algorithm, and language model all determine the emergence of hallucination heads. Below, we attempt to link the attention patterns of hallucination heads in L.La VA-BB 7- to those in its base language model, Vicuna-7B. We hypothesize that the over-reliance on text tokens in hallucination heads originates from the language model itself. To test this, we examine the attention map of Vicuna-7B, using the same question and response. Since Vicuna-7B cannot process image inputs directly, we use a placeholder token <image> to ensure contextual consistency, We then analyze the attention patterns within the text tokens (i.e., the responses). The results, displayed in Figure 4, show the similarity of attention patterns measured using cosine similarity.
*System tokens excluded in the calculation here, they often serve "attention sink" and lack specific semantic meanings (Xiao al., 2023). Thus, the sum of attentions weights in Figure 3 may not be 1.
5

---

Published as a conference paper at ICLR 2025
Hallucination Head (Top 1)
Non-Halluuination Head (Top 1]
LLavA 1.5-78
Vicuna-78
LLaVA 1.5-78
Vicuna-78
0.33
Cosine
Cosine
(8p
(8p
0.84
0.63
Figure 4: (Left and Middle): Attention maps of the top-1 hallucination heads and non -allluuccinnionn heads on generated text tokens of LVLM and its base LLM. Attention maps are downsampled for better visualization. (Right): Statistics over the top-20 attention heads.
Inherited Attention Patterns Hallucination Heads from Base Language Models. We observe notable similarity in the attention maps on generated text tokens between the hallucination heads of LLaVA-7B and Vicuna-7B, despite Vicuna-7B not processing actual image inputs. In contrast, non-hallucination heads do not exhibit such clear pattern, particularly in the most prominent hal- lucination and non-hallucination heads. This suggests that hallucination heads may inherit much of their behavior from the base language model's next-token prediction.
Some Hallucination Heads Show Slow Changes in Attention Maps During Visual Instruction Tun- ing. The above results imply that hallucination heads likely inherit much of their behavior from the base language model, despite undergoing extensive visual instruction tuning (e.g., 665k samples for LLaVA- 7B). To provide further evidence for this, we repli- cate the visual instruction uniig process of LLaVA- 7B and calculate the Jensen-Shannon Shannon (JS) divergence between the attention maps before and after tun- ing. As shown in Figure 5, find that top hal- lucination heads are "lazy", displaying noticeably slower changes in attention maps compared non- hallucination heads. This insight could be valuable for the future development of LVLMs and warrants further investigation. For our work, this finding mo-
the Peetraieed Nonel
0.36
Halluciation Heads (Top 31 Nen-Htlllaiinaion Heads (Tp
0.30
0.08
0.02
2 0.000
560 1962
3000 1000 4000 Instruction Tuning Steps
2000
Figure JS divergence of the attention map from the initial model (before visual instruc- tion tuning) throughout the tuning process. tivates the design of targeted fine- fine-tuning strategies, rather than full- -allmmereetter tuning, mitigate hallucinations in the next section.
4.3 MITIGATING HALLUCINATION THROUGH MODULAR INTERVENTION
The findings in Section 4.2 lead us to explore a practical question: can hallucination be reduced attention heads place less emphasis on text tokens, or alternatively, if they place more emphasis on image tokens? To investigate this, we adjust the text generation process by downscaling text attention and upscaling image attention through multiplication by scaling factor. We randomly sample 500 COCO images from the validation set, and prompt the LVLM with the instruction: "Please describe the image detail". The CHAIR metric used to measure the hallucination rate by computing the proportion of objects mentioned in the generated description that absent from the ground-trun truth labels. Furthermore, the BLEU score is used to assess the quality of the generated text. The results, presented in Figure 6, offer three interesting insights.
First, reducing text attention weights is more effective than increasing image attention weights." 6
Second, targeted intervention of text tokens on hallucination heads is more important than apply- ing changes the other attention heads.
"We provide an explanation Appendix A.2. We examine the linear spaces spanned by feature representa- tions of text tokens and image tokens, respectively. We find that some directions in the text space cannot be represented by the image space, changing image attention weights is sufficient.

---

Published as a conference paper at ICLR 2025
14
CHAiR (60
CHAIR (86
12
17.8
21
17.6
10-
10
Halludiaatin Heads
Halludination Heads
17.4
Random
Random
Halluchation
1.0
0.8
0.6
0.4
0.2
1.0
2.0
2.5
3.0
3.5
1.0
0.8
0.6
0.4
0.2 0
cownssalte weight on text attention
Upscale weight on image attention
Downscale weight on text attention
(a)
(b)
(c)
Figure 6: (a) Downscaling text attention weights reduce hallucination rate. (b) Upscaling image
attention weights does not work. (c) Downscaling text attentions weights zero lead to the drop in generation quality of BLEU.
Last, simply downscaling text attention on hallucination heads to zero could hurt generation qual- ity, as reflected the BLEU score drops Figure (c).
Based on these insights, propose two strategies mitigate hallucination: adaptively deactivate text attention weights during the decoding stage (Section 4.3.1) and fine-tuning hallucination heads specifically correct their attention patterns (Section 4.3.2).
4.3.1 ADAPTIVE DEACTIVATION OF HALLUCINATION HEADS
According to results presented in Figure 6, pruning the text attention weights of hallucination heads proves to be an effective decoding- strategy but at the cost of the quality text generation, potentially leading to less coherent outputs. To address this, we propose more adaptive strategy, to deactivate text attention only for those hallucination heads that demonstrate excessive reliance on text attention during the decoding phase.
The adaptive deactivation mechanism operates as follows: at each decoding step t. for each halluci- nation head h Hhallucination: first compute the causal self-attention weights A, by applying the softmax function to the dot product of the query and key matrices for that head. To evaluate whether the head over-relies on text information in the current decoding step, we calculate an indicator value He that accumulate the attention weights allocated to text tokens. If this indicator 1t exceeds predefined threshold т, deactivate the head H, for that particular decoding step, i.c. setting the text attention weights to zero. Otherwise, the head remains active. See Algorithm 1.
Algorithm Adaptive Deactivation of Halluci- nation Heads (AD-HH)
Algorithm 2 Targeted Fine-Tuning Tuning of Halluci- nation Heads (TF-HH)
Require:
Hallucination Head Set Haallncinationn
Require:
Hallucination Head Set Haallucination dataset D 1: for component c do
Threshold 7
1: for decoding time step do
2:
if attention head in Hastiuciiaaion then
2:
if c in Haallucination language head then
3:
H <- sum of text attention weights
3:
c.requires.grad
4:
if 1est > then
4: else
5:
Set the attention weights
c.requires.gras grad false for fine-tuning steps do Calculate the loss in Equation (3) for samples in D
zero
6:
6:
Self-attention calculation
7:
else
8:
Self-attention calculation
Perform gradient descent update
We would like to note that Algorithm 1 requires single generation forward process in the decoding stage. This differs from contrastive decoding methods (Chuang et al., 2024; Leng et al., 2024), which require two passes, methods (Huang al., 2024; Zhou al., 2024) that rely beam search and retrospection. As a result, Algorithm runs faster in practice compared to these baselines. However, key limitation of Algorithm it incompatible with memory-efficient mechanisms like FlashAttention (Dao, 2023). We address this issue by introducing another fine-tumin tun-ttning method below.
that it requires the explicit calculation of attention weights, making

---

Published as a conference paper at ICLR 2025
4.3.2 TARGETED FINE-TUNING OF HALLUCINATION HEADS
In this section, we propose targeted fine-tuning method (Algorithm 2) to mitigate hallucination. This approach directly addresses hallucination prone attention heads in LVLMs and ensures strong performance with greedy decoding during the generation process. Given the desired goals mentioned in the beginning of Section 4.3, we propose a simple training objective that combines the standard -eax-ttoen-preiitiion next-token- prediction objective for maintaining generation quality and text-attention penalty for reducing hallucinations. For a particular training sample (v, ø, /1:T), where v, æ, and V1rT denote the visual input, textual instruction and response, define the loss function:
((:T)
Σ
log Par (ye 2,, Mee) +
Σ
log A
x, 2.V33) (U,,,,))
(3)
hE Houtoxination
next-tokct--reciicion
text-cllancoereduction
where À > 0 controls the strength of the penalize for text reliance, and A is the summed attention weights on text tokens of hallucination head h. A key design consideration in our method is the choice of the optimization variables. Following the guidance from previous results, we only fine- tune the hallucination heads rather than full heads. Additionally, we fine-tune the final layer of the language model's prediction head (referred to as the language head Algorithm 2) to enhance the model's 's capacity in fine-tur fine-tuning. Because our fine-tuning method is targeted, it is highly effective in correcting hallucination heads. A small number of optimization steps is sufficient to achieve significant improvements. In practice, we found that 200 optimization steps for LLaVA-7B yielded good results, requiring less than 3% of the compute used for instruction tuning in VL-7B
5 EXPERIMENTS
5.1 EXPERIMENT SETUP
In addition to the previously examined VLaVA model, we also investigate the well-known LVLM model MiniGPT-4 (Zhu et al., 2023), which has 7B parameters as LLaVA.
Proposed Methods. For the decoding-time. method AD-HH in Algorithm I, we select the top 20 at- tention heads for VLa and the top 10 heads for MiniGPT-4. We use fewer heads for MiniGPT- 4 because fewer heads are remarkable in the contrastive influence map (see Appendix A. For the threshold 5 to control the when to deactivate text attention decoding, we find that setting T around 0.5 provides good balance between generation quality and hallucination reduction. Specifically, we set T 0.4 for VA-7B and т 0.5 for MiniGPT-4. For the fine-tuning method TF-HH in Algorithm 2, we slightly increase the number of attention heads to 30 and 20, to increase the representation power. For the penalty weight À in fine-tuning, we set it as 2. Both of our methods based on greedy decoding to generate samples for evaluation.
Baselines. Alongside proposed methods for mitigating hallucination, we also study baseline ap- proaches from prior literature, including the standard greedy decoding method and several state- of-the-art techniques: OPERA (Huang et al., 2024), VCD (Leng et al., 2024), LURE (Zhou et al., 2024), and HALC (Chen et al., 2024). Additionally, we include DoLA (Chuang et al., 2024), which was originally designed to enhance factuality in language models and has also been studied in previ- ous literature. saper-parameters of these basclines follow from previous literature and are provided in Appendix B for reference.
Datasets. We mainly focus on hallucination in open-ended generation tasks assess the effec- tiveness of our methods. To evaluate object hallucination in visual caption, we use images from the COCO validation and Nocaps (Agrawal et al., 2019) datasets. For Nocaps, we use the out-of-domain version. We randomly select 500 samples from each and prompt the LVLMs with the query, "Please describe this image in detail". We also examine the performance of LVLMS on complex tasks us- ing the MM-Vet (Yu al., 2024) dataset, which assesses integrations of six core vision-language capabilities: recognition, OCR, knowledge, language generation, spatial awareness, and math. The maximum token limit is set to 128 by default.
Evaluation Metrics. To evaluate object hallucination in image caption, we employ CHAIR met- rics (Rohrbach et al., 2018), designed for automatic hallucination assessment. CHAIR measures the hallucination rate by computing the proportion of objects mentioned in generated description that absent from the ground-truth labels. The metric is split into components: sentence-level
8

---

Published as a conference paper at ICLR 2025
Table 1: Hallucination rates terms of CHAIRs and CHAIR; on COCO and Nocaps (Out-of-
Domain) image captioning tasks, with lower value indicating better performance. The subscript indicates that this method training-free, while means that this method is training-based
COCO
Nocaps Cout-of-Domain
Methods
LL
MiniGPT-
LLaVA-78 7B
MiniGPT-4
CHAIRs
CHAIR
CHAIRs
CHAIR,
CHAIRs
CHAIR
CHAIRs
CHAIR;
Greedy?
51.8
13.3
40.6
13.7
43.2
14.3
57.4
20.0
DoLA
53.8
13.9
41.0
13.8
42.0
13.7
57.2
20.4
OPERA
35.2
12.8
44.2
14.4
46.2
16.2
VCD
55.4
15.7
38.8
14.8
43.6
14.4
48.2
17.5
LURE!
51.2
13.4
46.4
14.2
41.8
14.4
55.8
19.6
HALC
50.2
12.4
36.4
11.8
40.2
12.2
53.0
18.0
AD-HH (Ours)
29.6
8.0
35.2
11.7
35.6
9.4
46.8
16.2
TH-HH" (Ours)
35.0
8.7
32.0
11.4
35.4
11.1
45.2
16.8
Input Image
Original Model
Fine-tuned Model
detadidd polioo lesellgrroe con's
inaellyrooe twe's constoan-lSS... USS.l: Biieenn Eaos duactioe thie tsnge to detall Adbic LAKE
lodufal dotalld
quarchen USER clesgeo Paa doocilee cas aoxav in deted, ASSISTAN1)
smage Dapaita bay
dbotis nedd
snage depers teay
with
drinag
fow whllee Then loleed the anather fooe it and olees coutoied avarel vouse
willie There
can bokine lev, conelve from of and for ofhere wartood aoseer vocz. laaede
dan
sna arttkdal Intelanacs aubdack
betnas
antional cat aitclteen
wokttee glow helelk, drtalied polloo comeerr can's
sobitte soon fetutal detddd pollie
standing
guoday
beagy Saatows
anboce stanciig
greu They an postioeel is has. adice the let wita, seadle ts
greas 11yy see pooticcal in his, witll setre
antill tou murr cade Sold
the riae vale weil the toooeheet thsight de bol rend, cheo pronkding sbude for ailoce, Book
the mublle and
salids
ahko alace
and the toi effere on the right disct Aegoreed, then proniiie
Figure 7: Visualization of attention weights for the top hallucination head when predicting the next token (shown blue). We accumulate the attention weights on a total of 576 image tokens into the placeholder <image> simplify visualization. Redder values in the context indicate larger attention weights. The original LLaVA-7B model significantly relies on the previously generated tokens, resulting in hallucination. Our model through targeted fine-tuning does not have this issue.
hallucination (CHAIRs) and image-level hallucination (CHAIR)). Specifically:
CHAIRs |captions with hallucinated objects
|hallucinated objects
CHAIR, |all mentioned objects)
(4)
all captions)
A smaller value of CHAIRs and CHAIR, indicate a lower hallucination rate.
5.2 MAIN RESULTS
Modular Intervention Can Reduce Hallucination by up to 1.7x. We present the evaluation re- sults COCO in Table 1 (left). Our methods-eithh decoding fine-tuiing-achivee consistent improvements for both models, reducing hallucination rates by up to 1.7 times compared to greedy decoding for LLa VA-7B and 1.3 times for MiniGPT-4. We also note that reducing hallucinations does not compromise text generation quality, which is shown in Table 4 in the Appendix A.3, We find that DoLA ineffective in this scenario, finding also observed in the HALC paper. This likely because DoLA is designed to elicit factual knowledge from the model, which may uninten- tionally amplify language biases inherited from the base language model, making unsuitable for mitigating hallucinations in LVLMs.
To illustrate the effectiveness of our methods, we present case studies in Figure 7. The original model demonstrates a strong reliance text tokens, which leads to the generation of hallucination objects that have close semantic relations with the focus token. We then provide the same prompt before the hallucinated token to our fine-tuned model. We observe that the fine-tuned model shifts attention more towards the image tokens, resulting in the generation of maage-consisten objects.
Modular Attribution and Invention Show Transferbility. Recall that the hallucination heads in our methods were identified using the COCO training dataset, and the above evaluation is with the COCO validation dataset. To examine the transferability and robustness across datasets, evaluate

---

Published as a conference paper at ICLR 2025
performance the Nocaps (out-of-domain) dataset, which includes objects not present in COCO. We report the results in Table I (right). We observe similar conclusions: our method maintains strong performance out-of-domain datasets, indicating that modifications to the hallucination heads have a broad impact across tasks and implying the generalizability of our method.
Performance in complicated multimodal tasks from MM- Vet (Yu et al., 2024), with higher Table 2:
value indicating better performance.
LLa 7B
MiniGPT-
Methods
Rec
OCR
Know Gen Spat Math
Total
Rec
OCR
Know
Gen
Spat
Math Total
Greedy
36,3 21.8
17.0
18.9
24.8
7.7
31.4
26.5
13.3
17.5
13.9
22.3
8.1
22.2
DoLA
37.2 22.1
17.9
21.0
26.3
7.7
31.7
24.9
12.9
18.5
12.0
21.7
7.7
21.6
OPERA
35.4 25.6
20.5
22.9
30.9
11.5
32.0
28.2
15.0
16.5
11.4
21.9
11.5
23.6
VCDİ
33.0 23.6
16.0
19.4
25.6
3.8
29.4
25.3
14.8
17.4
15.0
20.3
20.9
HALC
36.2 21.5
17.5
20.1
23.5
7.7
30.8
24.9
15.7
15.2
10.7
23.2
7.7
21.7
AD-HH (Ours)
38.4
26.0
21.2
21.9
30.3
7.7
34.3
28.2
16.6
16.1
13.7
26.1
12.0
23.8
TF-HH (Ours)*
36.6
24.1
17.9
19.0
27.2
11.5
32.5
31.9
18.1
22.3
16.6
26.5
18.5
27.3
Modular Intervention Benefits Complicated Multimodal Tasks. To further validate our method's effectiveness on complex open-ended multimodal tasks, we evaluated performance on the MM-Vet dataset, which assesses six multimodal capabilities: recognition, OCR, knowledge, language gen- eration, spatial awareness, and math. Table 2 shows that our method, either through decoding or fine-tuning, also improves a range of multimodal capacibilites. For instance, our decoding method boosts LLaVA-7B's scores in OCR and spatial awareness by 4.2 and 5.5 points, respectively. Sim- ilarly, our fine-tuning method enhances MiniGPT-4's performance in recognition and math, with improvements of 5.4 and 10.4 points. These results demonstrate that modifying hallucination heads benefits not only tasks focused on hallucination reduction but also general multimodal tasks.
Generation Time Comparison. We compare the gener- ation time of our decoding method AD-HH with existing decoding-time hallucination mitigation methods in Figure 8. For OPERA and our method that requires explicit attention weights use the standard self-attention implementation. For Greedy, DoLA, VCD, and HALC, where explicit attention weights not needed, employ Flash-Attention, which generally faster than standard self-attention. All methods
30
Docoding time per sample (Sconnds)
28
26
were tested on single A100-80GB GPU. We observe that our method achieves similar decoding times to greedy decoding. This is because we intervene on the attention weights on- on-the- fly during the generation process. In contrast, other methods inevitably introduce computational overhead. For instance, VCD requires double inference process for contrastive de- coding, and OPERA requires retrospecting to previous steps when knowledge aggregation happens.
Gready DoLA OPÈRA
VCD
HALC
Ours
Figure 8: Generation time of a sin- gle response.
CONCLUSION
In this work, we explore why vision- -aiguale models generate hallucinated words by examining their internal structures. Our analysis reveals that multi-head attention modules are key contributors, with certain heads in the middle and deeper layers disproportionately focusing on text tokens over visual inputs. These attention patterns appear to be inherited from the base language model, with some hallucination heads showing slow adaptation during instruction tuning. To address this, propose both decoding-tin and tin-tunnng strategies to mitigate hallucinations. Our modular ap- proach offers practical and efficient post-training solution for managing the behavior of LVLMs, in contrast to full parameter tuning and intervention. This approach is especially valuable given the trend of increasing model sizes in LVL.ss.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENT
Tianyun Yang would like to thank Zeyu Qin and Yunke Wang for helpful discussion about the algo- rithm design. This work supported in part by the Australian Research Council under Projects DP240101 04400018881000 and FT230100549, China Scholarship Council, and Strategic Priority Research Pro- gram of the Chinese Academy of Sciences (No. XDB0680202).
REFERENCES
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1148-8957, 2019.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Hlbus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wist wise relevance propagation. PloS one, 301)4,0330140, 2015.
Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond. 2023.
Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404. 18930, 2024.
Satanjeev Banerjee and Alon Lavie. Meteor:
An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in lan- guage models. URL https://ppnaippubicc blob. core. windows. neffneewoneeplaiierhggeenindd haml.((atee accessed: 14.05. 2023), 2, 2023.
Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Hale: Object hallucination reduction via adaptive focal-contrast decoding. ICML, 2024.
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. In ICLR, 2024.
Ian Covert, Scott Lundberg, and Su-In Lec. Explaining by removing: A unified framework for model explanation. Journal of Machine Learning Research, 22(209):1-90, 2021.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023.
Ailin Deng, Zhirui Chen, and Bryan Hooi. Seeing is believing: Mitigating hallucination in large vision-language models via clip-guided decoding. arXiv preprint arXiv:2402. 15300, 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: comprehensive evaluation benchmark for multi- modal large language models. corr abs/2306. 13394 (2023). 2023.
Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Interpreting clip's image representation via text-based decomposition. arXiv preprint arXiv:2310.05916 2023.
Raymond Hicks and Dustin Tingley. Causal mediation analysis. The Stata Journal, 11(4):605-619, 2011.
Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and citispectioo-alloatttn In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13418-13427, 2024.

---

Published as a conference paper at ICLR 2025
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large lision-language models through visual contrastive de- coding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- fion, pp. 13872-13882, 2024.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023a.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint PrXiv:230.10335 2023b.
Chin-Yew Lin and FJ Och. Looking for few good metries: Rouge and its evaluation. In Nicir workshop, 2004.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hal- lucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in neural information processing systems, volume 36, 2024.
Neel Nanda and Lee Sharkey. former.
https://www Polysemantic attention head in a 4-layer trans- TStp://ppst///uuttssttttt a
2023.
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217, 2023.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint 2rXiv:2209.11888 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8766 PMLR, 2021.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. arXiv preprint wrXiv:/809.02//111 2018.
Harshay Shah, Andrew Ilyas, and Aleksander Madry. Decomposing and editing predictions by modeling model computation. arXiv preprint arXiv:2404. 1534, 2024.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint
arXiv:2405.098/1 09818. 2024.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal 1lms. arXiv preprint arXiv:2406. 16860, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint arXiv:1706.0376. 10, 2017.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv: 190.00990 2019.
12

---

Published as a conference paper at ICLR 2025
Kevin Wang, Alexandre Variengien, Arthur Commy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: a circuit for indirect object identification gpt-2 small. ICLR, 2023.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming
language models with attention sinks. arXiv preprini arXiv:2309. 17453, 2023.
Qinan Yu, Jack Merullo, and Ellie Pavlick. Characterizing mechanisms for factual recall in language models. arXiv preprint arXiv:2310.159/00 2023.
Weihao Yu, Zhengyuan Yang, Linjie Li. Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In ICML,, 2024.
Jinrui Zhang, Teng Wang, Haigang Zhang, Ping Lu, and Feng Zheng. Reflective instruction tuning: Mitigating ballucinations in large vision-language models. In ECCV, 2024.
Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. How language model hallucinations can snowball. arXiv preprint arXiv:2305. 1 13534, 2023.
Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. Explainability for large language models: A survey. ACM Trans- actions on Intelligent Systems and Technology, 15(2):1-38, 2024.
Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination large vision-language models. ICLR, 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
13

---

Published as a conference paper at ICLR 2025
A
ADDITIONAL RESULTS
A.1 ADDITIONAL RESULTS ON COMPONENT ATTRIBUTION
On the importance of contrastive influence metric. To validate the proposed contrastive influence metric, which isolates a head's effect on generating hallucinated versus non-hallucinated objects, we compare the non-contrastive influence, i.e., directly use hallaciaatooo for identifying hallucina- tion heads. In Table 3, we report the evaluation results for VLaVA-7B on the COCO captioning task, where we adaptively deactivate the top-20 hallucination heads identified through contrastive and non-contrastive methods. As shown, deactivating heads identified by contrastive influence results significantly greater reduction hallucinations, highlighting its superior precision localizing hal laciiaantioon heads.
Table 3: Ablation study on constrative influence.
Method
CHAIRs
CHAIR
Greedy
51.8
13.3
AD-HH (Non-contrastive Influence)
11.0
AD-HH (Constrative Influence)
29.6
8.8
Sensitivity analysis of the causal mediation method. In Figure 9()), we plot the Spearman rank correlation between the contrastive influence scores of each attention head found using N samples and those using 1000 samples. Here, N represents the number of samples used for hallucination head attribution. Varying from 50 1000, we observe that when N reaches 500, the Spearman rank correlation is 0.93 compared to the results with N = 1000. Beyond N 500, increasing the number of samples results in minimal changes to the attribution outcomes. This indicates that 500 samples are sufficient for accurately identifying hallucination heads.
We also evaluate alternative "knock-out" techniques for identifying hallucination heads, including: 1) using log probabilities instead of probabilities for intervention effects, and 2) replacing the output of the target component with the mean value of the hidden state outputs. Figure 9b,c), we plot the Pearson correlation between these two alternative methods and default zero-ablation method using probabilities. The high correlation observed suggests that the intervention results (contrastive influence) of each head align closely with the default methods, indicating that our approach exhibits low sensitivity.
Spearman Rank Simlarity: 0.89
Spear man Rank Simiarily: 0.96
1.0
3.06
Secmma Batk Car Sarttttton
0.8
0.32
3.002
Log Iny
2.000
0.4
-3.044
-2.06
0.2
Prbbuiity Abretoo
(a)
(b)
(c)
Figure 9: (a) Ablation study on number of samples to identify hallucination heads. (b) Spearman rank similarity comparison between effects calculated log probability and probability. (c) Spear- man rank similarity comparison between mean-ablation and zero-ablation methods.
Component attribution results MiniGPT-4. In Figure 10, present the constrastive influence of attention heads in the MiniGPT-4 model on generating hallucinated and non- non-hallucinated objects. As indicated in the Figure, the hallucination heads also are distributed in the latter half of the model.
14

---

Published as a conference paper at ICLR 2025
1.0
Text Attention
Image Attention
0.8
0,003
Attention Weights
Head Index
0.6
00.05
0.4
24
-0.15
0.2
.0020
32
16
24
32
Layer
Hallucination Heads
Non-Hallucination Heads
Figure 10: Contrastive influence of attention heads in the MiniGPT-4 model. Bluer boxes indicate heads more responsible for generating hallucinated objects.
Figure 11: Averaged attention weights on text and image tokens for hallucination and non- hallucination heads identified in MiniGPT-4.
Hallucination Head (Top 1)
Non-Hallucinatio Head (Top 1)
MiniGPT4-78
Ulama-778
MiniGPT4 70
Llama2 78
MiniGPT4-73 & Ulmma277
0.25
0.30
0.75
Cosine Sim:
Casine
0.70
"Saacchaooo (eace
Von IM raner leads
0.93
0.59
Figure 12: Hallucination heads in MiniGPT-4 inherit the text attention pattern from the base LLM.
A.2 ADDITIONAL RESULTS ON BEHAVIOUR ANALYSIS OF HALLUCINATION HEADS
Attention bias of hallucination heads in MiniGPT-4. In Figure 11, we plot the averaged attention weights on text and image tokens for top-3 hallucination and non-hallucination heads. As shown in the Figure, the hallucination heads of MiniGPT-4 also demonstrate much stronger attention bias than non-hallucination heads.
Inherited attention pattern of hallucination heads in MiniGPT-4. In Figure 12, we compare the attention patterns of hallucination heads in MiniGPT-4 with those in base language model, Llama-2-7B. As shown, the text attention patterns in MiniGPT-4 are more aligned with Llama-2-7B for hallucination heads than for non-hallucination heads. Specifically, the top hallucination head with the highest contrastive influence shows cosine similarity as high as 0.93, in contrast to the top non-hallucination head, which only exhibits similarity 0.59.
Spanned linear space analysis. Our findings in Section 4.3 indicate that downscaling text attention weights more effective than upscaling image attention weights. We hypothesize that this can be explained by the linear spaces spanned by text tokens and image tokens. If the space spanned by text tokens is significantly larger than that of image tokens, simply adjusting image attention weights (i.e., modifying the linear combination in the self-attention mechanism) may be insufficient.
To test this, we construct a linear space for text tokens, denoted as Swat. and a linear space for image tokens, denoted Simage- We then calculate the projection distances:
d(projection Simage onto Stest) = (S - -T T || = 3636.3 d(projection Stex onto Simage) |Sinage T Image Simage) -1 T Sweul Gioulll- = 15688.5
image S
15

---

Published as a conference paper at ICLR 2025
The results suggest that certain directions in the text space cannot be linearly represented by the image token features. As a result, even with careful tuning of image attention weights, the output of the self-attention mechanism may still retain components of textual information, contributing to hallucination.
Hallucination Head (26, 9)
Non-Hallucination Head (31, 16)
10
00000
10
Sydem
Irage
Ouipet
9yow
Image
Cutpot
Question
Figure 13: Complete attention maps of hallucination and non-hallucination heads, encompassing system, image, question, and output tokens. One cell in the maps denotes 10 tokens.
Complete attention maps of hallucination and non-hallucination heads. In Figure 13, present the complete attention maps of two representative hallucination and non-hallucination heads on the COCO captioning task, encompassing system, image, question, and output tokens. For clarity, the maps are downscaled by a factor of 10. As illustrated in the figure, the hallucination head assigns significantly lower attention scores image tokens, suggesting a tendency overlook them. In contrast, the non-hallucination head demonstrates a more balanced attention distribution between image and text tokens.
Further, we would like to emphasize that this over-reliance is a symptom of hallucination heads, not criterion identify them. We also note that there may be other attention heads that also exhibit over- reliance on text tokens but do not influence hallucination behavior (e.g., layer 31, head 4 in LLaVA- 7B). This occurs because some heads may function as general purpose language head, ensuring fluency and coherence in text generation. Although these heads look like the left attention map in Figure 13 (left) and over-rely on text tokens, they are unrelated hallucination behavior. They qualify as hallucination heads only when they are causally responsible for hallucination behavior, indicated by large contrastive influence values.
A.3 ADDITIONAL RESULTS ON MODULAR INTERVENTION
On the effectiveness of adaptive deactivation. To illustrate the effectiveness of adaptive deac- tivation of hallucination heads in preserving the generation quality and mitigating hallucination, we compare the performance of our proposed adaptive deactivation method and full deactivation, which completely setting the text attention weights of hallucination heads to zero regardless of the input. Figure 14 shows the the hallucination rate (CHAIR)) and generation quality (BIEU) of the two methods the number of top-k hallucination heads to be deactivated varied. As illustrated in the Figure, adaptive deactivation yeilds more optimal hallucination reduction and generation quality maintaining with the same number of top-k hallucination heads to be deactivated. This indicates that context-aware pruning is more flexible than static pruning method.
On the top-k hallucination head selection. In Figure 15, we present the relationship between the hallucination rate (CHAIR) and generation quality (BLEU) as the number & of deactivated halluci- nation heads increases. As illustrated, increasing h leads to more effective hallucination reduction
16

---

Published as a conference paper at ICLR 2025
Full Deactivation
Top 35
Adaptive Deactivation
100 CHAIR, (+)
Top
Top 30
Top 30
Top 20
92
Top 25
20
91
16.8
17.0
17.2
17.4
17.6
17.8
(L)
Figure 14: Compare adaptive deactivation and full deactivation in reducing hallucination and main- taining generation quality.
MiniGPT-4
LLaVA-7B
16
14-
16
14
CHAR (%)
CHARR
10-
10
12
Top Hallucination Heads
Top Hallucination Heads
Figure 15: Ablation study the top-k hallucination head selection.
Table 4: Generation quality comparison, with higher value indicating better performance.
COCO ROUGH
Nocaps ROUGH
Dataset
BLEU
METEOR
BLEU
METEOR
Greedy
17.9
18.8
18.2
24.6
21.8
18.8
AD-HH (Ours)
17.8
19.1
18.1
23.1
21.3
18.3
TF-HH (Ours)
18.8
20.0
18.7
25.5
22.9
19.2
but is also associated with decline in generation quality after certain number of deactivated hallu- cination heads. We find that for LLaVA-7B, k: 20 yields a favorable trade-off between generation quality and hallucination reduction for LLLVVA-155, while k 10 is optimal for MiniGPT4, which we adopt as our default parameter in adaptive deactivation of hallucination heads.
On the importance of fine-tuning hallucination heads. To highlight the importance of targeting only hallucination heads, we compare the results in Table 5 for fine-tuning the full parameters, 30 randomly selected heads, and the top 30 hallucination heads on LLaVA-1.5. As shown in the table, fine-tuning only the hallucination heads achieves significantly more hallucination reduction compared both full fine-tume menntuning and fine-tuning random heads. This verifies that hallucination is mainly caused by only small portion (less than 3%) of attention heads, and focusing on them is crucial for reducing hallucination.
On the generation quality comparison. In Table 15 and Table 16, we visualize some image de- scription examples of the LLaVA-7B model. The hallucinated objects are highlighted in RED. The main results show that our method does not influences the the coherent and fluency of generated context. We qualitatively measure the generation quality in Table For BLEU, ROUGH (Lin & Och, 2004), and METEOR (Banerjee & Lavie. 2005), the adaptive deactivation method shows slight decrease on the Nocaps dataset. While the fine-tuning method maintains and even improve the quality, which indicates that fine-tuning provides robust and safe adjustment to the model.
17

---

Published as a conference paper at ICLR 2025
Table 5: Ablation study fine-tuned components.
Methods
CHAIRs
CHAIR
Original Model
51.8
13.3
Fine-tune Full Parameters
56.3
15.6
Fine-tune Random Heads
54.4
15.7
Fine-tune Hallucination Heads
35.0
8.7
Table Human eval results on non-hallucination score and generation quality score. Higher values indicate lower hallucination rate and better quality.
Methods
Non-Hallucination Score
Generation Quality Score
Greedy
3.25
3.99
AD-HH (Ours)
3.87
3.85
TF-HH (Ours)
3.78
4.01
Table 7: Results on the MME dataset.
Method
Score
Greedy
1791.64
AD-HH (Ours)
1812.36
TF-HH (Ours)
1813.06
Table 8: Comparison results with additional baseline methods.
Greedy
GCD
LRV
REVERIE
AD-HH (Ours)
TF-HH (Ours)
CHAIR S
51.8
39.2
39.4
49.6
29.6
35.0
CHAIR_
13.3
10.8
13.1
12.7
8.0
8.7
We also conduct a manual evaluation study in the help of two Ph.D. students and one undergraduate student evaluate the responses. They were instructed to score each response to based on two criteria: (1) non-hallucination performance, with higher scores reflecting fewer hallucinations, and (2) generation quality, with higher scores indicating more fluent and descriptive responses. For both the baseline and our proposed methods, the evaluators assessed total of 500 generated responses per method, resulting in 1500 responses overall. The evaluation results shown in Table 6. These results demonstrate that our methods effectively mitigate hallucination while maintaining high generation quality.
scale of
Evaluation on the MME dataset. We also evaluate on the MME benchmark Fu et al. (2023), which measures the perception and cognition abilities LVLMs. As demonstrated in Table 7, our two methods show generalization ability on the benchmark, improving the overall score by about 20 absolute points.
Comparison with additional baseline methods. We extend our comparison by evaluating our method against additional baseline methods on the COCO dataset using the VLLVVA-B model, as presented in Table 8. The additional baseline methods include GCD (Deng et al., 2024), LRV (Liu et al., 2023), and REVERIE (Zhang et al., 2024). The results demonstrate that our approach, which leverages targeted interventions, is significantly more effective in mitigating object hallucinations in open-ended generation tasks.
Note that re-implemented these baselines as there are differences in the models and evaluation datasets used. Specifically, the GCD paper evaluates the COCO Karpathy Test set, whereas the COCO validation set. Additionally,
model and dataset to ensure fair comparison,
18

---

Published as a conference paper at ICLR 2025
Table 9: Evaluation on larger and more recent LVLM models.
1aamaa3.2-11B---e LLaVA-v1.5-13B
Chameleon-30B
LLaVA-v1.6-34B
Methods
CHAIRs
CHAIR;
[CHAIRs | CHAIR; |CHIRRs CHAIR,
Greedy
28.4
7.4
48.6
12.4
38.0
12.6
23.2
6.4
AD-HH (Ours)
22.6
4.9
38.8
9.4
34.8
12.5
20.4
5.6
Table 10: Number and ratio of salient hallucination heads across different scales of LVLMs.
LLaVA-7B
LLaVA-13B
LLaVA-34B
Number of Salient Hallucination Heads Ratio of Salient Hallucination Heads
42
37
10
4.1%
2.3%
0.3%
the provided MiniGPT-4 checkpoint by REVERIE is pretrained on Llama-7B, while our experiments are based on Llama2-7B.
Evaluation larger and more recent LVLM models. We extend experiments larger and more recent models, including Llama-3.2-1 11- 11--iion--v I.LVAA-1..5-131 Chameleon-30B (Team, 2024), and LLaVA-v1.6-34BB These models are both modern and representative, with Llama-3.2- 1IB-Vision being released just two months ago, Chameleon pioneers an carly-fusion- modal training strategy, LLLVA-Next-34B increases the input image resolution 4x more pixels to grasp more visual details. Using the same settings on the COCO dataset, our method consistent improvements, as indicated Table 9, Our method reduces CHAIRs by approximately points for Llama-3.2 2-.211111 1--11B-Viie Biviio,,, 10 points for LLaVA-v1.5-13B, 3.2 points for Chameleon-300h and 2.8 points improvement for LLaVA-v1.6-34p
multi-
Hallucination behaviour across different scales of LVLMs. Inspired by the reviewer's advice, we conducted an empirical analysis to investigate whether hallucination heads exist across different scales of LVLM models. Specifically, examined various scales of LLaVA models, including LLaVA-7B, LLaVA-13B, and LLaVA-34B. Using the contrastive map derived from Equation 2 for these models, we identified salient hallucination heads whose contrastive influence values exceed- ing 25% of the maximum contrastive influence value. These heads are deemed most responsible for hallucination behaviors. We evaluate both their absolute numbers and their ratio relative to all attention heads. The results are presented in Table 10.
Our findings indicate that hallucination heads tend to diminish as model size increases and sufficient post-training is applied. While we cannot entirely disentangle the contributions of individual factors, such as the LLM backbone, data size, and data sources, or the image tokenizer, our observations align with our hypothesis: larger models exhibit stronger representational power to learn correct behaviors from data, whereas smaller models are more prone to language bias
B EXPERIMENT DETAILS
Table 11: DoLA Hyperparameters.
Parameters
Value
Adaptive Plausibility Threshold Early Exist Layers
0.1
[0. 2, 4, .., 32]
Implementation of Baselines. In Table 11, Table 12, Table 13, and Table 14, we present the hyper- parameters for the baseline methods: DoLA, OPERA, VCD, and HALC. These parameters follow
19

---

Published as a conference paper at ICLR 2025
LJama-3.22 (.tee-3.2-11 18-Vision
ULAVA-1.S-38
2.00
Head Indox
-19
24-
-2.60
-1.15
Layer Index
Layer Index
(a)
(b)
Chameleen--00
LLAVA-1 6-34B
0000
0,000
0.000
15
-0905
Index
Layer
(c)
(d)
Figure 16: Contrastive influence maps of attention heads in Llama-3.2-11 1--a-...--IBB-isiooo LLAVA-v].5 13B, Chameleon-30B, and LLaVA-v1.6-22 Bluer boxes indicate heads more responsible for gen- erating hallucinated objects, and redder boxes indicate those more responsible for generating non- hallucinated objects.
Table 12: OPERA Hyperparameters.
Parameters
Value
Self-attention Weight Scale Factor
50
Attending Retrospection Threshold
15
Beam Size
5
Penalty Weights
1.0
Table 13: VCD Hyperparameters.
Parameters
Value
Amplification Factor
1
Adaptive Plausibility Threshold Diffusion Noise Step
0.1
500
the official configurations provided by their respective sources. For the LURE baseline, we use the MiniGPT-4 13B checkpoint provided in the official repository as the revisor. We observed that, after generating the revised response, LURE's final step involves splitting the response into two parts based the "(n" symbol and only retaining the first part. However, as most responses generated by LLaVA-7B contain "nn" in the middle, this split function significantly shortens the response length, reducing it by nearly 40% when the maximum generated token length is 128. This could compromise the fairness the comparison. Therefore, we bypassed this post-processing step and directly used the full output response from the revisor the final prediction for LURE.
20

---

Published as a conference paper at ICLR 2025
Table 14: HALC Hyperparameters.
Parameters
Value
Detector
Dino
JSD Buffer Size m Beam Size
Number of Sampled FOVs
Exponential Ratio Growiing Contextual Field
Bounding Box Threshold for Dino
0.4
Adaptive Plausibility Threshold
0.1
Implementation details of Algorithm 1. We select the top 20 attention heads as hallucination heads for LLaVA-7B and the top 10 heads for MiniGPT-4. For the threshold J to control the when deactivate text attention in decoding,
sweep search find an optimal value. Based this, we set = 0.4 for LLaVA-7B and r = 0.5 for MiniGPT-4.
Implementation details of Algorithm 2. For LLaVA-7B model, dataset and fine-tuning codes from the offical Github repo The learning rate is set to 2 x 10-5, and the global batch size is 128. We fine-tune the model for 200 steps, selecting the top 30 hallu- cination heads for tunent For the MiniGPT-4 model, we use the dataset and fine-tuning codes from the offical Github repo ".' The learning rate set to 10-5 and the global batch size is 128. We fine-tune the model for 200 steps, selecting the top 20 hallucination heads for fine-tuning. For both models, the penalty weight À set to We only fine-tune the Query and Key matrices of the attention heads, as this operation modifies how values are linearly combined in self-attention without altering the basis of the linear space. We find that additionally fine-tuning the value matrices inef- fective (i.e., resulting in a higher hallucination rate of CHIRS; 20.86% on LLaVA-7B and lower generation quality). We conjecture that this is because fine-tuning the value matrices significantly disrupts the linear space of the original model.
the instruction-tuningg
21

---

Published as a conference paper at ICLR 2025
Table 15: Generated images description examples.
Instruction Greedy
Please describe this image in detail.
The image features white dog playing with frisbee in a grassy field. The dog holding the frishee in its mouth, running and enjoying the game. The frishee positioned towards the right side of the dog. and the dog appears to be the middle of the field. 'n',n In the background, there is bench located on the left side of the field. possibly for spectators to sit and watch the dog play.
AD-HH (Ours)
The image features white dog playing with frishee in grassy field. The dog holding the frisbee in its mouth, running and playing with the frishee the field. The dog appears to be enjoying the game and actively engaged in the activity.
TF-HH (Ours)
The image features white dog playing with frisbee in field. The dog is in the process of catching the frisbee, which is positioned in the middle of the scene. The dog is running lowards the frisbee, displaying its agility and excitement. \n'n The field spacious, providing ample room for the dog to play and enjoy its frishee game. The dog's enthusiasm and the frisbee's 's position in the air create a lively and engaging scene.
Instruction
Please describe this image in detail.
Greedy
The image features a man standing next to a parked motorcycle, which is prominently displayed in the scene. The man is wearing a helmet and appears to be the owner of the motorcycle. The motorcycle is positioned in the foreground, occupying signif- icant portion of the image. (n',n In the background, there is bench situated near the motorcyele, providing place for the man to sit and relax. The scene also includes backpack placed the ground, possibly belonging the man someone else in the area.
AD-HH (Ours)
The image features a man standing next to a parked motorcycle, which is parked on dirt road. The man is wearing helmet and appears to be the owner of the motorcycle. The motorcycle is positioned on the left side of the image, with the man standing on the right s In addition the motoreycle, there is bench located in the background, providing place for the man to rest or take a break. The scene appears to be set in wooded area, with trees surrounding the area.
TF-HH (Ours)
The image features a man standing next to parked motorcycle, wearing black jacket and sunglasses. He is posing for the camera, possibly in a wooded area. The motorcycle is parked the side of the road, and the man is standing close it. i There are several trees in the background, and bench can he seen further back in the scene. The man appears be the main focus of the image, with the motorcycle and the surrounding environment serving as the backdrop.
22

---

Published as a conference paper at ICLR 2025
Table 16: Generated images description examples.
Instruction Greedy
Please describe this image detail.
The image features a bathroom with two toilets placed side by side. Both toilets white and have their lids up, revealing their bowls. The toilets positioned close each other, with slightly closer the left side of the room and the other slightly closer the right side. n'The bathroom also contains sink, which is located wards the left side the room. The sink accompanied by mirror, which placed above it. The overall scene gives the impression public restroom or a shared hath- room space.
AD-HH (Ours)
The image features bathroom with two toilets placed side by side in bathroom stall. Both toilets white and appear be in good condition. The toilets are positioned close each other, with one toilet slightly closer the left side the bathroom and other toilet slightly closer to right side, n'TTe hathroom appears to clean and well- well-maintained, with the toilets being the main focus of the image. The bathroom stall appears to be public restroom, as it is designed to accommodate multiple users at.
TF-HH (Ours)
The image features a bathroom with two toilets placed side by side. Both toilets are white and appear to he in good condition. One of toilets positioned closer to left side of the image, while the other is situated more towards the right side. \nin The bathroom also features a tiled floor, which adds to the overall clean and well- maintained appearance of the space.
Instruction
Please describe this image in detail.
Greedy
The image features blue plate filled with delicious meal consisting of pasta and broccoli. The pasta is mixed with creamy sauce, and the broccoli pieces are scattered throughout the dish. The plate is placed on dining table, and there sandwich on the table well. \n',nl addition to the main dish, there one located near top left corner and the other near the top right corner. keyboard can seen the background, suggesting that the dining area part home office or work
buwls in the scene,
AD-HH (Ours)
The image features blue plate filled with delicious meal consisting of and bruccoli. The plate is placed dining lable, and the plate is filled with generous portion of pasta and broccoli. The pasta appears to mix of macaroni and shells, main dish, of hread on the table, likely to be used a side or to accompany the meal. The dining table occupies most of the image, and the dining
and broccoli scattered throughout the dish. A nn addition there is
TF-HH (Ours)
The image features blue plate filled with delicious meal consisting of pasta and broccoli. The plate is placed piece bread. The pasta and broccoli are arranged a visually appealing manner, creating a colorful and appetizing presentation. The dish appears to be a healthy and nutritious meal, perfect for meal or a light lunch.
wooden table, and the dish accompanicd by
23


Reviews:
[{"review_id": "tZ1oyGKmjn", "reviewer": "Reviewer_pX1o", "summary": "The paper observes that some attention heads cause more changes after deactivating them, and name them hallucination heads. There are several characteristics of these heads: (1) they attend more on the images and (2) their parameters move slower than others. Then, the paper proposes one training-free and one training approach to reduce the attention of the hallucination heads on text parts. The experiments on several benchmarks show the benefits of the proposed approach.", "questions": "1. How this method can extend beyond object-level hallucination? For example, do equation (1) and (2) only tell if the existence of the objects but cannot capture if the model incorrectly counts the objects? Then the proposed method can not detect the heads which make hallucination on counting.\n\n2. How many samples is required to compute equation (1)? And what data split do they come from?\n\n3. What's the time required to compute equation (2) to get the scores for all heads?\n\n4. Why the Algorithm 1 directly deactivated the whole text attention weights for the hallucination heads? Isn't it too aggressive as we can see the BLEU scores drop by a lot in Figure 6 (c)? Moreover, why the accuracy on general tasks somehow improves when you apply Algorithm 1? I thought the performance should drop based on the observation in Figure 6 (c).\n\n5. What is the difference between downscaling weight on text attention and upscaling the weight on image attention (Figure 6 (a) and (b))? I thought they meant the same thing as the softmax in attention would keep the sum of text attention and image attention to be one.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The observation and analysis on the hallucination heads are interesting and well-motivated.\n\n2. The proposed solutions don't require expensive decoding time.\n\n3. The results on several benchmarks look promising.", "weaknesses": "1. In Section 4.3, the metrics (CHAIR) and the dataset that is evaluated should be explained.\n\n2. In Tables 1 and 2, it would be better if there is an additional column indicating which method is training-free and which is not.\n\n3. The rest please refer to Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JUsR9A8oEm", "reviewer": "Reviewer_JmFc", "summary": "This work empirically studies the hallucination problem of LVLMS via counterfactual analysis. This work reveals multiple interesting findings on why hallucination occurs and how to mitigate it. The analyses are performed with two models (LLaVA-7B and MiniGPT-4) on two benchmark datasets (COCO and MM-Vet).", "questions": "1. More in-depth analysis would be required to discuss why the fine-tuning approach is not as good as the training-free one.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This work shows several interesting observations about hallucination, as summarized in the sentences in the bold font in section 4. \n\n2. Based on the findings, this work proposes two simple ideas to mitigate hallucination – adaptive deactivation and targeted fine-tuning of hallucination heads.", "weaknesses": "1. This work fails to cite several related recent works as follows, to name a few.\n- They may need to be cited and compared in experiments. \n- A. Deng et al., Seeing is Believing: Mitigating Hallucination in Large Vision-Language Models via CLIP-Guided Decoding, arXiv 2024. \n- F. Liu et al., Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning, ICLR 2024. \n- J. Zhang et al., Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Models, ECCV 2024. \n\n2. The evaluation only depends on two CHAIR metrics. \n\n3. Some findings look obvious.\n- It is not so surprising that the multi-head-attention is more critical than feed-forward network, since the former takes a majority of parameters and does much more things than the latter in the transformer.\n- It is a well-known fact that hallucination is more related to language bias (rather than) image bias and multimodal models often ignore the information from images compared to that from text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1zol9TanLN", "reviewer": "Reviewer_Ex6k", "summary": "The paper addresses the issue of hallucination in Large Vision-Language Models (LVLMs), specifically focusing on why these models generate content that deviates from the provided image information in open-ended tasks like image captioning. The authors conduct a systematic investigation using causal mediation analysis and counterfactual edits to identify the internal components responsible for hallucination. They find that Multi-Head Attention (MHA) modules contribute more to hallucination than Multi-Layer Perceptron (MLP) modules, and within MHAs, certain attention heads—termed \"hallucination heads\"—are primarily responsible. To mitigate hallucination, the paper proposes two methods: (1) an adaptive deactivation of hallucination heads during decoding, which is training-free and can be applied directly, and (2) targeted fine-tuning of hallucination heads to reduce their reliance on text tokens. Both methods demonstrate significant reductions in hallucination rates on benchmark datasets like COCO captioning and Nocaps, outperforming existing baselines.", "questions": "- Can you provide more details on why certain hallucination heads exhibit slow changes during instruction tuning? What factors contribute to this \"laziness,\" and how might future work address this issue?\n- Have you evaluated how the proposed interventions affect the overall language generation quality of the models? Specifically, does reducing reliance on text tokens in hallucination heads impact the fluency, coherence, or descriptiveness of the generated captions? It would be helpful to see metrics (human eval) or analyses addressing potential trade-offs.\n- Do you think \"hallucination heads\" exist in the same way in larger scale (eg. 70b) VLMs? Would the same training-free method work similarly on them? Would be interesting to see.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The paper deploys a systematic step-by-step approach to identify the components responsible for hallucinations. The insights are valuable and the analysis seems sensible.\n- Identifying that only specific attention heads seem to contribute to hallucinations is a novel finding. The additional analysis they did (fig 5) is really interesting.\n- The mitigation strategies shown (both training-based and training-free) seem sensible and seem to work well on the benchmarks.\n- They show the method working with multiple datasets and multiple models.", "weaknesses": "- There is limited discussion on how the proposed interventions affect the model's overall language generation capabilities. Potential trade-offs between reducing hallucination and maintaining fluency or coherence are not thoroughly examined.\n- The paper is almost entirely focused on object hallucination.\n- The experiments are conducted on 7B parameter models (LLaVA-7B and MiniGPT-4). Given the trend towards larger models in the field, it would be valuable to assess whether the identified hallucination heads and mitigation strategies are applicable to larger models (e.g., 70B parameters) and whether similar patterns emerge at different scales.\n- There are minor issues with the writing and presentation that could be improved for clarity and professionalism. For example, phrases like \"Our Method Run Fast in Generation\" could be rephrased for better readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0HrGIGFpqO", "reviewer": "Reviewer_xVEm", "summary": "This study explores the causes of hallucination in large visual language models (LVLMs) during complex visual tasks and proposes mitigation measures. First, a causal analysis of specific modules of LVLM based on counterfactual editing found that the multi-head attention (MHA) module contributed more to the generation of hallucinatory words than the multi-layer perceptrons module. The study further identified attentional heads associated with hallucination, which are concentrated in the middle and deep layers of the model and show a strong attentional bias towards text markers. In addition, the patterns of these attentional heads remain relatively similar to the underlying language model and change more slowly during instruction tuning.", "questions": "(1) If it is a task like VQA, the generated text token is only one, and only ABCD is answered, whether this hallucination pattern does not exist. If it is the SFT method, then this result should also be applicable to VQA tasks?\n\n(2) It is written in the paper that the author has hallucinatory attention-head in the middle layer or deep layer. Why is this? The complete distribution of 32 heads was not seen in the supplementary materials.\n\n(3) Has it analyzed the difference between system_token+image token+prompt+output_token hallucination attention head and regular attention head? Is there any difference? Or can it only be observed through output-tokens?\n\n(4)What about the distribution of attention when judging LLMs hallucinations? LLMs unable to generate a caption to an image, right? Therefore, I don't think this kind of attention head observation at the decoder terminal is very reasonable.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) It is the first work to analyze the attention map of the output text token of LVLMs and the attention-map of the NLP model, which is enlightening.\n\n(2) This paper proposes two methods to alleviate the hallucination of multi-modal large model. One is to restrict the probability of generating tokens by improving transformer's decoder, and the other is to reduce over-reliance on text tokens by SFT. And the experiment proves that the two methods proposed in this paper are excellent.", "weaknesses": "(1) The training-free method is plug and play, but the methods applied in this paper are too few, only LLaVA and Minigpt4 are included, and the results of other models should be supplemented.\n\n(2) The paper discusses the attention-map relationship of text tokens between LVLMs/LLM. Although the setting is a hallucination problem, this mechanism should be attributed to the cause of LLM. So can datasets be extended to other datasets like ScienceQA, GQA,textVQA, POPE, mmbench, etc.?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)