Understanding and Mitigating Hallucination in Large Vision-Language Models via Modular Attribution and Intervention

Authors: Tianyun Yang, Ziniu Li, Juan Cao, Chang Xu
Primary area: foundation or frontier models, including LLMs
Keywords: Large Vision-Language Models, Hallucination, Interpretability

Abstract:
Large Vision-Language Models (LVLMs) exhibit impressive capabilities in complex visual tasks but are prone to hallucination, especially in open-ended generation tasks. This paper explores why LVLMs tend to hallucinate and how to mitigate it. First, we conduct causal mediation analysis through counterfactual edits on specific modules in LVLMs. Our results disclose that Multi-Head Attention (MHA) modules contribute more to the probability of generating hallucination words than multi-layer perceptron modules. We then identify specific heads that are responsible for hallucination, referred to as hallucination heads. Second, we examine the behavior of hallucination heads. We find that they are concentrated in the middle and deeper layers, displaying a strong attention bias toward text tokens. Further, we show that the attention patterns of certain hallucination heads exhibit greater similarity to the base language model and change slowly during the instruction tuning process. Finally, we propose two simple yet effective methods to mitigate hallucination: one is training-free and can be applied directly during decoding, while the other involves fine-tuning. Both methods are targeted for hallucination heads to reduce their reliance on text tokens. Notably, our methods achieve up to 1.7x reduction in hallucination rate for the LLaVA-v1.5-7B model in COCO captioning task, outperforming existing baselines. Overall, our findings suggest that hallucinations in LVLMs are likely to stem from certain modules, and targeted interventions can effectively mitigate these issues.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 8

Reviews and discussion:
<thread>
creation date:
1729223374524

thread_type:
official_review

<thread_object>
creator: reviewer_xvem

<content>
summary:
This study explores the causes of hallucination in large visual language models (LVLMs) during complex visual tasks and proposes mitigation measures. First, a causal analysis of specific modules of LVLM based on counterfactual editing found that the multi-head attention (MHA) module contributed more to the generation of hallucinatory words than the multi-layer perceptrons module. The study further identified attentional heads associated with hallucination, which are concentrated in the middle and deep layers of the model and show a strong attentional bias towards text markers. In addition, the patterns of these attentional heads remain relatively similar to the underlying language model and change more slowly during instruction tuning.

strengths:
(1) It is the first work to analyze the attention map of the output text token of LVLMs and the attention-map of the NLP model, which is enlightening.

(2) This paper proposes two methods to alleviate the hallucination of multi-modal large model. One is to restrict the probability of generating tokens by improving transformer's decoder, and the other is to reduce over-reliance on text tokens by SFT. And the experiment proves that the two methods proposed in this paper are excellent.

weaknesses:
(1) The training-free method is plug and play, but the methods applied in this paper are too few, only LLaVA and Minigpt4 are included, and the results of other models should be supplemented.

(2) The paper discusses the attention-map relationship of text tokens between LVLMs/LLM. Although the setting is a hallucination problem, this mechanism should be attributed to the cause of LLM. So can datasets be extended to other datasets like ScienceQA, GQA,textVQA, POPE, mmbench, etc.?

questions:
(1) If it is a task like VQA, the generated text token is only one, and only ABCD is answered, whether this hallucination pattern does not exist. If it is the SFT method, then this result should also be applicable to VQA tasks?

(2) It is written in the paper that the author has hallucinatory attention-head in the middle layer or deep layer. Why is this? The complete distribution of 32 heads was not seen in the supplementary materials.

(3) Has it analyzed the difference between system_token+image token+prompt+output_token hallucination attention head and regular attention head? Is there any difference? Or can it only be observed through output-tokens?

(4)What about the distribution of attention when judging LLMs hallucinations? LLMs unable to generate a caption to an image, right? Therefore, I don't think this kind of attention head observation at the decoder terminal is very reasonable.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730697894261

thread_type:
official_review

<thread_object>
creator: reviewer_ex6k

<content>
summary:
The paper addresses the issue of hallucination in Large Vision-Language Models (LVLMs), specifically focusing on why these models generate content that deviates from the provided image information in open-ended tasks like image captioning. The authors conduct a systematic investigation using causal mediation analysis and counterfactual edits to identify the internal components responsible for hallucination. They find that Multi-Head Attention (MHA) modules contribute more to hallucination than Multi-Layer Perceptron (MLP) modules, and within MHAs, certain attention heads—termed "hallucination heads"—are primarily responsible. To mitigate hallucination, the paper proposes two methods: (1) an adaptive deactivation of hallucination heads during decoding, which is training-free and can be applied directly, and (2) targeted fine-tuning of hallucination heads to reduce their reliance on text tokens. Both methods demonstrate significant reductions in hallucination rates on benchmark datasets like COCO captioning and Nocaps, outperforming existing baselines.

strengths:
- The paper deploys a systematic step-by-step approach to identify the components responsible for hallucinations. The insights are valuable and the analysis seems sensible.
- Identifying that only specific attention heads seem to contribute to hallucinations is a novel finding. The additional analysis they did (fig 5) is really interesting.
- The mitigation strategies shown (both training-based and training-free) seem sensible and seem to work well on the benchmarks.
- They show the method working with multiple datasets and multiple models.

weaknesses:
- There is limited discussion on how the proposed interventions affect the model's overall language generation capabilities. Potential trade-offs between reducing hallucination and maintaining fluency or coherence are not thoroughly examined.
- The paper is almost entirely focused on object hallucination.
- The experiments are conducted on 7B parameter models (LLaVA-7B and MiniGPT-4). Given the trend towards larger models in the field, it would be valuable to assess whether the identified hallucination heads and mitigation strategies are applicable to larger models (e.g., 70B parameters) and whether similar patterns emerge at different scales.
- There are minor issues with the writing and presentation that could be improved for clarity and professionalism. For example, phrases like "Our Method Run Fast in Generation" could be rephrased for better readability.

questions:
- Can you provide more details on why certain hallucination heads exhibit slow changes during instruction tuning? What factors contribute to this "laziness," and how might future work address this issue?
- Have you evaluated how the proposed interventions affect the overall language generation quality of the models? Specifically, does reducing reliance on text tokens in hallucination heads impact the fluency, coherence, or descriptiveness of the generated captions? It would be helpful to see metrics (human eval) or analyses addressing potential trade-offs.
- Do you think "hallucination heads" exist in the same way in larger scale (eg. 70b) VLMs? Would the same training-free method work similarly on them? Would be interesting to see.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720126008

thread_type:
official_review

<thread_object>
creator: reviewer_jmfc

<content>
summary:
This work empirically studies the hallucination problem of LVLMS via counterfactual analysis. This work reveals multiple interesting findings on why hallucination occurs and how to mitigate it. The analyses are performed with two models (LLaVA-7B and MiniGPT-4) on two benchmark datasets (COCO and MM-Vet).

strengths:
1. This work shows several interesting observations about hallucination, as summarized in the sentences in the bold font in section 4. 

2. Based on the findings, this work proposes two simple ideas to mitigate hallucination – adaptive deactivation and targeted fine-tuning of hallucination heads.

weaknesses:
1. This work fails to cite several related recent works as follows, to name a few.
- They may need to be cited and compared in experiments. 
- A. Deng et al., Seeing is Believing: Mitigating Hallucination in Large Vision-Language Models via CLIP-Guided Decoding, arXiv 2024. 
- F. Liu et al., Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning, ICLR 2024. 
- J. Zhang et al., Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Models, ECCV 2024. 

2. The evaluation only depends on two CHAIR metrics. 

3. Some findings look obvious.
- It is not so surprising that the multi-head-attention is more critical than feed-forward network, since the former takes a majority of parameters and does much more things than the latter in the transformer.
- It is a well-known fact that hallucination is more related to language bias (rather than) image bias and multimodal models often ignore the information from images compared to that from text.

questions:
1. More in-depth analysis would be required to discuss why the fine-tuning approach is not as good as the training-free one.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730755372732

thread_type:
official_review

<thread_object>
creator: reviewer_px1o

<content>
summary:
The paper observes that some attention heads cause more changes after deactivating them, and name them hallucination heads. There are several characteristics of these heads: (1) they attend more on the images and (2) their parameters move slower than others. Then, the paper proposes one training-free and one training approach to reduce the attention of the hallucination heads on text parts. The experiments on several benchmarks show the benefits of the proposed approach.

strengths:
1. The observation and analysis on the hallucination heads are interesting and well-motivated.

2. The proposed solutions don't require expensive decoding time.

3. The results on several benchmarks look promising.

weaknesses:
1. In Section 4.3, the metrics (CHAIR) and the dataset that is evaluated should be explained.

2. In Tables 1 and 2, it would be better if there is an additional column indicating which method is training-free and which is not.

3. The rest please refer to Questions.

questions:
1. How this method can extend beyond object-level hallucination? For example, do equation (1) and (2) only tell if the existence of the objects but cannot capture if the model incorrectly counts the objects? Then the proposed method can not detect the heads which make hallucination on counting.

2. How many samples is required to compute equation (1)? And what data split do they come from?

3. What's the time required to compute equation (2) to get the scores for all heads?

4. Why the Algorithm 1 directly deactivated the whole text attention weights for the hallucination heads? Isn't it too aggressive as we can see the BLEU scores drop by a lot in Figure 6 (c)? Moreover, why the accuracy on general tasks somehow improves when you apply Algorithm 1? I thought the performance should drop based on the observation in Figure 6 (c).

5. What is the difference between downscaling weight on text attention and upscaling the weight on image attention (Figure 6 (a) and (b))? I thought they meant the same thing as the softmax in attention would keep the sum of text attention and image attention to be one.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732354193765

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response and Summary of Changes

comment:
We sincerely thank all reviewers and area chairs for their efforts in reviewing our paper and providing valuable feedback. We have carefully addressed each concern raised by the reviewers and made corresponding revisions to our paper. We highlight the revision in blue. Below, we summarize the key changes in our revision:
- **Section 2 (Page 2)**: We added a discussion of related literature, including [Deng et al., 2024], [Liu et al., 2024], and [Zhang et al., 2024]. Different from these works, our work investigates hallucination through lens of attribution and intervention and designs targeted mitigation strategies. 
- **Section 4.3 (Page 6)**: We included descriptions of the metrics (CHAIR) and the dataset used to produce Figure 6, enhancing the figure's readability.
- **Section 5.2 (Table 1,2)**: We added symbols to denote whether methods are training-free or training-based, improving the clarity of the tables.
- **Appendix A.2**:
  - Complete attention maps (Page 16, Figure13): Figure 13 compares the complete attention maps of two typical hallucination and non-hallucination heads across system, image, question, and output tokens. The hallucination head exhibits a clear over-reliance on text tokens. We also add an discussion on the relationship between the text-token over-reliance behaviour and hallucination heads: while hallucination heads frequently show over-reliance on text tokens, not all heads that over-rely on text tokens are hallucination heads. 
- **Appendix A.3**: 
  - Human-evaluated generation quality (Page 18, Table 6). Table 6 presents the results of a human-based evaluation, focusing on hallucination and generation quality. In this evaluation, our method maintains consistent generation quality comparable to the baseline, while also showing improvements in hallucination reduction.
  - Comparison with additional baselines (Page 18, Table 8). Table 8 provides comparisons with additional three baseline methods, including [Deng et al., 2024], [Liu et al., 2024], and [Zhang et al., 2024]. Although these baselines also achieve a reduction in hallucination rates compared to greedy decoding, our approach, leveraging targeted interventions, demonstrates a greater reduction in hallucinations compared to these baselines.
  - Evaluation on larger and more recent LVLM models (Page 19, Table 9). Table 9 extends the evaluation to larger and more recent models, including Llama-3.2-11B-Vision, LLaVA-v1.5-13B,  Chameleon-30B, and LLaVA-v1.6-34B. Our method achieves up to 10 points in hallucination reduction on these models.
  - Hallucination behaviour across different scales of LVLMs (Page 19, Table 10). Table 10 provides an empirical results on the number and ratio of salient hallucination heads across different scales of LVLMs. The findings indicate that hallucination heads tend to diminish as model size increases and sufficient post-training is applied.

We believe the revisions outlined above have significantly enhanced the quality of our paper, thanks to the insightful feedback from the reviewers. We hope these updated results address the reviewers' concerns and further strengthen the contributions of our work. Thank you once again for your valuable feedback.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper investigates hallucination in Large Vision-Language Models (LVLMs) through the lens of causal attribution and intervention. The work identifies specific "hallucination heads" within the multi-head attention mechanism and proposes two methods to mitigate hallucination: a training-free adaptive deactivation approach and a targeted fine-tuning strategy. The paper has solid experiments, and provides a systematic analysis for hallucination in VLMs. The paper also had good responses to questions presented by reviewers. For these reasons I vote to accept this paper!