Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering

Authors: Sheng Liu, Haotian Ye, James Zou
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Large Vision-Language Models, Multimodal large language model, Hallucination

Abstract:
Hallucination poses a challenge to the deployment of large vision-language models (LVLMs) in applications. Unlike in large language models (LLMs), hallucination in LVLMs often arises from misalignments between visual inputs and textual outputs. This paper investigates the underlying mechanisms of hallucination, focusing on the unique structure of LVLMs that distinguishes them from LLMs. We identify that hallucinations often arise from the sensitivity of text decoders to vision inputs, a natural phenomenon when image encoders and text decoders are pre-trained separately. Inspired by this, we introduce Visual and Textual Intervention (VTI), a novel technique designed to reduce hallucinations by steering latent space representations during inference to enhance the stability of vision features. As a task-agnostic test-time intervention, VTI can be easily applied to any problem without additional training costs. Extensive experiments demonstrate that it can effectively reduce hallucinations and outperform baseline methods across multiple metrics, highlighting the critical role of vision feature stability in LVLMs.

Decision: accept (spotlight)
Reviewer scores: 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1730357405094

thread_type:
official_review

<thread_object>
creator: reviewer_etnv

<content>
summary:
The authors propose that hallucinations in LVLMs are typically caused by misalignment between visual and text representations. Consequently, the authors demonstrate this conclusion by perturbing the input image to obtain a more robust visual representation. Based on this experimental approach, the authors introduce a comprehensive method for intervening in both image and text representations to mitigate hallucinations in LVLMs. Furthermore, the authors substantiate the effectiveness of their method through extensive experiments.

The paper makes the following contributions:

1. Analyzes the correlation between the stability of visual representations and hallucinations in LVLMs.

2. Proposes a method for separately perturbing text and image representations to achieve a more robust representation space.

3. Demonstrates the effectiveness of their method through extensive experiments.

strengths:
1. The writing of this paper is exceptionally clear. Additionally, the motivation is thoroughly argued, and the analysis of the method is comprehensive.

2. It is indeed a novel attempt to mitigate hallucinations in LVLMs by perturbing the representation space through a training-free approach.

3. The experiments in this paper are very comprehensive, demonstrating the effectiveness of the proposed method.

weaknesses:
1. Using variance changes to explain changes in representation stability is somewhat one-sided. For instance, in PCA, PC1 is typically the direction with the largest variance, and it seems inevitable that the variance would decrease after using VTI. It seems more fundamental to examine the stability of the representation distribution for images with different noises or masks.

2. There are issues with the comparison methods used in the experiments. OPERA and VCD are both methods based on decoding to mitigate hallucinations, with the former optimizing beam decoding and the latter optimizing nucleus sample decoding. However, the paper does not specify the exact settings for the decode methods of vanilla and VTI.

3. Some components need to undergo ablation experiments.
  
    3.a. The role of PCA lacks ablation experiments. In Figure 8, it appears to be comparing different methods of representation updating, but what would be the impact if average were used instead of PCA?

    3.b. In line 713, the mask ratio is set to 0.99, which is an unusually high value, but the author does not provide a detailed analysis of this parameter.

questions:
1. Are there other more fundamental methods to prove representation stability? For example, calculating the KL divergence of representations after adding various perturbations to the image.

2. To demonstrate that VTI, as an inference intervention method, is superior to OPERA and VCD, a fairer experimental setup would be to use VTI separately with beam search and nucleus sample to compare against OPERA and VCD, respectively. This would be similar to the experimental setup in PAI[1].

3. What would be the effect of using average to calculate $d_{l, t}$ compared to PCA?

4. When the mask ratio is set to 0.99, almost no image information is retained. Performing PCA on the representation of such input images seems to be pointless. Therefore, I believe it would be beneficial to include a comparison of what happens when the mask ratio is set to different values and when a fully masked image is directly set as the intervention direction.

[1] Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs
</content>
</thread_object>

</thread>
<thread>
creation date:
1730598018935

thread_type:
official_review

<thread_object>
creator: reviewer_cwnc

<content>
summary:
The hallucination problem of LVLM refers to the phenomenon where the text generated by the model does not align with the content of the image. Previous methods mainly focus on issues that do not distinguish between LVLMs and LLMs. The authors first discovered that hallucination mainly stems from the sensitivity of the text decoder (LLM backbone) to vision input.
- By adding perturbations (Gaussian noise or random masking) to the input images, the authors found that the image encoder's generated embeddings exhibited some unstable features.
 - After perturbing the images, the model's hallucination increased, but if the embeddings were averaged over multiple perturbed images, stabilizing the features, the hallucination was reduced.

Inspired by this, the authors proposed a test-time intervention method called Visual and Textual Intervention (VTI).
  - For a given model, the method requires a sample set where each sample includes an image, a hallucination-free caption, and a hallucinated caption.
  - The images in the sample set are perturbed and sent to the vision encoder, calculating the visual shifting vectors.
  - Additionally, the two captions are fed into the text decoder, calculating the textual shifting vectors.
  - During testing, a new sample is processed by the model as usual, but the shifting vectors are added to the latent hidden states of the vision encoder and text decoder, respectively. 

VTI effectively reduced hallucinations and outperformed previous methods across multiple benchmarks.

strengths:
- The motivation and the method are explained smoothly and easy to understand.
- The experiments cover different hallucination benchmarks and LVLMs with different vision encoders.
- As a task-agnostic test-time intervention, the proposed method, VTI, outperforms previous baseline methods.

weaknesses:
### Major
- The description of method in Section 3 is kind of too general. 
How many LVLMs are tested here? Which LVLM(s) is used? 
Are the test images from COCO? 
Are the observations generalized to different LVLMs?
- VCD [1], which has been cited and compared in the paper, also adds Gaussian noise to distort visual input. They mention “visual uncertainty amplifies hallucination” in their Figure 2, which might be related to “vision feature instability” explored in the paper. I’m wondering if the authors are inspired by VCD. It would be better if the authors can explain the differences between their exploration and VCD.
- I’m concerned with the generalization ability of VTI, which may be a common problem of existing hallucination mitigation methods. LVLMs are expected to do a wide range of tasks. When image or task domains are changed, will the current methods damage model performance? Only hallucination benchmarks are covered in the experiments, so I'm not sure if VTI will damage model performance on other tasks. 

### Minor
- It would be better to early explain the structure of LVLM in abstract or introduction, like “A large vision language model typically consists of a vision encoder and a large language model as a text decoder” in Lines 143 – 144. Currently, the paper refers to vision encoder and text decoder directly at the beginning, which I feel may be confusing to readers.
- The authors claim that “VTI can be easily applied to any problem *without additional cost*” in the abstract. I feel “*without additional cost*” is kind of confusing, because VTI actually needs additional data and tune its shifting vectors.
- Lines 285-286, the authors mention that “we use 50 examples with paired images, hallucinated and non-hallucinated responses to …”
I’m wondering where the authors get the images. Are the images from COCO?
- If the authors use more dimensions of PCA in calculating the shifting vectors, will the method get better or worse performance?

- Typos

In the abstract, the abbreviation “(LLMs)” is shown twice. In the main content, no “(LLMs)”. The instruction needs to borrow one from the abstract.

Lines 151-152, “… between the vision and text components of large vision-language models (LVLMs):” It is not necessary to emphasize the abbreviation here now that there is "LVLMs" in the previous content.

Line 214 “As the desirable visual direction should be readily applied to new image queries.” The sentence has no main clause. I guess the authors wanted to use “as”, following the previous equation. The same with Line 278.

Missing “,” in Equation (1)



----------------------------------------------------------
[1] Mitigating object hallucinations in large vision-language models through visual contrastive decoding. CVPR2024.

questions:
Please see the Weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685043143

thread_type:
official_review

<thread_object>
creator: reviewer_ea1m

<content>
summary:
The paper introduces a heuristic approach of reducing hallucination in Visual Language Models by linearly offsetting the latent representation of both image and language modalities. The key to this heuristics is to pre-compute a set of offset vectors for image/text token embeddings that are the results of data augmentation. In particular, for image modality, the author introduced perturbations to the image input and computed the average change (averaged across all perturbed image instances) in token embedding across each layer of the image encoder. At inference time, the first PC of these change vectors across multiple images were then added to the embedding of any query image. Similar procedure was repeated for language modality where the “perturbation” is hallucinatory captions generated by another LLM. The authors show that the method, while simple, is able to significantly reduce the degree of hallucination in several benchmarks.

strengths:
1. The problem of hallucination in VLM is significant and the method proposed is highly intuitive (and appears effective). It uses ideas from data augmentation and applies it directly to inference time without the need to retrain the model. 
2. The analysis in the paper is thorough. Not only does it include good ablation of the various aspects of the method (granted there aren’t many moving parts to begin with), it also included discussions of how the method could reduce hallucination (i.e. impact on attention)

weaknesses:
All in all, this was a good paper and a pleasure to read. To nitpick, one of the biggest concerns of heuristic methods of this kind is its general applicability in practice. For example, 
1. what type of perturbations are most important in image/text modalities? 
2. how are the vision/language directions differ between models? 
3. What about more general open-ended QAs that go beyond captioning tasks? How would you even generate these hallucinatory language examples when the task is open QA?

questions:
Beyond addressing some of the questions I raised under "Weakness" above, here are a few other changes/questions.

Minor:
1. In Figure 2, CHAIRs was first mentioned. It should be described in the caption. Furthermore, the three subplots don’t seem to correspond to one another. It’ll be helpful to highly what experiments were shown in left/middle plots (sigma value and ratio value) to connect them to the third figure.
2. Discussion of the methodology could be improved. In particular, the description of the “visual direction” around equation 1 is confusing. 

Questions:
1. What do the embedding changes actually look like? How much explained variance is the first PC? 
2. I don’t really understand how this linear offset can improve “feature stability”. If the embedding is to be viewed as a random variable whose variance is caused by input perturbation, then doing a linear offset _cannot_ change the variance of such random variable. Perhaps I missed something.
</content>
</thread_object>

</thread>
<thread>
creation date:
1743199498636

thread_type:
public_comment

<thread_object>
creator: ~kai_zhao1

<content>
title:
The sign of principal direction

comment:
Dear Authors,

Congratulations on your excellent work!

I have a question regarding the vision steering vector $d_{l, t}^{vision}$, which is the first principal component derived from the stack of vision shifting vectors:
$$
\[\Delta_{l, t}^{v1}, Delta_{l, t}^{v2}, ..., \Delta_{l, t}^{vN}\].
$$

Since the principal component defines only the **direction** of maximal variation in the visual shifting vectors  $\Delta_{l, t}^v$,
its sign is arbitrary. I mean, if $d_{l, t}^{vision}$ is the first principal component, $-d_{l, t}^{vision}$ also is.

However, in your implementation, you applied **only positive scaling factors**  $\alpha\in\\{0.1, 0.2, ..., 1.0\\}$
to shift the image embeddings.

I'm very curious about the results of using negative scaling factors, or alternatively using  $-d_{l, t}^{vision}$ as the principal component.

Looking forward to your insights!

Best regards,
Kai Zhao
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes VTI (Visual and Textual Intervention), a novel method to reduce hallucinations in Large Vision-Language Models (LVLMs) by steering the latent space representations. The key idea is that hallucinations often arise from the sensitivity of text decoders to vision inputs, which can be mitigated by stabilizing vision features through latent space interventions.

### Strengths:

1. Novel and Effective Solution
> "The problem of hallucination in VLM is significant and the method proposed is highly intuitive (and appears effective). It uses ideas from data augmentation and applies it directly to inference time without the need to retrain the model." - EA1M

2. Comprehensive Technical Analysis  
> "The analysis in the paper is thorough. Not only does it include good ablation of the various aspects of the method (granted there aren't many moving parts to begin with), it also included discussions of how the method could reduce hallucination" - EA1M

3. Strong Experimental Validation
> "VTI effectively reduced hallucinations and outperformed baseline methods across multiple benchmarks" - Cwnc

### Weaknesses:

1. Limited Analysis of Feature Stability
> "Using variance changes to explain changes in representation stability is somewhat one-sided. For instance, in PCA, PC1 is typically the direction with the largest variance" - eTNv

2. Experimental Setup Concerns
> "OPERA and VCD are both methods based on decoding to mitigate hallucinations [...] However, the paper does not specify the exact settings for the decode methods of vanilla and VTI" - eTNv

3. Implementation Details Need Clarification
> "The description of method in Section 3 is kind of too general. How many LVLMs are tested here? Which LVM(s) is used?" - Cwnc


### Justification:
Despite some technical concerns, this paper makes a significant contribution to addressing hallucinations in LVLMs. The strengths clearly outweigh the weaknesses for several reasons:

1. The method is practically valuable - it provides a training-free approach to reducing hallucinations, making it immediately applicable to existing models.

2. The technical foundation is solid - while there are some methodological concerns, the authors have demonstrated strong theoretical understanding and provided additional analyses during discussion.

3. The experimental validation is comprehensive - the method shows consistent improvements across different models and benchmarks.

The authors have also shown strong engagement during the discussion period and commitment to addressing all raised concerns in the final version.