Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations

Authors: Nicholas Jiang, Anish Kachinthaya, Suzanne Petryk, Yossi Gandelsman
Primary area: interpretability and explainable AI
Keywords: Vision language models, hallucinations, logit lens, interpretability

Abstract:
We investigate the internal representations of vision-language models (VLMs) to address hallucinations, a persistent challenge despite advances in model size and training. We project VLMs’ internal image representations to their language vocabulary and observe more confident output probabilities on real objects than hallucinated objects. We additionally use these output probabilities to spatially localize real objects. Building on this approach, we introduce a knowledge erasure algorithm that removes hallucinations by linearly orthogonalizing image features with respect to hallucinated object features. We show that targeted edits to a model’s latent representations can reduce hallucinations by up to 25.7% on the COCO2014 dataset while preserving performance. Our findings demonstrate how a deeper understanding of VLMs’ latent representations can enhance reliability and enable novel capabilities, such as zero-shot segmentation.

Decision: accept (poster)
Reviewer scores: 5, 8, 6, 5

Reviews and discussion:
<thread>
creation date:
1730555945369

thread_type:
official_review

<thread_object>
creator: reviewer_ll1m

<content>
summary:
The paper explores the internal representations of Vision-Language Models (VLMs) to address the persistent issue of hallucinations. The authors project VLMs' internal image representations onto their language vocabulary to identify differences in token output probabilities between real and hallucinated objects. They introduce a knowledge erasure algorithm, PROJECTAWAY, which removes hallucinations by linearly orthogonalizing image features with respect to hallucinated object features. The study demonstrates that targeted edits to a model's latent representations can reduce hallucinations while preserving performance. Additionally, the paper presents a method for zero-shot segmentation using the logit lens technique, showing comparable performance to state-of-the-art methods.

strengths:
- The paper presents a newmethod for reducing object hallucinations in VLMs by editing their latent representations and the introduction of PROJECTAWAY offers a new technique for selectively removing hallucinated objects from VLMs' outputs.

- The authors provide a thorough analysis of the internal confidence values for object presence and absence, offering empirical evidence that supports their claims.

weaknesses:
- While the paper focuses on object hallucinations, it does not explore the applicability of the methods to other elements of visual scenes, such as people, attributes, or actions. The editing approach may struggle with abstract or complex sentences involving object attributes or interactions, which are not explicitly addressed in the paper.

- Could the authors elaborate on the potential impact of their editing techniques on other aspects of model performance, such as accuracy in non-hallucination tasks?

- The paper's reliance on LLaVA and InstructBLIP as baseline MLLMs does not provide a comprehensive comparison with the latest state-of-the-art models.

questions:
-  Would the authors consider including comparisons with the latest MLLMs, such as those incorporating more advanced architectures or larger datasets, to validate the robustness of their approach?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730630131090

thread_type:
official_review

<thread_object>
creator: reviewer_4zea

<content>
summary:
This paper presents a novel approach to understanding and editing vision-language models' (VLMs) internal representations through vocabulary projection and linear orthogonalization. By introducing a knowledge erasure algorithm PROJECTAWAY, the authors demonstrate significant improvements in hallucination reduction (up to 25.7%) and achieve competitive performance in zero-shot segmentation, while providing new insights into how VLMs process visual information.

strengths:
1. The paper presents a novel approach to interpreting and editing VLM representations through vocabulary projection and linear orthogonalization, requiring no model retraining or external components.
2. The work provides insights into VLM behavior by revealing the relationship between internal confidence scores and object presence.

weaknesses:
1. The paper's main analysis and evaluations (Sections 3 and 4) are predominantly conducted under the assumption that hallucinated objects are known beforehand using ground truth annotations. While Section 5 addresses this limitation with a more realistic approach using internal confidence thresholds, this should have been the primary evaluation framework. The current structure potentially overestimates the method's effectiveness by evaluating under idealized conditions.
2. The paper's structure is suboptimal, with the main analysis focusing on scenarios using ground truth annotations while relegating the more realistic approach to the applications section. 
3. The choice to use the last token for multi-token object representations (e.g., "hot dog", "dining table") lacks sufficient justification and empirical validation. The paper does not analyze potential issues with this approach, such as cases where the last token might not be the most semantically meaningful (e.g., "traffic light" where "light" alone might be ambiguous) or how this choice affects the method's performance compared to alternatives like averaging all tokens or using the first token.

questions:
1. The paper uses the model's unembedding matrix to interpret intermediate layer representations, but this matrix is trained for the final output layer. Have you conducted any layerwise probing or training of separate unembedding matrices for intermediate layers? This could affect the reliability of interpreting earlier layer representations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691442925

thread_type:
official_review

<thread_object>
creator: reviewer_bjvo

<content>
summary:
The authors use Logit Lens to interpret the intermediate image representations in LVLMs. For a given image embedding, they extract the latent representation of the image embedding at a specific layer, taking the logit lens to get the probability distribution over the vocabulary. 
- The highest probability of an object across image representations and layers, can act as the internal confidence of VLMs. The confidences for objects present are significantly higher than those of objects not present in the image.
- The authors propose an algorithm, ProjectAway, erasing objects from image representations.
- Moreover, they find that, using the internal confidence values, they can localize the objects in the image patches.

The authors show three applications of their findings and the algorithm: hallucination detection, hallucination mitigation, and zero-shot segmentation.

strengths:
- The findings are well-written and easy to understand.
- The experiments are comprehensive, exploring different aspects of internal visual information and covering different tasks.
- The proposed approach achieves significant improvements or comparable performance to SoTA on three applications.

weaknesses:
### Major
- Is the unembedding matrix for image representations directly from the LVLM last layer, or trained by the authors?
- Previous papers report the modality gap between language and vision in VLMs. In my experiments, I also notice that the distribution of vision tokens are significantly different from that of textual tokens. So I’m surprised that the logit lens can be directly used in image representations. 
I’m curious about the classification accuracy of logit lens. For example, if we feed a patch of cat, how accurate is the logit lens method to identify it is cat.
- Lines 200-202, the authors “randomly sample a subset of” objects not present. I’m wondering if this random sampling will choose some objects “obviously” not present in the image, making the comparison of the internal confidence too easy. It might be better if the authors can show: the confidence distribution of objects that commonly appear with objects in the image but not present this time.
- Section 5.3, I think LLaVA tends to generate some very general class when classifying an image, like predicting "dog" instead of “husky”. Are the authors using the generated class name from LLaVA no matter what it is or using the ground truth label?

### Minor
- InstructBLIP and LLaVA are representative LVLMs, but recent LVLMs are using more complicated vision embedding techniques [1, 2]. I’m wondering if the proposed method can still work with these new architectures.
- If we want to detect or remove the hallucinated objects, the propose method needs to know the object name. I'm wondering if the proposed method can work on a popular hallucination benchmark POPE [3]? In POPE, every sample is a "yes or no" question, like "Is there a person in the image?"
- Other limitations like handling multi-token classes have been mentioned in the paper.

[1] LLaVA-NeXT. https://llava-vl.github.io/blog/2024-01-30-llava-next/

[2] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs https://arxiv.org/abs/2406.16860

[3] Evaluating Object Hallucination in Large Vision-Language Models. https://arxiv.org/abs/2305.10355

questions:
Please see the Weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695542350

thread_type:
official_review

<thread_object>
creator: reviewer_9mwt

<content>
summary:
The paper addresses the issue of hallucinations in Vision-Language Models (VLMs) by interpreting and editing their internal representations. The authors apply the logit lens technique to project image representations onto the language vocabulary, discovering that objects present in the image have higher internal confidence scores compared to hallucinated objects. Utilizing this insight, they propose a method to detect hallucinations within VLMs. Furthermore, they introduce a knowledge erasure algorithm called PROJECTAWAY, which linearly orthogonalizes image features with respect to hallucinated object features to remove hallucinations from the model's output. The method is evaluated on two state-of-the-art VLMs, LLaVA 1.5 and InstructBLIP, showing a reduction in hallucinations by up to 25.7% on the COCO2014 dataset while preserving overall performance. Additionally, the authors demonstrate that their approach enables zero-shot segmentation by spatially localizing objects using internal confidence scores.

strengths:
- The paper introduces a novel application of the logit lens technique to interpret the internal image representations of VLMs, providing new insights into how these models process visual information.
- The proposed knowledge erasure algorithm, PROJECTAWAY, is a simple yet effective method that relies solely on manipulating the internal features of the VLMs without requiring additional training or external modules.
- The approach enables zero-shot segmentation by leveraging internal confidence scores to spatially localize objects
- The paper seems clear and well-written.

weaknesses:
- The proposed method requires specifying weight factors and selecting specific layers to retrieve text representations and apply edits. These hyperparameters are determined through ablation studies and do vary between models, and likely between datasets as well, requiring cumbersome ablation process to find good numbers.
- The experiments focus primarily on object hallucinations in image captioning tasks. It is unclear how the method performs on other types of hallucinations (e.g., action or attribute hallucinations) or on other tasks such as visual question answering (VQA).
- The impact of the method on overall caption quality is not thoroughly evaluated quantitatively. While the authors mention that the method preserves performance and provide some qualitative examples, additional quantitative evaluations would be interesting to see.
- The authors only seem to test their model on COCO2014.

questions:
- How sensitive is the proposed method to the selection of weight factors and layers across different models and datasets? Is there a way to generalize these hyperparameters or make the method more robust to their selection?
- How does the method perform on other tasks, such as visual question answering (VQA) or on other datasets beyond COCO2014? Have you considered testing the method on benchmarks like LLaVA Bench or MM-Vet?
- Is there a way to automate or simplify the selection of hyperparameters (e.g., layers, weight factors) to make the method more practical for real-world applications?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731719871890

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
To all reviewers

comment:
We thank all the reviewers for providing valuable comments and feedback on our paper. The reviewers describe our application of the logit lens technique on VLMs to be a “novel approach” that provides “new insights on how these models process visual information” (4Zea03; 9Mwt03). They mention how our editing technique, ProjectAway, is a “simple yet effective method” that requires “no model retraining or external components”(9Mwt03; 4Zea03). Our methods produce 3 applications that “achieve significant improvements or comparable performance to SoTA” (bJVo03). The reviewers find our experiments to be “comprehensive…and covering different tasks” and a “thorough analysis of the internal confidence values for object presence and absence” (bJVo03; LL1m02). Many of the reviewers also praise the clarity of our findings, stating they are “well-written” and “easy to understand” (9Mwt03; bJVo03).

There are 3 common concerns the reviewers raised that we hope we addressed in this rebuttal:

### Evaluating our methods on more advanced, recent VLMs (LL1m02, bJVo03)

While LLaVA and InstructBLIP follow a similar architecture as most VLMs today, we conduct additional evaluations on more recent VLMs like LLaVA-NeXT and Cambrian-1, which are trained with more advanced techniques on better datasets. Our results in Appendix A.5 show that our editing technique, paired with model confidences, is able to significantly reduce hallucinations (>25%) consistent with our other results. They also demonstrate that our method is robust to different hyperparameter selections as we did not run ablation studies to optimize them. 

### Applying our method beyond object hallucinations (9Mwt03, LL1m02)

A few reviewers wanted to see our method applied beyond object hallucinations, such as to attribute (color, relation, object number) hallucinations. As it is difficult to get precise quantitative numbers due to the lack of standard benchmarks for attribute hallucinations, we instead provide qualitative examples in Appendix A.7 from a VQA task to demonstrate that our method can accurately detect and correct wrong answers to attribute-related (ex. “What color is <blank>?”) questions.

### Lack of justification for using last tokens for multi-token text representations (4Zea03, bJVo03)

Our editing technique, ProjectAway, uses text embeddings pulled from the last token of multi-token objects, and a few reviewers wanted further justification for this design choice. We primarily use the last token because past works have found models tend to store information about multi-token entities in the last token for later use. For example, [1] finds that the last subject token encodes crucial factual associations, and [2] demonstrates that information is carried over to the last token position through relation propagation and attribute extraction.

We hope that our additional evaluations and results address the concerns of the reviewers. We will incorporate the feedback into our paper and would be happy to hear of any further ways to strengthen our claims.

[1] https://rome.baulab.info/

[2] https://arxiv.org/abs/2304.14767
</content>
</thread_object>

</thread>
<thread>
creation date:
1732212493391

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We appreciate the time and valuable feedback provided by all the reviewers on our work. We are thankful that the paper has been positively received overall. As the discussion period is nearing its end, we kindly request that all reviewers confirm whether our rebuttal has addressed their concerns and allow us the chance to respond to any additional follow-up. Thank you once more for your participation.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes an algorithm to erase spurious knowledge from VLMs. The algorithm, coined ProjectAway, relies on Logit Lens to remove information about objects from image representations. The proposed approach is evaluated on several applications: hallucination detection, hallucination removal, as well as zero-shot segmentation. While reviewers raised some concerns regarding the practical applicability of the proposed approach (too much manual work), and some concerns regarding the evaluation, the work constitutes a good piece of work in the space of VLM interpretability. For the above reason, despite borderline ratings, I recommend accepting this paper.