Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations

## Abstract

Abstract We investigate the internal representations of vision-language models (VLMs) to address hallucinations, a persistent challenge despite advances in model size and training. We project VLMs’ internal image representations to their language vocabulary and observe more confident output probabilities on real objects than hallucinated objects. We additionally use these output probabilities to spatially localize real objects. Building on this approach, we introduce a knowledge erasure algorithm that removes hallucinations by linearly orthogonalizing image features with respect to hallucinated object features. We show that targeted edits to a model’s latent representations can reduce hallucinations by up to 25.7% on the COCO2014 dataset while preserving performance. Our findings demonstrate how a deeper understanding of VLMs’ latent representations can enhance reliability and enable novel capabilities, such as zero-shot segmentation. 1 1 1 Code: https://github.com/nickjiang2378/vl-interp

## 1 Introduction

Vision-Language Models (VLMs) have recently emerged as powerful tools for understanding images via text (Dai et al., ; Liu et al., ). They have demonstrated remarkable capabilities across multimodal tasks such as image captioning (Li et al., ), visual question answering (Ye et al., ),
and complex multimodal reasoning (Bai et al., ). Despite their capabilities, VLMs tend to hallucinate content that does not appear in the images (Ji et al., ), which poses serious concerns for the reliability of these models in real-world applications (Hu et al., ; Luo et al., ).

Widespread belief has been that scaling to larger models and more training data will naturally mitigate hallucinations. However, recent studies have shown that hallucinations persist even in larger and more advanced models (Rohrbach et al., ; Li et al., ), suggesting that this issue cannot be solved by scale alone. Current methods reduce hallucinations by applying external interventions (e.g. object detectors; Yin et al. ()) or additional model fine-tuning (e.g. on hallucination examples; Zhou et al. (); Zhang et al. ()).
Nevertheless, these methods often struggle to distinguish between subtle hallucinations and existing details, requiring new models or updated model parameters.

In this paper, we aim to introduce fine-grained edits directly to the image latent representations of VLMs to reduce hallucinations without hindering their performance, an approach that has had some success in large language models  (Zhang et al., ; von Rutte et al., ). To edit the latent representations of VLMs, we first explain their role via text. We employ the logit lens technique (nostalgebraist, ) to directly interpret the spatial VLM image representations with VLM text vocabulary. Surprisingly, the characteristics of these image representations are different for real objects that appear in the image and objects that are hallucinated. Moreover, the logit lens enables spatially localizing objects within the input image.

Relying on the ability to detect hallucinated objects, we edit them out by intervening in their internal representations. We introduce a knowledge erasure algorithm, ProjectAway, to target and remove objects by linearly orthogonalizing image features with respect to the text features of target objects. We find that ProjectAway can erase both real and hallucinated objects with high rates of removal.

Figure: Figure 1: Interpreting VLM internal image representations. (a) Given a VLM, (b) we unembed the latent representations from image embeddings to the vocabulary and classify hallucinations. We remove hallucinations by (c) linearly editing them out of the latent representations.
Refer to caption: x1.png

We use our interpretation and editing approach for three tasks. First, we utilize the logit lens on image features to detect hallucinations in the image. We find that it improves mAP by 22.45% and 47.17% in two VLMs. Then, we combine our editing and detection method to erase hallucinations from the VLM’s internal representations, reducing hallucinations up to 25.7% on standard benchmarks, while preserving accuracy in image captioning. Finally, we use the logit lens to localize objects in the image features. We find that our spatial mapping provides comparable performance to state-of-the-art zero-shot segmentation methods. Our results indicate that understanding the internal representations of VLMs can be achieved and used to repair model hallucinations and introduce new capabilities.

## 2 Related work

### 2.1 Interpreting Latent Representations in Language Models

Interpreting the inner workings of large language models enables fine-grained improvement of the language model behavior. Recent work involves utilizing the model’s attention maps (Kobayashi et al., ; Chefer et al., ), activation patterns (Conmy et al., ; Meng et al., ; Bronzini et al., ), and latent representations (Ghandeharioun et al., ; Cunningham et al., ; Bricken et al., ) to understand their behavior with applications such as early exiting (Halawi et al., ) and editing or erasing the model’s knowledge (Dai et al., ; Ravfogel et al., ). One class of methods probe the VLMs knowledge with linear classifiers (Hewitt & Manning, ; Tucker et al., ; Li et al., ; Belrose et al., ). The logit lens method (nostalgebraist, ), which we will use in our analysis, finds the output distribution over the vocabulary of the language model at intermediate layers with the model’s own unembedding matrix. We apply this approach to VLMs to interpret the model’s understanding of visual information in the model’s textual vocabulary.

### 2.2 Interpreting latent representations in Vision Models

Understanding the internal dynamics of vision models is critical for ensuring safety and reliability in multimodal systems. Early works in this area focused on producing saliency maps (Petsiuk et al., ), analyzing individual neurons (Bau et al., ; ; Dravid et al., ), and training networks to map latent representations to concepts (Esser et al., ). With the emergence of transformer-based vision models like CLIP (Radford et al., ), recent methods explain latent tokens  (Chen et al., ) and the roles of attention heads and neurons with natural language (Gandelsman et al., ; ; Shaham et al., ). Few works currently interpret the internal computation of VLMs: Palit et al. () develop a neuron causal tracing tool; Schwettmann et al. () identifies multi-modal neurons; and Huo et al. () ablates domain-specific neurons to improve vision question-answering. Whereas past papers have primarily studied the mechanisms (e.g. neuron analysis) that drive VLMs, we focus on interpreting and editing their latent representations for real-world applicability.

### 2.3 Detecting and reducing VLM hallucinations

While VLM performances on image caption and visual question answering are continually improving, they continue to hallucinate facts that are not supported by the visual input. Existing methods for detecting hallucinations in language models during inference utilize latent representations (He et al., ; Su et al., ), activations (Chen et al., ), and output logit values (Varshney et al., ). SAPLMA (Azaria & Mitchell, ) trains a hallucination classifier on the internal latent representations. LUNA (Song et al., ) learns a transition function on latent representations and identifies abnormal transitions. Varshney et al. () uses the final layer logits to score the model’s confidence in an entity or keyword and intervenes by instructing the model to either repair or remove the hallucinated information. Among VLMs, LURE (Zhou et al., ) is a fine-tuned revisor model to detect and reduce hallucinations. OPERA (Huang et al., ) uses the model’s internal attention weights to detect and suppress patterns that align with the beginning of hallucinated phrases. In contrast to these methods, we leverage the internal image representations in the VLMs for hallucination reduction and for zero-shot segmentation.

## 3 Extracting Knowledge from VLMs

We start by introducing VLMs and the general framework of their architectures in most recent work. We then describe our approach for decoding the features in intermediate image representations in VLMs into text, and apply it to two types of VLMs. Surprisingly, this approach effectively probes the knowledge about objects present in images and can localize objects within the image.

### 3.1 Preliminaries

Vision-Language Models. The architecture of recent state-of-the-art VLMs for text generation typically involves three main components: a vision encoder to process image inputs, a mapping network to map image features to image embeddings, and an autoregressive language model to process the image embeddings and prompt embeddings to generate text. We focus on two recent state-of-the-art VLMs: LLaVA 1.5 (Liu et al., ) and InstructBLIP (Dai et al., ). We use 7B versions of both these models. LLaVA utilizes a frozen CLIP vision encoder and an MLP as a mapping network to project the vision encoder outputs into image embeddings for the language model. The MLP is pre-trained on a large vision-language dataset and both the MLP and the language model are fine-tuned on an instruction-focused dataset. In contrast, InstructBLIP freezes both the vision encoder and the language model and only trains the mapping network.

Notations. For the purposes of our work, we define the VLM architecture as follows. The vision encoder processes an input image to produce $n$ image features. These image features are projected to embedding space via the mapping network, resulting in $n$ $d$-dimensional image embeddings $\{k_{i}:k_{i}\in\mathbb{R}^{d},i=1,...,n\}$. For the language model, the entire set of text tokens constitutes the vocabulary $V$ with vocabulary size $|V|$. The image embeddings, followed by $m$ text embeddings $\{t_{i}:t_{i}\in\mathbb{R}^{d},i=1,...,m\}$ of the prompt tokens, are input to the language model through $L$ decoder layers. For an input embedding $x\in\mathbb{R}^{d}$, we define $h_{l}(x)\in\mathbb{R}^{d}$ to be the latent representation for embedding $x$ at layer $l\in\{1,...,L\}$, the output of the decoder layer, which is conditioned on previous tokens of the input sequence. An unembedding matrix $W_{U}\in\mathbb{R}^{|V|\times d}$ maps the last latent representation $h_{L}(t_{m})$ to a probability distribution over the vocabulary for the next token $t_{m+1}$.

Logit Lens. Logit Lens is an interpretability method for intermediate language model representations introduced in Section 2.1. The logit lens technique applies the unembedding matrix $W_{U}$ to latent representations $h_{l}(x)$ in the $L$ intermediate layers in the language model to retrieve the logit distributions over the vocabulary.

$$ $f_{l}(t_{m})=W_{U}\cdot h_{l}(t_{m})=[\text{logit}_{1},\text{logit}_{2},\text{ logit}_{3},\ldots,\text{logit}_{|V|}]$ (1) $$

This is the logit distribution representing the predictions of the model after $l$ layers, where $\text{logit}_{j}$ corresponds to the token $j$ in the vocabulary.

### 3.2 Applying Logit Lens on VLMs

We apply the logit lens to probe the language model as it processes the image representations. This enables us to interpret the image features’ output distributions as they are transformed by the layers of the language model and localize objects spatially within the image.

Extracting probability distributions from intermediate image representations. We apply logit lens on the image representations in the VLM. For a given image embedding $k_{i}$, we find the latent representation of the image embedding at layer $l$, $h_{l}(k_{i})$, taking the logit lens to get the probability distribution over the vocabulary, $\text{softmax}(f_{l}(k_{i}))$. We define an object $o$, an object word composed of tokens from the vocabulary. We inspect the probability of a specific object $o$, $\text{softmax}(f_{l}(k_{i}))_{o}$. For multi-token objects, we take the maximum probability value over the object tokens. This provides a generalizable framework for analyzing specific latent image representations via text, with respect to specific objects. Next, we find the maximum probability over all image representations over all layers. For object $o$, we compute:

$$ $c_{o}=\max_{\begin{subarray}{c}1\leq l\leq L\\ 1\leq i\leq n\end{subarray}}\{\text{softmax}(f_{l}(k_{i}))_{o}\}$ (2) $$

We define $c_{o}$ as the VLMs internal confidence of an object $o$ existing in the image: the highest probability of object presence across $n$ image representations through $L$ layers of the language model.

Comparing the internal confidence of present and not present objects. To determine if internal confidence provides meaningful information about objects in the image, we examine $c_{o}$ for objects present and not present in an image. We use InstructBLIP and LLaVA to caption 5000 random COCO2014 images in the Karpathy validation split (Lin et al., ) and determine $c_{o}$ for all 80 COCO objects, only a few of which are present in each image. Since there are many more objects not present than present, we randomly sample a subset of the internal confidences for objects not present. Figure 2 exhibits the internal confidences for objects present and not present in the image. We empirically find that the VLMs’ internal confidences are higher for present objects than not present ones. We use this claim later to classify objects as hallucinations in Section 5.1.

Figure: Figure 2: Comparison of internal confidence in objects present and not present in the image. We examine the internal confidence of COCO objects that exist and do not exist in the image within intermediate VLM image representations. We observe that objects that do not exist in the image have lower internal confidence.
Refer to caption: x2.png

Figure: Figure 3: Localizing objects using internal confidence values. We find the probabilities of objects through layers of the language model for every image embedding in LLaVA. We use the highest layer probability per image embedding to localize an object within the image.
Refer to caption: x3.png

Object localization. Given that the language model can distinguish between objects present and not present in an image, we examine whether it can attribute high object internal confidence to specific patches in an image. For each image embedding $k_{i}$ in $n$ image embeddings, we find the maximum softmax probability of an object within the layers of the model, $\max_{1\leq l\leq L}\{\text{softmax}(f_{l}(k_{i}))_{o}\}$. Using these internal confidence values, we localize the objects in the image patches, each of which maps to an image embedding. We focus on LLaVA for this task, since its image encoder preserves the spatial mapping of image patches to image features.

We observe that image representations that exhibit higher internal confidence for specific objects correspond to the image patches in which those objects are visually present (examples in Figure 3). Building on our previous observation, we see that the intermediate image representations semantically align with latent token representations of objects present in them while maintaining their spatial locality. We use this unique finding for zero-shot segmentation in Section 5.3.

While the model is not directly trained to map the image representations closer to the text representations of objects within them, we can unembed the image representations in the text vocabulary for localization and find differences in internal confidence for present and hallucinated objects. In Section 5.1, we will use this observation for various applications including hallucination detection and zero-short segmentation.

## 4 Erasing knowledge from VLMs

Recognizing that image embeddings are directly interpretable (Section 3.2), we edit these embeddings to erase the presence of objects from image captions. We propose a linear editing algorithm that subtracts the text embedding of a target object from all image embeddings. When applied on singular and multiple object removals, we find that it erases hallucinated objects more effectively than correctly detected (CD) objects (i.e. real objects that the model correctly detects).

### 4.1 Erasing objects from image representations

Figure: Algorithm 1 ProjectAway

We present an algorithm, ProjectAway (Figure 4), that orthogonalizes image representations with respect to text representations in order to erase objects in image captions, applying it to remove objects one at a time and all at once.

Given an image and an object to remove, we edit the latent representations $h_{l^{I}}(k_{i})$ at a hidden layer $l^{I}$ across all image embeddings $k_{i}$. We do not modify any latent representations outside of those belonging to image features. We compute the dot product, $p$, of $h_{l^{I}}(k_{i})$ and the object’s text embedding $\vec{t}$, subtracting a weighted $\vec{t}$ from $h_{l^{I}}(k_{i})$ only if the dot product is positive. At $\alpha=1$, ProjectAway is equivalent to orthogonalizing the image representations with respect to the text representation. To compute text representation $\vec{t}$, we pass the object (e.g. “hot dog”) into the VLM’s text model and extract $h_{l^{T}}(t_{\text{-1}})$ at hidden layer $l^{T}$, where $t_{\text{-1}}$ is the last token of the object. We use the last token of the object to capture the whole of the object’s meaning.

#### 4.1.1 Removing objects one by one

We evaluate the ProjectAway algorithm’s effectiveness at erasing individual objects from captions across multiple images and objects.

Experimental setting. We apply ProjectAway on 5000 random images from the COCO2014 training set on all mentioned COCO objects (i.e. hallucination and CD) individually and measure the removal rate at which objects no longer appear in the caption. For InstructBLIP, we set $(l^{I},l^{T},\alpha)=(1,2,1.5)$. For LLaVA, we set $(l^{I},l^{T},\alpha)=(19,21,3.5)$. These parameters are fixed irrespective of image and are chosen for their maximal effect (see ablations in Section 4.2). To differentiate hallucinations from CD, we compute CHAIR (Rohrbach et al., ), an evaluation criteria that compares model-generated captions to ground-truth human annotations. CHAIR provides two main scores, $\text{CHAIR}_{I}$ and $\text{CHAIR}_{S}$, that quantify hallucinations for instances and sentences, respectively:

$$ $\text{CHAIR}_{S}=\frac{|\{\text{captions with hallucinated objects}\}|}{|\{ \text{all captions}\}|},\text{CHAIR}_{I}=\frac{|\{\text{hallucinated objects} \}|}{|\{\text{all objects mentioned}\}|}$ (3) $$

Results. Table 1 shows that ProjectAway is significantly more effective in erasing individual hallucinated objects at an individual level than CD objects for both InstructBLIP and LLaVA. Along with the insight that hallucinated objects have lower softmax scores (Figure 2), these results suggest that hallucinated objects manifest more weakly in image embeddings and are hence easier to remove than CD objects.

#### 4.1.2 Mass-removing objects

We iteratively apply ProjectAway to a set of objects, following the same experimental setup and observing similarly different removal rates for hallucinated objects and CD objects.

Mass-removing hallucinations. We mass-remove hallucinations identified with ground truth annotations using ProjectAway. Table 1 shows that editing out all the hallucinations of an image yields a similar removal rate as individually editing out and, importantly, that erasing hallucinated objects together does not interfere with each other. We achieve a hallucination reduction rate of 41.3% for InstructBLIP and 23.3% for LLaVA (see Table 4). Recall count slightly increases for both models, indicating that caption accuracy is preserved. This may be because removed hallucinations are replaced with objects the model is more confident in. Qualitative results are in Figure 5.

**Table 1: Removing mentioned objects individually & in-mass. Using ProjectAway, we remove hallucinated objects and observe high hallucination reduction with CHAIR, mass-removal rate (Mass RR), and individual removal rate (Individual RR). We also remove correctly detected (CD) objects but find that they are more resistant to linear editing. Denote CHAIRS as $C_{S}$ and CHAIRI as $C_{I}$.**
| Edit Scope | Model | Individual RR (%) | Mass RR (%) | CD change (%) | Ci $\downarrow$ | Cs $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| No edits | InstructBLIP | - | - | - | 15.0 | 54.1 |
| LLaVA | - | - | - | 14.6 | 51.1 |  |
| Hallucinations | InstructBLIP | 83.3 | 74.3 | +0.07 | 8.94 | 33.2 |
| LLaVA | 86.0 | 72.8 | +0.01 | 11.2 | 35.5 |  |
| CD | InstructBLIP | 16.2 | 15.0 | -2.2 | 17.3 | 58.3 |
| LLaVA | 6.9 | 8.3 | -1.6 | 15.2 | 52.4 |  |

Figure: Figure 5: Qualitative results for mass object removal. We present example images and their captions after mass-removing hallucinations (red) with ProjectAway., which can effectively remove hallucinations while preserving, even increasing, correctly detected objects (green).
Refer to caption: x4.png

Mass removing CD. We similarly find that applying ProjectAway can successfully remove CD objects when edited all together in Table 1. Furthermore, CHAIR scores minimally change, which indicates that this mass-removal merely erases object presence without eroding caption accuracy. While the removal rate is lower than for hallucinated objects, this insight proves useful when we apply ProjectAway for hallucination reduction in Section 5.2.

### 4.2 Ablation Study: mass-removing hallucinations

We perform ablations on parameters of ProjectAway to improve object removal rate for erasing hallucinations in-mass.

Experimental setting. We ablate the three parameters of ProjectAway: layer $l^{I}$ to edit at, layer $l^{T}$ to retrieve the text representation, and weight factor $\alpha$. At $l^{T}=-1$, we average together the object’s constituent token embeddings. At $l^{I}=-1$, we edit the image embeddings directly inputted to the text model. We evaluate across 500 training samples from COCO 2014 that have at least one hallucination.

Hidden layers. Figure 7 shows hallucination reduction rate on LLaVA from mass-removing hallucinations on every combination of $l^{I}$ and $l^{T}$ (each from -1 to 31). As a core concern is that editing erodes caption accuracy, we gray out any combination that reduces CD objects. For InstructBLIP (see Figure 11), the best parameters $(l^{I}=1,l^{T}=2)$ reduces hallucinations by 38.5%. For LLaVA, our best parameters $(l^{I}=19,l^{T}=21)$ reduce hallucinations by 25.7%, and the middle layers are the best to edit and extract latent text embeddings from. Our results also provide a wide range of reasonable parameter alternatives to use if this reduction rate does not generalize beyond our samples.

Figure: Figure 6: Hidden layer ablations for LLaVA. We track hallucination reduction (%) across different layers to edit at and extract latent embeddings for the text embedding, crossing out (red) parameters from consideration where there is a decrease in correctly detected objects.
Refer to caption: x5.png

Weight factor. Using the best-reduced hidden layers, we ablate the weight factor $\alpha$ for ProjectAway across the same 500 randomly selected COCO images. Figure 7 shows that as $\alpha$ increases, hallucinations are removed at a higher rate, and the overall hallucination count drops significantly. At high $\alpha$, we observe through anecdotal examples that captions become nonsensical, as quantitatively shown by the complete loss of both correctly detected and hallucinated objects from the caption.
Therefore, as a pre-caution, we only select weight factors that do not reduce CD objects when we apply ProjectAway to erase hallucinated objects.

## 5 Applications

### 5.1 Hallucination Detection

When extracting knowledge from VLMs in Section 3.2, we found that applying logit lens on in-context image representations exhibit useful information about visual objects present in the image. Using these observations, we present an approach for object presence classification that only relies on the VLMs own parameters. We utilize the internal confidence $c_{o}$ value to classify object presence, since the internal confidence for objects that are not present in the image, or hallucinated, are lower within the image representations.

Experimental setting. We evaluate the strength of the internal confidence $c_{o}$ as an indicator of object presence. We sample 5000 images from the MSCOCO training set, using the image captioning objective to caption methods with both InstructBLIP and LLaVA. We use the $c_{o}$ for present objects and hallucinations within the captions generated by each VLM. We assess how well the internal confidence aligns with the ground truth labels of object presence, where a negative sample is a hallucination and a positive sample is a present object.

Baseline. As a baseline, we use the maximum output probability of the object’s tokens. This is the confidence of the model prediction. Previous works such as  Zhou et al. () have found that hallucinations occur more frequently on objects characterized by high uncertainty during generation.

Results. We present quantitative results in Figure 8 and Table 5. We show qualitative results for LLaVA (Figure 12) and InstructBLIP (Figure 13) in the Appendix. We find that utilizing internal confidence to classify object hallucinations provides a 47.17% improvement in mAP in InstructBLIP and 22.45% in LLaVA. Furthermore, the ROC AUC improves over the baseline by 50.10% in InstructBLIP and 44.68% in LLaVA, indicating stronger object presence classification.

Figure: Figure 8: Object Presence Classification Curves for InstructBLIP and LLaVA. We show the Precision-Recall and ROC curves of our confidence measure for present object-hallucination classification on the COCO training subset. Classifying object presence with the internal confidence outperforms the baseline, indicating that the language model’s image representations know which objects are hallucinations and which are truly present.
Refer to caption: x7.png

### 5.2 Hallucination Removal

We use the mass editing technique to remove hallucinations detected by the prior method. Section 4.1.2 successfully removes a significant portion of hallucinations but presupposes a knowledge of what the hallucinations are. We threshold on the internal confidence of each object to identify hallucinations and mass-remove them using ProjectAway. Our chosen threshold prioritizes precision over recall (i.e. we allow classification of some CD objects as hallucinations) because CD objects are less affected by the removal method, as shown in Section 4.1.2.

Experimental setting. We threshold hallucinations as $c_{o}<0.2$ for InstructBLIP and $c_{o}<0.1$ for LLaVA. Based on prior ablations (Section 4.2), we select $(l^{I}=1,l^{T}=2,\alpha=1.5)$ for InstructBLIP and $(l^{I}=19,l^{T}=21,\alpha=3.5)$ for LLaVA. Our prompt is “Please describe this image in detail.”

Baselines. Since our method intervenes during the decoder step, we compare our method with 3 standard decoding algorithms. Greedy decoding predicts the next token based on the highest logit probability. Beam search maintains a tree of beams and selects the best beam at generation end. Nucleus sampling selects the next token from a set of high probability tokens whose cumulative probability reaches a threshold $p$. We also evaluate against OPERA (Huang et al., ), which mitigates hallucinations by adding an overtrust penalty during decoder generation. We set $p=0.9$ for nucleus sampling. We use beam search in our method and unify $N_{\text{beam}}=5$ for the baseline.

Results. We apply these parameters to 500 COCO images from the Karpathy validation set. We provide qualitative results in Figure 15 and Figure 14. Quantitative results in Table 2 show that we outperform our baselines and reduce hallucinations by 25.7% on InstructBLIP and 23.8% on LLaVA compared to beam search. Our approach achieves a similar hallucination reduction rate as Section 4.1.2, despite not precisely differentiating hallucinations and some CD objects being incorrectly edited out. Notably, our method relies on no training or external models, effectively offering a “free lunch.”

**Table 2: Hallucination intervention performance. We mass-remove hallucinations detected by the method in Section 5.1 and outperform other baselines. We observe a considerable drop in the raw count of hallucinated objects.**
| Model | Method | CHAIR${}_{i}\downarrow$ | CHAIR${}_{s}\downarrow$ | Hallucinated Objects $\downarrow$ |
| --- | --- | --- | --- | --- |
| InstructBLIP | Greedy | 57.0 | 23.3 | 512 |
| Nucleus | 58.0 | 24.0 | 508 |  |
| Beam Search | 53.4 | 14.6 | 564 |  |
| OPERA | 45.6 | 13.9 | 472 |  |
| Ours | 43.8 | 12.5 | 419 |  |
| LLaVA | Greedy | 49.2 | 14.2 | 532 |
| Nucleus | 55.8 | 17.1 | 618 |  |
| Beam Search | 52.4 | 15.0 | 583 |  |
| OPERA | 44.8 | 12.8 | 462 |  |
| Ours | 42.0 | 12.2 | 444 |  |

### 5.3 Zero-shot Segmentation

Building upon our findings in Section 3.2, we utilize the internal confidence per image feature for zero-shot image segmentation. This application leverages the spatial information encoded in the image representations and demonstrates how VLMs internally represent and localize objects within images.

Method. Our approach leverages the spatial correspondence between image patches and their associated image embeddings. We use LLaVA to generate the name of the class in the image and we focus on the internal confidence of that class per image patch. We take the mean internal confidence for tokens comprising a class word. We resize the set of $24\times 24$ internal confidence values per image patch back into a fixed image size of $336\times 366$ pixels. We then apply a threshold to these confidence values to binarize them into a foreground/background segmentation for the object in the image.

Figure: Figure 9: Zero-shot segmentation. Warmer areas indicate higher internal confidence for the class at that image patch. We binarize these values with a threshold to generate segmentations.
Refer to caption: x8.png

Baseline. As a baseline, we extract the attention values of generated tokens with the image embeddings from LLaVA. We also compare to the segmentation method introduced by Gandelsman et al. (), which utilizes the attention heads of the image encoder without the additional VLM processing, using the same image encoder (CLIP-ViT-L/14 at 336px).

Results. We evaluate our method on the Imagenet validation set. Qualitative results are shown in Figure 9 and quantitative comparisons with the baselines in Table 3. We improve mAP by 8.03% over using the VLMs raw attention values and provide better and/or comparable performance to other state-of-the-art methods that utilize just the image encoder. While the VLM is not directly trained for segmentation, our technique reveals that it still encodes significant spatial information about objects within its intermediate image representations.

**Table 3: Segmentation Performance on ImageNet-segmentation. Localizing objects using their probabilities within the image representations results in more accurate zero-shot segmentation than previous vision-encoders-based and VLM-based methods.**
| Model | Method | Pixel Acc. $\uparrow$ | mIoU $\uparrow$ | mAP $\uparrow$ |
| --- | --- | --- | --- | --- |
| raw attention (CLIP) | Image Encoder | 69.81 | 45.19 | 77.30 |
| TextSpan (Gandelsman et al., ) | Image Encoder | 75.57 | 53.60 | 80.22 |
| raw attention (VLM) | VLM | 67.28 | 39.27 | 73.96 |
| Ours | VLM | 76.16 | 54.26 | 79.90 |

## 6 Discussion and limitations

We interpreted VLMs’ image representations through the language model layers and discovered that linear editing of these representations can selectively remove object information via a simple orthogonalization. Our findings enabled hallucination reduction and improved zero-shot segmentation. We present two limitations of our work and conclude with future directions.

Multi-token objects. Our method simplifies the use of object words that may be composed of multiple tokens, such as by taking the max internal confidence over object tokens or utilizing the average token embedding for editing. This can introduce noise to the internal confidence if certain tokens are common in multiple different words and lead to an approximation of the object’s latent representations when editing.

Fine-grained edits. The editing approach may struggle with highly abstract or longer sentences that involve attributes or interactions of objects. Removing a full sentence, for example, is not something we assessed in this paper, since our focus is on the removal of individual objects.

Future work. While our focus was on interpreting objects and object hallucinations in VLMs, we believe that our approach can be extended to other key elements of visual scenes, such as people, attributes, and actions. We also focused on object removal, but we believe that editing can also be extended to inject objects into a caption (by adding instead of subtracting the text embedding). We hope to explore the applications of our approach in other multimodal architectures. Our insights may help design better VLMs that are more robust to hallucinations and have improved spatial understanding. We plan to explore these directions in our future work.

### 6.1 Acknowledgments

We thank Kayo Yin for her comments and feedback on our paper. YG is supported by the Google Fellowship. Authors, as part of their affiliation with UC Berkeley, were supported in part by the the Berkeley Artificial Intelligence Research (BAIR) commons program.

## Appendix A Appendix

### A.1 Mass-removing objects

We mass-remove mentioned objects (hallucinations and correctly detected) with ProjectAway and tally up the total number of unique hallucinated and CD objects in Table 4.

**Table 4: Supplemental metrics for Table 1. We measure unique hallucinated and correctly detected (CD) objects.**
| Edit Scope | Model | Hallucinations | CD |
| --- | --- | --- | --- |
| No edits | InstructBLIP | 4545 | 14178 |
| LLaVA | 4372 | 15053 |  |
| Hallucinations | InstructBLIP | 2672 | 14189 |
| LLaVA | 3348 | 15061 |  |
| CD | InstructBLIP | 5078 | 13864 |
| LLaVA | 4583 | 14826 |  |

### A.2 Ablations for InstructBLIP

We show hidden layer and weight ablations for mass-removing hallucinations in InstructBLIP referenced in Section 4.2. The hidden layer ablations indicate that most of the parameter space is too sensitive to edit and leads to losses in correctly detected objects. We find that smaller $l^{T}$ and $l^{I}$ parameters are the most effective for reducing hallucinations. Our best parameters $(l^{I}=1,l^{T}=2)$ reduce hallucinations by 38.5%. It is not fully understood why the majority of the parameter search space is invalid in comparison with LLaVA in Figure 7. It is possible that the fine-tuning step in LLaVA semantically aligns hidden image representations with text embeddings more than InstructBLIP, allowing linear edits to have the precise, intended effect.

Figure: Figure 10: Hidden layer ablations for InstructBLIP. We track hallucination reduction (%) across different layers to edit at and extract latent embeddings for the text embedding, crossing out (red) parameters from consideration where there is a decrease in correctly detected objects.
Refer to caption: x9.png

### A.3 Hallucination Detection

We show quantitative comparisons from our hallucination detection approach using internal confidence (Section 5.1) to the baseline in Table 5. We also show qualitative examples for LLaVA in Figure 12 and for InstructBLIP in Figure 13. These samples exhibit model-generated captions, parsed objects, and whether they are classified as hallucinated or correctly detected based on their internal confidence score.

### A.4 Hallucination Reduction

We exhibit sample results from our hallucination reduction approach (Section 5.2), which linearly removes text representations of hallucinations from image representations, in Figure 15 for InstructBLIP and Figure 14 for LLaVA. We show the image caption before and after our linear editing method, removing objects detected as hallucinations.

### A.5 Object Localization

We show qualitative examples for localization with internal confidence for specific image representations, specifically for the LLaVA model, in Figure 16.

**Table 5: Object Presence Classification performance. We use internal confidence $c_{o}$ as a confidence score to classify whether the object is present in the image. We evaluate the mAP and ROC AUC of our classification method against the baseline for both the InstructBLIP and LLaVA models.**
|  | InstructBLIP | LLaVA |  |  |
| --- | --- | --- | --- | --- |
| Method | mAP $\uparrow$ | ROC AUC $\uparrow$ | mAP $\uparrow$ | ROC AUC $\uparrow$ |
| Baseline | 0.53 | 0.55 | 0.49 | 0.47 |
| Ours | 0.78 | 0.83 | 0.60 | 0.68 |

Figure: Figure 12: LLaVA Object Presence Classification. Sample image captions from LLaVA and the internal confidence scores for objects in the caption used for classification as correctly detected objects or hallucinations.
Refer to caption: x11.png

Figure: Figure 13: InstructBLIP Object Presence Classification.
Refer to caption: x12.png

Figure: Figure 14: Qualitative results for LLaVA hallucination intervention. Our algorithm removes hallucinations and, at times, adds correctly detected objects.
Refer to caption: x13.png

Figure: Figure 15: Qualitative results for InstructBLIP hallucination intervention.
Refer to caption: x14.png

Figure: Figure 16: Object Localization Samples.
Refer to caption: x15.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: - How sensitive is the proposed method to the selection of weight factors and layers across different models and datasets? Is there a way to generalize these hyperparameters or make the method more robust to their selection?
- How does the method perform on other tasks, such as visual question answering (VQA) or on other datasets beyond COCO2014? Have you considered testing the method on benchmarks like LLaVA Bench or MM-Vet?
- Is there a way to automate or simplify the selection of hyperparameters (e.g., layers, weight factors) to make the method more practical for real-world applications?
soundness: 3
strengths: - The paper introduces a novel application of the logit lens technique to interpret the internal image representations of VLMs, providing new insights into how these models process visual information.
- The proposed knowledge erasure algorithm, PROJECTAWAY, is a simple yet effective method that relies solely on manipulating the internal features of the VLMs without requiring additional training or external modules.
- The approach enables zero-shot segmentation by leveraging internal confidence scores to spatially localize objects
- The paper seems clear and well-written.
summary: The paper addresses the issue of hallucinations in Vision-Language Models (VLMs) by interpreting and editing their internal representations. The authors apply the logit lens technique to project image representations onto the language vocabulary, discovering that objects present in the image have higher internal confidence scores compared to hallucinated objects. Utilizing this insight, they propose a method to detect hallucinations within VLMs. Furthermore, they introduce a knowledge erasure algorithm called PROJECTAWAY, which linearly orthogonalizes image features with respect to hallucinated object features to remove hallucinations from the model's output. The method is evaluated on two state-of-the-art VLMs, LLaVA 1.5 and InstructBLIP, showing a reduction in hallucinations by up to 25.7% on the COCO2014 dataset while preserving overall performance. Additionally, the authors demonstrate that their approach enables zero-shot segmentation by spatially localizing objects using internal confidence scores.
weaknesses: - The proposed method requires specifying weight factors and selecting specific layers to retrieve text representations and apply edits. These hyperparameters are determined through ablation studies and do vary between models, and likely between datasets as well, requiring cumbersome ablation process to find good numbers.
- The experiments focus primarily on object hallucinations in image captioning tasks. It is unclear how the method performs on other types of hallucinations (e.g., action or attribute hallucinations) or on other tasks such as visual question answering (VQA).
- The impact of the method on overall caption quality is not thoroughly evaluated quantitatively. While the authors mention that the method preserves performance and provide some qualitative examples, additional quantitative evaluations would be interesting to see.
- The authors only seem to test their model on COCO2014.

---

confidence: 5
contribution: 3
initial_rating: 3
presentation: 4
questions: Please see the Weaknesses section.
soundness: 4
strengths: - The findings are well-written and easy to understand.
- The experiments are comprehensive, exploring different aspects of internal visual information and covering different tasks.
- The proposed approach achieves significant improvements or comparable performance to SoTA on three applications.
summary: The authors use Logit Lens to interpret the intermediate image representations in LVLMs. For a given image embedding, they extract the latent representation of the image embedding at a specific layer, taking the logit lens to get the probability distribution over the vocabulary. 
- The highest probability of an object across image representations and layers, can act as the internal confidence of VLMs. The confidences for objects present are significantly higher than those of objects not present in the image.
- The authors propose an algorithm, ProjectAway, erasing objects from image representations.
- Moreover, they find that, using the internal confidence values, they can localize the objects in the image patches.

The authors show three applications of their findings and the algorithm: hallucination detection, hallucination mitigation, and zero-shot segmentation.
weaknesses: ### Major
- Is the unembedding matrix for image representations directly from the LVLM last layer, or trained by the authors?
- Previous papers report the modality gap between language and vision in VLMs. In my experiments, I also notice that the distribution of vision tokens are significantly different from that of textual tokens. So I’m surprised that the logit lens can be directly used in image representations. 
I’m curious about the classification accuracy of logit lens. For example, if we feed a patch of cat, how accurate is the logit lens method to identify it is cat.
- Lines 200-202, the authors “randomly sample a subset of” objects not present. I’m wondering if this random sampling will choose some objects “obviously” not present in the image, making the comparison of the internal confidence too easy. It might be better if the authors can show: the confidence distribution of objects that commonly appear with objects in the image but not present this time.
- Section 5.3, I think LLaVA tends to generate some very general class when classifying an image, like predicting "dog" instead of “husky”. Are the authors using the generated class name from LLaVA no matter what it is or using the ground truth label?

### Minor
- InstructBLIP and LLaVA are representative LVLMs, but recent LVLMs are using more complicated vision embedding techniques [1, 2]. I’m wondering if the proposed method can still work with these new architectures.
- If we want to detect or remove the hallucinated objects, the propose method needs to know the object name. I'm wondering if the proposed method can work on a popular hallucination benchmark POPE [3]? In POPE, every sample is a "yes or no" question, like "Is there a person in the image?"
- Other limitations like handling multi-token classes have been mentioned in the paper.

[1] LLaVA-NeXT. https://llava-vl.github.io/blog/2024-01-30-llava-next/

[2] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs https://arxiv.org/abs/2406.16860

[3] Evaluating Object Hallucination in Large Vision-Language Models. https://arxiv.org/abs/2305.10355

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. The paper uses the model's unembedding matrix to interpret intermediate layer representations, but this matrix is trained for the final output layer. Have you conducted any layerwise probing or training of separate unembedding matrices for intermediate layers? This could affect the reliability of interpreting earlier layer representations.
soundness: 3
strengths: 1. The paper presents a novel approach to interpreting and editing VLM representations through vocabulary projection and linear orthogonalization, requiring no model retraining or external components.
2. The work provides insights into VLM behavior by revealing the relationship between internal confidence scores and object presence.
summary: This paper presents a novel approach to understanding and editing vision-language models' (VLMs) internal representations through vocabulary projection and linear orthogonalization. By introducing a knowledge erasure algorithm PROJECTAWAY, the authors demonstrate significant improvements in hallucination reduction (up to 25.7%) and achieve competitive performance in zero-shot segmentation, while providing new insights into how VLMs process visual information.
weaknesses: 1. The paper's main analysis and evaluations (Sections 3 and 4) are predominantly conducted under the assumption that hallucinated objects are known beforehand using ground truth annotations. While Section 5 addresses this limitation with a more realistic approach using internal confidence thresholds, this should have been the primary evaluation framework. The current structure potentially overestimates the method's effectiveness by evaluating under idealized conditions.
2. The paper's structure is suboptimal, with the main analysis focusing on scenarios using ground truth annotations while relegating the more realistic approach to the applications section. 
3. The choice to use the last token for multi-token object representations (e.g., "hot dog", "dining table") lacks sufficient justification and empirical validation. The paper does not analyze potential issues with this approach, such as cases where the last token might not be the most semantically meaningful (e.g., "traffic light" where "light" alone might be ambiguous) or how this choice affects the method's performance compared to alternatives like averaging all tokens or using the first token.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: -  Would the authors consider including comparisons with the latest MLLMs, such as those incorporating more advanced architectures or larger datasets, to validate the robustness of their approach?
soundness: 3
strengths: - The paper presents a newmethod for reducing object hallucinations in VLMs by editing their latent representations and the introduction of PROJECTAWAY offers a new technique for selectively removing hallucinated objects from VLMs' outputs.

- The authors provide a thorough analysis of the internal confidence values for object presence and absence, offering empirical evidence that supports their claims.
summary: The paper explores the internal representations of Vision-Language Models (VLMs) to address the persistent issue of hallucinations. The authors project VLMs' internal image representations onto their language vocabulary to identify differences in token output probabilities between real and hallucinated objects. They introduce a knowledge erasure algorithm, PROJECTAWAY, which removes hallucinations by linearly orthogonalizing image features with respect to hallucinated object features. The study demonstrates that targeted edits to a model's latent representations can reduce hallucinations while preserving performance. Additionally, the paper presents a method for zero-shot segmentation using the logit lens technique, showing comparable performance to state-of-the-art methods.
weaknesses: - While the paper focuses on object hallucinations, it does not explore the applicability of the methods to other elements of visual scenes, such as people, attributes, or actions. The editing approach may struggle with abstract or complex sentences involving object attributes or interactions, which are not explicitly addressed in the paper.

- Could the authors elaborate on the potential impact of their editing techniques on other aspects of model performance, such as accuracy in non-hallucination tasks?

- The paper's reliance on LLaVA and InstructBLIP as baseline MLLMs does not provide a comprehensive comparison with the latest state-of-the-art models.

Meta-review: This paper proposes an algorithm to erase spurious knowledge from VLMs. The algorithm, coined ProjectAway, relies on Logit Lens to remove information about objects from image representations. The proposed approach is evaluated on several applications: hallucination detection, hallucination removal, as well as zero-shot segmentation. While reviewers raised some concerns regarding the practical applicability of the proposed approach (too much manual work), and some concerns regarding the evaluation, the work constitutes a good piece of work in the space of VLM interpretability. For the above reason, despite borderline ratings, I recommend accepting this paper.

Decision: Accept (Poster)