Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models

## Abstract

Abstract Hallucination remains a significant challenge in Large Vision-Language Models (LVLMs).
To alleviate this issue, some methods, known as contrastive decoding, induce hallucinations by manually disturbing the raw vision or instruction inputs and then mitigate them by contrasting the outputs of the original and disturbed LVLMs.
However, these holistic input disturbances sometimes induce potential noise and also double the inference cost.
To tackle these issues, we propose a simple yet effective method named Self-Introspective Decoding (SID).
Our empirical investigations reveal that pre-trained LVLMs can introspectively assess the importance of vision tokens based on preceding vision and text (both instruction and generated) tokens. Leveraging this insight, we develop the Context and Text-aware Token Selection (CT 2 S) strategy,
which preserves only the least important vision tokens after the early decoder layers, thereby adaptively amplify vision-and-text association hallucinations during auto-regressive decoding.
This strategy ensures that multimodal knowledge absorbed in the early decoder layers induces multimodal contextual rather than aimless hallucinations, and significantly reduces computation burdens.
Subsequently, the original token logits subtract the amplified fine-grained hallucinations, effectively alleviating hallucinations without compromising the LVLMs’ general ability.
Extensive experiments illustrate that SID generates less-hallucination and higher-quality texts across various metrics, without much additional computation cost. 1 1 1 Codes are in the appendix and also available at https://github.com/huofushuo/SID .

## 1 Introduction

Recent advancements in Large Language Models (LLMs) (Touvron et al., ; Bai et al., ; Chiang & Li, ; Touvron et al., ; Meta, ) have demonstrated great success over the past few years. Many efforts have been made to extend LLMs to Large Vision-Language Models (LVLMs) (Ye et al., ; Li et al., ; Bai et al., ; Li et al., ; Dai & et al, ; Liu et al., ; Bavishi et al., ; Young et al., ; Li et al., ), achieving impressive performance across various vision tasks (Li et al., ; Zhang et al., ) as well as more complex tasks like content comprehension (Lai et al., ) and generation (Geng et al., ).

Despite their extraordinary versatility, LVLMs face a significant challenge known as the ‘hallucination’. Concretely, hallucinated texts are fluent and semantically coherent but contain incorrect or non-exist statements about the given image, e.g., generating irrelevant or meaningless responses, identifying inaccurate colors, numbers, and locations of objects not present in the image (Huang et al., ).
This flaw in LVLMs poses a significant risk for real-world applications to become trustworthy AI assistants. For instance, in model-assisted computer-aided diagnosis scenarios (Wang et al., ), such misinterpretation of medical images could lead to serious medical accidents.

Figure: Figure 1: Contrastive Decoding strategies: (a) Visual Contrastive Decoding (VCD) (Leng et al., ) manually distort vision inputs. (b) Instruction Contrastive Decoding (ICD) (Wang et al., ; Kim et al., ) also manually design noisy instruction (negative prompt). Detailed analyses are in Sec. 3.2. We ablate other modules like the vision encoder and tokenizer for clarity. t: ‘Please describe this image in detail.’; sys.: system prompt. $g$: generated text tokens. $\alpha$ in Eq. 2 defaults to 1.
Refer to caption: x1.png

One mainstream approach to alleviating hallucinations in LVLMs involves developing training-free decoding strategies known as Contrastive Decoding (CD) (Leng et al., ; Favero et al., ; Wang et al., ; Kim et al., ), which adjusts the next-token logits in a contrastive manner.
Concretely, Vision CD (VCD) manipulates vision inputs with Gaussian noise (Leng et al., ) or directly ablates visual inputs (Favero et al., ) to amplify language priors. Instruction CD (ICD) (Wang et al., ; Kim et al., ) designs negative prompt.(^2^22negative prompts like ‘You are a confused object detector.’ and ‘Always respond with the opposite of what you’re asked.’ for different tasks.)
The rationale is that disturbed inputs significantly exacerbate hallucinations, and CD subtracts hallucinated concepts from the original distribution to mitigate hallucinations.
However, input disturbances require elaborate designs for various downstream tasks, and the inference cost is inevitably doubled. Moreover, the contrastive distributions are vision-and-text agnostic, not necessarily amplify desired hallucinations but sometimes induce potential uncertainty noise for CD. Intuitive examples are illustrated in Fig. 1, and detailed analyses are in Sec. 3.2.
In Fig. 1 (a) and (b), LVLMs directly infer the correct next token from multimodal inputs. For Vision CD, distorted vision input exacerbates hallucinated object logits such as football and basketball, while the holistic noise suppresses baseball to a low logit value. Consequently, VCD might compromise normal decoding. Similarly, for Instruction CD, LVLMs tend to refuse to answer negative prompts in open-end generation task (as seen in Fig. 5 and 10), and also suffer from potential uncertainty noise similar to VCD.

To address the aforementioned issues, we propose a novel decoding strategy called Self-Introspective Decoding (SID).
Our empirical investigations reveal that pre-trained LVLMs can introspectively assess the importance of vision tokens adaptively,
based on preceding vision and text (both instruction and generated) tokens.
SID leverages this capability to amplify and then subtract vision-and-text association hallucinations by proposing token-level disturbances named Context and Text-aware Token Selection (CT^2S) strategy. This strategy induces multimodal contextual hallucinations, rather than aimless ones, by conducting token selection in the early decoder layers.
In summary, our main contributions are three-fold:

•  We re-think CD methods in LVLMs and attribute their failure cases to vision-and-text agnostic input distributions that induce potential uncertainty noise.

•  To address this, we propose Self-Introspective Decoding (SID), where the CT^2S strategy adaptively amplifies and then subtracts vision-and-text association hallucinations. This approach is grounded in our investigations that pre-trained LVLMs can introspectively assess visual importance informed by preceding tokens.

•  Through comprehensive comparisons, we demonstrate that SID generates high-quality texts with fewer hallucinations. Additionally, SID significantly reduces inference cost of contrastive decoding.

## 2 Related Work

We review recent advances in Hallucination in Foundation Models. More backgrounds about Large Vision-Language Models and Decoding Strategy in LLMs are in Appendix A.1.

Hallucination in Foundation Models.
Hallucination, defined as the generation of irrelevant, factually incorrect, or meaningless text in a given context (Rohrbach et al., ; Zhang et al., ; Guan et al., ; Wu et al., ), is a significant bottleneck in current foundation models.
This issue can stem from overfitting specific patterns in the training data, a lack of understanding world knowledge, or an inability to effectively contextualize a given input (Ji et al., ). In the context of LLMs, hallucinations often manifest as generated content that conflicts with world knowledge or common sense. For LVLMs, the primary concern is whether the generated answer conflicts with the provided images. To mitigate the hallucination issue,
several solutions have been proposed, including robust instruction tuning with curated datasets (Lee et al., ; Gunjal et al., ; Liu et al., ; Zhao et al., ; Jiang et al., ; Yu et al., ; ; Ma et al., ; Yue et al., ), post-hoc utilizing auxiliary analysis networks (Manakul et al., ; Zhou et al., ; Yin et al., ; Chen et al., ; Wu et al., ; Feng et al., ), and various decoding strategies (Li et al., ; Chuang et al., ; Liu et al., ; Leng et al., ; Favero et al., ; Wang et al., ; Kim et al., ; Zhu et al., ). However, robust instruction tuning requires massive high-quality datasets and advanced GPU clusters, making it resource-intensive; Post-hoc utilizing auxiliary networks heavily rely on the auxiliary network, leading to high inference costs.
As for decoding strategies, representative LVLMs hallucination alleviation methods (Leng et al., ; Favero et al., ; Wang et al., ) manually disturb raw inputs to induce hallucinations and then contrast them to alleviate the issue. However, holistic disturbing raw inputs might bring additional noise during contrastive decoding, and double the inference cost.
In this paper, we propose an efficient Self-Introspective Decoding (SID) that induces and then mitigates vision-and-text association hallucination by token-level disturbances, greatly reducing the inference cost.

## 3 Preliminary and Motivation

In the following, we first illustrate the generation paradigm of LVLMs to facilitate the understanding of SID. We then re-think the contrastive decoding in LVLMs and propose our motivation for SID.

### 3.1 Paradigm of LVLMs Generation

Vision and Language Inputs.
The inputs of LVLMs consist of both image ($v$) and text ($t$). Generally, the raw images are commonly fed to the visual encoder, and then the cross-model projection module maps vision information into LLMs’ input space, which is denoted as vision tokens $v=\left\{v_{1},v_{2}...v_{n}\right\}$ ($n$ is the length of vision tokens). Similarily, text is processed by tokenizer and embedding modules, which is denoted as text tokens $t=\left\{t_{1},t_{2}...t_{m}\right\}$ ($m$ is length of text tokens). Then, the image ($v$) and text ($t$) tokens are concatenated as the final input of LLMs.

LVLMs Forward.
The backbone networks of LVLMs are pre-trained LLMs like Vicuna (Chiang & Li, ) and LLaMA 2 (Touvron et al., ), parameterized by $\theta$. Given multimodal tokens $\left\{v,t\right\}$, LVLMs predict the next token probability ($y_{i}$) at $i$ time step in an auto-regressive manner following the methodology of LLMs, over the vocabulary set $\nu$:

$$ $p(y_{i}|v,t,y_{<i})=\mathrm{softmax}(logit_{\theta}(y_{i}|v,t,y_{<i})),y_{i}\in\nu$ (1) $$

Next Token Decoding. After obtaining the next token probability $p(y_{i}|v,t,y_{<i})$, different decoding strategies (Appendix A.1) are proposed to predict next token.
The decoded token is concatenated to the last original input token, for the next round of generation until the end of the generation process.

### 3.2 Re-thinking Contrastive Decoding in LVLMs

**Table 1: Efficacy Analyses on CD strategies on MSCOCO dataset. The Random setting means objects absent from the image are chosen randomly, while the Adversarial setting prioritizes co-occurring objects which are not present in the image. Results are from (Leng et al., ) or the average of three running times conducted on LLaVA-1.5 7B for fair comparisons. $\alpha$ in Eq. 2 defaults to 1 and Eq. 3 has no effect on the greedy decoding.**
|  | Greedy | Sampling |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Setting | Method | Accuracy $\uparrow$ | F1 Score $\uparrow$ | Accuracy $\uparrow$ | F1 Score $\uparrow$ |
| Random | Normal | 88.8±0.05 | 88.6±0.08 | 84.9±0.03 | 83.2±0.01 |
| \cdashline2-6 | VCD | 87.8±0.02 | 87.9±0.06 | 87.73 | 83.28 |
|  | w/o Eq. 3 | - | - | 83.3±0.04 | 82.2±0.02 |
| \cdashline2-6 | ICD | 87.9±0.04 | 88.1±0.02 | 86.9±0.03 | 85.2±0.04 |
|  | w/o Eq. 3 | - | - | 82.7±0.02 | 81.8±0.03 |
| \cdashline2-6 | Ours | 89.3±0.08 | 89.5±0.02 | 88.8±0.03 | 88.7±0.02 |
|  | w/o Eq. 3 | - | - | 87.2±0.01 | 88.0±0.02 |
| Adversarial | Normal | 79.3±0.05 | 80.9±0.09 | 78.7±0.03 | 78.9 ±0.02 |
| \cdashline2-6 | VCD | 80.9±0.06 | 81.0±0.04 | 80.88 | 81.33 |
|  | w/o Eq. 3 | - | - | 76.2±0.04 | 76.0±0.04 |
| \cdashline2-6 | ICD | 80.2±0.03 | 81.3±0.01 | 79.1±0.02 | 80.4±0.04 |
|  | w/o Eq. 3 | - | - | 75.4±0.02 | 76.4±0.04 |
| \cdashline2-6 | Ours | 83.3±0.07 | 82.5±0.06 | 82.6±0.05 | 82.1±0.06 |
|  | w/o Eq. 3 | - | - | 82.2±0.03 | 81.9±0.01 |

Following the seminal works (Li et al., ) in natural language processing, which introduced the Contrastive Decoding (CD) mechanism to enhance coherence and informativeness by considering the differences between expert and amateur models, various studies have adapted this strategy to LVLMs by distorting the visual or instruction inputs for contrastive purposes.
As the vision and instruction contrastive processes are symmetrical, we use visual contrastive decoding as an example. The contrastive decoded probability of next-token ($p_{cd}$) can be generally formulated as follows:

$$ $p_{cd}(y_{i}|v,v_{d},t,y_{<i})=\mathrm{softmax}[(1+\alpha)logit_{\theta}(y_{i} |v,t,y_{<i})-\alpha logit_{\theta}(y_{i}|v_{d},t,y_{<i})]$ (2) $$

where $d$ and $\alpha$ indicate distortion operation and hyperparameter, respectively. Generally, CD methods employ an adaptive plausibility constraint to calibrate the entire output distribution, preventing implausible outputs from the augmented distribution (Li et al., ; Chuang et al., ; Leng et al., ; Favero et al., ; Wang et al., ; Kim et al., ; Zhu et al., ):

$$ $\begin{split}\nu_{token}(y_{<i})=\left\{y_{i}\in\nu:p_{\theta}(y_{i}|v,t,y_{<i })\geq\beta\max_{\omega}p_{\theta}(\omega|v,t,y_{<i})\right\},\\ p_{cd}(y_{i}|v,v_{d},t,y_{<i}))=0,\;\mathrm{if}\;y_{i}\notin\nu_{token}(y_{<i} )\end{split}$ (3) $$

where $\nu$ and $\nu_{token}$ are the output vocabulary and selected tokens.
$\beta$ controls the strength of truncation, with larger $\beta$ indicating more aggressive truncation that retains only high-probability tokens.

However, we argue that manually disturbing raw inputs might not trigger the desired hallucinations, while holistic disturbances will bring uncertainty noise that compromises the normal decoding. To validate our claim, we analyze the performances of normal decoding, VCD, and ICD using the POPE (Li et al., ) metric, under both sampling and greedy decoding settings.
POPE quantitatively converts the hallucination evaluation into a binary classification problem by using the question format to prompt the model: ‘Is there a <object> in the image?’, with expected answers being ‘Yes’ or ‘No’.
From Table 1,
under the greedy decoding setting, CD methods improve performance in the adversarial setting, which are more challenging as they prioritize co-occurring confusing objects. CD methods achieve this by exacerbating and subtracting hallucinated concepts from the original distribution. However, in random settings, where objects absent from the image are chosen randomly and are easily recognized, CD methods slightly underperform normal greedy decoding, which indicates that the correct token logit is somewhat compromised during contrastive decoding. In the sampling decoding setting, CD methods clearly outperform the normal sampling decoding.
However, CD methods rely on the adaptive plausibility constraint (Eq. 3) to filter out low-probability tokens. Without Eq. 3, CD methods are inferior to normal decoding in both random and adversarial settings, validating that vision-and-text agnostic input distributions induces potential uncertainty noise after Eq. 2. To address these issues, we propose a decoding strategy named Self-Introspective Decoding (SID). SID adaptively amplifies vision-and-text association hallucinations informed by generated tokens to guide LVLMs in exploring factualness. Details are illustrated in the Sec. 4 and Fig. 2.

Figure: Figure 2: Overview of Self-Introspective Decoding (SID). CT^2S: Context and Text-aware Token Selection strategy. LLaVA-1.5 7B is utilized as an example to visualize visual tokens with low and high scores (Eq. 5).
Refer to caption: x2.png

## 4 Methodology

### 4.1 Understanding the Self-Introspective Pre-trained LVLMs.

LLMs (Bai et al., ; Chiang & Li, ; Touvron et al., ; Meta, ) have been scaled up to billions of paramters and pre-trained on trillions of tokens, endowing LLMs with encyclopedic ability like in-context learning (Wang et al., ), zero (Kojima et al., )/few-shot (Brown et al., ) ability. LVLMs extend LLMs to multimodal understanding capabilities by visual instruction tuning. Some works (Cao et al., ; Shang et al., ; Chen et al., ) pointed out that vision information is redundant in LVLMs, and develop vision token reduction technologies to prune (Rao et al., ) and merge (Bolya et al., ) tokens guided by importance metrics without further re-training.
Regarding the hallucination issue, we argue that vision tokens with low attention scores induce vision-and-text association hallucination.
Formally, for the transformer block (Vaswani et al., ) in the auto-regressive decoder (^3^33Here we illustrate the transformer block without KV Cache for better understanding.), vision ($v$), text instruction ($t$), and generated tokens ($g$) are concatenated and projected into three distinct vectors: the query vector Q, the key vector K, and the value vector V, utilizing three linear transformations $W_{q}$, $W_{k}$, and $W_{v}$. The self-attention ($SA$) mechanism computes the relevance of each item to other items as follows:

$$ $\begin{split}\mathbf{R}=SA(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathbf{A}\cdot \mathbf{V},\\ \mathbf{A}=\mathrm{softmax}(\frac{\mathbf{Q}\cdot\mathbf{K}^{T}}{\sqrt{d_{l}}} +M)\end{split}$ (4) $$

where $d_{l}$ represents the dimension of Q, K, V, $M$ represents the casual mask. $\mathbf{A}\in R^{(b,h,n,n)}$, where $b$, $h$, and $n$ denote batch size, number of key-value heads, and total token number, respectively. We denote the $\mathbf{A}_{i}$ as the attention matrix after Layer $i$ of LVLMs. We then calculate vision token importance scores ($\mathrm{Score}_{i}(v)$) as shown in Fig. 2 (Selector) based on $\mathbf{A}_{i}$:

$$ $\mathrm{Score}_{i}(v)=\frac{1}{h}\sum_{j=1}^{h}\mathbf{A}_{i}^{(\cdot,j,\cdot, \cdot)}[-1]$ (5) $$

where $v$ means vision token indexes. Contrary to token pruning/merging (Rao et al., ; Bolya et al., ), we preserve a certain number of the least important vision tokens based on Eq. 5.

Analyses. Fig. 3 and 4 preliminarily valid the efficacy of $\mathrm{Score}_{i}(v)$ qualitatively. In Fig. 3, the preserved least important tokens mainly reflect areas opposite to the query. For instance, when querying ‘cup’ in Fig. 3 (left), LVLMs focus on ‘cup’ in the foreground, thus preserving background tokens with low $\mathrm{Score}_{i}(v)$.
Conversely, LVLMs pay attention to background items when querying ‘couch’. When querying existing items in Fig. 3 (right), vision tokens of unrelated regions are mainly preserved. For open-end generative tasks in Fig. 4, auto-regressive decoded tokens are generated based on preceding vision ($v$), instruction ($t$), and generated text ($g$) tokens. The preserved vision tokens are adaptively adjusted according to preceding tokens at each decoding step, primarily focusing on spurious related regions.
More quantitative analyses are provided in Appendix A.6,
where Table 8 illustrates that vision tokens with high $\mathrm{Score}_{i}(v)$ greatly maintain original ability, while tokens with low $\mathrm{Score}_{i}(v)$ reach nearly $50\%$ accuracy in binary classification.
Above evaluations suggest that Eq. 5 effectively assesses the importance of vision tokens.

Figure: Figure 3: Visualization Results of the least important vision tokens on discrimination tasks informed by preceding vision and text tokens. LLaVA-1.5 7B with Layer $i=3$ is utilized.
Refer to caption: x3.png

Figure: Figure 4: Visualization Results of Adaptively Selecting the least important vision tokens on open-end generative tasks informed by preceding vision and text tokens. LLaVA-1.5 7B with Layer $i=3$ is utilized.
Refer to caption: x4.png

We further demonstrate the open-end generated hallucinations induced by ours, Vision Disturbance (VD) (Leng et al., ), and Instruction Disturbance (ID) (Wang et al., ) in Fig. 5 and 10. The hallucinations we amplified are more vision-and-text association compared to VD, while LVLMs usually refuse to response to ID.
Additionally, we demonstrate the quantitative results for discrimination and generation tasks with VD and ID as inputs in Appendix Table 8. Interestingly, VD and ID do not degrades much, especially in discrimination tasks. Experiments imply that disturbed target logits still have the highest probability in most cases, and therefore, contrastive decoded target logits are not enhanced much after Eq. 2, while CD methods are susceptible to potential uncertainty noise.

Figure: Figure 5: Instance Illustration of Different Disturbance Results. Examples are from MSCOCO inferred by LLaVA-1.5 7B with $i=3$ and Top-k=50. Hallucinations are marked in red.
Refer to caption: x5.png

### 4.2 Context and Text-aware Token Selection (CT 2 S) Strategy.

Based on the above investigations, we argue that to induce context- and text-aware hallucinations for contrastive decoding, only a small percentage
of vision tokens with low attention scores should be preserved after the early decoder layers.
To validate our claims, we conduct the following experiments: 1) In Vision Encoder (VE), we preserve tokens with low attention values between the $\mathbf{[CLS]}$ token and vision tokens in the penultimate layer, calculated as: $\mathbf{A}=\mathrm{softmax}(\frac{\mathbf{[CLS]}\cdot\mathbf{K^{T}}}{\sqrt{d_{
k}}})$. 2) In the LLM decoder, we preserve tokens with low importance score (Eq. 5) across varying layers ($i$). Additionally, we adjust the number of preserved vision tokens. As shown in Fig. 6, firstly, pruning vision tokens in VE based on $\mathbf{[CLS]}$ may not always yield positive gains, as the $\mathbf{[CLS]}$ token lacks information about instructions and generated texts, which are crucial for multimodal understanding.
Specifically, pruning all vision tokens resembles VIG (Favero et al., ), which contrastively amplifies the vision importance over the language prior by ablating vision inputs. Secondly,
aggressive pruning vision tokens (i.e., 0$\%$) after the $Layer_{i=1}$ is not optimal. The loss of vision information for subsequent decoder layers results in losing the visual contexts, leading to aimless hallucinations without sufficient vision grounding. Thirdly, selecting tokens in the late decoder layers degrades contrastive decoding to normal decoding, as preceding layers of LVLMs already decode and understand multimodal information, which is consistent with LLMs’

Figure: Figure 6: Analyses of varying $i$ and preserved ratios in CT^2S. VE: vision encoder; $i$: $i$-th decoder layer.
Refer to caption: x6.png

early-exiting mechanisms (Schuster et al., ; Elhoushi et al., ). In summary, the proposed CT^2S strategy selects Top-k least important vision tokens after the early layers based on attention score (Eq. 5), where the induced hallucinations are aware of both visual contexts and text information. Finally, following CD methods (Sec. 3.2), we contrastively subtract amplified vision-and-text association hallucinations for the next token prediction.

Discussion. Based on analyses of the self-introspective pre-trained LVLMs, could we enhance vision information informed by the proceeding vision and text tokens rather than utilizing the contrastive decoding? To explore this, we rewrite Eq. 2 as follows:

$$ $p_{add}(y_{i}|v,v\uparrow,t,y_{<i})=\mathrm{softmax}[(1-\alpha)logit_{\theta}( y_{i}|v,t,y_{<i})+\alpha logit_{\theta}(y_{i}|v\uparrow,t,y_{<i})]$ (6) $$

where we preserve vision tokens with high importance scores (Eq. 5) denoted as $v\uparrow$. From Table 2,

**Table 2: Analyses of Contrastive Decoding Mechanisms on the POPE metric. Hyperparameters are consistent with CD settings.**
| Setting | Random | Adversarial |
| --- | --- | --- |
| Sampling | 84.7 | 78.7 |
| VCD | 87.7 | 80.9 |
| Add | 88.9 | 79.4 |
| Ours | 88.8 | 82.6 |
| Greedy | 88.8 | 79.1 |
| VCD | 87.8 | 80.9 |
| Add | 89.1 | 80.1 |
| Ours | 89.3 | 83.3 |

we observe that enhancing vision information (i.e., Add) alleviates hallucinations to some extent, which also implicitly validates the efficacy of Eq. 5.
However, in the adversarial setting, enhancing vision information does not bring much benefits compared to ours.
Because our amplified hallucinations effectively associate co-occurring objects,
reflected in high logit values of hallucination token, and then contrastively suppress them.
In contrast, enhancing vision information primarily boosts the original prediction’s target logits grounded in attention scores, which does not significantly improve discrimination, especially in the adversarial setting.

## 5 Experiments

### 5.1 Experimental Settings

Models and Baselines. We utilize four representative LVLMs: InstructBLIP (Dai & et al, ), Shikra (Chen et al., ), LLaVA-1.5 (Liu et al., ) at the 7B scale, and LLaVA-NeXT (Li et al., ) at the 8B scale.
For detailed model descriptions and results at larger scales, please refer to Appendix A.2 and A.6.
Since our method aims to propose training-free LVLM decoding strategies, we compare six decoding methods: Sampling (Top-p=1), Greedy, Dola (Chuang et al., ), and LVLM decoding strategies (VCD (Leng et al., ), ICD (Wang et al., ), and OPERA (Huang et al., )). For comprehensive comparisons, we apply VCD and ICD in both sampling (Top-p=1) and greedy decoding settings. Additionally, we validate SID with different decoding strategies, as detailed in Appendix A.6.

Implementation Details. As analyzed in Sec. 4.2, we set Layer $i$=3 and preserve top 10$\%$ least important vision tokens for Shikra, LLaVA-1.5, and LLaVA-NeXT and $i$=5 and top 10$\%$ least important vision tokens for Q-former based LVLMs (InstructBLIP) to induce fine-grained hallucinations. Hyperparameters in Eq. 2 and 3 follow VCD and ICD. More details are in Appendix A.2.

### 5.2 Evaluation Results

In this section, we follow previous methods (Leng et al., ; Wang et al., ; Huang et al., ) to evaluate the SID on CHAIR (Rohrbach et al., ) and POPE (Li et al., ) metrics. Besides manually designed metrics, we also leverage GPT-4 assisted benchmark (Zhao et al., ) to evaluate attribute, location, and relation hallucinations. MME (Fu et al., ) and MMBench (Liu et al., ) benchmarks are employed to assess the LVLM’s general ability. Moreover, GPT4-V assisted evaluation (Appendix A.4) on both hallucination alleviation and generated text quality and case studies (Appendix A.5) on LLaVA-Bench-in-the-Wild are in Appendix.

**Table 3: Results on the CHAIR metric. ^∗ and ^⋆ denote adopting the same sampling and greedy decoding strategies, respectively.**
| Setting | LLaVA-1.5 | InstructBLIP | Shikra | LLaVA-NeXT |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $C_{S}$$\downarrow$ | $C_{I}$$\downarrow$ | $C_{S}$$\downarrow$ | $C_{I}$$\downarrow$ | $C_{S}$$\downarrow$ | $C_{I}$$\downarrow$ | $C_{S}$$\downarrow$ | $C_{I}$$\downarrow$ |  |
| Sampling | 51.3 | 16.8 | 51.0 | 24.2 | 48.9 | 14.7 | 42.6 | 14.1 |
| ICD^∗ | 48.7 | 13.9 | 48.3 | 16.7 | 47.8 | 14.5 | 42.7 | 13.6 |
| VCD^∗ | 48.0 | 14.3 | 47.9 | 17.2 | 48.1 | 13.8 | 41.3 | 12.9 |
| Ours^∗ | 45.0 | 11.7 | 43.6 | 13.1 | 46.0 | 12.9 | 38.4 | 11.4 |
| \cdashline1-9<br>Greedy | 49.6 | 14.4 | 54.6 | 13.6 | 47.1 | 13.9 | 42.9 | 13.2 |
| Dola^⋆ | 47.1 | 13.8 | 52.7 | 14.0 | 46.8 | 14.2 | 40.9 | 13.1 |
| OPERA | 45.2 | 12.7 | 47.4 | 12.9 | 44.4 | 13.6 | 39.4 | 11.8 |
| ICD^⋆ | 47.4 | 13.9 | 46.3 | 15.3 | 47.3 | 14.1 | 42.1 | 12.6 |
| VCD^⋆ | 46.8 | 13.2 | 44.0 | 13.6 | 47.8 | 14.0 | 41.1 | 12.9 |
| Ours^⋆ | 44.2 | 12.2 | 42.3 | 12.4 | 44.8 | 12.8 | 38.1 | 11.3 |

CHAIR and POPE Evaluations.
CHAIR (Rohrbach et al., ) and POPE (Li et al., ) are quantitative metrics to assess objection hallucinations of VLMs. Please refer to the Appendix A.3 for detailed descriptions of CHAIR and POPE.
As for CHAIR, Following (Wang et al., ; Huang et al., ; Yue et al., ), we randomly select 500 images from the validation set of the MSCOCO (Lin et al., ) dataset and query different LVLMs with the prompt: ‘Please describe this image in detail.’. We set the max new tokens to 512 to generate responses for fair comparisons. As shown in Table 3, our method outperforms other baselines in most cases, validating the effectiveness of SID in open-end generation tasks. Compared to CD methods, SID online adaptively prunes attention-important vision tokens informed by instruction and generated text to induce fine-grained hallucinations for contrastive decoding during open-end text generations.
For the POPE metric, which comprises three datasets, we average the results in Table 4. Our method performs best overall in random, popular, and adversarial sampling settings. Specifically, in the sampling decoding setting, SID surpasses the normal sampling decoding by a large margin in a train-free manner. SID also clearly outperforms CD methods (Dola, ICD, and VCD) because the self-introspective decoding strategy amplifies vision-and-text association hallucinations then subtracts them, rather than coarsely disturbing raw inputs. Additionally, owing to the context and text-aware token selection strategy, SID is more computation-efficient than CD methods, as analyzed in Table 6. Note that beam-search based OPERA (Huang et al., ) shows almost no gain in the POPE metric, primarily because answering the binary classification only requires a few tokens and selecting the best beam score in a decoded sequence ($N$=5) brings little improvement.

**Table 4: Average results on the POPE metric. ^∗ and ^⋆ denote adopting the same sampling and greedy decoding strategies, respectively. Results are from the papers or re-implemented based on official codes.**
| Setting | Random | Popular | Adversarial |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Decoding | Accuracy$\uparrow$ | F1 Score$\uparrow$ | Accuracy$\uparrow$ | F1 Score$\uparrow$ | Accuracy$\uparrow$ | F1 Score$\uparrow$ |
| LLaVA-1.5 | Sampling | 84.77 | 82.28 | 79.98 | 79.34 | 76.03 | 76.26 |
| ICD^∗ | 87.51 | 83.28 | 83.15 | 83.91 | 79.13 | 80.41 |  |
| VCD^∗ | 86.84 | 86.83 | 82.65 | 83.37 | 77.31 | 79.28 |  |
| Ours^∗ | 88.91 | 88.84 | 83.97 | 85.42 | 82.54 | 81.98 |  |
| \cdashline2-8 | Greedy | 88.81 | 88.52 | 82.76 | 83.36 | 79.11 | 80.92 |
|  | Dola^⋆ | 87.94 | 87.97 | 83.87 | 84.68 | 80.35 | 81.21 |
|  | OPERA | 88.85 | 88.67 | 82.77 | 83.40 | 79.16 | 80.93 |
|  | ICD^⋆ | 87.97 | 87.84 | 84.03 | 84.22 | 80.21 | 80.97 |
|  | VCD^⋆ | 87.02 | 86.96 | 83.53 | 84.56 | 78.12 | 80.16 |
|  | Ours^⋆ | 89.46 | 89.62 | 85.13 | 85.94 | 83.24 | 82.21 |
| InstructBLIP | Sampling | 80.42 | 80.94 | 76.09 | 77.65 | 72.37 | 75.42 |
| ICD^∗ | 85.78 | 85.73 | 81.12 | 82.25 | 76.82 | 78.99 |  |
| VCD^∗ | 84.11 | 84.13 | 79.96 | 80.80 | 76.32 | 78.08 |  |
| Ours^∗ | 86.56 | 85.94 | 80.26 | 81.75 | 77.64 | 80.41 |  |
| \cdashline2-8 | Greedy | 84.56 | 83.75 | 78.23 | 79.16 | 74.58 | 76.34 |
|  | Dola^⋆ | 84.67 | 83.38 | 78.21 | 79.19 | 75.69 | 77.98 |
|  | OPERA | 84.57 | 83.74 | 78.24 | 79.15 | 74.59 | 76.33 |
|  | ICD^⋆ | 84.36 | 83.82 | 77.88 | 78.70 | 75.17 | 77.23 |
|  | VCD^⋆ | 84.52 | 83.63 | 78.04 | 78.45 | 75.95 | 77.76 |
|  | Ours^⋆ | 87.23 | 86.90 | 81.16 | 82.57 | 78.51 | 81.26 |
| Shikra | Sampling | 81.42 | 82.46 | 79.60 | 80.78 | 73.85 | 76.39 |
| ICD^∗ | 82.34 | 82.82 | 78.17 | 80.43 | 74.96 | 77.68 |  |
| VCD^∗ | 82.31 | 82.73 | 79.34 | 80.93 | 75.61 | 77.96 |  |
| Ours^∗ | 83.87 | 83.94 | 80.26 | 82.07 | 77.85 | 78.94 |  |
| \cdashline2-8 | Greedy | 83.00 | 83.19 | 81.39 | 81.90 | 76.69 | 78.31 |
|  | Dola^⋆ | 82.87 | 82.98 | 82.42 | 82.50 | 76.85 | 78.09 |
|  | OPERA | 83.05 | 83.20 | 81.40 | 81.89 | 76.73 | 78.31 |
|  | ICD^⋆ | 82.67 | 82.64 | 80.73 | 81.58 | 75.98 | 78.43 |
|  | VCD^⋆ | 82.96 | 82.63 | 80.68 | 81.27 | 76.94 | 78.32 |
|  | Ours^⋆ | 84.46 | 84.62 | 82.38 | 82.73 | 78.67 | 79.34 |
| LLaVA-NeXT | Sampling | 86.32 | 83.11 | 82.27 | 81.03 | 77.32 | 76.96 |
| ICD^∗ | 87.32 | 84.03 | 83.62 | 83.54 | 80.31 | 80.41 |  |
| VCD^∗ | 86.97 | 86.71 | 83.07 | 83.65 | 79.42 | 80.28 |  |
| Ours^∗ | 89.16 | 88.92 | 84.38 | 85.76 | 82.95 | 81.98 |  |
| \cdashline2-8 | Greedy | 89.37 | 88.82 | 83.68 | 84.62 | 80.08 | 80.74 |
|  | Dola^⋆ | 88.73 | 88.67 | 84.56 | 84.96 | 80.32 | 80.68 |
|  | OPERA | 89.36 | 88.80 | 83.65 | 84.60 | 80.10 | 80.75 |
|  | ICD^⋆ | 87.40 | 87.96 | 84.11 | 83.79 | 80.94 | 80.67 |
|  | VCD^⋆ | 87.83 | 87.09 | 82.68 | 83.55 | 79.61 | 81.20 |
|  | Ours^⋆ | 90.05 | 89.97 | 86.13 | 85.69 | 84.06 | 82.95 |

GPT-4 Assisted Benchmark. While CHAIR and POPE evaluate object-existence-level hallucinations, these metrics are unable to identify other types of hallucination, such as positional, relational, and attribute hallucinations. Therefore, the GPT-4 assisted benchmark (Zhao et al., ) utilizes the fine-grained object-level descriptions in the Visual Genome (VG) dataset (Krishna et al., ) as ground-truth and relies on the advanced GPT-4 to judge the fine-grained hallucinations and calculate Sentence-level Hallucination Ratio (SHR). Please refer to the Appendix A.3 for detailed implementations.
Moreover, we employ n-gram fluency (n = 1 and 2) metrics to measure the smoothness of generated text, and the number of generated words/sentences per image (WPI/SPI) to compare the detailedness of generated texts.
As shown in Fig. 7, SID achieves the best results in the SHR metric among the four LVLMs, outperforming others by a clear margin.
Regarding the quality of the generated texts,
Sampling decoding outperforms ours slightly in terms of 1-gram fluency and WPI. However, compared to other baselines, our approach alleviates hallucinations with minimal sacrifice in text generation quality regarding smoothness and detailness. For instance, OPERA generates text with fewer words and sentences due to penalization of the over-trust mechanism, and VCD impairs text fluency, possibly arising from the holistic and fixed disturbance of contrastive inputs.

Figure: Figure 7: GPT-4 assisted benchmark (Zhao et al., ). Hallucination (SHR), fluency (1$\&$2-gram), and detailness (WPI and SPI) aspects are compared. Larger areas mean better performances. VCD and ours adopt the same sampling decoding. Please zoom in for details.
Refer to caption: x7.png

MME and MMBench Evaluations.
Besides, we test different decoding strategies on two popular LVLMs benchmark: MME (Fu et al., ) and MMBench (Liu et al., ). MME comprises ten subtasks to evaluate
models’ perceptual capabilities and four subtasks for assessing cognitive abilities, in the form of the yes/no question. MMBench systematically evaluates twenty ability dimensions of LVLMs. We
present the results of LLaVA-1.5 7B as a representative in Table 5, SID can maintain and improves the multimodal ability on LVLMs benchmarks. In contrast, other CD methods tend to compromise the general multimodal ability.

**Table 5: LVLM benchmark evaluations. ICD, VCD, and SID employ the same greedy decoding.**
|  | Greedy | Sampling | DoLa | ICD | VCD | OPERA | SID |
| --- | --- | --- | --- | --- | --- | --- | --- |
| MME | 1510.7 | 1471.9 | 1480.1 | 1473.6 | 1488.7 | 1515.4 | 1520.2 |
| MMbench | 64.3 | 64.0 | 63.8 | 63.1 | 63.9 | 64.4 | 65.1 |

## 6 Ablation Analyses

In this section, we conduct ablation analyses on the computation efficiency and hyperparameter sensitivity. More analyses about Token Selection Strategies Analyses, Larger-scale Backbone and Other Decoding Strategies are illustrated in Appendix A.6.

Figure: Table 6: Efficiency Comparisons on NVIDIA V100. 10% and 40% mean tokens preserved ratios.
Refer to caption: x8.png

Computation Efficiency.
One primary concern of hallucination alleviation decoding methods is the computational burden. We evaluate inference time (in seconds) and peak GPU memory (in MB) on the LLaVA-1.5 7B under the POPE adversarial setting, as shown in Table 6. Contrastive Decoding (CD) methods (Leng et al., ; Wang et al., ) involve constructing distorted raw inputs, resulting in twice the inference complexity. OPERA (Huang et al., ) is based on beam-search decoding and maintains a set of beams to enlarge the candidate range. Additionally, roll back mechanism in the retrospection-reallocation strategy further exacerbates computational complexity. Our SID induces vision-and-text association hallucinations by pruning large-ratio attention-important tokens in the early layers,
which greatly reduces the inference time of CD up to $\sim$30$\%$.

Hyperparameter Sensitivity. Beyond the sensitivity analyses in Fig. 6, we validate the robustness of SID concerning $\alpha$ and $\beta$ of Eq. 2 and 3, compared to the contrastive decoding methods (i.e., VCD) on LLaVA-1.5 7B. From Fig. 8 (left), it is evident that as $\alpha$ decreases, the contrastive decoding mechanism diminishes. However, SID still achieves pleasant results, while VCD degrades close to Sampling when $\alpha$=0.1, as the CT^2S strategy induces informative vision-and-text association hallucinations. When $\alpha$ increases, VCD degrades to some extent because holistic input disturbance does not always trigger contextual-related hallucination and might exacerbate uncertainty noise. Regarding $\beta$, a larger $\beta$ indicates more aggressive truncation of the output vocabulary. Fig. 8 (right) shows that VCD’s performance heavily relies on large $\beta$ to retain only high-probability tokens. With mild or no adaptive plausibility constraint (Eq. 3), VCD performs worse than the sample decoding strategy due to output logits influenced by distorted visual inputs. Ours is robust to the $\beta$ setting as the CT^2S strategy induces discriminative contrastive logits to generate plausible tokens.

## 7 Conclusion and Future Work

In this paper, we firstly re-think contrastive decoding in LVLMs and empirically find that vision-and-text-agnostic input disturbances in CD do not always amplify desired hallucinations rather than induce potential uncertainty noise. To mitigate these issues, we propose a training-free decoding strategy named Self-Introspective Decoding (SID).
By developing Context and Text-aware Token Selection (CT^2S) strategy,
SID amplifies vision-and-text association hallucinations to guide LVLMs in contrastive decoding, thereby improving faithfulness. Extensive experiments validate the effectiveness and robustness of SID.(^4^44Codes are available at https://anonymous.4open.science/r/SID-1795.)

As for Future Work: 1) As the pruning ratios and layer are set manually, we consider training the external network to automatically determine optimal hyperparameters, inspired by (Chen et al., ). 2) Moreover, given that SID amplifies fine-grained hallucinations, we consider leveraging the CT^2S strategy to automatically construct high-quality negative instruction for robust visual instruction tuning rather than relying on expensive GPT-4 (Liu et al., ; Zhao et al., ).

## Appendix A Appendix

Overview. More related work are described in Appendix A.1, including backgrounds about Large Vision-Language Models and Decoding Strategy in LLMs. The detailed experimental settings are in Appendix A.2 including Model Details and Implementation Details. Evaluation metric details, including CHAIR Evaluations, POPE Evaluations, and GPT-4 Assisted Evaluations are introduced in Appendix A.3. We demonstrate the GPT4-V Assisted Evaluation in Appendix A.4 for comprehensive evaluations. We visualize some case study from the LLaVA-Bench-in-the-Wild dataset for qualitative analysis in Appendix A.5. More ablation analyses including Quantitative Attention Scores Analyses, Token Selection Strategies Analyses, Larger-scale LVLM Backbone, and Adopting Other Decoding Strategies are illustrated in Appendix A.6

### A.1 More backgrounds

Large Vision-Language Models. Motivated by the success of Large Language Models (LLMs) (Touvron et al., ; Bai et al., ; Chiang & Li, ; Taori et al., ; Touvron et al., ; Meta, ), recent studies have extended LLMs to multimodal regions and provided Large Vision-Language Models (LVLMs) (Liu et al., ; Zhu et al., ; Chen et al., ; Ye et al., ; Li et al., ; Bai et al., ; Li et al., ; Dai & et al, ; Liu et al., ; Bavishi et al., ; Chen et al., ; Li et al., ) powered by pre-trained LLMs. LVLMs understand and generate diverse content in a more comprehensive way by integrating user instruction and vision inputs. LLaVA (Liu et al., ) connects open-set vision encoder with LLMs (i.e., Vicuna (Chiang & Li, )) by instruction tuning with elaborated language-image instruction-following data. Then, LLaVA-1.5 (Liu et al., ) develops the vision-language connector that is data-efficient and powerful for better multimodal understanding. Shikra (Chen et al., ) further incorporates grounding data and trains the model to understand the grounding knowledge in the given images. BLIP-2, InstructBLIP, and MiniGPT-4 (Li et al., ; Dai & et al, ; Zhu et al., ) introduce a learnable querying transformer to fusion multimodal features and largely reduce image tokens. Fuyu (Bavishi et al., ) proposes a vanilla decoder-only architecture without the vision encoder and adapter that makes it easier to understand, scale, and deploy. InternVL (Chen et al., ) three simple but effective improvements, including a strong vision encoder, dynamic high-resolution, and high-quality bilingual dataset. Recently, built on SOTA open-source LLaMA 3 (Meta, ) and increasing the input vision resolution to 4$\times$ more pixels, LLaVA-NeXT (Li et al., ) exhibits excellent multimodal capabilities. Despite the impressive results, all of the above LVLMs suffer from serious hallucination problems, and we mainly conduct experiments on advanced LVLMs, including InstructBLIP, Shikra, LLaVA-1.5, and LLaVA-NeXT.

Decoding Strategy in LLMs.
Selecting decoding strategies in language models is crucial, as it determines how models generate text.
Greedy decoding selects the highest probability next token at each step but might lead to less varied text.
Beam search (Graves, ) is an accumulated-score-based decoding strategy. It maintains a set of beams to enlarge the candidate range and finally selects the best one in beams, which is more sophisticated than greedy decoding.
Sampling decoding generates the next words by randomly selecting from the output distribution, where Top-k sampling (Fan et al., ) samples from Top-k likely tokens (Fan et al., ) and brings diversity but sometimes induces less coherent outputs.
Top-p (Nucleus) sampling (Holtzman et al., ) improves Top-k sampling that considers the dynamic number of words that reach the probability p, achieving a balance between randomness and relevance. Recently, to alleviate the hallucination issue, DoLa (Chuang et al., ) decoding emphasizes the knowledge of mature layers and downplays that of pre-mature layers.
OPERA (Huang et al., ) is established on beam-search decoding strategy and finds the interesting phenomenon of high-probability co-occurrence between the hallucination and the knowledge aggregation patterns. OPERA penalizes ‘Over-Trust Logit’ in the beam score to alleviate aggregation patterns.
Self-Introspective Decoding (SID) can be seamlessly integrated into different decoding strategies to mitigate hallucinations without sacrificing text generation quality, such as diversity, coherence, and relevance.

### A.2 Detailed experimental settings

Models Details.
To validate the effectiveness of our SID, we conduct experiments on four representative LVLMs: InstructBLIP (Dai & et al, ), Shikra (Chen et al., ), LLaVA-1.5 (Liu et al., ), and LLaVA-NeXT (Li et al., ).
InstructBLIP employs Q-former (Li et al., ) to encode 32 image tokens to feed into LVLMs, while Shikra, LLaVA-1.5, and LLaVA-NeXT directly leverage linear projection layers as vision-language connectors to align multimodal features. Shikra and LLaVA-1.5 encode 256 and 576 image tokens to LVLMs. LLaVA-NeXT increases the input vision resolution by 4$\times$ to capture more visual details, resulting in  4$\times$ more encoded vision tokens than LLaVA-1.5. All LVLMs utilize pre-trained vision encoders like CLIP (Radford et al., ) vision encoder, as well as pre-trained LLMs as language decoders, such as Vicuna v1.1 (Chiang & Li, ), LLaMA 2 (Touvron et al., ), and recently released LLaMA 3 (Meta, ). We provide results at the 7 Billion (B) scale, and larger-scale results are in the Appendix A.6.

Implementation Details.
For sampling and greedy decoding, we adopt the default hyperparameter settings. As for Dola (Chuang et al., ), it is designed to alleviate hallucinations (i.e., improve factuality) of LLM by contrasting the differences in logits obtained from projecting the later layers versus premature layers. Dola is sensitive to the premature layer selection, we adapt Dola to LVLMs, following OPERA (Huang et al., ) to utilize “0,2,4,6,8,10,12,14” as the indexes of candidate premature layers and “32” as the index of the mature layer. The repetition penalty is set to 1.2, as Dola suggests. OPERA, VCD, and ICD are proposed for LVLMs and we adopt the default settings. For fair comparisons, SID’s hyperparameters of Eq. 2 and 3 follow VCD and ICD. Moreover, we apply SID, VCD, and ICD in both sampling (Top-p=1) and greedy decoding settings for comprehensive comparisons.
Note that due to amplified fine-grained hallucinations, SID is more robust to hyperparameters compared to other CD methods (Sec. 6). Experiments are performed on NVIDIA V100/A100 GPUs.

### A.3 evaluation metric details

CHAIR Evaluations.
The Caption Hallucination Assessment with Image Relevance (CHAIR) (Rohrbach et al., ) metric is specially designed to assess objection hallucinations in the image caption tasks. Concretely, CHAIR quantifies the degree of hallucinations in a generated image caption by calculating the proportion of all objects mentioned in the caption that are not present in the ground truth label pool. There are two common variants of CHAIR: CHAIRI ($C_{I}$) and CHAIRS ($C_{S}$), which evaluate the degree of object hallucination in the instance and sentence level, respectively. These two metrics are formulated as follows:

$$ $C_{I}=\frac{|\{\mathrm{hallucinated\ objects\}}|}{|\{\mathrm{all\ mentioned\ objects}\}|},\ C_{S}=\frac{|\{\mathrm{captions\ with\ hallucinated\ objects}\} |}{|\mathrm{\{all\ captions}\}|}$ (7) $$

The smaller the value of $C_{I}$ and $C_{S}$, the better the hallucination alleviation performance.

POPE Evaluations.
The Polling-based Object Probing Evaluation (POPE) (Li et al., ) was recently developed to assess hallucination problems in LVLMs. POPE queries the LVLMs with the template: Is there a <object> in the image? The ratio between queries about existing and no-existing objects is balanced (i.e., 50$\%$-50$\%$). This benchmark consists of three sampling settings: random, popular, and adversarial, each differing in the construction of negative samples. Specially, in the random setting, objects that are not present in the image are selected at random. The popular setting selects missing objects from the high-frequency pool, whereas in the adversarial setting, co-occurring objects that are not present in the image are prioritized. POPE consists of three different datasets, including MSCOCO (Lin et al., ), A-OKVQA (Schwenk et al., ), and GQA (Hudson & Manning, ). POPE involves 500 images from each dataset with six questions each, ultimately yielding 27,000 query-answer pairs. Accuracy and F1 score are chosen as evaluation metrics. The larger the value of Accuracy and F1 score, the better the hallucination alleviation performance.

GPT-4 Assisted Evaluations. Besides object-existence-level hallucinations evaluated by CHAIR and POPE, GPT-4 assisted benchmark (Zhao et al., ) utilizes the fine-grained object-level description in the Visual Genome (VG) dataset (Krishna et al., ) as ground-truth and relies on the advanced GPT-4 to judge the detailed (such as positional, relational, and attribute) hallucinations and calculate Sentence-level Hallucination Ratio (SHR). With the generated sentences and manually annotated factual information, GPT-4 is prompted to evaluate whether existing hallucinations sentence by sentence. The prompt template is provided in Appendix Fig. 11. Following (Zhao et al., ), we utilize 200 images from the VG dataset and set max new tokens to 512, with the prompt of ‘Please describe this image in detail.’ We conduct experiments on sampling decoding strategies and representative LVLMs decoding strategies: VCD (Leng et al., ) and OPERA (Huang et al., ).

### A.4 GPT4-V Assisted Evaluation.

To further analyze the hallucinations and text quality for open-end generation tasks, following (Huang et al., ; Yin et al., ), we utilize the strong multi-modal assistant GPT4-V, which simultaneously processes input from vision and text modalities.
We strictly follow (Huang et al., ), which utilizes 500 images from the MSCOCO dataset and prompts LVLM:‘Please describe this image in detail.’ with the maximum number of 512.
To mitigate the impact of the sequential order fed to GPT4-V, we simultaneously compare the generated texts obtained from two decoding methods and instruct GPT4-V to judge the correctness and detailedness score on a scale of 0-10 based on the input image. The detailed GPT4-V prompt is in Appendix Fig. 12. We set up three representative pairs of comparison experiments: greedy decoding and ours, CD-based VCD (Leng et al., ) and ours, and OPERA (Huang et al., ) and ours. As shown in Table 7, our SID achieves the best results in terms of most metrics. Concretely, our method improves correctness by about 15-20$\%$ compared to sampling decoding while not compromising the detailedness level. Compared to advanced hallucination mitigation methods VCD and OPERA, SID generates text with obvious more details and better mitigates the hallucination issue. Since the perceptual and reasoning capabilities of GPT4-V are very close to those of humans, the results of the GPT4-V evaluation reflect, to some extent, the strong performance of the compared methods in terms of mitigating hallucinations and generating text quality from a human perceptual perspective.

**Table 7: GPT4-V assisted hallucination evaluations (Huang et al., ; Yin et al., ). VCD and ours adopt the same sampling decoding strategy. $C$: correctness; $D$: detailedness**
| Setting | LLaVA-1.5 | InstructBLIP | Shikra | LLaVA-NeXT |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C$\uparrow$ | D$\uparrow$ | C$\uparrow$ | D$\uparrow$ | C$\uparrow$ | D$\uparrow$ | C$\uparrow$ | D$\uparrow$ |  |
| Sampling | 5.18 | 5.79 | 4.73 | 5.10 | 5.03 | 5.17 | 5.34 | 5.67 |
| Ours | 5.97 | 6.01 | 5.62 | 5.16 | 5.78 | 5.10 | 6.47 | 5.85 |
| VCD | 5.46 | 5.63 | 4.98 | 5.21 | 5.31 | 5.24 | 5.92 | 5.47 |
| Ours | 6.16 | 5.94 | 5.37 | 5.46 | 5.61 | 5.29 | 6.12 | 5.78 |
| OPERA | 6.16 | 5.57 | 5.29 | 4.86 | 5.34 | 4.87 | 6.11 | 5.24 |
| Ours | 6.15 | 5.94 | 5.76 | 5.42 | 5.97 | 5.88 | 6.63 | 6.23 |

### A.5 Case Study.

In addition to using crafted metrics (CHAIR and POPE), GPT-4/GPT4-V-aided evaluations, and MME (Fu et al., ) and MMBench (Liu et al., ) benchmarks, we qualitatively present several case studies of SID’s hallucination alleviation ability from LLaVA-Bench-in-the-Wild dataset (Liu et al., ). As illustrated in Appendix Fig. 13, 14, and 15, SID effectively mitigates hallucination in these challenging scenes by dynamically amplifying vision-and-text association hallucinations. Meanwhile, it preserves the detailness of each image.

### A.6 More ABLATION ANALYSES

Attention Scores and Token Selection Strategies Analyses.

Quantitative Attention Scores Analyses.
Here we quantitatively analyze vision attention score in terms of POPE Li et al. () and CHAIR Rohrbach et al. () metrics. Concretely, we select the
top-100 and least-100 important vision tokens out of a total of 576 vision tokens of LLaVA-1.5 7B based on attention score based on Eq. 5. Visual and Instruction Disturbance (VD and ID) are also employed as inputs for analyses. Quantitative results in Table 8 illustrate that 100 out of 576 vision tokens with high attention scores greatly maintain original ability, while low attention score tokens reach almost 50$\%$ accuracy for the binary classification problem, which indicates attention scores are a good indicator for
vision token importance. As for VD and ID, disturbance in raw input does not obviously harm the LVLMs’ discrimination ability, as indicated by the POPE metric. However, VD and ID significantly compromise the open-end generation tasks reflected by the CHAIR metric (LVLMs tend to refuse to ID as shown in Fig. 5 and 10).

**Table 8: Efficacy Analyses on Vision Token Attention Scores with POPE (binary classification problem) on MSCOCO dataset and CHAIR metrics. We select the Top-100 and Least-100 important vision tokens out of a total of 576 vision tokens of LLaVA-1.5 7B, based on Eq. 5 ($i$=3). VD: Visual Disturbance; ID: Instruction Disturbance.**
| Setting | Random | Adversarial | CHAIRs $\downarrow$ | CHAIRi $\downarrow$ |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Accuracy $\uparrow$ | F1 Score $\uparrow$ | Accuracy $\uparrow$ | F1 Score $\uparrow$ |  |  |  |
| Greedy | 88.8 | 88.6 | 79.3 | 80.9 | 49.6 | 14.4 |
| \cdashline1-7<br>+Top-100 | 85.6 | 83.9 | 77.1 | 76.3 | 52.7 | 15.2 |
| +Least-100 | 55.3 | 66.1 | 54.0 | 65.3 | 63.2 | 38.7 |
| \cdashline1-7<br>+VD | 88.0 | 87.6 | 78.9 | 79.8 | 56.7 | 16.9 |
| +ID | 88.2 | 87.7 | 79.1 | 80.1 | - | - |

Token Selection Strategies Analyses.
To validate the effectiveness of SID in selecting low attention scores to induce vision-and-text association hallucination, we further conduct quantitative experiments under different vision token selection strategies with the same preserved vision token number and Layer $i$=3 as ours. Table 9 shows that vision tokens with high attention scores degrade obviously, as it does not amplify contextual hallucinations rather than retain original important information. Contrastive decoding does not benefit from subtracting hallucinations amplified by the disturbed inputs rather than suffers from the potential disturbance noise.
Selecting random vision tokens brings improvements in the adversarial setting because randomly selected vision tokens amplify the over-reliance on statistical bias and language priors, similar to Vision CD (Leng et al., ) and Instruction CD (Wang et al., ). However, token-level random disturbance also induces uncertainty noise, resulting in the inferior performance in the random setting to greedy decoding. Overall, these experiments further validate the rationality of our token selection strategy based on attention sores.

**Table 9: Analyses of Different Token Selection Strategies with POPE on MSCOCO dataset and CHAIR metrics. We select the high importance scores (Eq. 5) of vision tokens (-Top) and random vision tokens (-Random) for contrastive decoding. Experiments are conducted on LLaVA-1.5 7B.**
| Setting | Random | Adversarial | CHAIRs $\downarrow$ | CHAIRi $\downarrow$ |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Accuracy $\uparrow$ | F1 Score $\uparrow$ | Accuracy $\uparrow$ | F1 Score $\uparrow$ |  |  |  |
| Greedy | 88.8 | 88.6 | 79.3 | 80.9 | 49.6 | 14.4 |
| \cdashline1-7<br>Ours | 89.3 | 89.5 | 83.3 | 82.5 | 44.2 | 12.2 |
| -High | 87.0 | 87.3 | 76.5 | 79.4 | 57.9 | 25.6 |
| -Random | 88.4 | 87.2 | 80.9 | 81.5 | 48.6 | 13.5 |

Larger-scale LVLM Backbone.

**Table 10: Results on Larger-scale Backbones. Sampling decoding is adopted and results average of three running times.**
|  | POPE | CHAIR |  |  |
| --- | --- | --- | --- | --- |
| Methods | Accuracy | F1 Score | CS | CI |
| LLaVA-1.5 | 81.60 | 80.31 | 49.6 | 16.1 |
| +VCD | 82.67 | 81.46 | 46.7 | 16.4 |
| +OPERA | 82.32 | 81.10 | 43.3 | 13.6 |
| +Ours | 84.75 | 83.17 | 43.5 | 12.7 |
| InstructBLIP | 77.26 | 79.23 | 50.8 | 19.7 |
| +VCD | 79.77 | 80.27 | 47.9 | 17.6 |
| +OPERA | 80.31 | 80.91 | 42.5 | 14.3 |
| +Ours | 81.97 | 82.21 | 41.7 | 13.3 |

We validate the effectiveness of SID in terms of 13B scale backbones on LLaVA-1.5 and InstructBLIP architectures. We choose POPE (Li et al., ) and CHAIR (Rohrbach et al., ) to validate the hallucination issues in both discrimination and open-end generation tasks. Table 10 shows that SID remains effective as backbone networks scale up.

Adopting Other Decoding Strategies.

Meanwhile, besides direct sampling and greedy decoding, we conduct experiments on LLaVA-1.5 7B using the MSCOCO dataset with various decoding strategies, including Top-p sampling ($p$=0.9), Top-k sampling ($k$=50), Top-k sampling with varying temperature ($k$=50, $t$=1.5 and 0.8). Figure 9 shows that, regardless of the sampling strategy adopted, the application of SID consistently helps to alleviate hallucinations and improve the overall performance of LVLMs. This consistency highlights the versatility and effectiveness of SID across different sampling strategies.

Figure: Figure 9: Results of different decoding strategies.
Refer to caption: extracted/5910451/others.png

Figure: Figure 10: Instance Illustration of different disturbance results. Examples are from MSCOCO (Lin et al., ) inferred by LLaVA-1.5 7B with $i=3$ and Top-k=50. Hallucinations are marked in red.
Refer to caption: x9.png

Figure: Figure 11: Prompts of GPT-4 for evaluations.
Refer to caption: x10.png

Figure: Figure 12: Prompts of GPT4-V for evaluations.
Refer to caption: x11.png

Figure: Figure 13: Case Study from LLaVA-Bench-in-the-Wild on LLaVA-1.5 7B. Hallucinations are marked in red.
Refer to caption: x12.png

Figure: Figure 14: Case Study from LLaVA-Bench-in-the-Wild on LLaVA-1.5 7B. Hallucinations are marked in red.
Refer to caption: x13.png

Figure: Figure 15: Case Study from LLaVA-Bench-in-the-Wild on LLaVA-1.5 7B. Hallucinations are marked in red.
Refer to caption: x14.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Major comments:
1. Table 1 is not explained clear enough, according to the results whatever methods used, greedy decoding setting always performed better, than what is the significance of considering the constraints in Equ 3?
2. What about the possibility of integrating CT2S with other hallucination alleviation strategies?

Minor comments:
1. Better to keep names of the metrics consistent, e.g., CHAIRI and CHAIRS might be written as CHAIRi and CHAIRs, to be the same as Table 8-9.
soundness: 3
strengths: 1. The proposed SID with its key component: Context and Text-aware Token Selection (CT2S) strategy is a Training-Free decoding strategy which efficiently and effectively solves hallucination problem in LVLMs without additional costs.
2. Extensive experiments were conducted and several evaluation metrics were applied to verify the effectiveness of the SID in various aspects and the superiority over existing methods.
3. Hyperparameter sensitivity evaluation showed the proposed SID’s robustness to different hyperparameter settings.
summary: This manuscript proposes to solve the hallucination issue in Large Vision-Language Models (LVLMs). The proposed method named Self-Introspective Decoding (SID) aims to solve the issue with a different decoding strategy compared to existing ones. The Context and Text-aware Token Selection (CT2S) strategy within SID preserves only unimportant vision tokens after the early layers of LVLMs, in order to amplify text-informed hallucinations during the auto-regressive decoding process and guide the LVLMs to produce more accurate outputs. Evaluation was conducted using four representative LVLMs: InstructBLIP, Shikra, LLaVA-1.5, and LLaVA-NeXT. Evaluation metrics include CHAIR, POPE, GPT-4 Assisted Evaluations and MME and MMBench Evaluations. Performance of the proposed SID was compared with Sampling, Greedy, Dola, and LVLM decoding strategies (VCD, ICD, and OPERA). Experiment results demonstrate the effectiveness of SID to generates less-hallucination and higher-quality texts, with lower additional computation cost.
weaknesses: 1. The proposed SID is performed on pre-trained LVLMs, so it’s possible that the performance of SID is limited by those pre-trained models.
2. And currently there’s no solution to specifically tune SID for different LVLMs.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. How does the performance of the proposed approach vary across different LVLM sizes? Do we have any results for larger and smalled LVLMs? (other than 7B)
2. Where does the proposed approach fail to mitigate hallucinations?
soundness: 2
strengths: - The proposed approach does not require additional computational resources
- GPT-4 assisted analyses were done to calculate Sentence-level Hallucination Ratio
- The paper is well written
summary: The paper introduces Self-Introspective Decoding (SID) to alleviate hallucinations in large vision language models. Other available methods either introduce noise or lead to double inference costs. SID offers an alternative using a Context and Text-aware Token Selection (CT2S) strategy that selectively attenuates less important vision tokens in early decoder layers. This reduces irrelevant hallucinations in the generations. The approach requires minimal additional computational resources. The paper presents multiple empirical results to show that the method reduces hallucinations while preserving text quality.
weaknesses: - The paper does not present the results on different benchmarks to show the preservation of LVLM ability. LVLMs should be extensively tested on a variety of benchmarks that test different skills, like - MathVision and Mathvista for mathematical reasoning, MMMU for college-level knowledge on various subjects, MM-Vet/v2
- The paper does not cover a comprehensive set of baselines - Woodpecker (https://arxiv.org/abs/2310.16045), LRV (https://www.researchgate.net/publication/375596083), LURE (https://arxiv.org/pdf/2310.00754)
- "Hallucination, defined as the generation of irrelevant, factually incorrect, or meaningless text in a given context." The approach aims to tackle hallucinations in LVLMs as a whole but does not mention or tackle style hallucinations or biases introduced when LVLMs are instruction-tuned.

-Minor:
(typo) Figure 2 - Gnerated

#256 validate

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. In Table 1, the authors demonstrate that removing the adaptive plausibility constraint resulted in lower performance degradation for the proposed method on the POPE benchmark. Does this also hold for the open-ended CHAIR benchmark and the more comprehensive MME benchmark?
2. How do you define vision-and-text association hallucination?
3. What is the specific experimental setup in Figure 5? Why does ID produce entirely inconsistent responses?
4. How do the proposed method and other baselines perform on the Existence, Count, Position, and Color subsets of the MME benchmark?
5. The experiment in Lines 346-362 is interesting. However, the rationale behind why adding underperforms the proposed method is unclear. From my understanding, boosting the target logits of the original prediction could also enhance discrimination.
soundness: 3
strengths: 1. The proposed method seems novel and interesting. The motivation is also clear.
2. Through empirical experiments, the authors highlight that current vision-and-text agnostic contrastive decoding methods can introduce uncertainty noise and degrade performance. This provides valuable insights for future research.
3. The experimental results are comprehensive and promising, effectively validating the effectiveness of the proposed approach. Efficiency comparisons further highlight the proposed method’s advantage over other contrastive decoding approaches.
4. The authors provide the code implementation for this work, enhancing its reproducibility (though I have not run the code myself).







5. The writing and presentation of this paper is good.
summary: This paper proposes Self-Introspective Decoding (SID) for mitigating hallucinations in Large Vision-Language Models (LVLMs). The authors specifically observe that LVLMs are capable of introspectively assessing the importance of vision tokens based on preceding tokens. Based on this, the authors propose a Context and Text-aware Token Selection ($\text{CT}^2\text{S}$) strategy to amplify fine-grained hallucinations, which are ultimately mitigated using a contrastive decoding method. Comprehensive experimental results across various benchmarks demonstrate that the proposed SID outperforms other contrastive decoding baselines in alleviating hallucinations in LVLMs, while also achieving lower computational cost.
weaknesses: 1. The proposed method shares some similarities with AVISC [1], which also identifies less-important visual tokens for contrastive decoding. A detailed discussion of these similarities, along with a comparison of experimental results, would strengthen this paper.
2. In Figure 6, pruning all vision tokens results in less than a 0.5% performance difference from the optimal setting, suggesting that the proposed strategy of selecting the top-k least important tokens provides only marginal performance gains. Additionally, could you clarify the statement, “*The loss of vision information for subsequent decoder layers results in losing the visual context, leading to aimless hallucinations without sufficient vision grounding*”? Are there any examples or experiments that illustrate this?
3. The proposed method achieves marginal performance improvements on the MME and MMBench benchmarks. Could you please include the standard deviations for the two benchmarks, as their performance is sensitive to random seeds?
4. The proposed method may lack interpretability, as the selected less important tokens (shown in Figures 3 and 4) do not carry true semantic meaning, unlike other contrastive decoding approaches such as HALC [2]. Besides, the proposed method seems not very reasonable when applied to simple LVLMs such as InstructBLIP, which only has 32 vision tokens. In this case, preserving only the least important tokens provides a very coarse measure of vision importance and is unlikely to effectively induce vision-and-text association hallucinations.

Minor issues:
1. In Table 6, is the inference time calculated per instance or for the entire benchmark?
2. Wrong citation format in Line 910.
3. In Table 1, why are there no standard deviations reported for the VCD method in the Sampling setting? Additionally, are the three experiments conducted using three different random seeds? The reported deviations are lower than expected.

[1] Woo, S., Kim, D., Jang, J., Choi, Y., & Kim, C. (2024). Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models. *arXiv preprint arXiv:2405.17820*.

[2] Chen, Z., Zhao, Z., Luo, H., Yao, H., Li, B., & Zhou, J (2024). HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding. In *International Conference on Machine Learning*.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: Refer to weakness section.
soundness: 3
strengths: 1) Presentation of the paper is clear and straightforward.
2) The paper does a good set of experiments to support their argument that low importance score tokens induce hallucinations.
3) The paper does sufficient ablations and evaluations to prove its approach.
summary: 1) The main contribution os the paper is proposing that some vision tokens with low attention scores can induce hallucinations while decoding. 
2) The paper backs this up by conducting experiments to prove that low score vision tokens focus mainly on unrelated regions in the image. 
3) The paper does extensive ablations on till which layer to prune vision tokens and comparison with other baselines.
weaknesses: 1) An important baseline the paper has missed is a simple but effective one, how would the comparison look if we append the image description before asking a question? Ask the LVLM to describe the image and append it to the question and pass through the model again and see if it has any improvement. Some research on this includes - Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMs (Ghosh et al, 2024).

2) A few more important benchmark comparisons are needed - MathVista, MMVet, LLaVA-Bench, MMMU

Meta-review: This work brings a new method, named Self-Introspective Decoding (SID), that alleviates hallucinations in large vision language models. Existing approaches either introduce noise or need double inference costs, while this paper provides an alternative that uses a Context and Text-aware Token Selection scheme to selectively attenuate less important tokens in early decoder layers, and thus mitigates irrelevant hallucinations. The designed SID and the Context and Text-aware Token Selection scheme is a Training-Free strategy that effectively handles the hallucination issue without additional costs. All reviewers acknowledge the contributions of this work. In the camera ready version, authors still need to carefully improve the work following reviewers' comments.

Decision: Accept (Poster)