SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defense

## Abstract

Abstract Large Vision-Language Models (LVLMs) excel in diverse cross-modal tasks. However, object hallucination, where models produce plausible but inaccurate object descriptions, remains a significant challenge. In contrast to previous work focusing on LLM components, this paper is the first to trace LVLM hallucinations to visual encoders and identifies three key issues: statistical bias, inherent bias, and vulnerability. To address these challenges, we propose SHIELD, a training-free framework that mitigates hallucinations through three strategies: re-weighting visual tokens to reduce statistical bias, introducing noise-derived tokens to counter inherent bias, and applying adversarial attacks with contrastive decoding to address vulnerability. Experiments demonstrate that SHIELD effectively mitigates object hallucinations across diverse benchmarks and LVLM families. Moreover, SHIELD achieves strong performance on the general LVLM benchmark, highlighting its broad applicability. Code will be released.

## 1 Introduction

Large Vision-Language Models (LVLMs) [2, 23, 8] combine visual and textual information and have advanced significantly in cross-modal tasks. Despite these advances, they suffer from object hallucination, generating object descriptions that may appear reasonable but misrepresent the image, either by misidentifying attributes (e.g., color, quantity, position) or by introducing non-existent objects. This issue poses reliability and safety risks in domains such as healthcare [12, 35], autonomous systems [5, 37], and robotics [24, 21].

Figure: Figure 1: Key issues underlying object hallucinations in LVLMs. Statistical bias: the visual encoder overemphasizes frequent visual patterns, distorting fine-grained perception. Inherent bias: the encoder produces erroneous representations of dominant objects in the pretraining data, regardless of input. Vulnerability: the encoder is sensitive to minor perturbations, yielding unreliable features.
Refer to caption: x1.png

Various approaches have been proposed to mitigate object hallucinations. Early efforts, such as fine-grained modality alignment [3] and data augmentation to reduce co-occurrence bias [16, 30], were designed for small-scale VLMs but fail to generalize to LVLMs [15, 36]. More recent research falls into two categories: training-required and training-free methods. Training-required methods, including preference optimization [26], post-hoc revisers [39], and RLHF [31], improve factual consistency but demand substantial human and computational resources. In contrast, training-free methods, such as contrasting outputs from distorted inputs [17] or applying over-trust penalties during decoding [13], offer a more efficient alternative.
However, these approaches primarily focus on the LLM component, leaving the role of visual encoders underexplored.

This paper is the first to trace LVLM hallucinations to visual encoders, filling this gap by identifying three key issues: statistical bias, inherent bias, and vulnerability, as illustrated in Figure [1](https://arxiv.org/html/2510.16596v1#S1.F1). Despite large-scale pretraining, these encoders remain affected by imbalanced distributions of visual concepts in the pretraining data, resulting in statistical and inherent biases. Statistical bias leads the visual encoder to overemphasize tokens related to frequent visual patterns, thereby distorting the perception of details. Inherent bias leads the visual encoder to produce representations of dominant objects in the pretraining data, regardless of the input, even when it is meaningless. Furthermore, vulnerability, arising from insufficient robustness to noise and perturbations during pretraining, leads the encoder to produce inaccurate visual representations even with small perturbations.

To address bias and vulnerability in visual encoders that hinder feature extraction and amplify hallucinations in LVLMs, we propose SHIELD, a training-free method combining token re-weighting, token subtraction, and contrastive decoding. Specifically, token re-weighting alleviates statistical bias by distributing attention across more tokens relevant to the ground-truth objects, thus avoiding fine-grained distortion from overemphasized tokens. In parallel, token subtraction mitigates inherent bias by estimating erroneous representations related to dominant objects in pretraining data via noise input and eliminating them through token-level subtraction. To address vulnerability, contrastive decoding exposes hallucinations with a perturbed image and suppresses them by contrasting with outputs from a natural image.

Experiments demonstrate that SHIELD consistently improves performance on object hallucination benchmarks, including CHAIR [30], POPE [19], the hallucination subset of MME [10], and GPT4o-aided evaluations on LLaVA-Bench [23]. Moreover, these improvements are observed across diverse LVLM families, such as LLaVA [23], InstructBLIP [8], and Qwen-VL [2]. Beyond object hallucination mitigation, SHIELD also enhances general perception capabilities, as evidenced by improvements on the full MME benchmark [10], highlighting its broader applicability.

Our contributions are summarized as follows:

- •
We analyze the role of visual encoders in contributing to object hallucinations in LVLMs, focusing on statistical bias, inherent bias, and vulnerability.
- •
We propose SHIELD, a training-free method that mitigates object hallucinations by reducing statistical bias via token re-weighting, alleviating inherent bias using token subtraction, and addressing vulnerability through contrastive decoding.
- •
Comprehensive experiments validate SHIELD’s effectiveness in mitigating object hallucinations across diverse benchmarks and LVLM families. Moreover, its strong performance on the general LVLM benchmark highlights its broad applicability.

## 2 Related work

### 2.1 Large Vision Language Models (LVLMs)

Recent advances in large-scale foundation models and multimodal learning have accelerated the development of Large Vision-Language Models (LVLMs). By combining Large Language Models (LLMs) [1, 4, 6, 7, 11, 29, 32, 33, 34] with cross-modal frameworks such as CLIP [28] and BLIP [18], LVLMs integrate visual and textual information for more comprehensive understanding. Nevertheless, LVLMs across different architectures, including LLaVA-1.5 [22], InstructBLIP [8], and Qwen-VL [2], still suffer from hallucinations, particularly in fine-grained object recognition and challenging visual grounding. Such errors, often involving non-existent objects or misidentified attributes, remain a key challenge for reliability in real-world applications.

### 2.2 Object Hallucination in LVLMs

Approaches to mitigating object hallucinations in LVLMs can be grouped into training-required and training-free methods. Training-required methods reduce hallucinations by optimizing model parameters or training auxiliary modules. Prominent methods include CLIP-DPO [26], leveraging CLIP-based similarity ranking for preference optimization; LURE [39], using post-hoc revisers to align text with visual input; and LLaVA-RLHF [31], incorporating human feedback through reinforcement learning. Training-free methods improve decoding without modifying model. Representative approaches include Visual Contrastive Decoding (VCD) [17], which contrasts outputs from natural and blur inputs to mitigate hallucinations, and OPERA [13], which reduces overconfidence through penalty mechanisms and token-level adjustments. While effective, these methods rarely address the bias and vulnerability of visual encoders, which this work aims to address.

## 3 Method

Figure: (a) Token Overemphasis
Refer to caption: x2.png

### 3.1 Hallucinations Stem from Visual Encoder

Accurate visual feature extraction is crucial for LVLMs to generate reliable outputs. However, bias and vulnerability in visual encoders distort features, intensifying object hallucinations. This section delves into these challenges.

#### 3.1.1 Statistical and Inherent Bias in Visual Encoder

Most LVLMs adopt visual encoders derived from pretrained CLIP models. Although these encoders benefit from large-scale pretraining, they are influenced by the imbalanced distribution of visual concepts in the pretraining data. Specifically, certain visual concepts appear far more frequently than others, while rare or context-dependent elements are severely underrepresented [27]. As a result, the model develops a strong inductive bias toward frequent patterns and dominant objects, giving rise to both statistical and inherent bias.

Statistical bias denotes the visual encoder’s over-reliance on frequent visual patterns in the pretraining data, causing overemphasis on the corresponding tokens with disproportionately high L2 activation values [9]. This overemphasis distorts the downstream LLM’s perception of fine-grained details by directing attention to overweighted tokens [14], often resulting in hallucinations. Analysis of LLaVA-1.5’s responses and visual tokens on the POPE COCO subset (Figure [2(a)](https://arxiv.org/html/2510.16596v1#S3.F2.sf1)) shows that the proportion of hallucinated samples grows with stronger token overemphasis, measured by the peak-to-average L2 ratio (the deviation of the highest-norm token from the mean among visual tokens).

Inherent bias is the visual encoder’s overdependence on dominant objects in the pretraining data, leading it to generate erroneous representations of these objects regardless of the input, even when meaningless. As shown in Figure [2(b)](https://arxiv.org/html/2510.16596v1#S3.F2.sf2), analysis of LLaVA-1.5’s responses to the POPE COCO random split questions with meaningless images (random noise) as input shows frequent hallucinations of dominant objects such as cars, chairs, and tables, defined as cases where the model incorrectly predicts the presence of queried objects.

#### 3.1.2 Vulnerability in Visual Encoder

The vulnerability of visual encoders is another key factor contributing to object hallucinations. It arises from their limited robustness to noise and subtle perturbations [25], making them susceptible to constructing inaccurate visual representations under such disturbances. As shown in Figure [2(c)](https://arxiv.org/html/2510.16596v1#S3.F2.sf3), the performance of LLaVA-1.5 drops sharply on the POPE COCO subset even with a few attack steps, demonstrating that minor perturbations can exploit this weakness and yield unreliable features.

Figure: Figure 3: Illustration of the proposed SHIELD framework. Given an input image and a query text, the visual encoder produces tokens affected by statistical bias (overemphasized tokens ), inherent bias (erroneous representations ), and vulnerability (inaccurate features ). SHIELD addresses these issues through three modules: (i) Token Re-weighting, which redistributes attention to more ground-truth-object relevant tokens to alleviate overemphasis (); (ii) Token Subtraction, which estimates and removes erroneous representations () via noise-derived tokens; and (iii) Contrastive Decoding, which exposes inaccurate features () using attacked images and suppresses corresponding outputs by contrasting them with those from the natural image.
Refer to caption: x5.png

### 3.2 SHIELD: suppressing hallucinations in lvlm encoders via bias and vulnerability defense

Building on these observations, we propose SHIELD, a training-free method to mitigate object hallucinations by addressing statistical bias, inherent bias, and vulnerability in visual encoders, as illustrated in Figure [3](https://arxiv.org/html/2510.16596v1#S3.F3). SHIELD integrates three strategies. Token re-weighting distributes attention across more tokens relevant to ground-truth objects, thereby reducing fine-grained distortion from overemphasized tokens and alleviating statistical bias. Token subtraction estimates erroneous representations of dominant objects in the pretraining data using noise input and removes them through token-level subtraction, thus mitigating inherent bias. Finally, contrastive decoding applies perturbations to the input image to expose hallucinations and suppresses them by contrasting outputs with those from the natural image, countering vulnerability.

#### 3.2.1 Formulation of LVLM Inference

A LVLM’s inference can be described in three stages. First, the visual encoder $E(\cdot)$ extracts $N$ visual tokens from the raw image $\mathbf{v}$:

$$ $\mathbf{x}^{v}={x_{0},x_{1},\ldots,x_{N-1}}=E(\mathbf{v}).$ (1) $$

Next, given $\mathbf{x}^{v}$ and the query text $\mathbf{t}$, the LLM computes the output logits for token $y_{i}$ at step $i$, conditioned on the preceding sequence $y_{<i}$:

$$ $\operatorname{logit}(y_{i}\mid\mathbf{x}^{v},\mathbf{t},y_{<i})=\operatorname{LLM}(\mathbf{x}^{v},\mathbf{t},y_{<i}).$ (2) $$

Finally, the logits are transformed into a probability distribution over the vocabulary via softmax, from which the next token is selected according to the decoding strategy:

$$ $p(y_{i}\mid\mathbf{x}^{v},\mathbf{t},y_{<i})=\operatorname{softmax}\left[\operatorname{logit}(y_{i}\mid\mathbf{x}^{v},\mathbf{t},y_{<i})\right].$ (3) $$

Autoregressive repetition of the second and third stages produces the final textual output.

Figure: Figure 4: Mitigating Statistical Bias. Visual tokens are re-weighted via a similarity matrix between visual tokens and naive caption tokens, emphasizing more ground-truth-object relevant tokens and reducing overemphasis.
Refer to caption: x6.png

#### 3.2.2 Mitigating Statistical Bias

As discussed in Section [3.1.1](https://arxiv.org/html/2510.16596v1#S3.SS1.SSS1), statistical bias causes the visual encoder to overemphasize tokens associated with frequent visual patterns, distorting fine-grained perception. To address this, token re-weighting is applied based on the similarity between visual tokens and naive caption tokens, encouraging the model to attend to more tokens relevant to ground-truth objects.

As shown in Figure [5](https://arxiv.org/html/2510.16596v1#S3.F5), token re-weighting begins with generating a naive caption $\mathbf{c}^{\text{naive}}$ using the vanilla LVLM for the given image $\mathbf{v}$:

$$ $\mathbf{c}^{\text{naive}}=\operatorname{VanillaLVLM}\big(\mathbf{v},\textit{``Please describe this image''}\big).$ (4) $$

The CLIP text encoder $E_{t}(\cdot)$ (paired with $E(\cdot)$ during CLIP pretraining) then encodes the caption into $P$ tokens:

$$ $\mathbf{c}=\{c_{0},c_{1},\ldots,c_{P-1}\}=E_{t}(\mathbf{c}^{\text{naive}}).$ (5) $$

Given the caption tokens $\mathbf{c}$ and the visual tokens $\mathbf{x}^{v}$, a similarity matrix $\mathbf{M}\in\mathbb{R}^{N\times P}$ is computed:

$$ $\mathbf{M}=\frac{\mathbf{x}^{v}\mathbf{c}^{\top}}{\|\mathbf{x}^{v}\|_{2}\cdot\|\mathbf{c}\|_{2}}.$ (6) $$

From $\mathbf{M}$, weights $\mathbf{W}^{v}$ are obtained by taking the maximum along the caption dimension and normalizing to $[0,1]$:

$$ $\mathbf{W}^{v}=\operatorname{norm}(\max_{j}\mathbf{M}_{i,j}),\quad\mathbf{W}^{v}\in\mathbb{R}^{N}.$ (7) $$

Finally, the weights are applied via residual addition ($\odot$: element-wise multiplication) to emphasize visual tokens corresponding to captioned objects, yielding statistical-bias-corrected tokens ${\mathbf{x}^{v}}^{\prime}$:

$$ ${\mathbf{x}^{v}}^{\prime}={\mathbf{x}^{v}}+{\mathbf{x}^{v}}\odot{\mathbf{W}^{v}}.$ (8) $$

Although the naive caption $\mathbf{c}^{\text{naive}}$ may introduce hallucinations, they do not affect re-weighting, as hallucinated objects fail to match any visual tokens with high similarity during similarity matrix $\mathbf{M}$ computation. Thus, token re-weighting remains focused on ground-truth objects.

#### 3.2.3 Reducing Inherent Bias

As in Section [3.1.1](https://arxiv.org/html/2510.16596v1#S3.SS1.SSS1), inherent bias leads the visual encoder to produce erroneous representations of dominant objects in the pretraining data, regardless of the input. To counter this, token subtraction introduces noise inputs to estimate such erroneous features and removes them from the visual tokens.

As shown in Figure [5](https://arxiv.org/html/2510.16596v1#S3.F5), $K$ random noise inputs $\mathbf{n}_{i}$ (with the same size as the image) are passed through the visual encoder. The resulting tokens are averaged to estimate erroneous representations, which are then subtracted from the statistical-bias-corrected tokens ${\mathbf{x}^{v}}^{\prime}$, yielding bias-reduced tokens:

$$ ${\mathbf{x}^{v}}^{\prime\prime}={\mathbf{x}^{v}}^{\prime}-\frac{1}{K}\sum_{i=1}^{K}E(\mathbf{n}_{i}).$ (9) $$

Since inherent bias depends only on visual encoder parameters, the estimation of erroneous representations from noise inputs can be pre-calculated for each model to improve efficiency.

The bias-reduced tokens ${\mathbf{x}^{v}}^{\prime\prime}$, together with the query text $\mathbf{t}$, are subsequently fed into the LLM to produce bias-reduced logits:

$$ $\operatorname{logit}\left(y_{i}\mid{\mathbf{x}^{v}}^{\prime\prime},\mathbf{t},y_{<i}\right)=\operatorname{LLM}\left({\mathbf{x}^{v}}^{\prime\prime},\mathbf{t},y_{<i}\right).$ (10) $$

Figure: Figure 6: Addressing Vulnerability. An attack tensor constructed from the input image and its naive caption via adversarial learning is applied to reveal objects likely to be hallucinated, followed by contrastive decoding to suppress their generation.
Refer to caption: x8.png

#### 3.2.4 Address Vulnerability

As noted in Section [3.1.2](https://arxiv.org/html/2510.16596v1#S3.SS1.SSS2), the visual encoder lacks robustness to subtle perturbations and noise, making it susceptible to inaccurate representations, especially when key pixels are disturbed. To counter this vulnerability, a two-step strategy is adopted: adversarial attack is first applied to reveal objects likely to be hallucinated, followed by contrastive decoding to suppress the probability of generating the corresponding outputs during inference.

To expose vulnerability-induced hallucinations, an attack tensor is constructed from the input image $\mathbf{v}$ and its naive caption $\mathbf{c}^{\text{naive}}$ (Equation [4](https://arxiv.org/html/2510.16596v1#S3.E4)) using the visual encoder $E(\cdot)$ and its paired text encoder $E_{t}(\cdot)$. As illustrated in Figure [6](https://arxiv.org/html/2510.16596v1#S3.F6), a learnable perturbation $\mathbf{\delta}$ is added to the input image and refined via backpropagation. The adversarial loss is defined as the cosine similarity between the global representation of the perturbed image and that of the naive caption:

$$ $l_{\text{adv}}=\cos\big(E(\mathbf{v}+\mathbf{\delta}),\;E_{t}(\mathbf{c}^{\text{naive}})\big).$ (11) $$

The final attack tensor $\mathbf{\delta}^{*}$ is obtained by minimizing $l_{\text{adv}}$ via gradient descent with learning rate $l$:

$$ $\mathbf{\delta}^{*}=\arg\min_{\mathbf{\delta}}\;l_{\text{adv}}.$ (12) $$

During inference, the attack tensor $\mathbf{\delta}^{*}$ is added to the input image to produce vulnerability-induced inaccurate visual representations:

$$ $\overline{\mathbf{x}}^{v}=\{\bar{x}_{0},\bar{x}_{1},\ldots,\bar{x}_{N-1}\}=E(\mathbf{v}+\mathbf{\delta}^{*}).$ (13) $$

These inaccurate representations are then used to produce adversarial logits:

$$ $\operatorname{logit}\big(y_{i}\mid\overline{\mathbf{x}}^{v},\mathbf{t},y_{<i}\big)=\operatorname{LLM}\big(\overline{\mathbf{x}}^{v},\mathbf{t},y_{<i}\big).$ (14) $$

To suppress hallucinations revealed by the attack tensor, contrastive decoding is applied. At each decoding step $i$, SHIELD contrasts the bias-reduced logits with the adversarial logits to adjust the output probability distribution, where $\alpha$ controls the impact of contrastive decoding:

$$ $p_{shield}(y_{i})=\operatorname{softmax}\!\Big[(1+\alpha)\,\operatorname{logit}(y_{i}\mid{\mathbf{x}^{v}}^{\prime\prime},\mathbf{t},y_{<i})-\alpha\,\operatorname{logit}(y_{i}\mid\overline{\mathbf{x}}^{v},\mathbf{t},y_{<i})\Big],$ (15) $$

Following [17], an adaptive plausibility constraint is introduced to avoid implausible outputs. Only tokens with probabilities no smaller than a fraction $\beta$ of the maximum are retained:

$$ $\nu_{\text{token}}(y_{i})=\{\,y_{i}\in\nu:p(y_{i})\geq\beta\max_{\omega}p(\omega)\,\},$ (16) $$

where $\nu$ is the vocabulary and $\nu_{\text{token}}(y_{i})$ is the valid subset at step $i$. The threshold $\beta$ determines truncation aggressiveness. For tokens not in $\nu_{\text{token}}(y_{i})$, probabilities are set to zero.

## 4 Experiments

### 4.1 Implementation Details

To evaluate the effectiveness of SHIELD in mitigating hallucinations, three representative LVLMs were selected: LLaVA-1.5 [22], InstructBLIP [8], and Qwen-VL [2]. SHIELD was compared against the corresponding vanilla LVLMs and two recent training-free methods, VCD [17] and OPERA [13]. Following their original setups, vanilla LVLMs and VCD adopted sampling-based decoding, while OPERA employed beam search decoding with a penalty term on logits to reduce overconfidence. For SHIELD, sampling-based decoding was used, drawing from the modified post-softmax distribution. Unless otherwise specified, $\alpha=2$, $\beta=0.35$, $K=32$, and $l=0.02$ were applied across all LVLMs, where $\alpha$ controls the strength of contrastive decoding, $\beta$ sets the truncation threshold in the plausibility constraint, $K$ denotes the number of noise inputs for estimating inherent bias, and $l$ is the learning rate for optimizing the attack tensor. All experiments were conducted on a single RTX A6000 GPU.

### 4.2 Quantitative Results

This section evaluates the effectiveness of SHIELD in mitigating hallucinations for both detailed descriptions and simplified VQA answers.

CHAIR Evaluation.
The Caption Hallucination Assessment with Image Relevance (CHAIR) metric [30] assesses object hallucinations in image captioning by calculating the proportion of object references absent from the ground-truth annotations. It comprises two levels of evaluation, $C_{S}$ (sentence-level) and $C_{I}$ (instance-level):

$$ $C_{S}=\frac{|\text{sentences with hallucinated object}|}{|\text{all sentences}|},\quad C_{I}=\frac{|\text{hallucinated objects}|}{|\text{all objects mentioned}|}.$ $$

Following the setup in [13], we conduct evaluations on 500 randomly sampled images from the COCO 2014 validation dataset [20], using the prompt “Please describe this image in detail.” For fair comparison, all generated captions are truncated to a maximum length of 512 tokens.

As shown in Table [2](https://arxiv.org/html/2510.16596v1#S4.T2), SHIELD consistently outperforms all previous training-free decoding methods on both $C_{S}$ and $C_{I}$, achieving up to 18% improvement over the second-best method, OPERA, on LLaVA-1.5. This performance gain stems from SHIELD’s ability to counter biases and vulnerability in the visual encoder, thereby reducing hallucination risk in detailed descriptions.

**Table 1: CHAIR Hallucination Evaluation**
|  | LLaVA-1.5 | InstructBLIP | Qwen-VL |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method | $C_{S}$$\downarrow$ | $C_{I}$$\downarrow$ | $C_{S}$$\downarrow$ | $C_{I}$$\downarrow$ | $C_{S}$$\downarrow$ | $C_{I}$$\downarrow$ |
| Vanilla | 48.8 | 14.2 | 54.6 | 24.8 | 49.2 | 13.1 |
| VCD | 46.8 | 13.2 | 44.0 | 13.6 | 46.4 | 11.9 |
| OPERA | 44.6 | 12.8 | 46.4 | 14.2 | 34.6 | 9.5 |
| \rowcolor[HTML]C0C0C0<br>Ours | 36.6 | 10.3 | 40.4 | 10.9 | 28.9 | 9.2 |

GPT4o Assisted Evaluation.
While CHAIR effectively evaluates object-level hallucinations, it fails to capture errors in attributes, locations, or relations. To complement this, we employ GPT4o, a strong multi-modal assistant, to assess LVLM outputs on the LLaVA-Bench dataset. GPT4o scores responses on correctness (C) and detailedness (D) from 0–10, with higher correctness indicating fewer hallucinations. The evaluation explicitly targets objects mentioned but absent from the image, as well as errors in attributes, colors, positions, or relationships. Further details are provided in the Appendix [A.1](https://arxiv.org/html/2510.16596v1#A1.SS1).

As shown in Table [2](https://arxiv.org/html/2510.16596v1#S4.T2), SHIELD achieves substantial gains in correctness, confirming its effectiveness in mitigating hallucinations. In contrast, improvements in detailedness are modest, since the method primarily addresses bias and vulnerability in the visual encoder rather than enhancing fine-grained descriptive coverage.

**Table 3: POPE Hallucination Evaluation on COCO subset**
|  |  | Random | Popular | Adversarial | Average |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LVLM | Method | Accuracy $\uparrow$ | F1 $\uparrow$ | Accuracy $\uparrow$ | F1 $\uparrow$ | Accuracy $\uparrow$ | F1 $\uparrow$ | Accuracy $\uparrow$ | F1 $\uparrow$ |
|  | Vanilla | 83.2 | 81.3 | 81.8 | 80.0 | 78.9 | 77.5 | 81.3 | 79.6 |
|  | VCD | 87.7 | 87.1 | 85.3 | 85.0 | 80.8 | 81.3 | 84.6 | 84.4 |
|  | OPERA | 89.1 | 89.0 | 86.0 | 86.3 | 79.1 | 80.9 | 84.7 | 85.4 |
| LLaVA-1.5 | \cellcolor[HTML]C0C0C0Ours | \cellcolor[HTML]C0C0C091.3 | \cellcolor[HTML]C0C0C091.1 | \cellcolor[HTML]C0C0C087.4 | \cellcolor[HTML]C0C0C087.6 | \cellcolor[HTML]C0C0C082.5 | \cellcolor[HTML]C0C0C083.6 | \cellcolor[HTML]C0C0C087.0 | \cellcolor[HTML]C0C0C087.4 |
|  | Vanilla | 80.7 | 80.4 | 78.2 | 78.3 | 75.8 | 76.5 | 78.2 | 78.4 |
|  | VCD | 84.5 | 83.6 | 81.4 | 81.0 | 79.5 | 79.5 | 81.8 | 81.3 |
|  | OPERA | 89.8 | 89.6 | 83.4 | 84.0 | 80.7 | 81.8 | 84.6 | 85.1 |
| InstructBLIP | \cellcolor[HTML]C0C0C0Ours | \cellcolor[HTML]C0C0C088.2 | \cellcolor[HTML]C0C0C087.6 | \cellcolor[HTML]C0C0C084.6 | \cellcolor[HTML]C0C0C084.3 | \cellcolor[HTML]C0C0C082.2 | \cellcolor[HTML]C0C0C082.4 | \cellcolor[HTML]C0C0C085.0 | \cellcolor[HTML]C0C0C084.8 |
|  | Vanilla | 84.7 | 82.6 | 84.1 | 82.0 | 82.2 | 80.3 | 83.6 | 81.6 |
|  | VCD | 88.6 | 87.8 | 87.1 | 86.4 | 84.2 | 83.9 | 86.6 | 86.0 |
|  | OPERA | 86.1 | 84.2 | 85.7 | 83.8 | 83.9 | 82.1 | 85.2 | 83.3 |
| Qwen-VL | \cellcolor[HTML]C0C0C0Ours | \cellcolor[HTML]C0C0C089.2 | \cellcolor[HTML]C0C0C088.6 | \cellcolor[HTML]C0C0C087.6 | \cellcolor[HTML]C0C0C087.1 | \cellcolor[HTML]C0C0C084.3 | \cellcolor[HTML]C0C0C084.2 | \cellcolor[HTML]C0C0C087.0 | \cellcolor[HTML]C0C0C086.6 |

POPE Evaluation.
Similar to CHAIR, the Polling-based Object Probing Evaluation (POPE) [19] assesses existence-level hallucinations in LVLMs. It adopts a VQA-style format (e.g., “Is there a {object} in the image?”) to test whether models correctly associate images with specific objects. POPE includes three splits: “random” for random objects, “popular” for frequent objects, and “adversarial” for objects semantically related to those in the image. The evaluation is conducted on three subsets: COCO, A-OKVQA, and GQA. Additional results are provided in Appendix [B.1](https://arxiv.org/html/2510.16596v1#A2.SS1).

As shown in Table [3](https://arxiv.org/html/2510.16596v1#S4.T3), SHIELD outperforms previous training-free methods across most splits of the COCO subset. Although all methods show performance drops from Random to Adversarial, SHIELD more effectively mitigates hallucinations in the challenging Adversarial split, highlighting that biases and vulnerability in visual encoders are major contributors to hallucinations. For InstructBLIP, however, the improvements are limited since its Q-Former module constrains the use of modified visual features, thereby diminishing the benefits of SHIELD.

**Table 4: MME Hallucination Evaluation**
|  |  | Object-level | Attribute-level |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| LVLM | Method | Existence Score $\uparrow$ | Count Score $\uparrow$ | Position Score $\uparrow$ | Color Score $\uparrow$ | Total Score $\uparrow$ |
|  | Vanilla | 175.6 | 124.6 | 114.0 | 151.0 | 565.3 |
|  | VCD | 184.6 | 138.3 | 128.6 | 153.0 | 604.6 |
|  | OPERA | 180.6 | 133.3 | 123.3 | 155.0 | 592.3 |
| LLaVA-1.5 | \cellcolor[HTML]C0C0C0Ours | \cellcolor[HTML]C0C0C0195.0 | \cellcolor[HTML]C0C0C0141.6 | \cellcolor[HTML]C0C0C0148.3 | \cellcolor[HTML]C0C0C0183.3 | \cellcolor[HTML]C0C0C0668.3 |
|  | Vanilla | 141.0 | 75.3 | 66.6 | 97.3 | 380.3 |
|  | VCD | 168.3 | 92.3 | 64.0 | 123.0 | 447.6 |
|  | OPERA | 156.0 | 78.3 | 55.0 | 95.0 | 384.3 |
| InstructBLIP | \cellcolor[HTML]C0C0C0Ours | \cellcolor[HTML]C0C0C0170.0 | \cellcolor[HTML]C0C0C075.0 | \cellcolor[HTML]C0C0C088.3 | \cellcolor[HTML]C0C0C0128.3 | \cellcolor[HTML]C0C0C0461.6 |
|  | Vanilla | 155.0 | 127.6 | 131.6 | 173.0 | 587.3 |
|  | VCD | 156.0 | 131.0 | 128.0 | 181.6 | 596.6 |
|  | OPERA | 165.0 | 145.0 | 133.3 | 180.0 | 623.3 |
| Qwen-VL | \cellcolor[HTML]C0C0C0Ours | \cellcolor[HTML]C0C0C0180.0 | \cellcolor[HTML]C0C0C0170.0 | \cellcolor[HTML]C0C0C0128.3 | \cellcolor[HTML]C0C0C0190.0 | \cellcolor[HTML]C0C0C0668.3 |

**Table 5: MME Full Set Evaluation**
| Method | Perception$\uparrow$ | Cognition$\uparrow$ | Total Score$\uparrow$ |
| --- | --- | --- | --- |
| Vanilla | 1279.2 | 352.9 | 1632.1 |
| VCD | 1363.9 | 353.2 | 1717.1 |
| OPERA | 1413.0 | 304.2 | 1717.2 |
| \rowcolor[HTML]C0C0C0<br>Ours | 1473.0 | 337.8 | 1810.8 |

Figure: Figure 7: Evaluation on the full MME. Larger radar indicate better performance.
Refer to caption: x9.png

MME Hallucination Subset Evaluation.
Although POPE adopts a VQA format effective for evaluating object-existence-level hallucinations, it does not capture attribute-level aspects such as count, position, and color. To address this limitation, the MME hallucination subsets [10] provide a more comprehensive benchmark. Following [38], we evaluate object-level hallucinations using the existence and count subsets, and attribute-level hallucinations using the position and color subsets. Performance is reported using the combined metrics of accuracy and accuracy+ as defined in the official implementation.

As shown in Table [4](https://arxiv.org/html/2510.16596v1#S4.T4), SHIELD achieves consistent improvements across all models, leading to higher total scores. By correcting statistical bias in visual encoders, SHIELD reduces the impact of overemphasized tokens on fine-grained perception, thereby significantly mitigating attribute-level hallucinations.

LVLM General Evaluation.
To evaluate the overall performance of SHIELD-enhanced LVLMs, we conduct experiments on the full MME benchmark [10] using the LLaVA-1.5 7B model. The benchmark covers ten perception-related subtasks (including four hallucination-related) and four cognition-oriented ones. Performance is reported using both accuracy and accuracy+ as defined in the official implementation.

As shown in Table [6](https://arxiv.org/html/2510.16596v1#S4.T6) and Figure [7](https://arxiv.org/html/2510.16596v1#S4.F7), SHIELD not only improves hallucination-related performance but also yields notable gains in perception tasks such as OCR and Posters, thereby enhancing the overall capability of the model. Further details are provided in the Appendix [B.2](https://arxiv.org/html/2510.16596v1#A2.SS2).

### 4.3 Ablation Study

Module Ablation.
To assess the effectiveness of SHIELD, we performed ablation studies on each module using the CHAIR benchmark with the LLaVA-1.5 7B model. The adaptive plausibility constraint, a key element of contrastive decoding, was also ablated to evaluate its role in mitigating hallucinations. As shown in Table [6](https://arxiv.org/html/2510.16596v1#S4.T6), all modules contribute notably to reducing hallucinations.

Although integral to contrastive decoding, the adaptive plausibility constraint alone was less effective, indicating that filtering low-probability candidates cannot fully suppress hallucinations. In contrast, each SHIELD module individually reduced hallucination frequency, with their full combination achieving the greatest improvement. Notably, after addressing vulnerability, adding the statistical bias mitigation module yielded a further 13% reduction, highlighting statistical bias as a major source of hallucinations, particularly in longer descriptions.

Figure: (a) Token Re-weighting
Refer to caption: x10.png

Visualization.
Figure [8](https://arxiv.org/html/2510.16596v1#S4.F8) illustrates SHIELD’s effectiveness in mitigating biases and reducing vulnerability in visual encoders. Figure [8(a)](https://arxiv.org/html/2510.16596v1#S4.F8.sf1) demonstrates that re-weighting visual tokens alleviates statistical bias by distributing attention across more object-relevant tokens and reducing overemphasis on specific ones, thereby improving fine-grained perception. Figure [8(b)](https://arxiv.org/html/2510.16596v1#S4.F8.sf2) illustrates SHIELD’s effect on the POPE COCO subset with ambiguous inputs, showing that by leveraging noise-derived tokens to remove inaccurate representations, SHIELD significantly reduces hallucinations of dominant objects in pretraining data. Finally, Figure [8(c)](https://arxiv.org/html/2510.16596v1#S4.F8.sf3) highlights SHIELD’s robustness against perturbations, where SHIELD-enhanced LLaVA-1.5 exhibits substantially less performance degradation under increasing attack steps.

## 5 Conclusion

This paper investigates object hallucinations in LVLMs, tracing their origin to visual encoders. Despite large-scale pretraining, these encoders suffer from three issues: statistical bias, which overemphasizes frequent patterns and distorts fine-grained perception; inherent bias, which induces erroneous representations related to dominant objects in pretraining data; and vulnerability, which makes encoders sensitive to minor perturbations and results in inaccurate features.
To address these challenges, we propose SHIELD, a training-free framework that integrates token re-weighting, token subtraction, and contrastive decoding. Token re-weighting alleviates statistical bias by distributing attention to more ground-truth-relevant tokens. Token subtraction mitigates inherent bias by estimating and removing erroneous dominant-object representations using noise-derived tokens. Contrastive decoding counters vulnerability by exposing hallucinations via perturbed image and suppressing them through contrast with natural inputs.
Extensive experiments demonstrate that SHIELD not only achieves significant improvements on hallucination benchmarks but also enhances general perception tasks, highlighting its effectiveness and broad applicability.

## References

- [1]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan,
Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji
Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men,
Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang,
Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang,
Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan,
Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou,
Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu.
Qwen technical report.
CoRR, abs/2309.16609, 2023.
- [2]
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang
Lin, Chang Zhou, and Jingren Zhou.
Qwen-vl: A frontier large vision-language model with versatile
abilities.
CoRR, abs/2308.12966, 2023.
- [3]
Ali Furkan Biten, Lluís Gómez, and Dimosthenis Karatzas.
Let there be a clock on the beach: Reducing object hallucination in
image captioning.
In WACV, pages 2473–2482, 2022.
- [4]
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan,
Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda
Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom
Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens
Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott
Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec
Radford, Ilya Sutskever, and Dario Amodei.
Language models are few-shot learners.
In NeurIPS, 2020.
- [5]
Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James
Willmott, Danny Birch, Daniel Maund, and Jamie Shotton.
Driving with llms: Fusing object-level vector modality for
explainable autonomous driving.
In ICRA, pages 14093–14100, 2024.
- [6]
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin
Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al.
Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt
quality.
See https://vicuna. lmsys. org (accessed 14 April 2023),
2(3):6, 2023.
- [7]
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra,
Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian
Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez,
Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran,
Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob
Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm
Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia,
Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David
Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David
Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai,
Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica
Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi
Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei,
Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah
Fiedel.
Palm: Scaling language modeling with pathways.
J. Mach. Learn. Res., 24:240:1–240:113, 2023.
- [8]
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao,
Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi.
Instructblip: Towards general-purpose vision-language models with
instruction tuning.
In NeurIPS, 2023.
- [9]
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski.
Vision transformers need registers.
In ICLR, 2024.
- [10]
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin,
Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong
Ji.
MME: A comprehensive evaluation benchmark for multimodal large
language models.
CoRR, abs/2306.13394, 2023.
- [11]
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli.
Chatgpt outperforms crowd-workers for text-annotation tasks.
CoRR, abs/2303.15056, 2023.
- [12]
Mingzhe Hu, Shaoyan Pan, Yuheng Li, and Xiaofeng Yang.
Advancing medical imaging with language models: A journey from
n-grams to chatgpt.
CoRR, abs/2304.04920, 2023.
- [13]
Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua
Lin, Weiming Zhang, and Nenghai Yu.
OPERA: alleviating hallucination in multi-modal large language
models via over-trust penalty and retrospection-allocation.
In CVPR, pages 13418–13427, 2024.
- [14]
Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang.
See what you are told: Visual attention sink in large multimodal
models.
In ICLR, 2025.
- [15]
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon
Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.
Scaling laws for neural language models.
CoRR, abs/2001.08361, 2020.
- [16]
Jae-Myung Kim, A. Sophia Koepke, Cordelia Schmid, and Zeynep Akata.
Exposing and mitigating spurious correlations for cross-modal
retrieval.
In CVPR Workshops, pages 2585–2595, 2023.
- [17]
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and
Lidong Bing.
Mitigating object hallucinations in large vision-language models
through visual contrastive decoding.
In CVPR, pages 13872–13882, 2024.
- [18]
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi.
BLIP: bootstrapping language-image pre-training for unified
vision-language understanding and generation.
In ICML, pages 12888–12900, 2022.
- [19]
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen.
Evaluating object hallucination in large vision-language models.
In EMNLP, pages 292–305, 2023.
- [20]
Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona,
Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.
Microsoft COCO: common objects in context.
In ECCV, pages 740–755, 2014.
- [21]
Haokun Liu, Yaonan Zhu, Kenji Kato, Izumi Kondo, Tadayoshi Aoyama, and Yasuhisa
Hasegawa.
Llm-based human-robot collaboration framework for manipulation tasks.
CoRR, abs/2308.14972, 2023.
- [22]
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.
Improved baselines with visual instruction tuning.
In CVPR, pages 26286–26296, 2024.
- [23]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning.
In NeurIPS, 2023.
- [24]
Jinjie Mai, Jun Chen, Bing Li, Guocheng Qian, Mohamed Elhoseiny, and Bernard
Ghanem.
LLM as A robotic brain: Unifying egocentric memory and control.
CoRR, abs/2304.09349, 2023.
- [25]
Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick.
Understanding zero-shot adversarial robustness for large-scale
models.
In ICLR. OpenReview.net, 2023.
- [26]
Yassine Ouali, Adrian Bulat, Brais Martínez, and Georgios Tzimiropoulos.
CLIP-DPO: vision-language models as a source of preference for
fixing hallucinations in lvlms.
In ECCV, pages 395–413, 2024.
- [27]
Shubham Parashar, Zhiqiu Lin, Tian Liu, Xiangjue Dong, Yanan Li, Deva Ramanan,
James Caverlee, and Shu Kong.
The neglected tails in vision-language models.
In CVPR, pages 12988–12997, 2024.
- [28]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
Gretchen Krueger, and Ilya Sutskever.
Learning transferable visual models from natural language
supervision.
In ICML, pages 8748–8763, 2021.
- [29]
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael
Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.
Exploring the limits of transfer learning with a unified text-to-text
transformer.
J. Mach. Learn. Res., 21:140:1–140:67, 2020.
- [30]
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate
Saenko.
Object hallucination in image captioning.
In EMNLP, pages 4035–4045, 2018.
- [31]
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen,
Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and
Trevor Darrell.
Aligning large multimodal models with factually augmented RLHF.
In ACL (Findings), pages 13088–13110, 2024.
- [32]
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos
Guestrin, Percy Liang, and Tatsunori B. Hashimoto.
Stanford alpaca: An instruction-following llama model.
[https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023.
- [33]
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang,
Hyung Won Chung, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou,
Neil Houlsby, and Donald Metzler.
UL2: unifying language learning paradigms.
In ICLR, 2023.
- [34]
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne
Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric
Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave,
and Guillaume Lample.
Llama: Open and efficient foundation language models.
CoRR, abs/2302.13971, 2023.
- [35]
Sheng Wang, Zihao Zhao, Xi Ouyang, Qian Wang, and Dinggang Shen.
Chatcad: Interactive computer-aided diagnosis on medical image using
large language models.
CoRR, abs/2302.07257, 2023.
- [36]
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian
Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H.
Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William
Fedus.
Emergent abilities of large language models.
Trans. Mach. Learn. Res., 2022.
- [37]
Zhenyu Wu, Ziwei Wang, Xiuwei Xu, Jiwen Lu, and Haibin Yan.
Embodied task planning with large language models.
CoRR, abs/2307.01848, 2023.
- [38]
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang
Shen, Ke Li, Xing Sun, and Enhong Chen.
Woodpecker: Hallucination correction for multimodal large language
models.
CoRR, abs/2310.16045, 2023.
- [39]
Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn,
Mohit Bansal, and Huaxiu Yao.
Analyzing and mitigating object hallucination in large
vision-language models.
In ICLR, 2024.

## Appendix A Detailed Experimental Setup

**Table 7: The Prompt used for GPT4o-aid evaluation**
| GPT-4o Prompt |
| --- |
| You are required to score the performance of four AI assistants in describing a given image. You should pay extra attention to the hallucination, which refers to the part of descriptions that are inconsistent with the image content, such as claiming the existence of something not present in the image or describing incorrectly in terms of the counts, positions, or colors of objects in the image. Please rate the responses of the assistants on a scale of 1 to 10, where a higher score indicates better performance, according to the following criteria: |
| 1: Correctness: whether the response is accurate with respect to the image content. Responses with fewer hallucinations should be given higher scores. |
| 2: Detailedness: whether the response is rich in necessary details. Note that hallucinated descriptions should not count as necessary details. |
| Please output the scores for each criterion, containing only four values indicating the scores for Assistant 1, 2, 3 and 4, respectively. The four scores are separated by a space. Following the scores, please provide an explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment. |
| [Assistant 1] |
| {} |
| [End of Assistant 1] |
| [Assistant 2] |
| {} |
| [End of Assistant 2] |
| [Assistant 3] |
| {} |
| [End of Assistant 3] |
| [Assistant 4] |
| {} |
| [End of Assistant 4] |
| Output format: |
| Correctness: <Scores of the four answers> |
| Reason: |
| Detailedness: <Scores of the four answers> |
| Reason: |

### A.1 GPT4o Assisted Evaluation

Following [38], we use GPT4o to evaluate the vanilla LVLM, VCD, OPERA, and our proposed SHIELD model. For each LVLM and image, we generate a description using the prompt “Please describe this image in detail.” Using the evaluation prompt in Table [7](https://arxiv.org/html/2510.16596v1#A1.T7), GPT4o scores the four descriptions from 0 to 10 on two aspects: correctness, which measures how well the description matches the image, giving higher scores to accurate descriptions and lower scores to those with hallucinated content, and detailedness, which assesses how fully the description captures image details. The prompt instructs GPT4o to ignore bias from the order of inputs and focus on inconsistencies such as objects mentioned but not present, or incorrect colors, positions, and relationships.

**Table 8: Comparison on POPE A-OKVQA and GQA Subset using LLaVA-1.5**
|  |  | Random | Popular | Adversarial | Average |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dataset | Method | Accuracy↑ | F1↑ | Accuracy↑ | F1↑ | Accuracy↑ | F1↑ | Accuracy↑ | F1↑ |
|  | Vanilla | 83.4 | 82.5 | 79.9 | 79.5 | 74.0 | 75.1 | 79.1 | 79.0 |
|  | VCD | 86.1 | 86.3 | 81.8 | 82.8 | 74.9 | 77.7 | 80.9 | 82.2 |
|  | OPERA | 88.1 | 88.5 | 83.3 | 84.5 | 73.8 | 77.7 | 81.7 | 83.5 |
| AOKVQA | \cellcolor[HTML]C0C0C0Ours | \cellcolor[HTML]C0C0C090.3 | \cellcolor[HTML]C0C0C089.9 | \cellcolor[HTML]C0C0C085.3 | \cellcolor[HTML]C0C0C085.5 | \cellcolor[HTML]C0C0C077.2 | \cellcolor[HTML]C0C0C079.1 | \cellcolor[HTML]C0C0C084.2 | \cellcolor[HTML]C0C0C084.8 |
|  | Vanilla | 83.7 | 82.9 | 78.1 | 78.3 | 75.0 | 76.0 | 78.9 | 79.0 |
|  | VCD | 86.6 | 86.9 | 80.7 | 82.2 | 76.0 | 78.7 | 81.1 | 82.6 |
|  | OPERA | 88.6 | 89.1 | 79.8 | 82.1 | 75.0 | 78.8 | 81.1 | 83.3 |
| GQA | \cellcolor[HTML]C0C0C0Ours | \cellcolor[HTML]C0C0C090.5 | \cellcolor[HTML]C0C0C090.3 | \cellcolor[HTML]C0C0C084.2 | \cellcolor[HTML]C0C0C084.8 | \cellcolor[HTML]C0C0C079.5 | \cellcolor[HTML]C0C0C081.0 | \cellcolor[HTML]C0C0C084.7 | \cellcolor[HTML]C0C0C085.3 |

**Table 9: Results on MME perception-related tasks using LLaVA-1.5**
| Method | Existence | Count | Position | Color | Posters | Celebrity | Scene | Landmark | Artwork | OCR |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Vanilla | 175.67 | 124.67 | 114.00 | 151.00 | 127.82 | 113.59 | 148.30 | 129.95 | 102.20 | 92.00 |
| OPERA | 180.67 | 133.33 | 123.33 | 155.00 | 136.39 | 128.53 | 154.25 | 154.25 | 122.25 | 125.00 |
| VCD | 184.66 | 138.33 | 128.67 | 153.00 | 132.11 | 120.94 | 152.20 | 140.45 | 109.60 | 104.00 |
| \rowcolor[HTML]C0C0C0<br>Ours | 195.00 | 141.67 | 148.33 | 183.33 | 139.46 | 118.24 | 153.25 | 140.50 | 118.25 | 135.00 |

**Table 10: Results on MME cognition-related tasks using LLaVA-1.5**
| Method | Common Sense Reasoning | Numerical Calculation | Text Translation | Code Reasoning |
| --- | --- | --- | --- | --- |
| Vanilla | 106.43 | 72.50 | 95.50 | 78.50 |
| OPERA | 114.29 | 40.00 | 87.50 | 62.50 |
| VCD | 111.29 | 68.50 | 89.50 | 84.00 |
| \rowcolor[HTML]C0C0C0<br>Ours | 122.86 | 57.50 | 82.50 | 75.00 |

## Appendix B More Results

### B.1 POPE Evaluation on AOKVQA & GQA

To further validate the effectiveness of SHIELD, we conducted experiments on POPE using AOKVQA and GQA datasets under random, popular, and adversarial settings with the LLaVA-1.5 7B model. As shown in Tables [8](https://arxiv.org/html/2510.16596v1#A1.T8), SHIELD significantly reduces hallucinations compared to previous methods. On average, SHIELD achieves an absolute improvement of 2.5 in Accuracy and 1.3 in F1 score on AOKVQA, and 3.6 in Accuracy and 2.0 in F1 score on GQA. Notably, SHIELD is particularly effective under the challenging Adversarial setting, underscoring biases and vulnerability in visual encoders as key contributors to object hallucination in these scenarios.

### B.2 Detailed Results on MME

Table [9](https://arxiv.org/html/2510.16596v1#A1.T9) presents the results on perception tasks of the MME benchmark using the LLaVA-1.5 7B model. Compared to the vanilla LVLM, SHIELD achieves overall improvements, highlighting its ability to reduce hallucinations and enhance perception capabilities. This improvement likely stems from SHIELD’s effectiveness in mitigating biases and vulnerabilities, thereby recalibrating the LVLM’s visual feature extraction. When compared to previous methods, SHIELD achieves a higher total perception score (Table [6](https://arxiv.org/html/2510.16596v1#S4.T6)) but exhibits limited improvements on tasks requiring external knowledge beyond the input image (e.g., Celebrity, Scene, Landmark, Artwork). This limitation may result from contrastive decoding in SHIELD, which directs the LVLM to prioritize visual inputs over leveraging prior knowledge embedded in the LLM.

Furthermore, Table [10](https://arxiv.org/html/2510.16596v1#A1.T10) presents the results on cognition tasks of the MME benchmark using the LLaVA-1.5 7B model. The findings indicate that applying SHIELD improves the LVLM’s recognition on complex visual scenes, such as common sense reasoning, but performs poorly on simpler visual scenes, including numerical calculation, text translation, and code reasoning. This may be because simpler visual inputs are less likely to induce hallucinations, while contrastive decoding in SHIELD may limit the utilization of prior knowledge embedded in the LLM.

**Table 11: Inference Overhead on POPE COCO using LLaVA-1.5**
| Method | Accuracy $\uparrow$ | F1 $\uparrow$ | Time/Sample (ms) $\downarrow$ | Relative $\downarrow$ |
| --- | --- | --- | --- | --- |
| Vanilla | 81.3 | 79.6 | 128 | 1.0× |
| VCD | 84.6 | 84.4 | 190 | 1.5× |
| OPERA | 84.7 | 85.4 | 1586 | 12.4× |
| \rowcolor[HTML]C0C0C0<br>Ours | 87.0 | 87.4 | 980 | 7.6× |

### B.3 Inference Overhead

Table [11](https://arxiv.org/html/2510.16596v1#A2.T11) reports the per-sample inference time on the POPE COCO subset with the LLaVA-1.5 7B model. SHIELD introduces more overhead than lightweight methods such as VCD but remains faster than OPERA. Considering its strong reduction of hallucinations, this trade-off is reasonable. The overhead can be further alleviated by shortening naive captions or reducing adversarial steps.

## Appendix C Additional Hyper-Parameters Ablation

**Table 12: $\alpha$ Ablation**
| $\alpha$ | $C_{S}$$\downarrow$ | $C_{I}$$\downarrow$ |
| --- | --- | --- |
| 1.0 | 41.6 | 11.6 |
| 1.5 | 40.2 | 11.2 |
| 2.0 | 36.6 | 10.3 |
| 2.5 | 38.4 | 10.7 |

### C.1 α \alpha Ablation

Table [15](https://arxiv.org/html/2510.16596v1#A3.T15) summarizes the ablation results for the parameter $\alpha$, which regulates the influence of contrastive decoding when integrated with adversarial attack strategies. The study shows a significant reduction in hallucinations on the CHAIR benchmark as $\alpha$ increases from 1.0 to 2.0, demonstrating the effectiveness of addressing vulnerabilities.

### C.2 β \beta Ablation

Table [15](https://arxiv.org/html/2510.16596v1#A3.T15) reports the ablation results for the parameter $\beta$, which governs the adaptive plausibility constraint. A higher value of $\beta$ enforces stronger truncation, preserving only tokens with high generation probabilities. As $\beta$ increases, changes in hallucination reduction exhibit minor fluctuations, suggesting that while the adaptive plausibility constraint, as an integral part of contrastive decoding, prevents the generation of implausible content, it plays a limited role in alleviating object hallucinations.

### C.3 K K Ablation

Table [15](https://arxiv.org/html/2510.16596v1#A3.T15) reports the ablation results for the parameter $K$, which determines the number of noise samples employed to estimate and subsequently eliminate inherent bias from the visual tokens. Increasing $K$ improves the accuracy of inherent bias estimation, resulting in more effective hallucination mitigation. However, when $K$ becomes excessively large, the estimated bias converges toward zero, limiting its impact on mitigating hallucinations.

### C.4 l l Ablation

Table [15](https://arxiv.org/html/2510.16596v1#A3.T15) reports the ablation results for the learning rate $l$ used in the vulnerability addressing, which regulates the granularity of adversarial attack tensor calculations. When the learning rate is too large, the generated attack tensor fails to adapt effectively to image details, reducing its effectiveness. Consequently, the subsequent contrastive decoding process cannot adequately minimize hallucinations caused by vulnerabilities in the visual encoder.

Figure: (a) Case Study on LLaVA-Bench No.14
Refer to caption: x13.png

## Appendix D Case Study

Figures [9](https://arxiv.org/html/2510.16596v1#A3.F9) present two case studies demonstrating how vanilla LVLMs and previous methods, given identical prompts and images, can produce object hallucinations due to biases and vulnerabilities in the visual encoder. For instance, in Figure [9(a)](https://arxiv.org/html/2510.16596v1#A3.F9.sf1), shadows and blurriness along the tray’s edges expose visual encoder vulnerabilities, leading the vanilla LVLM to misidentify fried chicken as fried fish. Similarly, in Figure [9(b)](https://arxiv.org/html/2510.16596v1#A3.F9.sf2), statistical bias causes the vanilla LVLM to overemphasize tokens associated with frequent visual concepts in CLIP’s pre-training data, thereby distorting detail perception and incorrectly identifying the ironing board as being the same blue color as the shirt. In contrast, SHIELD effectively mitigates hallucinations while preserving the coherence and informativeness of the generated text.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: See weakness
soundness: 2
strengths: - The paper presents a reasonable motivation and a clearly described method.
- The proposed method is training-free, making it easy to integrate without retraining overhead.
- Experiments on LLaVA‑1.5, InstructBLIP, and Qwen‑VL showed improvement in reducing hallucinations.
summary: The paper claims that object hallucinations in LVLMs largely originate in the visual encoder and proposes a training‑free, three‑module wrapper SHIELD. The method includes token re-weighting to balance attention across visual tokens, token subtraction using noise-derived features, and contrastive decoding that contrasts outputs from natural and perturbed images. Authors presents correlational evidence for encoder statistical bias, inherent bias, and vulnerability and reports gains on CHAIR, POPE, and MME across LLaVA‑1.5, InstructBLIP, and Qwen‑VL.
weaknesses: - Encoder-level causes of visual hallucination have been extensively studied in earlier publications and the proposed methods are largely adapted from existing ideas. For example, strategies like token re-weighting and contrastive decoding are incremental given VCD has proposed such contrastive approaches.
- Related to the first point, there is a lack of discussion on directly related works [1-5]. There have been a lot of existing work on training-free contrastive methods for hallucination reduction.
- Experiments are done with outdated models. The selected three LVLMs were the very early versions, despite substantial advancements in multimodal capabilities since then. For example, newer models such as Qwen2-VL, Qwen2.5-VL, and Qwen3-VL have been released. As a result, the current experimental setup appears outdated and limited in scope.
- Improvements in Table 2 and 3 are small.
- Many citation formats in this paper are incorrect. For example, line 322-323.

[1] HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding
[2] Brave: Broadening the Visual Encoding of Vision-Language Models
[3] Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
[4] Reducing Hallucinations in Vision-Language Models via Latent Space Steering
[5] Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: See Above.
soundness: 2
strengths: 1. This paper propose SHIELD, a training-free method that mitigates object hallucinations by reducing statistical bias via token re-weighting, alleviating inherent bias using token subtraction, and addressing vulnerability through contrastive decoding.
2. Comprehensive experiments validate SHIELD’s effectiveness in mitigating object hallucinations
across diverse benchmarks and multiple LVLMs.
summary: This paper propose SHIELD, a training-free framework that mitigates hallucinations through three strategies: re-weighting visual tokens to reduce statistical bias, introducing noise-derived tokens to counter inherent bias, and applying adversarial attacks with contrastive decoding to address vulnerability.
weaknesses: 1. This method is limited to object-level hallucination. Comprehensive multimodal hallucination includes attribute level, relation level etc, as well. 
2. Although a training free method, the paper should present comparison with SOTA hallucination mitigation methods as well, either traning-free or training needed.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Can the authors provide a quantitative analysis of the inference-time latency (e.g., in seconds per image) and memory overhead of SHIELD compared to the vanilla model and the baselines (VCD, OPERA)? How much does each of the three modules (re-weighting, subtraction, contrastive decoding) contribute to this overhead?

2. The paper lacks a sensitivity analysis for the key hyperparameters $\alpha$, $\beta$, and $K$. How much does performance change when varying these? How were the final values chosen?

3. Regarding the "Token Re-weighting" module: What happens if the "naive caption" contains a plausible hallucination (e.g., seeing a "boat" in a wide shot of a lake)? Can you provide an analysis of this failure case?

4. Why was the specific PGD-style attack chosen for the "Address Vulnerability" module? How does the method perform if a different type of attack is used (e.g., FGSM) or if no attack is used at all (i.e., only using modules 1 and 2)?

5. For the CHAIR evaluation, the paper follows prior work using 500 randomly selected images. How is the balance between coverage (ensuring all present objects are in the ground-truth) and faithfulness (ensuring no non-existent objects are in the ground-truth) managed for this label set? This seems critical for a metric that punishes “hallucinated objects”.

6. For the GPT-4o assisted evaluation, why was a 0-10 human-like scoring system for “correctness” and “detailedness” chosen? Given recent work on automated evaluation (e.g., Valor-eval: Holistic coverage and faithfulness evaluation of large vision-language models. https://arxiv.org/pdf/2404.13874), what are the advantages of this 0-10 scale over more objective, feature-extraction-based or model-based evaluation methods?
soundness: 3
strengths: - The paper's focus on diagnosing hallucination sources within the visual encoder is a valuable contribution. The categorization into statistical bias, inherent bias, and vulnerability provides a structured way to think about the problem, moving beyond the more common focus on the LLM component.

- The design of SHIELD is logical, with each of its three components clearly mapped to one of the three identified problems. This modularity makes the approach easy to understand.

- As a post-hoc, training-free method, SHIELD offers a practical advantage, as it can be applied to any pre-trained LVLM without requiring expensive re-training or fine-tuning.

- The experiments are thorough, covering multiple model families and benchmarks. The results demonstrate that SHIELD consistently improves over the vanilla models and outperforms other training-free baselines like VCD and OPERA in reducing object hallucinations, especially on benchmarks like POPE.
summary: This paper identifies the visual encoder as a source of object hallucinations in Large Vision-Language Models (LVLMs), suggesting this stems from three issues: statistical bias, inherent bias, and vulnerability. The authors propose SHIELD, a training-free, post-hoc framework to mitigate these issues. SHIELD consists of three modules: (1) Token Re-weighting, which uses a naively generated caption to up-weight relevant visual tokens; (2) Token Subtraction, which estimates and removes a "bias" feature derived from noise inputs; and (3) Contrastive Decoding, which penalizes logits that are sensitive to a small adversarial perturbation on the input image. Experiments on several LVLMs (LLaVA, InstructBLIP, Qwen-VL) and hallucination benchmarks (CHAIR, POPE) show that SHIELD can reduce hallucination rates compared to other training-free methods.
weaknesses: 1. The primary weakness is the method's high computational cost at inference time. SHIELD requires multiple additional forward passes: one to generate the "naive caption," $K$ (e.g., 32) passes for the noise inputs, and another pass for the adversarially attacked image. This combined overhead likely makes the method prohibitively slow for any practical or real-time application, which severely limits its impact.

2. The method introduces several new and sensitive hyperparameters ($\alpha$ for subtraction, $\beta$ for contrastive decoding, $K$ for noise samples, and the attack parameters $l$ and step size). The paper provides very little ablation or sensitivity analysis for these. It's unclear how these were tuned or how performance would change on a different dataset or model, making the results potentially difficult to reproduce or generalize.

3. The idea of "inherent bias" being estimated by feeding pure noise to the encoder is a strong assumption. It is not convincingly demonstrated that the resulting feature vector is a meaningful representation of "bias" rather than just an artifact of the network's response to out-of-distribution data. Furthermore, using a large $K=32$ to get a stable estimate adds significantly to the computational burden.

4. The "Token Re-weighting" module fully depends on an initial caption from the vanilla LVLM, which is itself prone to hallucination. The paper's claim that this doesn't matter (because hallucinated words won't match visual tokens) seems overly optimistic and is not well-supported. A plausible hallucination could easily bias the re-weighting in the wrong direction, but this potential failure mode is not investigated.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: See weakness
soundness: 4
strengths: 1. Clear identification of encoder-level failure modes. The decomposition into statistical bias, inherent bias, and vulnerability is conceptually clean and well motivated by analyses (token peak-to-average ratios, dominant-object errors under noise, perturbation sensitivity). 

2. Strong token-level analysis. The paper’s quantitative and visualization evidence showing how individual visual tokens (peak norms / overemphasis) correlate with hallucination risk is compelling and valuable: it grounds the solution at the token representation level rather than treating hallucination as a black-box LLM issue. This token perspective is an important contribution for future encoder diagnostics. 

3. Modular, training-free design. SHIELD’s three modules are practical (can be combined or ablated) and the paper provides ablations showing complementary gains. Precomputable components (e.g., noise-derived estimates) make the approach more practical.
summary: This paper identifies three encoder-level sources of object hallucination—statistical bias, inherent bias, and vulnerability—and proposes SHIELD, a training-free pipeline that integrates token re-weighting, token subtraction (noise-derived removal of erroneous features), and adversarially informed contrastive decoding to mitigate these issues. Extensive experiments on CHAIR / POPE / MME evaluations demonstrate consistent gains across several LVLM families.
weaknesses: 1.	Generative / open-ended hallucination benchmarks.
The current evaluation focuses mainly on binary or existence-based benchmarks (POPE, MME). To better demonstrate robustness under real generative scenarios, the paper should include results on open-ended hallucination benchmarks such as AMBER [1], MMHal-Bench [2], which assess caption-level fidelity and factual consistency rather than yes/no predictions.

2.	Comparison with token pruning or sparse-token methods.
It would be useful to compare SHIELD with token pruning or sparse-token selection approaches that also modify the visual token set before decoding.

3.	Ablation on precomputation and inference overhead.
Since the paper mentions precomputing noise-derived erroneous representations, reporting runtime and memory trade-offs between precomputation and online computation would make the efficiency claims more convincing.

4.	Broader hallucination metrics and human evaluation.
For open-ended captioning, automatic metrics can be unreliable. Including limited human or GPT-4–based evaluation [2] focusing on whether hallucinated objects are introduced or removed would enhance credibility and interpretability.

[1] Amber: An llm-free multi-dimensional benchmark for mllms hallucination evaluation, 2023.

[2] Aligning large multimodal models with factually augmented rlhf, ACL 2024.


Decision: Accept (Poster)