SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defense
Authors: Yiyang Huang
Liang Shi
Yitian Zhang
Yi Xu
Yun Fu
Venue: ICLR 2026 Poster (2026)
Keywords: VLM
Hallucination
Training Free
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Large Vision-Language Models (LVLMs) excel in diverse cross-modal tasks. However, object hallucination, where models produce plausible but inaccurate object descriptions, remains a significant challenge. In contrast to previous work focusing on LLM components, this paper is the first to trace LVLM hallucinations to visual encoders and identifies three key issues: statistical bias, inherent bias, and vulnerability. To address these challenges, we propose SHIELD, a training-free framework that mitigates hallucinations through three strategies: re-weighting visual tokens to reduce statistical bias, introducing noise-derived tokens to counter inherent bias, and applying adversarial attacks with contrastive decoding to address vulnerability. Experiments demonstrate that SHIELD effectively mitigates object hallucinations across diverse benchmarks and LVLM families. Moreover, SHIELD achieves strong performance on the general LVLM benchmark, highlighting its broad applicability. Code is available at https://github.com/hukcc/SHIELD.

Published as a conference paper at ICLR 2026
SHIELD: SUPPRESSING HALLUCINATIONS IN LVLM ENCODERS VIA BIAS AND VULNERABILITY DEFENSE
Yiyang Huang' Liang Shi Yitian Zhang' Yi Xu' Yun Fu
'Department of Electrical and Computer Engineering, Northeastern University "Khoury College Computer Science, Northeastern University {huang. yiyan, shi. 1ia, zhang yitian, xu. sun.ottteeettthhheeeeeedddd
ABSTRACT
Large Vision-Language Models (LVLMss) excel in diverse cross-modal tasks. However, object hallucination, where models produce plausible but inaccurate object descriptions, remains significant challenge. In contrast to previous work focusing on LLM components, this paper is the first to trace LVLM hallucina- tions to visual encoders and identifies three key issues: statistical bias, inher- ent bias, and vulnerability. To address these challenges, we propose SHIELD, training-free framework that mitigates hallucinations through three strategies: re-weighting visual tokens to reduce statistical bias, introducing noise-derived to- kens counter inherent bias, and applying adversarial attacks with contrastive decoding to address vulnerability. Experiments demonstrate that SHIELD ef- fectively mitigates object hallucinations across diverse benchmarks and LVLM families. Moreover, SHIELD achieves strong performance on the general LVLM benchmark, highlighting its broad applicability. Code is available at https:
7/ //github.cc com wtt.://w..ccm//.cc////c//aa..... SHILL
INTRODUCTION
Large Vision-Language Models (LVLMs) (Bai et 2023b Liu al. 2023b Dai et al. 2023 com- bine visual and textual information and have advanced significantly in cross-modal tasks. Despite these advances, they suffer from object hallucination, generating object descriptions that may appear reasonable but misrepresent the image, either by misidentifying attributes (e.g., color, quantity, posi- tion) or by introducing non-existent objects. This issue poses reliability and safety risks domains such as healthcare (Hu et al. 2023 Wang et al. 2023b), autonomous systems (Chen et al. 2024a
Wu et 2023), and robotics MMi al. 2023 Liu et al. 2023a).
Various approaches have been proposed mitigate object hallucinations. Early efforts, such fine- grained modality alignment (Biten et al. 2022) and data augmentation to reduce co-occurrence bias (Kim et al. 2023 Rohrbach al. 2018). were designed for small-scale VLMs but fail to generalize to LVLMs Kaplan et al.) 2020: Wei et al. 2022). More recent research falls into two categories: training-require and training- free methods. Trainingrrquuie methods, including preference opti-
mization (Ouali et al. 2024 post-hoc revisers (Zhou et al. 2024), and RL.FF (Sun et al. 2024a)
Visual Encoder
Encoder
Encoder
Skier
What's in image?
Skier
What's the image?
What's the image?
Cabin
Tree
LLM
Tree Car
LLM
Cabin Tree Car
LLM
Car
Statistical Bias
Inherent Bias
Vulnerability
Figure 1: Key issues underlying object hallucinations in LVLMs. Statistical bias: the visual encoder overemphasizes frequent visual patterns, distorting fine-grained perception. Inherent bias: the en- coder produces erroneous representations of dominant objects in the pretraining data, regardless of input. Vulnerability: the encoder is sensitive minor perturbations, yielding unreliable features.

---

Published as a conference paper at ICLR 2026
improve factual consistency but demand substantial human and computational resources. In trast, training-free free methods, such as contrasting outputs from distorted inputs (Leng et al. 2024) or applying over-trust penalties during decoding (Huang et al. 2024). offer more efficient alterna- tive. However, these approaches primarily focus on the LLM component, leaving the role of visual encoders underexplored.
This paper is the first to trace LVLM hallucinations to visual encoders, filling this gap by identifying three key issues: statistical bias, inherent bias, and vulnerability, as illustrated in Figure [ Despite large-scale pretraining, these encoders remain affected by imbalanced distributions of visual con- cepts in the pretraining data, resulting in statistical and inherent biases. Statistical bias leads the visual encoder to overemphasize tokens related to frequent visual patterns, thereby distorting the perception of details. Inherent bias leads the visual encoder to produce representations of dominant objects in the pretraining data, regardless of the input, even when it is meaningless. Furthermore, vulnerability, arising from insufficient robustness to noise and perturbations during pretraining, leads the encoder to produce inaccurate visual representations even with small perturbations.
To address bias and vulnerability visual encoders that hinder feature extraction and amplify hal- lucinations in LVLMs, we propose SHIELD, training-free method combining token re-weighting, token subtraction, and contrastive decoding. Specifically, token re-weighting alleviates statistical bias by distributing attention across more tokens relevant to the ground-trute objects, thus avoiding fine-grained distortion from overemphasized tokens. In parallel, token subtraction mitigates inher- ent bias by estimating erroneous representations related to dominant objects in pretraining data via noise input and eliminating them through token-level subtraction. To address vulnerability, con- trastive decoding exposes hallucinations with a perturbed image and suppresses them by contrasting with outputs from a natural image.
demonstrate that SHIELD consistently improves performance on object hallucination benchmarks, including CHAIR (Rohrbach et al. 2018), POPE (Li et al. 2023). the hallucination subset of MME (Fu et al. 2023), y and GPT-4o-aided evaluations on LLa MaaVA--Bena Bench (Liu et al. 2023b). Moreover, these improvements are observed across diverse LVLM families, such as LLa VA (Liu jet al.) 2023b), InstructBLIIP (Dai et al.) (023), and Qwen-VL. VL. (Bai et al.) (0236). Beyond object hallucination mitigation, SHIELD also enhances general perception capabilities, evidenced by improvements on the full MME benchmark (Fu et al. 2023), highlighting its broader applicability.
Experiments
Our contributions are summarized as follows:
We analyze the role of visual encoders in contributing to object hallucinations in LVLMs, focusing statistical bias, inherent bias, and vulnerability.
We propose SHIELD, a training-free method that mitigates object hallucinations by reducing sta- tistical bias via token re- weighting, alleviating inherent bias using token subtraction, and address- ing vulnerability through contrastive decoding.
Comprehensive experiments validate SHIELD's effectiveness in mitigating object hallucinations across diverse benchmarks and LVLM families. Moreover, its strong performance on the general LVL.M benchmark highlights its broad applicability.
RELATED WORK
2.1 LARGE VISION LANGUAGE MODELS (LVLMs)
Recent advances in large-scale foundation models and multimodal learning have accelerated the
development of Large Vision-Language Models (LVLMs). By combining Large Language Models
(LLMMs) (Bai et al. 2023a Brown et al.] 2020 Chiang al. 2023; Chowdhery et al.) 2023 Gilardi
et al. 2023 Raffel et al.] 2020 Taori et 2003 Tay et al. 2023 Touvron et al. (023) with cross-
ei al. 2022). LVLMs integrate visual and textual information for more comprehensive understanding. Nevertheless, LVLMs across different architectures, including LLaVA-1.5 (Liu et al. 2024), InstructBLIP (Dai et al. 2023). ) and Qwen- VL (Bai et al. 2023b), still suffer from hallucmations, particularly in fine- aineggrined object recognition and challenging visual grounding. Such errors, often involving non-existent objects or misidentified attributes, remain a key challenge for reliability in real-world applications.
modal frameworks such as CLIP (Radford et al. (021) and BLIP
2

---

Published as a conference paper at ICLR 2026
0.3
677-00
78.5
Halluc ination Ratio
7
5254
35875
Hallucination Times
table
76.6
62568
Iose 75
dial
diiigg
F1 Score
702
60 6.0-225
6.577
7
pessin
H hyydem:
66.9
(dgg
64.2
b.
61.8
Peak-to-Average Ratio
Categones
Attack Steps
(a) Token Overemphasis
(h) Dominant Object Error
(c) Perturbation Vulnerability
Figure 2: Statistics show that hallucinations stem from bias and vulnerability. (a) The X-axis shows the peak-to-average 1.2 norm ratio of visual tokens, measuring the deviation of the highest-norm token from the average, and the Y-axis shows the proportion of hallucinating samples at cach level. Stronger overemphasis leads to higher hallucination rates. (b) The X-axis lists objects, and the Y- axis shows hallucination occurrences under meaningless inputs. Dominant objects more likely to be falsely perceived as present. (c) The X-axis denotes the number attack steps, and the Y-axis shows the Fl score. Even small perturbations increase hallucinations and degrade performance.
2.2 OBJECT HALLUCINATION IN LVLMS
Approaches to mitigating object hallucinations in LVLMs can be grouped into training-required and training-free methods. Braining-required methods reduce hallucinations by optimizing model pa- rameters or training auxiliary modules. Prominent methods include CLIP-DPO (Ouali et al. leveraging CLIP-based based similarity ranking for preference optimization; LURE (Zhou et al. 2024
2024).
using post-hoc revisers to align text with visual input; and LLaVA-RIHF (Sun et al., 2024a), incor- porating human feedback through reinforcement learning. Training-free methods improve decoding without modifying model. Representative approaches include Visual Contrastive Decoding (VCD) (eng et al., 22024) which contrasts outputs from natural and blur inputs to mitigate hallucinations, OPERA (Huang et al. 2024), which reduces overconfidence through penalty mechanisms and token- level adjustments, HALC (Chen al., 2024b), which employs adaptive focal-contred decoding to provide token-wise visual grounding during inference, MARINE (Zhao et al. (025a). which incor- porates mmage-grounded guidance from external vision models, and VTI (Liu et al. 2025), which steers latent representations to stabilize vision features at test time. While effective, these methods rarely address the bias and vulnerability of visual encoders, which this work aims address.
3 METHOD
3.1 HALLUCINATIONS STEM FROM VISUAL ENCODER
Accurate visual feature extraction is crucial for LVLMs to generate reliable outputs. However, bias and vulnerability in visual encoders distort features, intensifying object hallucinations. This section delves into these challenges.
3.1.1 STATISTICAL AND INHERENT BIAS IN VISUAL ENCODER
Most LVLMs adopt visual encoders derived from pretrained CLIP models. Although these encoders benefit from large-scale pretraining, they are influenced by the imbalanced distribution of visual concepts in the pretraining data. Specifically, certain visual concepts appear far more frequently than others, while rare or context-dependent elements are severely underrepresented (Parashar et al., 2024]. As result, the model develops a strong inductive bias toward frequent patterns and dominant objects, giving rise to both statistical and inherent bias.
Statistical bias denotes the visual encoder's 's over-reliance on frequent visual patterns the pretrain- ing data, causing overemphasis on the corresponding tokens with disproportionately high L.2 activa- tion values (Darcet et al. 2024). This overemphasis distorts the downstream LLM's perception of fine-grained details by directing attention to overweighted tokens (Kang et al. 2025) often resulting in hallucinations. Analysis of LLaVA-1.55 responses and visual tokens on the POPE COCO subset (Figure 2ap shows that the proportion of hallucinated samples grows with stronger token overem- phasis, measured by the peak-to- po-veree L2 ratio (the deviation the highest-norm token from the mean among visual tokens).
3

---

Published as a conference paper at ICLR 2026
Input Image
Noise Input
Attacked Image
x K
Visual Encoder
Reduce Statistical Bias
Mitigate Inherent Bias
Token Re-weighting
Token Subtraction
Large Language Model
Address Vulnerability
Whatt's in the image?
skier & Tree
Contrestive Decoding
Cabin Tree Car
SaberCabin Tree
Sider Cabin Tree SiberCabin
Car
Query
Answer
Figure 3: Illusratioon of the proposed SHIELD framework. Given an input image and a query text, the visual encoder produces tokens affected by statistical bias (overemphasized tokens @), inherent bias (erroneous representations O), and vulnerability (inaccurate features @). SHIELD addresses these issues through three modules: (i) Token Re-weighting, which redistributes attention more ground-truth-ojjeet relevant tokens to alleviate overemphasis (D); (ii) Token Subtraction, which estimates and removes erroneous representations (O) via noise-derived tokens; and (iii) Contrastive Decoding, which exposes inaccurate features (0) using attacked images and suppresses correspond- ing outputs by contrasting them with those from the natural image.
Inherent bias the visual encoder's overdependence on dominant objects in the pretraining data, leading it to generate erroneous representations of these objects regardless of the input, even when meaningless. As shown in Figure analysis of LLa LLa responses to the POPE COCO random split questions with meaningless images (random noise) as input shows frequent hallucinations of dominant objects such as cars, chairs, and tables, defined as cases where the model incorrectly predicts the presence of queried objects.
3.1.2 VULNERABILITY IN VISUAL ENCODER
The vulnerability visual encoders is another key factor contributing to object hallucinations. arises from their limited robustness to noise and subtle perturbations (Mao al. 2023), making them susceptible to constructing inaccurate visual representations under such disturbances. As shown in Figure 2c the performance of LLa VA-1.5 drops sharply on the POPE COCO subset even with a few attack steps, demonstrating that minor perturbations can exploit this weakness and yield unreliable features.
3.2 SHIELD: SUPPRESSING HALLUCINATIONS LVLM ENCODERS VIA BLAS AND VULNERABILITY DEFENSE
Building on these observations, we propose SHIELD, a training-fire free method to mitigate object hal- lucinations by addressing statistical bias, inherent bias, and vulnerability visual encoders, as il- lustrated in Figure 3] SHIELD integrates three strategies. Token re- re-weighting distributes attention across more tokens relevant to ground-truth objects, thereby reducing fine-grained distortion from overemphasized tokens and alleviating statistical bias. Token subtraction estimates erroneous repre- sentations of dominant objects in the pretraining data using noise input and removes them through token-leve subtraction, thus mitigating inherent bias. Finally, contrastive decoding applies pertur- bations to the input image to expose hallucinations and suppresses them by contrasting outputs with those from the natural image, countering vulnerability.

---

Published as a conference paper at ICLR 2026
Vanilla LVLM
Naive Caption
Token Re-weighting
Token Subtraction
000 Mean
CLIP Text
Max
Norm
xK
Visual
xk
Encoder
Encoder
O00
Visual
000
Token Re-weighting
Encoder
Figure 4: Mitigating Statistical Bias. Visual tokens are re-weighted via a similarity matrix between sual tokens and naive caption tokens, emphasizing more ground-truth-objeet relevant tokens and reduc- ing overemphasis.
Figure 5: Reducing Inherent Bias. K noise inputs are used to estimate erroneous rep- resentations of dominant objects in the pre- training data, which are then removed from visual tokens via feature subtraction.
3.2.1 FORMULATION OF LVLM INFERENCE
A LVLM's inference can be described in three stages. First, the visual encoder E(-) extracts N visual tokens from the raw image
x" 20. IN-1 E(v).
(1)
Next, given x" and the query text t. the LLM computes the output logits for token yi at step conditioned on the preceding sequence Vest
logit(y; x". t,,)e) LLM(X", t.e).
Finally, the logits are transformed into probability distribution over the vocabulary via softmax, from which the next token is selected according to the decoding strategy:
p(U: x" Mevi = softmax [logit(y: x", V<:)]-
Autoregressive repetition of the second and third stages produces the final textual output.
3.2.2 MITIGATING STATISTICAL BIAS
As discussed in Section 3.1.1 statistical bias causes the visual encoder overemphasize tokens associated with frequent visual patterns, distorting fine-grained perception. To address this, token re-weighting is applied based the similarity between visual tokens and naive caption tokens, encouraging the model to attend to more tokens relevant to ground-truth objects.
As shown in Figure token re-weighting begins with generating naive caption caave using the vanilla LVLM for the given image v:
naive
VanillaLVIM (v, "Please describe this image").
(4)
The CLIP text encoder Ex(-) (paired with E(-) during CLIP pretraining) then encodes the caption into P tokens:
c = {co, Ci. cP-1) = (c naive
(5)
Given the caption tokens e and the visual tokens x", similarity matrix M NxP is computed:
cc
M
||||2
From M, weights W are obtained by taking the maximum along the caption dimension and nor- malizing to [0, 1]:
W norm(max M.,)), W N.N.
Finally, the weights are applied via residual addition (o: element- wise multiplication) to emphasize visual tokens corresponding captioned objects, yiclding statistical-bias corrected ciarccercte tokens x*
(8)
Although the naive caption miive may introduce hallucinations, they do not affect re- weighting, as hallucinated objects fail to match any visual tokens with high similarity during similarity matrix computation. Thus, token weiightting remains focused on ground-truth objects.
5

---

Published as a conference paper at ICLR 2026
3.2.3 REDUCING INHERENT BIAS
inherent bias leads the visual encoder to produce erroneous representations of dominant objects in the pretraining data, regardless of the input. To counter this, token subtraction introduces noise inputs estimate such erroneous features and removes them from the visual tokens.
As in Section 3.1.1
As shown in Figure K random noise inputs n. (with the same size as the image) are passed through the visual encoder. The resulting tokens are averaged to estimate erroneous representations, which are then subtracted from the sitiisicaa-biasccoeeee tokens x" yielding bias-reduced tokens:
x""
Σ E(n,).
(9)
K
Since inherent bias depends only on visual encoder parameters, the estimation of erroncous repre- sentations from noise inputs be pre-calculated for each model to improve efficiency.
The bias-reduced tokens x"", together with the query text t, are subsequently fed into the LLM to produce bias-reduced logits:
logit (y: | x"" .t, t.,))) = LLM (x"", ,t,ss))
(10)
3.2.4 ADDRESS VUINERABILITY
As noted in Section 3.1.2 the visual encoder lacks robustness to subtle perturbations and noise, making it susceptible to inac- curate representations, especially
Visual Encoder
What's
the image?
Visual Encoder
Adv Loss
Large Language Model
when key pixels are disturbed. To
counter this vulnerability, a two-
CLIP Text
Encoder
Vanilla
LVLM
Cabin
Cubbi
Sk
step strategy is adopted: adversar- ial attack is first applied to reveal objects likely to hallucinated, followed by contrastive decoding to suppress the probability of gen- erating the corresponding outputs during inference.
-Naive Caption
Car D
Tree Car
Treo
Car 0
Adversarial Attack
Contrastive Decoding
Addressing Vulnerability. An attack tensor con- structed from the input image and its naive caption via adversar- ial learning is applied to reveal objects likely to be hallucinated, followed by contrastive decoding to suppress their generation.
Figure 6:
To expose vulnerability-indue induced hallucinations, attack tensor is constructed from the input image v and its naive caption maive (Equation 4 using the visual encoder E(-) and its paired text encoder E.(.) As illustrated in Figure 6 learnable perturbation ö is added to the input image and refined via backpropagation. The adversarial loss defined as the cosine similarity between the global representation of the perturbed image and that of the naive caption:
lnav cos (E(v + 8), (e(cmive)).
(11)
The final attack tensor 8* is obtained by minimizing lav via gradient descent with learning rate I:
arg min laiv-
(12)
During inference, the attack tensor is added to the input image to produce vulnerability inaccurate visual representations:
x = (T Ei. IN-1} = E(v + 8*).
(13)
These inaccurate representations are then used to produce adversarial logits:
logit (y: x" vei) L.M (x", t,,,))
(14)
To suppress hallucinations revealed by the attack tensor, contrastive decoding is applied. At each decoding step SHIELD contrasts the bias-reduced logits with the adversarial logits adjust the output probability distribution, where controls the impact of contrastive decoding:
Pshieta(ys) = softmax (1 + +a) logit(yi | xx" te.))) logit(y; 1x", (.ii)) (15)

---

Published as a conference paper at ICLR 2026
Table 1: CHAIR Hallucination Evaluation
Table GPT40-aid Hallucination Evaluation
LLaVA-1
InstructBLIP
Qwen-VL
LLaVA-1.5
InstructBLIP
Qwen-VL
Method
Cs4
Cr4
Cs4
Cr4
Cs4
Cr4.
Method
Ct
D1
C1
17t
Ct
Vanilla
48.8
14.2
54.6
24.8
49.2
13.1
Vanilla
4.9
5.0
4.2
4.2
6.2
4.6
VCD
46.8
13.2
44.0
13.6
46.4
11.9
VCD
5.5
5.5
5.1
6.5
5.7
OPERA
44.6
12.8
46.4
14.2
34.6
OPERA
5.6
5.3
6.5
5.6
Ours
36.6
10.3
40.4
10.9
28,9
9.2
Ours
6.2
6.1
5.6
533
6.9
5.8
Following (Leng et al.) 2024), adaptive plausibility constraint is introduced to avoid implausible
outputs. Only tokens with probabilities no smaller than a fraction 33 of the maximum are retained:
(ys) = { ys E v p(3)) 2 ,3 max p(w))))))))))) },
(16)
w
where is the vocabulary and Maaeen (ys) is the valid subset at step The threshold / determines truncation aggressiveness. For tokens not in Mooen (V:), probabilities are set to zero.
4 EXPERIMENTS
4.1
IMPLEMENTATION DETAILS
To evaluate the effectiveness of SHIELD in mitigating hallucinations, three representative LVLMs were selected: LLaVA-1.5 (Liu et al., 2024). InstructBLIP (Dai et al., (023). and Qwen-VI. (Bai et al. 2023b) SHIELD was compared against the corresponding vanilla LVLMs and two recent training-free free methods, VCD (Leng et al. 2024) and OPERA (Huang et al. 2024). Following their original setups, vanilla LVLMs and VCD adopted sampling-based decoding, while OPERA em- ployed beam search decoding with penalty term on logits to reduce overconfidence, For SHIELD, sampling-base based decoding was used, drawing from the modified post-softmax distribution. Unless otherwise specified, a = 2, 3 = 0.35, K = 32, and = 0.02 were applied across all LVLMs. where controls the strength of contrastive decoding, B sets the truncation threshold the plausibility constraint, K denotes the number of noise inputs for estimating inherent bias, and is the learning rate for optimizing the attack tensor. apper-parameters ablation are provided in the Appendix C All experiments were conducted on single RTX A6000 GPU.
4.2 QUANTITATIVE RESULTS
This section evaluates the effectiveness SHIELD in mitigating hallucinations for both detailed descriptions and simplified VQA answers.
CHAIR Evaluation. The CHAIR metric (Rohrbach et al. 2018 measures object hallucination in image captioning by calculating the proportion of objects mentioned in captions that not present in the ground-truth label set. It consists of two dimensions: a sentence-level score C's and an instance-level score C,, which are defined
sentences with hallucinated objects|
C Ihallucinated objects)
Cs
all sentences]
all objects mentioned]
2024 evaluation is performed on 500 randomly selected images from the COCO 2014 validation set (Lin et al. 22014). using the prompt "Please describe this image in detail." " To ensure fairness, generated captions are truncated maximum of 512 tokens.
Following (Huang et
As shown in Table SHIELD consistently outperforms all previous training-free methods on both Cs and C1, achieving up 18% improvement over the second-best method, OPERA, on LLaVA- 1.5. This performance gain stems from SHIELD's ability to counter biases and vulnerability in the visual encoder, thereby reducing hallucination risk detailed descriptions.
GPT-4o Assisted Evaluation. While CHAIR effectively evaluates object-level hallucinations, it fails to capture errors in attributes, locations, or relations. To complement this, we employ GPT-4o, a strong multi-modal assistant, to assess LVLM outputs on the LLaVA-Bench dataset. GPT-4o scores responses on correctness (C) and detailedness (D) from 0-10, with higher correctness indicating fewer hallucinations. The evaluation explicitly targets objects mentioned but absent from the image, as well as errors in attributes, colors, positions, or relationships. Further details are provided in the Appendix A.
7

---

Published as a conference paper at ICLR 2026
Table 3: POPE Hallucination Evaluation on COCO subset
IVLM
Method
Random
Popular
Advensarial
Average
Accuracy
Accuracy
Accuracy
Accurncy
Vanilla
83.2
81.3
51.8
80.0
78.9
77.5
81.3
79.6
LLaVA-1.5
VCD
87.7
87.1
85,3
85.0
80.8
81.3
84.6
84.4
OPERA
Ours
91.3
91.1
87.4
87.6
82.5
83.6
87.0
87.4
Vanilla
80.7
80.4
78.2
78.3
75.8
76.5
78.2
78.4
InstructR1.IP
VCD
84,5
83.6
81.4
81.0
79.5
79.5
81.8
81.3
OPERA
89.8
89.6
83.4
84.0
80.7
81.8
84.6
85.1
88.2
84.3
82.2
82.4
84.8
Vanilla
84.7
82.6
84.1
82.0
82.2
80.3
83.6
81.6
Qwen VL.
VCD
88.6
87.8
87.1
86.4
84.2
83.9
86.6
86.0
OPERA
86.1
84.2
85.7
83.8
83.9
82.1
85.2
83.3
Ours
89.2
88.6
87.6
87.1
84.3
84.2
87.0
86.6
Table 4: MME Hallucination Evaluation
Object-level
Attribute-lovel
LVLM
Method
Total Score
Existence Score
Count Score
Position Score
Color
Vanilla
175.6
124.6
114.0
151.0
565.3
VCD
184.6
138.3
128.6
153.0
604.6
LLVA--15
OPERA
180.6
133.3
123.3
155.0
592.3
Ours
195.0
141.6
148.3
183.3
668.3
75.3
380.3
InstructBLIP
VCD
168.3
92.3
64.0
123.0
447.6
OPERA
156.0
78.3
$5.0
95.0
384.3
170.0
75.0
88.3
128.3
461.6
Vanilla
155.0
127.6
131.6
173.0
587.3
Qwen-VL
131.0
128.0
596.6
OPERA
165.0
145.0
133.3
180.0
623.3
Ours
170.0
128.3
190.0
668.3
As shown in Table SHIELD achieves substantial gains in correctness, confirming its effectiveness in mitigating hallucinations. In contrast, improvements in detailedness are modest, since the method primarily addresses bias and vulnerability in the visual encoder rather than enhancing fine-grained descriptive coverage.
POPE Evaluation. Similar to CHAIR, POPE (Li et al. 2023) evaluates existence-level hallucina- tions in LVLMs. It adopts a VQA-style format (e.g., "Is there {object} the image?") to test whether models correctly associate images with specific objects. POPE includes three splits: "ran- dom" for random objects, "popular" for frequent objects, and "adversarial" for objects semantically related those in the image. The evaluation is conducted on three subsets: COCO, A-OKVQA, and GQA. Additional results are provided Appendix B.T
As shown in Table SHIELD outperforms previous training-free methods across most splits of the COCO subset. Although all methods show performance drops from Random to Adversarial, SHIELD more effectively mitigates hallucinations in the challenging Adversarial split, highlighting that biases and vulnerability in visual encoders are major contributors to hallucinations. For In- StructBLIP, however, the improvements are limited since its Q-Former module constrains the of modified visual features, thereby diminishing the benefits of SHIELD.
MMEE Hallucination Subset Evaluation. Although POPE adopts a VQA format effective for eval- uating object- bbject-exiiteenc-leveel hallucinations, it does not capture attribute-level aspects such as count, position, and color. To address this limitation, the MME hallucination subsets (Fu al., 2023) provide a more comprehensive benchmark. Following (Yin et al.) (202)), we evaluate object-level hallucinations using the existence and count subsets, and attribute-level hallucinations using position and color subsets. Performance is reported using the combined metrics of accuracy and accuracy+ as defined the official implementation.
As shown in Table SHIELD achieves consistent improvements across all models, leading higher total scores. By correcting statistical bias visual encoders, SHIELD reduces the impact of overem- phasized tokens fine-gra fineggriined perception, thereby significantly mitigating attribute-level halluci- nations.
Attribute and Relation Level Hallucination Evaluation. To further evaluate hallucinations the attribute and relation level, we conduct experiments on the AMBER benchmark (Wang et al. (023a), comparing against series of baselines based on LLaVA-1.5 7B. AMBER offers comprehensive
8

---

Published as a conference paper at ICLR 2026
Table 5: AMBER Hallucciaaion Evaluation
Generative Task
Discriminative Task
AMBER
Method
Score
CHAIR
Covert
Hallucination.
Cognition
Accuracyt
Precisiont
Recalft
F11
Vanilla
41.3
29.2
3.7
65.7
83.2
64.7
73.2
82.0
VCD
8.1
44.2
28.6
3.1
68.3
85.8
65.2
74.0
82.9
31.2
Ours
6.4
46.1
25.1
1.8
78.3
89.1
76.6
82.4
88.0
Table 7: Module Ablation on CHAIR
Table 6: MME Full Set Evaluation
Module
Cs 48.8
Crl
Method
Perceptiont
Cognition
Total Scoret
Vanilla L.aVA-115
14.2
Vanilla
YCD
1363.9
352.9
353.2
1632.1
1717.2
1717.1
adaptive plausibility constraint
vullerrbiliiy (Ours)
46.4
50.2
13.8
12.8
Ours
1473.0
337.8
1810.8
+ matigate statistical bias (Ours)
40.4
11.0
reduce
(Ours)
suite of generative and discriminative metrics, explicitly covering existence, attribute, and relation hallucinations, making it strong benchmark for assessing multimodal hallucination across diverse tasks. As shown in Table 5 our method achieves the highest Amber Score (88.0) and consistently outperforms all baselines across metries, demonstrating effective mitigation of hallucinations.
LVLM General Evaluation. To evaluate the overall performance of SHIEL.D-enhanced LVLMs, we conduct experiments on the full MME benchmark (Fu et al. (223) using the VA-1.55 7B model. The benchmark covers ten perception- related subtasks (including four hallucination- related) and four cognition-oriented ones. Perfor- mance is reported using both accuracy and accu- racy+ as defined in the official implementation.
(Parc)
Colebity
Poctton
(Phr
(Parc)
Dount:
Landmaant
Eeaoaacc
As shown in Table 6 and Figure 7 SHIELD not only improves hallucination--rlatee performance but also yields notable gains in perception tasks such OCR and Posters, thereby enhancing the overall capability of the model. Further details are provided in the Appendix B.2
Artward
OCI
Teroldion
Sarae ROSSOEESS
Calouaiiie
Vanilla
VCD
OPERA
Ours
Figure Evaluation on the full MME. Larger radar indicate better performance.
4.3 ABLATION STUDY
Module Ablation. To assess the effectiveness of SHIELD, we performed ablation studies on each module using the CHAIR benchmark with the LLaVA-115 7B model. The adaptive plausibility constraint, a key element of contrastive decoding, was also ablated to evaluate its role in mitigating hallucinations. As shown in Table 7 all modules contribute notably reducing hallucinations.
Although integral to contrastive decoding, the adaptive plausibility constraint alone was less ef- fective, indicating that filtering low-probability candidates cannot fully suppress hallucinations. In contrast, each SHIELD module individually reduced hallucination frequency, with their full com- bination achieving the greatest improvement. Notably, after addressing vulnerability, adding the statistical bias mitigation module yielded further 13% reduction, highlighting statistical bias as major source of hallucinations, particularly in longer descriptions.
Visualization. Figure g illustrates SHIELD's effectiveness in mitigating biases and reducing vul- nerability in visual encoders. Figure Sa demonstrates that re-weighting visual tokens alleviates sta- tistical bias by distributing attention across more object-relevant tokens and reducing overemphasis on specific ones, thereby improving fine-grained perception. Figure 8b illustrates SHIELD's effect on the POPE COCO subset with ambiguous inputs, showing that by leveraging noise-derived tokens to remove inaccurate representations, SHIELD significantly reduces hallucinations of dominant ob- jects in pretraining data. Finally. Figure 8c highlights SHIELD's robustness against perturbations, where SHIELD-enhanced enhanced LLa VA-1.5 exhibits substantially less performance degradation under in- creasing attack steps.

---

Published as a conference paper at ICLR 2026
Ours
car
Vanilla Ours
828
81.8
Vanilla
Hallucination Times
78.2
78.5
crsing botte
F1 Score
78.5
73.8
76.6
70.4 69.7
88.7
72.7
70.2
0
64
8
10
61.8
Activabion Indensity
Categories
Attack Steps
(a) Token Re-weighting
(b) Erroncous Representation Removal
(c) Address Vulnerability
Figure 8: (a) Token Re-weighting highlights more object-relevant tokens. (b) Token Subtraction
removes erroneous representations, reducing hallucinations associated with dominant objects in the pretraining data. Blue-only categories indicate zero hallucinations with our method. (c) Contrastive Decoding improves robustness to perturbations, mitigating vulnerabilityinndeces hallucinations.
Table 9: Module Efficiency on CHAIR
Table 8: Efficiency Comparison on CHAIR
Methed
Cs
(wkemmple),
Memj,
Module
T (ssaample) L
Memj
Vanilla
2.39
13,69UB
Vanilla
48.8
2.59
15.69GB
w/ Maligate Statistical
16.56GB
VCD
OPERA
46.8
44.6
24.01
4.89
16.52GB
34.88GB
w/ Reduce Inherent Bias
2.63
16.50GB
w/ Address Vulnerability
7.30
18. 17GB
Ours
36.6
7.34
18.17GB
Ours (All Modules)
7.34
18.17GB
Inference Overhead Analysis. To assess the computational cost of SHIELD, we evaluate its hal- lucination ratio, runtime, and peak GPU memory usage on the CHAIR benchmark with VA-1.- 7B. As shown in Table 8 SHIELD achieves the lowest hallucination ratio (Cg 36.6) while main- taining acceptable inference overhead. In particular, it runs substantially faster than OPERA and provides a more favorable trade-off between hallucination reduction and computational efficiency compared to VCD. To further analyze the sources of computational cost, we conduct module wise ablation study using the CHAIR benchmark and VA-1.5 7B, as shown in Table 9 The majority of overhead arises from the Address Vulnerability and Mitigate Statistical Bias modules, both of which require caption generation. In particular, Address Vulnerability further involves adversarial tensor computation, increasing runtime to 7.30 seconds per sample and memory usage to 18.17 GB. The Mitigate Statistical Bias module, which only relies on caption generation, increases runtime to 4.64 seconds and memory 16.56 GB. For reference, caption generation alone takes 2.05 seconds and 16.56 GB. In contrast, the Reduce Inherent Bias module introduces minimal overhead, adding only 0.04 seconds and 0.81 GB. Notably, the full SHIELD configuration incurs negligible additional cost beyond Address Vulnerability. These results indicate that most overhead is concentrated in caption generation and adversarial tensor computation, and that the overall trade-off can be flexibly adjusted by tuning caption length or the number of adversarial optimization steps.
5 CONCLUSION
This paper investigates object hallucinations in LVLMs, tracing their origin to visual encoders. De- spite large-scale pretraining, these encoders suffer from three issues: statistical bias, which overem- phasizes frequent patterns and distorts fine-grained perception; inherent bias, which induces er- roneous representations related to dominant objects in pretraining data; and vulnerability, which makes encoders sensitive to minor perturbations and results in inaccurate features. To address these challenges, we propose SHIELD, a training- framework that integrates token re-weighting, to- ken subtraction, and contrastive decoding. Token re-weighting alleviates statistical bias by dis- tributing attention to more ground-truth trateee-att-releaant tokens. Token subtraction mitigates inherent bias by estimating and removing erroneous dominant-object representations using noise-derived tokens. Contrastive decoding counters vulnerability by exposing hallucinations via perturbed image and suppressing them through contrast with natural inputs. Extensive experiments demonstrate that SHIELD not only achieves significant improvements on hallucination benchmarks but also enhances general perception tasks, highlighting its effectiveness and broad applicability.
Future work could explore the design of more robust visual encoders to further reduce bias and vulnerability, as well as the optimization of adversarial attack mechanisms lower computational costs for real-time applications.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
In this paper, we propose SHIELD, a method that mitigates bias and vulnerability to reduce hallu- cinations in LVLMs, thereby enhancing their safety and reliability for the community. Outputs of SHIELD may occasionally contain inappropriate content inherited from the base model, and such outputs do not reflect the authors' views. We strictly adhere to the ICLR ethical research standards and applicable laws. To the best of our knowledge, this work complies with the General Ethical Principles.
REPRODUCIBILITY STATEMENT
We follow the ICLR reproducibility standards and ensure the reproducibility of our work. All datasets used for inference and evaluation are publicly available, and detailed experimental settings, including hyper-paramen parameters and implementation steps, are documented in the paper and Appendix. Furthermore, we will release our code upon acceptance, enabling other researchers and practitioners to easily reproduce and extend our results.
REFERENCES
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. CoRR, abs/2309.16609, 2023a.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. CoRR, abs/2308. 12966, 2023b.
Ali Furkan Biten, Lluis Gómez, and Dimosthenis Karatzas. Let there be clock on the beach: Reducing object hallucination in image captioning. In WACV, pp. 2473-2482, 2022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with llms: Fusing object-level vector modality for explainable autonomous driving. In /CRA, pp. 14093-14100 14100, 2024a.
Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Hale: object hal- lucination reduction via adaptive focal-contrast decoding. In Proceedings of the 41st International Conference Machine Learning, pp. 7824-7846, 2024b.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuaa Imsys. org (accessed 14 April 2023), 2(3);6, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsyyascchnkko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James
11

---

Published as a conference paper at ICLR 2026
Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. Mach. Learn. Res., 24:240:1 22:::140111 2023.
Douglas
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality large language models. In ICLR. OpenRe- view.net, 2024.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung. and Steven C. H. Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. In NeurIPS, 2023.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In ICLR, 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang. Xiawu Zheng. Ke Li, Xing Sun, and Rongrong Ji. MME: A comprehensive evaluation benchmark for multimodal large language models. CoRR, abs/2306. 2023.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd-workers for text- annotation tasks. CoRR, abs/2303.15056 2023.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR (Poster), 2015.
Mingzhe Hu, Shaoyan Pan, Yuheng Li, and Xiaofeng Yang. Advancing medical imaging with language models: A journey from n-grams to chatgpt. CoRR, abs/2304.04920, 2023.
Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang. Dahua Lin, Weiming Zhang, and Nenghai Yu. OPERA: alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-alllcattoo In CVPR, pp. 13418-13427, 2024.
Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang, Zhicheng Chen, and Peilin Zhao. Self- introspective decoding: Alleviating hallucinations for large vision-language models. In ICLR. OpenReview.net, 2025.
Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. See what you are told: Visual attention sink in large multimodal models. In ICLR, 2025.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, fiss200010886// 2020.
Jae-Myung Kim, A. Sophia Koepke, Cordelia Schmid, and Zeynep Akata. Exposing and mitigating spurious correlations for cross-modal retrieval. In CVPR Workshops, pp. 2585-2595, 2023.
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vasion-language models through visual contrastive de- coding. In CVPR, pp. 13872-13882, 2024.
Junnan Li, Dongxu Li. Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language- -mage pre-training for unified vasion-language understanding and generation. In ICMI, pp. 12888- 12900, 2022.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vasion-language models. In EMNLP,, pp. 292-305, 2023.
12

---

Published as a conference paper at ICLR 2026
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects context. In ECCV, pp- 740-755, 2014.
Haokun Liu, Yaonan Zhu, Kenji Kato, Izumi Kondo, Tadayoshi Aoyama, and Yasuhisa Hasegawa. Llm-based human-robot collaboration framework for manipulation tasks. CoRR, 0tth/2338.14972/ 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023b.
Haotian Liu, Chunyuan Li. Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning. In CVPR, pp. 36286-26296 2024.
Sheng Liu, Haotian Ye, and James Zou. Reducing hallucinations in large vision-language models via latent space steering. In The Thirteenth International Conference on Learning Representations, 2025.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR (Poster). OpenReview.net, 2018.
Jinjie Mai, Jun Chen, Bing Li, Guocheng Qian, Mohamed Elhoseiny, and Bernard Ghanem. LLM as A robotic brain: Unifying egocentric memory and control. CoRR, abs/2300.00939, 2023.
Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. Understanding zero-shot adversarial robustness for large-scale models. In ICLR. OpenReview.net, 2023.
Yassine Ouali, Adrian Bulat, Brais Martínez, and Georgios Tzimiropoulos. CLIP-DPO: vision- language models source of preference for fixing hallucinations in Ivims. In ECCV, pp. 395- 413, 2024.
Shubham Parashar, Zhiqiu Lin, Tian Liu, Xiangjue Dong, Yanan Li, Deva Ramanan, James Caver- lee, and Shu Kong. The neglected tails in vision-language models. In CVPR, pp. 12988-19888 12997,, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, pp. 8748-8766 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 11:140:1-140:6: 2020.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In EMNLP, pp. 4035 4045, 2018.
Zhiqing Sun, Sheng Shen. Shengcao Cao, Haotian Liu. Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented RLHF. In ACL (Findings), pp. 13088-13110, 2024a.
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang. Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 13088-13110, 2024b.
Feilong Tang, Zile Huang, Chengzhi Liu, Qiang Sun, Harry Yang, and Ser-Nam Lim. Interven- ing anchor token: Decoding strategy alleviating hallucinations for milms. In JCLR. OpenRe- view.net, 2025.
13

---

Published as a conference paper at ICLR 2026
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An cearuction-following llama model. https: /tttps://iitub... com/ wtt.e/oo//ttt.eee tatstt 1asessllla-aae stanford alpaca 2023.
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. UL2: unifying language learning paradigms. ICLR, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, 0bs/2302.139771 2023.
Junyang Wang. Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, et al. Amber: An Ilm-free multi-dimensional benchmark for milms hallucination evaluation. arXiv:2311.0739) 2023a.
Sheng Wang. Zihao Zhao, Xi Ouyang, Qian Wang, and Dinggang Shen. Chatcad: Interactive computer- aided diagnosis on medical image using large language models. CoRR, abs72220.020,,7 2023b.
Xintong Wang, Jingheng Pan, Liang Ding, and Chris Biemann. Mitigating hallucinations in large vision-language models with instruction contrastive decoding. In ACL (Findings), pp. 15840- 15853. Association for Computational Linguistics, 2024.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang. Jeff Dean, and William Fedus. Emergent abilities of large language models. Trans. Mach. Learn. Res., 2022.
Zhenyu Wu, Ziwei Wang, Xiuwei Xu, Jiwen Lu, and Haibin Yan. Embodied task planning with large language models. CoRR, abs/2307.01 01 848, 2023.
Yun Xing, Yiheng Li, Ivan Laptev, and Shijian Lu. Mitigating object hallucination via concentric causal attention. CoRR, abs/2410.159266, 2024.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large lan- guage models. CoRR, abs/2310. 2023.
Zihao Yue, Liang Zhang, and Qin Jin. Less is more: Mitigating multimodal hallucination from an EOS decision perspective. (C), pp. 11766-11781. Association for Computational Linguis- tics, 2024.
Linxi Zhao, Yihe Deng, Weitong Zhang, and Quanquan Gu. Mitigating object hallucination in large vision-language models via image-grounded guidance. In Forty-second International Conference on Machine Learning. 2025a.
Qiyan Zhao, Xiaofeng Zhang, Yiheng Li, Yun Xing. Xiaosong Yuan, Feilong Tang, Sinan Fan, Xuhang Chen, Xuyao Zhang, and Dahan Wang. Mca-llava: Manhattan causal attention for reduc- ing ballucination large vision-language models. CoRR, abs/2507.00 091500779914,,. 2025b.
Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vasion-language models. In ICLR, 2024.
14

---

Published as a conference paper at ICLR 2026
Table 10: The Prompt used for GPT4o-aid evaluation
GPT-4o Prompt
You required to scere the performance of four Al assistants in describing given image. You should pay extra attention to the hullacination, which refers to the part descriptions that ure inconsistent with image content, such as claiming the existencee of something not present in image deserihing incorreatly
of the the responses of the assistants on scale of to 10, where higher score indicates better performance, according to the following criteria: Correctness: whether the response accurate respect to the image content. Responses with ballucinations should given
positions, calors of objects in the image. Please
higher scores.
Detailedness: whether response
in necessary
Note hallucinated descriptions should not
necessary
detnils.
Please outpal the scores for each criterion, containing only four values indicating the scores Assisiant 1, and 4, respectively. The four
separated space. Following the please provide explanation of your evaliaatio, avoiding any potential bins and ensuring that the order in which the responses were presented does not affect your judgment.
[Assistaut {}
[End of Assistant 1]
[Assistant 2]
()
[End of Assistant
[Assistant 3]
{}
[End of Assistant 3]
[Assistant 4]
()
[End of Assistant 4]
Output format:
Correctness: Scorres of the four answers
Remsn::
Detailedness: Scoores of the four answers?>
Reason:
A DETAILED EXPERIMENTAL SETUP
A.1 GPT-40 ASSISTED EVALUATION
Following YYin et al. 2023), use GPT-4o to evaluate vanilla LVLM, VCD, OPERA, and our pro- posed SHIELD. Given a LVLM and an image, descriptions are generated using the prompt, "Please describe this image in detail". Using the evaluation prompt shown in Table TO GPT-4o rates the four descriptions scale of 0 to 10 across two aspects: Correctness, which measures the consistency between the description and the image, assigning higher scores to highly consistent descriptions and lower scores those with hallucinations, and Detailedness, which evaluates how comprehen- sively the description captures image details. The prompt instructs GPT-4o to disregard biases from sequential order and focus inconsistencies, such objects mentioned but absent in the image, including incorrect colors, positions, or relationships. Leveraging its analytical capabilities, GPT-4o performs thorough and detailed evaluation.
Table 11: Comparison on POPE A-OKVQA and GQA Subset using -La
Datasct
Method
Random
Popalar
Adversarial
Average
Accuracy
Accuracy
Accuracy1
F-7
Accuracy
Vanilla
83.4
82.5
79.9
79.5
74.0
75.1
79.1
79.0
AOKVQA
VCD
86.1
86.3
81.8
82.8
74.9
77.7
80.9
82.2
OPERA
88.1
88.5
83.3
84.5
73.8
77.7
81.7
83.5
Ours
90.3
89.9
85.3
85.5
77.2
79.1
84.2
84.8
82.9
76.0
78.9
79.0
VCD
86.6
86.9
80.7
82.2
76.0
78.7
81.1
82.6
GQA
OPERA
88.6
89.1
79.8
82.1
75.0
78.8
81.1
83.3
Ours
90.5
90.3
84.2
84.8
79.5
81.0
84.7
85.3
B MORE RESULTS
B. POPE EVALUATION ON AOKVQA & GQA
To further validate the effectiveness of SHIELD, we conducted experiments on POPE using AOKVQA and GQA datasets under random, popular, and adversarial settings with the LLaVA-1.5
15

---

Published as a conference paper at ICLR 2026
7B model. As shown in Tables 11 SHIELD significantly reduces hallucinations compared to previ- ous methods. On average, SHIELD achieves am absolute improvement of 2.5 in Accuracy and 1.3 in Fl score AOKVQA, and 3.6 in Accuracy and 2.0 in F1 score GQA. Notably, SHIELD is par- ticularly effective under the challenging Adversarial setting, underscoring biases and vulnerability in visual encoders key contributors to object hallucination in these scenarios.
Table 12: Results on MME perception-related tasks using LL
Methoul
Existence
Count
Position
Calor
Posters
Celebrity
Scene
Landmark
Artwork
OCR
Vanilla
175.67
124.67
114,00
151.00
127.82
113.59
148.30
129.95
102.20
92.00
OPERA
155.00
154.25
122.25
VCD
184.66
138.33
128.67
153.00
132.11
120.94
152.20
140.45
109.60
104.00
183.33
153.25
Table 13: Results on MME cognition-related tasks using LLaVA-1.5
Method
Common Reasoning
Numerical Calculation
Reasoning
Vanilla 106.43
72.50
95.50
78.30
OPERA
114.29
40.00
87.50
62.50
VCD
111.29
68.50
89.50
84,00
Ours
122.86
57.50
82.50
75.00
B.2 DETAILED RESULTS ON MME
Table 12 presents the results on perception-related tasks of the MME benchmark using the LLaVA- 1.5 7B model. Compared the vanilla LVLM, SHIELD achieves overall improvements, highlight- ing its ability to reduce hallucinations and enhance perception capabilities. This improvement likely stems from SHIELD's effectiveness in mitigating biases and vulnerabilities. thereby recalibrating the LVLM's visual feature extraction, When compared to previous methods, SHIELD achieves higher total perception score (Table ( but exhibits limited improvements on tasks requiring exter- nal knowledge beyond the input image (e.g., Celebrity, Scene, Landmark, Artwork). This limitation may result from contrastive decoding in SHIELD, which directs the LVLM to prioritize visual inputs over leveraging prior knowledge embedded in the LLM.
Furthermore, Table 13 showcases the results on cognition-related tasks within the MME benchmark using the LLaVA- 1.5 7B model. The results show that applying SHIELD improves the LVLM's recognition on complex visual scenes (e.g., Common Sense Reasoning) but performs poorly on simpler visual scenes (e.g., Numerical Calculation, Text Translation, and Code Reasoning). This may be because simpler visual inputs are less likely to induce hallucinations, while contrastive decoding SHIELD may limit the utilization of prior knowledge embedded in the LLM.
Table 14: GPT-4 Assisted Evaluation on MMHal-Bench
Method
Overall
Hallucination
Attr.
Adv. 1
Comp.
Cut. 1
Rel.
Eav. 1
Hol. 1
Other f
Vanilla
2.21
0.70
2.75
2.00
2.50
2.50
1.67
2.17
1.83
2.25
Ours
2.65
0.61
2.42
1.83
2.58
2.67
2.83
3.92
2.08
2,83
B.3 GPT-4 ASSISTED EVALUATION ON MMHAL-BENCH
Table 14 presents results on MMHal-Bench (Sun al. 2024b), a benchmark designed evaluate hallucinations in open-ended multimodal tasks. It spans eight hallucination-related categories, in- cluding attribute, adversarial object, comparison, counting, relation, environment, holistic descrip- tion, and other visually grounded errors. To ensure up-to-date evaluation, we re-assess both the vanilla LLaVA-1.5 7B and our method using the latest version of GPT-4, as the original benchmark relied the deprecated gpt 4-03--04444 4. Our method significantly reduces hallucination and im- proves performance across most categories, with the most notable gains observed relation and environment understanding.
16

---

Published as a conference paper at ICLR 2026
Table 15: Extended Evaluation on Recent VLMs Variants (POPE COCO Adversarial Split)
Qwen2-VL (7B)
Qwe.2.5-VL.CT (7B)
Qwen3-VL (8B)
Method
Accurncy (?)
Score (1)
Accuracy (1)
Score (1)
Accuracy (1)
Fl (t)
Vanilla
85.3
84.5
85.6
85,1
86.2
85,8
Ours
86.1
85.4
86.8
86.0
87.4
86.6
B.4 EXTENDED EVALUATION ON RECENT VLMS
The VLMs used in our main experiments (LLaVA-1.5, InstructBLIP, and Qwen-VLL VLV follow standard practice in recent multimodal hallucination studies, ensuring fair and reproducible comparison. To further evaluate the generalization of SHIELD, we extend our experiments to more recent Qwen- VI variants: Qwen2-VL VL. (7B)), Qwen2.5-VI. (7B), and the newly released Qwen3-VI. (8B). Notably, Qwen3-VI. 8B was released November 2025 and represents the latest generation of the Qwen-VI. family. We report results the POPE COCO adversarial split.
As shown in Table [5| SHIELD consistently improves both accuracy and Fl score across all models. These results demonstrate the effectiveness of our approach on more capable and recently released VLMs.
Table 16: Comparison with Training-Fres and Training- Based Methods on CHAIR (512 tokens)
Method
Training
Cs(J)
Cr(()
Vanilla
DoLa (Chuang al
2024
47.7
13.8
ICD Wang cl
2054
47.4
13.9
VCD
20A4
46.8
13.2
TAME
Jang el al 2125
45.2
14.0
OPERA
2024
44.8
12.8
SID [Tuo el al 1173
X
44.2
12.2
M.VAA-RLFF
(Sin
2024b
CCA-LLaVA XXing et al
NUZA
43.0
11.5
Less is More
Yue clald
2014
40.2
12.3
-LCA-LLVV (Zhoo
20255
38.0
10.9
SHIRL (Ours)
36.6
10.3
B.5 EXTENDED COMPARISON WITH EXISTING HALLUCINATION MITIGATION METHODS
To provide more comprehensive evaluation, we expand our comparison include wider set of hallucination mitigation methods on the CHAIR benchmark. These include both training- methods (DoLa, ICD, VCD, TAME, OPERA, SID, and Ours) and training-required approaches (LLaVA-RIHF, CCA-LLaVA, Less is More, and MCA-LLaVA). All methods are implemented with LLa 7B to ensure consistency in backbone and evaluation setup.
As shown in Table 16] our method outperforms all training-free baselines and remains competitive with training-based methods. Notably, SHIELD achieves the lowest hallucination rates on both sentence and instance levels while requiring additional VLM fine-tuning.
C ADDITIONAL HYPER-PARAMETER ABLATION
C.1 EFFECT OF IN CONTRASTIVE DECODING
Table 17 presents the results of an ablation study on a, which controls the impact of contrastive decoding combined with adversarial attacks. The study shows a significant reduction in hallucina- tions on the CHAIR benchmark as a increases from 1.0 to 2.0, demonstrating the effectiveness of addressing vulnerabilities.
C.2 EFFECT OF 3 IN ADAPTIVE PLAUSIBLE CONSTRAINT
Table 18 presents the results of an ablation study on 3, which controls the adaptive plausibility con- straint. A larger 33 indicates aggressive truncation, retaining only hiobbbrobability tokens. As
17

---

Published as a conference paper at ICLR 2026
Table 17: a Ablation
Table 18: / Ablation
Table 19: Ablation
Table 20: Ablation
Cs
C
8
Cs4
Cr4
K
Cs4
Ca4
Cr
10
41.6
11.6
0.20
36.8
11.2
39.6
11.3
0.01
37.8
11.2
1.5
40.2
11.2
0.25
38.0
11.0
16
38.2
10.8
36.6
10.3
2.0
36.6
10.3
0.30
36.2
10.3
32
36.6
10.3
0.03
38.0
11.2
38.4
10.7
0.35
36.6
10.3
64
38.4
11.5
43.2
11.6
increases, changes in hallucination reduction exhibit minor fluctuations, suggesting that while adaptive plausibility constraint, an integral part of contrastive decoding, prevents the generation of implausible content, plays a limited role in alleviating object hallucinations.
C.3 EFFECT OF K IN REDUCING INHERENT BIAS
Table 19 presents the results of an ablation study on K, which specifies the number of noise inputs used estimate inherent bias for subsequent removal from visual tokens. Increasing K improves the accuracy of inherent bias estimation, resulting in more effective hallucination mitigation. However, when becomes excessively large, the estimated bias converges toward zero, limiting its impact on mitigating hallucinations.
C.4 EFFECT OF I IN ADDRESSING VULNERABILITY
Table 20 presents the results of an ablation study the learning rate within Vulnerability Address- ing, which controls the granularity of adversarial attack tensor computation. When the learning rate large, the generated attack tensor fails to adapt effectively image details, reducing its effec- tiveness. Consequently, the subsequent contrastive decoding process cannot adequately minimize hallucinations caused by vulnerabilities in the visual encoder.
C.5 EFFECT OF ÁTTACK STEP IN ADDRESSING VULNERABILITY
Table 21: Attack Steps Ablation
Table presents an ablation study on the number of adversarial optimization steps in addressing vulnerability module. This pa- rameter controls how thoroughly the attack tensor explores en- coder weaknesses. The results show that performance saturates at 30 steps, with no further gains at 45 steps.
Steps
Cs 40.8
CjL
15
11.6
36.6
10.3
10.4
D ADDITIONAL ANALYSIS
Table 22: Class- wise hallucination rate (%) under different noise settings on POPE COCO
Input
person skis soowboard bird backpack skateboard bowl scissors knife keyboard carrot refrigerator remete sandwich sink
Noise 1mg
100
80
30
30
20
20
20
10
10
10
10
10
10
Noise Tok
10
20
D.1 ADDITIONAL INHERENT BIAS ANALYSIS
To further examine inherent bias the visual encoder, we conduct a controlled experiment using the POPE COCO subset. For each object category, we design 10 identical POPE-style questions (e.g., "Is there (ohject) the image?") and evaluate hallucination rates under two distinct noise settings. In the first setting, the visual encoder receives noise as image input (Noise Image), while in the second, the original image retained but its visual tokens are replaced with noise embeddings (Noise Visual Tokens).
We measure the proportion "Yes" responses, which indicate hallucination, for each object. As shown in Table 22 hallucinations under the Noise Image setting heavily skewed, with certain categories (c.g.. person) being disproportionately predicted. contrast, hallucination rates under Noise Visual Tokens remain uniformly across categories. These findings confirm that the visual
18

---

Published as a conference paper at ICLR 2026
encoder exhibits class class-specific inherent biases, which can be effectively revealed through noise- based evaluation.
To mitigate the impact of randomness from individual noise inputs, SHIELD averages responses over K noise inputs. This aggregation helps reduce variance and provides a more stable estimate of the encoder's s bias tendencies.
Raw Caption:
"Sggat accoss PEDu
In the image, there is microwave oven sitting top of counter. The microwave is placed next toaster oven, and both appliances appear
Par
clean well-maintained. Above the microwave and toaster oven, there is sign with suggestions to keep everyone
barddr
Hallucinated Caption:
BKC
The image features a kitchen with a microwave sitting on top of counter. The microwave is placed on a shelf, and there is sign above it, possibly providing instructions suggestions. The kitchen located on the left side of the scene.
has refrigerator, which
Figure 9: Example naive captions with and without plausible hallucination. The hallucinated version includes a refrigerator not present in the image.
Table 23: Impact of Plausible Hallucinations in Naive Caption
Model
Accuracy
FI-Score (f)
Vanilla
Ours w/ Row Caption
80.
79.14
Ours w/ Hallucinated Caption
79.83
78.81
D.2 PLAUSIBLE HALLUCINATION IN NAIVE CAPTION ANALYSIS
To assess the effect of plausible hallucinations in naive captions, conduct controlled experiment on the POPE COCO adversarial subset. We simulate plausible hallucinations by introducing mild noise into the visual input during naive caption generation. This can produce hallucinated objects that are contextually reasonable but not actually present in the image.
shows one representative example where the hallucinated caption mentions "refrigerator", which is plausible in the context but does not appear in the image. Despite this, grounded elements such as "microwave" and "sign" remain in the caption, which continue to support effective token re- re-weighting. The impact of such plausible hallucinations is limited because they typically have lower visual grounding and do dominate the token distribution.
Figure Shouss
To quantify this, we compare SHIELD's performance when using raw versus hallucinated captions. As shown Table 23 the drop in performance is minor, indicating robustness of SHIELD plau- sible caption noise.
Table 24: Effect of Attack Strategy on CHAIR
Configuration
Cs (
(U)
Vanilla
48.8
Ours w/ FGSM-based Vulocrability Module
39.2
11.5
Ours w/ PGD--bsed Vulnerability Module
37.2
11.3
Ours (Leamable Atlack)
36.6
10.3
D.3 ATTACK STRATEGY ANALYSIS
To evaluate the design of our vulnerability addressing module, compare our learnable perturba- tion with standard adversarial attacks that use fixed step sizes, such as FGSM (Goodfellow et al. 2015) and PGD (Madry et al. 2018). Instead of relying fixed-magnitude updates, method
19

---

Published as a conference paper at ICLR 2026
learns fine-grained perturbation through adversarial optimization between the visual encoder and its paired CLIP text encoder (see Section 3.2.4). Conditioned on both the image and naive caption, the perturbation is specifically optimized to expose vulnerabilities in the encoder, offering more flexible and precise control than fixed-step-size attacks.
To validate its effectiveness, ing variants where the vulnerability addressing module implemented with FGSM or PGD instead. As shown in Table 24 while all configurations incorporating the vulnerability module help reduce hallucinations, our learnable attack achieves the lowest scores on both Cs and Cr.
evaluate multiple configurations the CHAIR benchmark, includ-
Sometimes I jjust look at pictures the earth from space and marvel at how beautiful it all is.
Prompt: Please describe this image in detail.
Vanilla: The image features close-up view tray of the
pieces of battered and fried fish arranged to resemble
world. The fish tray contains many countries, showcasing the Iragrant appelizing meal.
chicken displayed across the tray. each figuratively representing country within the map. The arrangement provides an artistic representation of the world. crafted through the presenlation delicious, frosted friendly fish
The Image features a tray filled with various pieces of fried chicken, arranged to resemble a map of world, complete with continents, Bountris, and
OPERA:
The chicken cullets
logether
depict different regions, showcasing creativity and attention to detail. The Iray placed on dining table, ready to be enjoyed
The image showcases a delicious fried chicken dinner served on a metal tray. The chicken cut into pieces and arranged all over the tray. some pieces placed tray. The tray is filled with various pieces of fried chicken, making delightful sight for those who love chicken. The dish appears to be ready be eaten, inviting people indulge
VCD:
edges others in the middle the
appetizing meal.
SHIELD (Ours): The image features a tray with various pleces of fried chicken arranged to resemble map of the world The chicken pieces are spread across the tray, with some placed closer to top. middle, and bottom the tray, while others positioned lowards the edges. The chicken golden presentation of the fried chicken pieces adds a creative touch to the dish, making a unique and fun way to enjoy meal.
and crispy, making appetizing sight. The map-liie like
(a) Case Study on VAVA-Bench 1o.14
Prompt: Pieaze describo this Image In desall
Vanillac imssge, treee aucurna ad
abread. mree haldhing Biiian
bo the back of mooding senoll drowinng ngMht benlnd a ES figi Another car can soon kocaled swarrds the left dee of the soure Buskok lsmne le wadtie in Eucre cnt Waanss
in macale
willih
my
appenos
OPERA:
the image man standing on the baok yellno BEIENS Wiiking riding the back
He
appoars ng
the back the moment, 36 the man tariss advantage of the sanls space to get work dano the
tmaepe schomea mann aberadig Mileer + vulloe incring dottass aos inuning scilullly bolanding
The Bapooons
Boorii plaaoed
Tind offers awayy Aud tomallyy now inatie
backgguuns Ind caling that sonne loey tecing place on buty city streel
(Cuns): bway BURe The focuoed posallly the coethee quiccyy
yellow iininng
wnsinommet copp the hundin bueele
of city where people are always the move and muitlaskingg
(b) Case Study LLaVA-Bence No.10
Figure 10: Examples from LLaVA-Benchh demonstrate the effectiveness of our method in correcting hallucinations. Hallucinated content is highlighted in red, and key information is highlighted in green.
E
CASE STUDY
Figures 10 present two case studies demonstrating how vanilla LVLMs and previous methods, given identical prompts and images, can produce object hallucinations due to biases and vulnerabilities in the visual encoder. For instance, in Figure T0al shadows and blurriness along the tray's edges expose
20

---

Published as a conference paper at ICLR 2026
visual encoder vulnerabilities, leading the vanilla LVLM misidentify fried chicken fried fish. Similarly, in Figure [0b] statistical bias causes the vanilla LVLM to overemphasize tokens associated with frequent visual concepts in CLIP's pre-training data, thereby distorting detail perception and incorrectly identifying the ironing board as being the same blue color as the shirt. In contrast, SHIELD effectively mitigates halluciiations while preserving the coherence and informativeness of the generated text.
F THE USE OF LARGE LANGUAGE MODELS (LLMs)
We used GPT for two purposes: (i) polishing grammar and improving readability, and (ii) assisting in the evaluation of LVLM outputs. All research ideas and analyses were conducted by the authors, who take full responsibility for the content.
21


Reviews:
[{"review_id": "WnQwf5NoXg", "reviewer": "Reviewer_56NU", "summary": "The paper claims that object hallucinations in LVLMs largely originate in the visual encoder and proposes a training‑free, three‑module wrapper SHIELD. The method includes token re-weighting to balance attention across visual tokens, token subtraction using noise-derived features, and contrastive decoding that contrasts outputs from natural and perturbed images. Authors presents correlational evidence for encoder statistical bias, inherent bias, and vulnerability and reports gains on CHAIR, POPE, and MME across LLaVA‑1.5, InstructBLIP, and Qwen‑VL.", "questions": "See weakness", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper presents a reasonable motivation and a clearly described method.\n- The proposed method is training-free, making it easy to integrate without retraining overhead.\n- Experiments on LLaVA‑1.5, InstructBLIP, and Qwen‑VL showed improvement in reducing hallucinations.", "weaknesses": "- Encoder-level causes of visual hallucination have been extensively studied in earlier publications and the proposed methods are largely adapted from existing ideas. For example, strategies like token re-weighting and contrastive decoding are incremental given VCD has proposed such contrastive approaches.\n- Related to the first point, there is a lack of discussion on directly related works [1-5]. There have been a lot of existing work on training-free contrastive methods for hallucination reduction.\n- Experiments are done with outdated models. The selected three LVLMs were the very early versions, despite substantial advancements in multimodal capabilities since then. For example, newer models such as Qwen2-VL, Qwen2.5-VL, and Qwen3-VL have been released. As a result, the current experimental setup appears outdated and limited in scope.\n- Improvements in Table 2 and 3 are small.\n- Many citation formats in this paper are incorrect. For example, line 322-323.\n\n[1] HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding\n[2] Brave: Broadening the Visual Encoding of Vision-Language Models\n[3] Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance\n[4] Reducing Hallucinations in Vision-Language Models via Latent Space Steering\n[5] Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TKtopPSnT9", "reviewer": "Reviewer_D8oq", "summary": "This paper propose SHIELD, a training-free framework that mitigates hallucinations through three strategies: re-weighting visual tokens to reduce statistical bias, introducing noise-derived tokens to counter inherent bias, and applying adversarial attacks with contrastive decoding to address vulnerability.", "questions": "See Above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper propose SHIELD, a training-free method that mitigates object hallucinations by reducing statistical bias via token re-weighting, alleviating inherent bias using token subtraction, and addressing vulnerability through contrastive decoding.\n2. Comprehensive experiments validate SHIELD’s effectiveness in mitigating object hallucinations\nacross diverse benchmarks and multiple LVLMs.", "weaknesses": "1. This method is limited to object-level hallucination. Comprehensive multimodal hallucination includes attribute level, relation level etc, as well. \n2. Although a training free method, the paper should present comparison with SOTA hallucination mitigation methods as well, either traning-free or training needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lqyI1G5M8l", "reviewer": "Reviewer_85aF", "summary": "This paper identifies the visual encoder as a source of object hallucinations in Large Vision-Language Models (LVLMs), suggesting this stems from three issues: statistical bias, inherent bias, and vulnerability. The authors propose SHIELD, a training-free, post-hoc framework to mitigate these issues. SHIELD consists of three modules: (1) Token Re-weighting, which uses a naively generated caption to up-weight relevant visual tokens; (2) Token Subtraction, which estimates and removes a \"bias\" feature derived from noise inputs; and (3) Contrastive Decoding, which penalizes logits that are sensitive to a small adversarial perturbation on the input image. Experiments on several LVLMs (LLaVA, InstructBLIP, Qwen-VL) and hallucination benchmarks (CHAIR, POPE) show that SHIELD can reduce hallucination rates compared to other training-free methods.", "questions": "1. Can the authors provide a quantitative analysis of the inference-time latency (e.g., in seconds per image) and memory overhead of SHIELD compared to the vanilla model and the baselines (VCD, OPERA)? How much does each of the three modules (re-weighting, subtraction, contrastive decoding) contribute to this overhead?\n\n2. The paper lacks a sensitivity analysis for the key hyperparameters $\\alpha$, $\\beta$, and $K$. How much does performance change when varying these? How were the final values chosen?\n\n3. Regarding the \"Token Re-weighting\" module: What happens if the \"naive caption\" contains a plausible hallucination (e.g., seeing a \"boat\" in a wide shot of a lake)? Can you provide an analysis of this failure case?\n\n4. Why was the specific PGD-style attack chosen for the \"Address Vulnerability\" module? How does the method perform if a different type of attack is used (e.g., FGSM) or if no attack is used at all (i.e., only using modules 1 and 2)?\n\n5. For the CHAIR evaluation, the paper follows prior work using 500 randomly selected images. How is the balance between coverage (ensuring all present objects are in the ground-truth) and faithfulness (ensuring no non-existent objects are in the ground-truth) managed for this label set? This seems critical for a metric that punishes “hallucinated objects”.\n\n6. For the GPT-4o assisted evaluation, why was a 0-10 human-like scoring system for “correctness” and “detailedness” chosen? Given recent work on automated evaluation (e.g., Valor-eval: Holistic coverage and faithfulness evaluation of large vision-language models. https://arxiv.org/pdf/2404.13874), what are the advantages of this 0-10 scale over more objective, feature-extraction-based or model-based evaluation methods?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper's focus on diagnosing hallucination sources within the visual encoder is a valuable contribution. The categorization into statistical bias, inherent bias, and vulnerability provides a structured way to think about the problem, moving beyond the more common focus on the LLM component.\n\n- The design of SHIELD is logical, with each of its three components clearly mapped to one of the three identified problems. This modularity makes the approach easy to understand.\n\n- As a post-hoc, training-free method, SHIELD offers a practical advantage, as it can be applied to any pre-trained LVLM without requiring expensive re-training or fine-tuning.\n\n- The experiments are thorough, covering multiple model families and benchmarks. The results demonstrate that SHIELD consistently improves over the vanilla models and outperforms other training-free baselines like VCD and OPERA in reducing object hallucinations, especially on benchmarks like POPE.", "weaknesses": "1. The primary weakness is the method's high computational cost at inference time. SHIELD requires multiple additional forward passes: one to generate the \"naive caption,\" $K$ (e.g., 32) passes for the noise inputs, and another pass for the adversarially attacked image. This combined overhead likely makes the method prohibitively slow for any practical or real-time application, which severely limits its impact.\n\n2. The method introduces several new and sensitive hyperparameters ($\\alpha$ for subtraction, $\\beta$ for contrastive decoding, $K$ for noise samples, and the attack parameters $l$ and step size). The paper provides very little ablation or sensitivity analysis for these. It's unclear how these were tuned or how performance would change on a different dataset or model, making the results potentially difficult to reproduce or generalize.\n\n3. The idea of \"inherent bias\" being estimated by feeding pure noise to the encoder is a strong assumption. It is not convincingly demonstrated that the resulting feature vector is a meaningful representation of \"bias\" rather than just an artifact of the network's response to out-of-distribution data. Furthermore, using a large $K=32$ to get a stable estimate adds significantly to the computational burden.\n\n4. The \"Token Re-weighting\" module fully depends on an initial caption from the vanilla LVLM, which is itself prone to hallucination. The paper's claim that this doesn't matter (because hallucinated words won't match visual tokens) seems overly optimistic and is not well-supported. A plausible hallucination could easily bias the re-weighting in the wrong direction, but this potential failure mode is not investigated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q5AOshoHHW", "reviewer": "Reviewer_AwNy", "summary": "This paper identifies three encoder-level sources of object hallucination—statistical bias, inherent bias, and vulnerability—and proposes SHIELD, a training-free pipeline that integrates token re-weighting, token subtraction (noise-derived removal of erroneous features), and adversarially informed contrastive decoding to mitigate these issues. Extensive experiments on CHAIR / POPE / MME evaluations demonstrate consistent gains across several LVLM families.", "questions": "See weakness", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Clear identification of encoder-level failure modes. The decomposition into statistical bias, inherent bias, and vulnerability is conceptually clean and well motivated by analyses (token peak-to-average ratios, dominant-object errors under noise, perturbation sensitivity). \n\n2. Strong token-level analysis. The paper’s quantitative and visualization evidence showing how individual visual tokens (peak norms / overemphasis) correlate with hallucination risk is compelling and valuable: it grounds the solution at the token representation level rather than treating hallucination as a black-box LLM issue. This token perspective is an important contribution for future encoder diagnostics. \n\n3. Modular, training-free design. SHIELD’s three modules are practical (can be combined or ablated) and the paper provides ablations showing complementary gains. Precomputable components (e.g., noise-derived estimates) make the approach more practical.", "weaknesses": "1.\tGenerative / open-ended hallucination benchmarks.\nThe current evaluation focuses mainly on binary or existence-based benchmarks (POPE, MME). To better demonstrate robustness under real generative scenarios, the paper should include results on open-ended hallucination benchmarks such as AMBER [1], MMHal-Bench [2], which assess caption-level fidelity and factual consistency rather than yes/no predictions.\n\n2.\tComparison with token pruning or sparse-token methods.\nIt would be useful to compare SHIELD with token pruning or sparse-token selection approaches that also modify the visual token set before decoding.\n\n3.\tAblation on precomputation and inference overhead.\nSince the paper mentions precomputing noise-derived erroneous representations, reporting runtime and memory trade-offs between precomputation and online computation would make the efficiency claims more convincing.\n\n4.\tBroader hallucination metrics and human evaluation.\nFor open-ended captioning, automatic metrics can be unreliable. Including limited human or GPT-4–based evaluation [2] focusing on whether hallucinated objects are introduced or removed would enhance credibility and interpretability.\n\n[1] Amber: An llm-free multi-dimensional benchmark for mllms hallucination evaluation, 2023.\n\n[2] Aligning large multimodal models with factually augmented rlhf, ACL 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)