On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in Large Vision-Language Models
Authors: Hoigi Seo
Dong Un Kang
Hyunjin Cho
Joohoon Lee
Se Young Chun
Venue: NeurIPS 2025 poster ()
Keywords: Large Vision-Language Models
Object Hallucinations
Uncertain Visual Tokens
Primary area: deep_learning
TL;DR: This paper mitigates object hallucinations in Large Vision-Language Models by exploring the uncertain visual tokens.

Abstract: Large vision-language models (LVLMs), which integrate a vision encoder (VE) with a large language model, have achieved remarkable success across various tasks. However, there are still crucial challenges in LVLMs such as object hallucination, generating descriptions of objects that are not in the input image. Here, we argue that uncertain visual tokens within the VE is a key factor that contributes to object hallucination. Our statistical analysis found that there are positive correlations between visual tokens with high epistemic uncertainty and the occurrence of hallucinations. Furthermore, we show theoretically and empirically that visual tokens in early VE layers that exhibit large representation deviations under small adversarial perturbations indicate high epistemic uncertainty. 
Based on these findings, we propose a simple yet effective strategy to mitigate object hallucination by modifying the VE only. Our method comprises a proxy method with adversarial perturbations for identifying uncertain visual tokens efficiently and a method to mask these uncertain visual tokens during the self-attention process in the middle layers of the VE, suppressing their influence on visual encoding and thus alleviating hallucinations. Extensive experiments show that our method significantly reduces object hallucinations in LVLMs and can synergistically work with other prior arts.

On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in Large Vision-Language Models
Dong Un Kangl* Se Young Chun 1,2,3t 12,3t Dept. of ECE 'IPAI & IIMM, Scoul National University, Republic of Korea (seohoiki3215, qkrtnskfk23, jim0228, joohoonl, sychun)
Hoigi Seo
Hyunjin Cho
Joohoon Lee²
Abstract
Large vision-language models (LVL.Ms), which integrate a vision encoder (VE) with large language model, have achieved remarkable success across various tasks. However, there are still crucial challenges in LVLMs such as object halluci- nation, generating descriptions objects that are not in the input image. Here, we argue that uncertain visual tokens within the VE is key factor that contributes to object hallucination. Our statistical analysis found that there are positive corre- lations between visual tokens with high epistemic uncertainty and the occurrence hallucinations. Furthermore, we show theoretically and empirically that visual tokens in early VE layers that exhibit large representation deviations under small adversarial perturbations indicate high epistemic uncertainty. Based on these find- ings, we propose simple yet effective strategy mitigate object hallucination by modifying the VE only. Our method comprises a proxy method with adversarial perturbations for identifying uncertain visual tokens efficiently and method mask these uncertain visual tokens during the self-attention process in the middle layers of the VE, suppressing their influence on visual encoding and thus alleviating hallucinations. Extensive experiments show that our method significantly reduces object ballucinations LVLMs and can synergistically work with other prior arts.
Introduction
Large Vision-Language Models (LVLMs)) have demonstrated impressive capabilities across a range of multi-modal tasks, including image captioning [1 10
63] visual-question answering Despite these notable ad- 57 have reported that LVLMs are susceptible hallucination, generating textual descriptions that do not align with the input image. In particular, object hallucina- tion, where the model describes objects not present in the input image, significantly undermines the reliability and thus the practical utility of LVLMs 20 24
30
33
(VQA) 10 58), and multi-modal dialogue systems 14,
28
35
36
vancements, recent studies [19 31
27
To mitigate object hallucination in LVLMs, recent works 2 20
24
25
27
37
have explored
training-ffo free approaches including modifying the decoding strategy of the language model 2 20 27 37. modulating attention mechanisms [2 25 37 altering the input image (2) during inference. While these methods have shown effectiveness in reducing object hallucination, they often suffer limitations such requiring multiple inferences of the large language model, which is the most computationally expensive component of LVLMs, or yielding relatively small performance gains. In contrast, approaches for object hallucination mitigation that directly target the vision encoder, core component responsible for visual perception, have been relatively underexplored.
* equal contribution, corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In this work, we investigate how visual information contributes to object hallucination in LVLMs, with particular focus on the uncertainty of visual tokens introduced by the pre-trained vision encoder (i.e. epistemic uncertainty). Estimating this uncertainty typically requires intensive computation, such as Monte Carlo (MC) dropout [4]] which involves thousands of forward passes. To provide a more efficient alternative, we present a theoretical analysis showing that the deviation of visual token representations under adversarial perturbations is monotonically related to an upper bound of uncer- tainty for each visual token, particularly in the early layers of the vision encoder. Empirically, we find that the norm of representation deviation in visual tokens caused by adversarial perturbations closely aligns with uncertainty estimates obtained via MC dropout, enabling more efficient approximation of visual token uncertainty. Furthermore, we empirically demonstrate a strong positive correlation between visual token uncertainty and the occurrence of object hallucination of LVLMs.
Motivated by this observation, we propose a simple yet effective method to mitigate hallucination by intervening only in the vision encoder during inference. Specifically, we first identify uncertain visual tokens, defined as those whose representations exhibit significant deviation under PGD-based adversarial perturbations [40] which reflect high epistemic uncertainty. We then suppress their influence by masking these uncertain tokens in the self-attention layers of intermediate vision encoder blocks. This approach reduces the model's s dependence on uncertain visual features while preserving the global semantic structure of the image representation.
Extensive experiments demonstrate that our method effectively reduces object hallucination on benchmark datasets such as CHAIR 47 POPE [3]]
and AMBER [56]. We validate our approach 69, incorporating diverse vision encoders, language models, and training regimes to ensure generalizability. Notably, because our method exclusively modifies the vision encoder, it can be seamlessly combined with existing methods that adjust decoding strategies or attention mechanisms within the language model.
across a range of IVL.M architectures 9 35
Our contribution can be summarized as follows.
We theoretically and empirically demonstrate that the visual tokens exhibiting the representa- tion deviations under adversarial perturbations indicate upper bound of epistemic uncertainty, which is strongly correlated with object hallucination in LVLMs.
Motivated by this insight, we propose an efficient and effective method that mitigates hallucination by identifying uncertain visual tokens via adversarial perturbation and masking them in the self-attention layers of intermediate vision encoder blocks.
Our method is validated across multiple benchmarks and LVLM architectures, and is easily compatible with existing mitigation methods, enabling synergistic gains in performance.
2
Related Works
Large Vision-Language Models. Large Vision-Language Models (LVLMs) integrate visual and textual inputs for multi-modal reasoning and generation. Modern LVLMs typically consist of a vision encoder [55 17 22 46 655, a connector, and a language model K
54 62). Some use linear projections to align visual features with the language embedding space 9 36 while others adopt Q-Former modules [44 30 69) that use learnable queries to extract and compress visual information. Despite their remarkable performance on multi-modal tasks, LVLMS exhibit hallucination, generating output misaligned with visual content, raising concerns about their reliability in real-world usage.
II
Mitigating hallucinations in LVLMs. Hallucination in LVLMs refers to the phenomenon in which the output contradicts the visual input by fabricating visual information [5] 34]. Mitigation strategies fall into maining-based and training-free frre categories. Training- based methods optimize the
LVLMs [33 (4] or incorporate auxiliary modules for output guidance [16
39 68], but are often
computationally expensive. Training-fi free approaches modify logits of language models to suppress
hallucination- prone text tokens 2
20
21
27
29
37 55 77), adjust attention process 24] 25| 22]
37 60). modify inputs 2
42 66 However, the approaches overlook deficiencies in the vision
encoder. We instead propose an orthogonal and training-free free strategy: leverage adversarial attacks to identify uncertain visual tokens and suppress them, complementing language-level approaches.
Adversarial attack on LVLMs. Adversarial attack 18 40 53 introduces imperceptible pertur- bations in images to induce incorrect predictions by model. While early efforts focused on tasks

---

+VE(Jauu - Farial
Lyperi
|
factk
;: oprimizable
Foris
Vision Encoder (VE)
++
Origiaa
Vision Encoder (VE)
Laverr
Aggregen Tanahal
Mask
Attacked
Original
Artacked
(a) Adversarial attack process
(b) Uncertainty mask generation process
Figure I: Overall illustration of the adversaria attack and uncertainty mask generation process. (a) The original image processed by the vision encoder (VE) to obtain features forig- An adversarial image is created by adding optimizable noise, which is then encoded produce fank- The noise is optimized using Projected Gradient Descent (PGD) to maximize the mean squared error between (b) From layers i
fong and fank. as described in Eq. | f } The norm differences of corresponding features w-11). [u'....,u"1) These maps are min-max normalized, aggregated, and standardized to produce the final binary uncertainty mask M using a threshold Oh-
1, extract feature sets Fori
(Fong:..... ff-1) fongg and Fank {fakk
form layer-wise uncertainty maps 21 = {u',
recent work has extended attacks to to improve the robustness of the models. In image-targeted attacks, where input in discrete pixel space, Projected Gradient Descent (PGD) [40] remains dominant strategy due to its effectiveness. The optimization process of PGD is formalized as follows.
such classification 18 50
and object detection 6 32 61
LVLMs 45 49
8++1 II à + a sign 22, ((F()))
(1)
where a E the learning rate, F is a target neural network, x is the original image, à; denotes the perturbed image at iteration i, and II projects onto the constraint set || æ/I < k. LVLMs
show strong multi-modal capabilities but remain vulnerable adversarial attacks 7 59 67
which can target the entire model 7 45 67
or specifically the vision encoder [59]
3 Method
In this section, we present approach for identifying uncertain visual tokens within the vision encoder using adversarial perturbations, as detailed in S.1 We demonstrate that these tokens significantly contribute object hallucination in LVLMs through statistical analysis. Based these findings, we propose a masking strategy within the vision encoder to suppress the influence of uncertain tokens, resulting notable reduction in hallucinations, as described in Sec
3.1 Adversarial Attack to Vision Encoder Reveals Uncertain Visual Tokens
3.1.1 Efficient uncertainty approximation of visual token with adversarial attack
Estimating uncertainty induced by deep neural networks (i.e. epistemic uncertainty) commonly approached by approximating Bayesian inference using Monte Carlo (MC) dropout [26] 44 However, the approximation process introduces substantial overhead as a result of thousands of forward passes. In this work, we find that the epistemic uncertainty of individual visual tokens differs from each other, as perceived by the vision encoder for a given image, and their upper bound can be efficiently estimated via adversarial attacks. To support this claim, first introduce the following lemma.
(f+) be a smooth T-llayer neural network parameterized by O. For an input E N x3 define the hidden state at layer t as (9) = fr .... o fi (x). For a perturbed input x + E, with ||e||oo S k for sufficiently small > O. define the perturbed hidden state as z()) = .... fi(x + +). Then, under the assumption that the perturbation small and f E C 7(0) can locally approximated by a Gaussian centered 2 with third-ordes remainder in the tog-density.
Lemma 3.1 (Approximate local Gaussianity under small perturbation). Let
3

---

12.2s
2.45
12.4
2.5c
12.3
2.36
CHA
AP
CHA PI
CHA PL
BOS
BOS
BOS
Original image
MC Dropout
Map U (Ours)
Original image
MC Dropout
Map U (Ours)
Figure 2: Visual comparison of estimated uncertainty from MC dropout [44] and our method.
Our uncertainty map U identifies uncertain regions similar to the uncertainty map obtained via MC dropout. MC dropout was applied to the residuals of the LLa VA-AA 1.5 vision encoder with a dropout rate of p = 0.5 and the variance of each token was estimated over 1,000 forward passes. For the adversarial attack, we applied 100 iterations of PGD with k = 3. The MC-based uncertainty values were log-scaled for visualization clarity. The runtime for each example is shown in the top-left corner.
The proof of Lemma 3.1 can be found in the Appendix Sec. A. The lemma implies that the hidden states exhibit Gaussianity under small perturbation, which allows us to prove the following theorem.
Theorem 3.2 (Upper bound of differential entropy increases hidden state deviation increases under adversarial attack). Let x be an input image, and let be small adversarial perturbation. Define the perturbed input as X x + E. Let
(ftf be smooth L-block transformer that processes sequence of N input tokens. Let 2(t) ;= fo fi (x) NR and Z ;= f...-.o (X) IR x be the hidden states at layer t for the clean and perturbed inputs, respectively. Denote the i-th token (0) representation at layer as 2i E Rdd and 7,!) E Rd. 1f 75 changes smoothly with small then the upper bound of the differential entropy of 2 increases as [e -
(1 increases.
The proof of Theorem 3.2. provided in Appendix Sec. A.2 shows that under adversarial attack, the norm of hidden state deviation efficiently approximates the upper bound of visual token's 'e entropy.
Leveraging this insight from Theorem 3.2 we aim to obtain a mask that identifies uncertain visual tokens with an adversarial attack. Specifically, given an image and vision encoder Fv. we first obtain the feature Song. = (v) E xxd where N denotes the number of image tokens. We then generate adversarially perturbed image to by adding small noise to z such that ||e|| oc k. We then extract feature perturbed image with fank. = Fv (ão). We define the adversarial objective as the mean squared error between forig. and fank. and optimize with PGD for / iterations as specified in Eq. D to obtain the final attacked image & == f1. This attack process is illustrated in Fig. la
Next, we extract the hidden states from each layer of the Fv within the consecutive layer index set ji 1} for both the original image x and the perturbed image f. This results in two hidden states sets: Forig = (forig..... f from x and Fank = {fink f(-1)) from á. For each layer E S, we compute the norm of deviation between the corresponding hidden states defined as u' 11 fungll2. resulting in set of layer-wist uncertainty maps Ut {u'.... - V E S}.
We then aggregate the layer-wise uncertainty maps in l/ to produce the aggregated uncertainty map by applying min-max normalization to each u' and averaging across layers, as defined below:
u'
U U= リーコー
(2)
u
max
min
Finally, we normalize the uncertainty map U using its mean pu and standard deviation ou, and binarize it with threshold parameter ath to obtain the binary uncertainty mask M as follows:
M 2 sign (U-
- 0th + E 10.1)
aU
Here, a value of 0 in the mask M indicates an "uncertain" visual token, while I denotes a relatively "certain" one. Figure Tb ibiltusttss the mask generation process, and examples of M are shown in
Appendix See G.3 In Sec 3.2 we describe how M is used to mitigate object hallucination.

---

3.1.2 Empirical study on extracting uncertainty with adversarial attack
Comparison with uncertainty via MC dropout. We compare our uncertainty map U with MC dropout 44 to assess how well U ap- proximates epistemic uncertainty. As shown in 2 the results indicate that U closely aligns with the uncertainty estimated via MC dropout, demonstrating that U serves proximation. On average, it approximately times faster than MC dropout in practice. Addi- tional qualitative and computational cost com- parisons are provided in Appendix Sec E.1
Relative norm deviation
1.0
0.9
0.8
Selected range
0.7
0.6
0.5
efficient ap-
0.4
10
16 22
Layer index
Figure 3: Relative deviation between attacked and original features. We used 500 images from the MS-COCO 13 with VLa vision en- coder [3). Perturbations introduced through the vision encoder remain minimal early layers but intensify in later ones. We extract the mask from early layers where feature deviations are compara- tively small. Error bars denote the 2a range.
The range of layer indices set S of vision en- coder. As described in Sec. B.1.1 we extract hidden states from the consecutive layer index set S. Our Lemma 3. and Theorem 3.2 rely on the assumption that adversarially induced norm of visual feature deviations are small, requiring that perturbations remain limited. Fig shows these deviations are minor in early layers but increase in later ones. To ensure consistency with both the theoretical assumptions and empirical observations, we construct S from early layers of vision encoder. Further analyses on S, provided in Sec additionally support this theoretical and empirical alignment.
3.2 Mitigating Object Hallucination of LVLMs via Uncertain Visual Tokens
Building on the identification of uncertain visual tokens through adversarial perturbations in Sec. 3.1 we now investigate how these tokens can be utilized reduce object hallucination in LVLMs.
3.2.1 Uncertain visual tokens contribute to object hallucination
To assess the practical relevance of uncertain visual tokens in object hallucination, we con- ducted a preliminary study using LLaVA-1.55 7B (35] on 1,000 randomly sampled images from MS-COCO (33. We estimate the uncer- tainty map of each visual token via Monte Carlo (MC) dropout, by computing the token-level variance. Using Eq. Band a threshold of ahh = 1, we generate uncertainty mask and calculate the average variance across the uncertain visual tokens in each image. The resulting averaged variances are grouped into 10 bins, and for each bin, we evaluate the severity of hallucination using the CHAIR 47 benchmark.
Hallucination score
0.8
0.6
CHAIR
0.4
FI
Average variance (uncertainty of visual tokens)
Figure 4: Relationship between uncertain vi- sual tokens and object hallucination. The æ-axis represents the average variance within each bin, while the y-axis shows the corresponding met- ric scores. The results indicate that higher uncer- tainty is associated with more object hallucina- tion, with p-value 0.05. The trend line was fitted with quadratic function. Note that higher values of CHAIR, and CHAIR,, and lower score indicate more severe object hallucinations.
The experimental results presented in Fig. 4 Fig. ssowws that higher average uncertainty of visual tokens corresponds to more severe object hallucination. To statistically validate this mono- tonic trend, we performed Spearman's rank cor- relation analysis between the average uncer- tainty (measured via token-level variance) and each hallucination metric. The resulting corre-
lation coefficients were p = 0.794 (p-value = 0.006) for CHAIR,, P = 0.733 (p-value = 0.016) for CHAIR,, and p -0.745 (p-value 0.013) for the Fl score, all statistically significant at p-value < 0.05, and indicating strong monotonic relationships (48) (p) > 0.7). Through this statistical analysis, we confirm that uncertain visual tokens contribute to hallucination of LVLMs.
5

---

3.2.2 Masking uncertain visual tokens for training-fres free hallucination mitigation
Building on the findings in Sec. [.2.1] we propose a method to reduce object hallucination by leveraging the uncertainty mask M, which highlights uncertain visual tokens identified through adversarial perturbation. Instead of completely removing these tokens, we attenuate their influence during the self-attention process the intermediate layers of the vision encoder. The intermediate layers of vision encoder was selected on the basis of empirical evidence that indicates its superior effectiveness in mitigating object hallucination.
Let Q, K, V E R- be the query, key, and value matrices in a self- layer, where N denotes the number of tokens and d" the dimensionality of the hidden states within self-attention process. Then, our masking strategy
Let M E {0, 1] be the binary uncertainty mask obtained from Eq 3 modifies the attention computation as follows:
QKT Vd
Attention( (2, K, V, M) = Softmax
O M
44)
Here, @ denotes token-wist multipli- cation. This operation reduces the in- fluence of uncertain tokens in the at- tention output while keeping the at- tention weights and other token in- teractions intact, Since the masking applied within the residual connec- tion structure, the model retains sta- ble and meaningful visual representa- tions while suppressing the contribu- tion from uncertain visual tokens. We illustrate this masking strategy within the self-attention process of the vision encoder within LVLMs in Fig. 5
MHSA Block
The image depicts some shoes
+
M
LLM
0OOM
VE
Tokenizer
0 ØKT
USER:: Please describe the image in detail.
Taketwwise product Softmax operation
Original
Figure 5: Illustration of our attention masking method during inference. In the intermediate multi-head self- attention layers of the vision encoder, we apply binary uncertainty mask M to the attention outputs. This token- wise masking reduces the influence of uncertain visual to- kens, while preserving the meaningful visual representation.
Compared masking strategies ap- plied at the input or output of the vi- sion encoder, intervening during self- attention computation in intermediate layers of the vision encoder offers more balanced approach to reduce the effect of uncertain tokens without discarding potentially useful visual information, as shown in the ablation study in Sec. 4.3
3.2.3 Does our method reduce uncertainty and mitigate object hallucination? Yes.
Based on the relationship between uncertainty of visual tokens and object hallucination discussed in Sec. 3.2.1 we mitigate object hallucination using the method introduced in Sec. 3.2.2 To
1.0
Hallucination score
0.8
CHAIN CHAIK FI
CHAIR (Ours) CHAII (Ours) at (Ours)
evaluate the effectiveness of our method in re-
ducing visual token uncertainty, we conducted
0.4
the same experiment as shown Fig. 4
0.2
The results in Fig. 6 show that the average vari- ance in the bin with the highest uncertainty de- creases from 6.04 to 4.98, CHAIR, drops from 1.00 to 0.33, CHAIR, from 0.27 to 0.09, and the Fl score increases from 0.47 to 0.77. To evaluate statistical significance, we performed the Wilcoxon signed rank test [3]. which con- firmed significant reductions average vari- ance (p- = 0.002), CHAIR, (p = 0.002), and CHAIR; (p- = 0.004), all statistically significant p < 0.05. The Fl score was preserved. These results demonstrate that the uncertainty of visual tokens contributes to object hallucination, and that method effectively suppresses this uncertainty, thereby mitigating hallucinations in LVLMs.
0.0
Average variance (uncertainty of visual tokens) Figure 6: Impact of the proposed masking strat- egy on visual token uncertainty. Average token- level variance estimated via MC dropout decreases after applying our method, indicating reduced un- certainty. This reduction correlates with improved performance on object hallucination metrics. The trend line was fitted with quadratic function.

---

Table 1: Quantitative results on CHAIR and POPE benchmarks. Object hallucination is evaluated on the CHAIR and POPE benchmarks using three LVLMs and five decoding strategies, both with and without our method. POPE results are reported on three splits: Random, Popular, and Adversarial. The maximum token length is set to 512. A %. denotes the relative difference in performance. t indicate that higher/lower values are better. We highlight the best scores in bold.
Method
Greedy
OPERA
VCD
PAI
Devils
Orig.
+Ours
A%
Orig.
+Ours
A%
Orig.
+Ours
A%
Orig.
+Ours
A%
Orig.
+Ours
L.LaV-- (LaVA-L-3-77 1.5-78
C, b
47.4
29.2
138.4%
47.8
29.4
138.5%
53.8
35.2
134.6%
33.2
26.0
121.7%
27.0
23.0
[14.8%
G.L
12.2
123.8%
12.8
9.5
125.8%
15.2
10.7
109.6%
7.9
17.15
[15.28
77.9
10.4%
77.7
78.4
10.9%
75.2
75.2
70.0%
78.3
77.2
J1.4%
10.4%
Rand.
89.3
10.0%
89.2
88.6
10.7%
84.6
86.2
11.9%
89.4
89.2
J0.2%
90.0
10.4%
Pop.1 t
85.8
85.8
10.05
85.8
85.2
10.75
82.4
82.9
60.6%%
86.0
86.4
10.5%
86.4
87.2
10.99.
Adv.
79.3
80.0
10.9%
80.3
79.6
10.9%
77.0
78.1
71.4%
79.5
79.9
70.556
78.6
79.6
11.3%
C,
58.0
43.2
125,5%
34,8
28.8
117.2%
56.2
47.2
116.0%
32.4
22.2
131.5%
24.4
20.6
115.6%
15.6
11.7
125.0%
11.1
9.6
113.5%
16,1
12.8
L20,5%
7.8
6.1
121.8%
7.6
6.8
[10.5%
Shika-7B
F1
74.7
76.9
12.9%
74.2
74.2
10.0%
74.4
75.2
70.1%
76.7
75.1
12.11%
73.3
72.2
11.55
Rand.
83.2
85.1
12.3%
84.88
85.4
10.7%
82.1
82.7
70.7%
83.9
84.0
10.11%
83.8
82.5
11.0%
Pop. f
82.3
82.6
10.4%
82.8
82.1
10.8%
79.7
80.7
11.3%%
83.1
80.7
12.9%
79.9
78.2
12.1%
Adv: 1
78.2
78.8
10.85
79.2
79.7
10.65
77.3
77.1
J0.35
78.8
77.4
11.85%
77.7
76.7
11.3%
C,
28.6
27.4
14421%
23.8
22.6
15.0%
32.0
30.6
J4.4%
19.6
17.8
J9.216
21.6
20.8
13.7%
MiniGPT44
C,
8.5
8.3
12.4%
8.8
8.5
13.0%%
9.7
9.1
_6.2%%
6.2
6.0
13.2%
7.5
7.0
16.7%
F1
71.5
10.35
70.0
10.39
70.2
71.3
-1.75
71.7
70.0%
70.1
70.4
10.45
Rand.
82.8
10.45
74.2
74.4
10.35
59.2
59.3
-0.25
82.0
J0.15%
77.8
10.5%
Pop.1 1
75.1
10.7%
71.3
71.8
10.7%
54.9
55.0
-0.2%
75.8
75,2
10.8%
10.3%
Adv.
71.8
71.2
10.8%
69.7
69.4
10.41
53.8
54.2
01.1%
72.1
71.6
J0:216
65.2
65.3
10.2%
4
Experiments
4.1 Experimental Setup
Baselines and implementation details. We evaluate our method on diverse LVLMs differing in size, architecture, and vision encoders: LLaVA-115-7BB 335 with CLIP-L/336px 46 Shikra-7B [9] with CLIP-L, and MiniGPT-4 using EVACLIP-g g [5]] and a Q-Former for image-text alignment. To assess compatibility, we integrate our method with hallucination mitigation methods including OPERA (20), VCD 277. PAI [37], and Devils 24] Adversarial attacks are run with h: and 200 PGD steps. The uncertainty masks M are extracted from layers S = {1,.... 10} of the vision encoder, with masking applied to layers 13-17 for VL- and Shikra, and 9-16 for MiniGPT-4. S are tuned per baseline-
pair. Further details are provided in Appendix Sec. C and D.3
Benchmarks.
To measure object hallucination, we use three standard benchmarks. CHAIR measures sentence-level (C, := CHAIR,) and instance-level (C; == CHAIR;) hallucinations from generated descriptions with 500 prompts randomly sampled from COCO 33
CHAIR, [sentences with hallucinated objects]|
CHAIR, = { (ballucinated objects) (5)
[all sentences]]
|all mentioned objects]]
POPE 31 evaluates hallucination through binary object presence queries across three splits (Random,
Popular, Adversarial), total 9,000 prompts, reporting accuracy. AMBER (56) comprehensively evaluates hallucination in two settings: generative approach (Gen.) that assesses hallucination through image captioning and a discriminative approach (Disc.) that uses yes/no choices. To measure object ballucination with AMBER, we adopted the full set of Gen. and the 'Existence' subset of Disc., conducting with a total of 5,928 prompts. See Appendix D.I and G.2 for more details and results.
4.2 Experimental results
Quantitative Results. We evaluate the effectiveness of our method in mitigating object halluci-
nations in multiple LVLM using CHAIR (47 and POPE 31 benchmarks. As shown in Table [ our method consistently reduces hallucination rates (C., C₂) across -LLVA-11.-7B. Shikra-7B, and MiniGPT-4, while preserving caption quality (F1). For example, on L.L 1.5-B, C. drops from 47.4 to 29.2 and C, from 12.2 to 9.3, Although the improvement on MiniGPT-4 is smaller, this is
7

---

Table Quantitative results on AMBER benchmark for LLaVA-1.5-7B. We evaluate object hal- lucination using the AMBER benchmark under various mitigation methods, including combinations with our approach. AMBER measures hallucination in generative (Gen.) and discriminative (Disc.) settings, with its score offering a comprehensive assessment across both. The maximum token length set to 512 for generative task. A % denotes the relative difference in performance.
Groody
OPERA
VCD
PAI
Devils
Method
Orig.
+Ours
A%
Orig.
+Ours
Orig.
+Ours
A%
Orig.
+Ours
A%
Orig.
+Ours
A%
Gen.
CHAIR
5.1
133.9%
5.8
(31.6%
8.5
138.25
4.7
17.8%
3.9
14.9%
Hall 4
30,2
24.2
119.9%
33.0
23.3
38.4
28.6
135.5%
25.1
22.5
_10.4%
21.0
20.9
10.5%
Cog.
3,8
2.3
139.5%
3.7
2.1
.43.2%
4.4
2.3
147.7%
1.9
1.9
10.0%
1,4
1.5
17.1%
Disc.
Pre. f
100.0
100.0
10.0%
100.0
100.0
10.0%
100.0
100.0
10.0%
100.0
100.0
10.0%
100.0
100.0
10.0%
Roc. 1
71.2
78.0
19.6%
74.9
81.0
17.5%
67.3
75.7
112.5%
71.9
74.1
13.1%
72.5
75.2
13.7%
83.2
87.6
15.3%
85.6
89.5
14.0%
80.4
86.2
17.2%
83.6
85.1
1.88%
84.1
85.8
12.0%
AMBERT
88.2
91.2
13.4%
89.1
91.8
13.0%
86.0
90.1
14.8%
89.2
90.2
11.1%
90.0
91.0
11.15
Please deseribe image in detail.
USER
The image faatures a train station with two trains on the trackss. The red trinn is positioned closer the left side the image, while the white and blue train situated further back. There several people the scene, with some standing near the trains and others scattered throughout the station. A bench can be the middle of the station, providing place for passengers sit Additionally, there is handhag placed on the ground near the center of the scene, possibly belonging to one of the passengers.
LLaYA
The image fentures red train maveling down the taaks, with another train visible in the hackground. The red train is positioned in the middle of the scene, w while the other train located towards the right side of the image. The trains both moving slong tracks, creating sense motion activity.
LLaVA
Ours
Input image
Please deseribe the image in detail.
USFR
The image features a wac-way street sign promincully placed next to white brick building. A bench is situated close the sign, providing place for pedlesrrias sit and observe the surroudingss The seene also includes a few cars parked along the street, with one car closer to the left side of the image, another further back, and third hy drant visible on the right side of the image, near the edge of the frame.
ONE WA
Shikra
the right side the
Additionally, there fire
The image features a one-way street sign positioned in front of building. The sign is pointing the left, indicating that vahicles should travel in one direction on this particular street. Beside the sign, bench can be seen, providing place for pedestrians to and observe the surroundings. The overall scene suggests viow of's street corner with some graffit on the wall behind the sign.
Ours
Input image
Figure 7: Qualitative results of our method on LLaVA-1.5-7B and Shikra-7B, Greedy decoding leads to object hallucinations by describing non-existent objects in the image (e.g.. 'several people'. "bench", 'handbag', 'passengers' in LLaVA; faw cars', "ar',, fare hydrant' in Shikra). In contrast, our method, which modifies only the vision encoder, substantially reduces such hallucinations.
likely due to its Q-Former module between the vision encoder and LLM, which limits the effect of our method modifying the vision encoder. In POPE, our method yields comparable or slightly im- proved performance across all models, indicating robustness under discriminative evaluation settings. Furthermore, it integrates well with existing mitigation methods such as OPERA, VCD, PAI, and Devils, providing additional gains without compromising caption quality. We also present results on newer models (DeepSeek- VL. (38), Qwen2.5-VL.
4 44) and larger models (LLaVA-1.5-133) are
provided in Appendix Sec. G.2
We further evaluate our method on the AMBER benchmark 56) using LLaVA- 1.5-7B across five strategies as depicted in Table 2 Our approach substantially reduces object hallucinations in both generative and discriminative tasks, achieving up to a 28.2% reduction in CHAIR and a 7.2% improvement in F1, resulting in consistently higher AMBER scores across all settings.
Qualitative results. We provide qualitative examples to demonstrate the effectiveness of our method. As shown in Fig. greedy decoding with vanilla LVLMs leads to object hallucinations, generating descriptions that mention non-existent objects such as several people, bench, car, or a a.fire hydrant. In contrast, our method substantially reduces such hallucinations in the generated outputs. Notably, in the case of Shikra integrated with our method, the model is able to correctly identify previously overlooked objects like graffiti, reflecting improved visual grounding and descriptiveness. We provide further qualitative results for various combinations of models and methods in the Appendix Sec. G.3
8

---

Table 3: Impact of vision encoder layers on generating the uncertainty mask M. Using early layers of vision encoder (1-10) to compute M yields the most effective object hallucination mitigation performance.
Table 4: Effeet applying the uncertainty mask M to different layers in the vision encoder. Ap- plying the mask middle layers of vision encoder (13-17) results in the most effective performance.
Masking Layer Range
C, +
C,
+
1
Mask Source Layer
0,4
C,
+
F1
Greedy Layers 1-8
47.4
12.2
77.9
Greedy
47.4
12.2
77.9
45.0
12.6
77.9
Layers 1-10
29.2
9.3
78.2
Layers 8
55.8
15.5
75.7
Layers 11-20
44.2
12.7
77.4
Layers 13-17
29.2
9.3
78.2
Layers 21-22
41.8
12.1
77.9
Layers
45.8
13.0
77.7
Table 5: Comparison of masking strategies for uncertain visual tokens. We compare our attention- level masking method with alternatives applied at different stages of the vision encoder (VE). S.M. denotes soft masking, which attenuates uncertain tokens by small factor (e-g., 0.1 or 0.2).
Strategy
Greedy
Input VE
Output of VE
MLP Layer
S.M. (0.1 /.2))
Ours
C. 47.4
47.4
34.4
$1.0
35.0/ / 40.0
29.2
C.
12.2
12.5
10.0
13.5
10.4 11.5
9.3
FI f
77.9
77.5
74.7
77.9
78.3 / 78.1
78.2
4.3 Ablation Study and Analysis
To assess the impact of each component on reducing object hallucination, we perform ablation studies on the LLa (35 model. We examine two key factors in the vision encoder: (1) uncertain visual token estimation and (2) training-free masking strategy. Each experiment isolates one variable to ensure fair comparison. Limitations of our method are discussed in Appendix J
Uncertainty estimation of visual tokens from early layers of vision encoder. We examine which layers of vision encoder are most effective for generating the binary uncertainty mask A using PGDD-ased adversarial attacks. As shown in Table 3 extracting uncertainty from early layers (1 to 10) leads to the largest reduction in hallucinations (C., C,) and the highest F1 score, outperforming intermediate or deeper layers. This result aligns with 3.c and Fig where early layers exhibit smaller adversarial feature shifts, making them more suitable for uncertainty estimation.
Masking uncertain visual tokens in intermediate layers of vision encoder. We investigate the effect of applying the binary uncertainty mask M to different layers of self-attention process within the vision encoder. As shown in Table 4 masking at intermediate layers (13 to 17) yields the best performance, significantly reducing hallucination (C., C₂) and achieving the highest Fl score. In contrast, masking in earlier layers shows limited benefit, and deeper layers provide minimal gains.
Comparative analysis of masking strategies for uncertain visual tokens. We compare several masking strategies using the binary uncertainty mask M, including masking at the input image, the output of the vision encoder, the MLP layer before the residual connection in the transformer block, and soft masking applied to the self-attention that attenuates uncertain visual tokens by small factor. As shown in T our method, which applies hard masking within the self-attention mechanism using M, achieves the best hallucination scores while maintaining competitive F1 score.
5 Conclusion
We present a simple yet effective approach for mitigating object hallucination in Large Vision- Language Models (LVLMs) by identifying uncertain visual tokens within the vision encoder and reducing their influence through masking these tokens in their self-attention layers. Our theoretical and empirical analyses show that adversarial perturbations efficiently approximate an upper bound of epistemic uncertainty, which we confirm to be strongly correlated with object hallucination in LVLMss Extensive experiments demonstrate that our approach consistently reduces object hallucination across diverse models and integrates seamlessly with other prior arts to improve performance.

---

Acknowledgement
This work was supported in part by Institute of Information & communications Technology Planning & Evaluation (ITTP) grants funded by the Korea governmen((MSIT) INO.RS-2021-1121 1132111111 Artifi- cial Intelligence Graduate School Program (Seoul National University)], (No.RS 202S-2025-000 2025-02314125 Teaming With Multimodal Hazy Oracle Models) and by the National
Effective Nlmman-Machinn Research Foundation of Korea(NRF) grant funded by the Korea govermeen(MMSIT) (No. RS-2025- 02263628). Also, the authors acknowledged the financial support from the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Seoul National University.
References
[1] Harsh Agrawal, Karan Desai, Yufei Wang. Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In ICCV, pages 2019.
[2] Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, QianYing Wang, Guang Dai, Ping Chen, and Shijian Lu. Agla: Mitigating object hallucinations in large dision-languaa models with assembly of global and local attention. arXiv preprint arXiv:2406. 12718, 2024.
[3] Jinze Bai, Shuai Bai, Yunfci Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mci Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men. Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang. Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang. Jian Yang. Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint Ar3i 230. 2023.
[4] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
[5] Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongho Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404. 2ee::2404.8880 229444884300 18930, 2024.
[6] Zikui Cai, Yaoteng Tan, and M Salman Asif. Ensemble-based blackbox attacks on dense prediction. In CVPR, pages
2023.
[7] Nicholas Carlini, Milad Nasr, Christopher A Choquette- Matthew Jagielski, Irena Gao, Pang Wei W Koh, Daphne Ippolito, Florian Tramer, and Ludwig Schmidt. Are aligned neural networks adversarially aligned? NeurIPS, 36:61478 61500, 2023.
[8] Liwei Che, Tony Qingze Liu, Jing Jia, Weiyi Qin, Ruixiang Tang, and Vladimir Paylovic. Eazy: Eliminating ballucinations in Ivlms by zeroing out hallucinatory image tokens. arXiv preprint arXiv:2503.077: 2025.
[9] Keqin Chen. Zhao Zhang, Weili Zeng. Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal i1m's referential dialogue magic. arXiv preprint arXiv:2306.1511 15195, 2023.
[10] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language- image model. In ICLR, 2023.
[11] Wei-l in Chiang, Zhuohan Li, 7i Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Liammin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
[12] dung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim. James R. Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. In ICLR, 2024.
[13] William Jay Conover. Practical nonparametric statistics. john wiley & sons, 1999.
[14] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards veneral-purposs lasion-language models with instruction tuning. In NeurIPS, 2023.
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer. Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. ICLR, 2021.
10

---

[16] Jinhao Duan, Fei Kong, Hao Cheng, James Diffenderfer. Bhavya Kailkhura, Lichao Sun, Xiaofeng Zhu, Xiaoshuang Shi, and Kaidi Xu. Truthprint: Mitigating Ivlm object hallucination via latent truthful-guided pre-intervention. arXiv preprint urXiv:2503. 10602, 2025.
[17] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang. and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In CVPR, pages 19358-19388 19369, 2023.
[18] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. ICLR, 2015.
[19] Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In AAAL, volume 38, pages 18135-18143, 2024.
[20] Qidong Huang. Xiaoyi Dong. Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in mulli-modal large language models via over-trust penalty and retrospeciionaalloctioo In CVPR, pages 13418-13427, 2024.
[21] Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang. Zhicheng Chen, and Peilin Zhao. Self- introspective decoding: Alleviating hallucinations for large vision- -aiguaee models. arXiv preprint arXiv:2408. 02032, 2024.
[22] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. July 2021. If you use this software, please cite it as below.
[23] Chaoya Jiang. Haiyang Xu, Mengfan Dong. Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language 2024.
model. In CVPR, pages
[24] Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens. arXiv preprint arXiv:2411.16724 2024.
[25] Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. See what you are told: Visual attention sink in large multimodal models. In ICLR, 2025.
[26] Max-Heinrich Laves, Sontje Ihler, Karl-Philipp Kortmann, and Tobias Ortmaier. Calibration of model uncertainty for dropout variational inference. arXiv preprint arXiv:2006. 11584, 2020.
[27] Sicong Leng. Hang Zhang. Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object ballucinations in large vision- languaee models through visual contrastive decoding. In CVPR. pages 13872 13882, 2024.
[28] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang. Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.033: 033266. 2024.
[29] Jiaming Li, Jiacheng Zhang, Zequn Jie, Lin Ma, and Guanbin Li. Mitigating hallucination for large vision language model by inter-modality correlation calibration decoding. arXiv preprint arXiv:2501.0192 2025.
[30] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman -mmgg pre-training with frozen image encoders and large language models. In JCML, pages 17730-19742. PMLR, 2023.
[31] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Xin Zhao, and Ji- R--Rngg Wen. Evaluating object hallucination in large Aision-language models. In EMNLP, 2023.
[32] Siyuan Liang, Baoyuan Wu, Yanbo Fan, Xingxing Wei, and Xiaochun Cao. Parallel rectangle flip attack: A query-based black-box attack against object detection. In ICCV, pages 7677-7687. IEEE Computer Society, 2021.
[33] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740-755. Springer, 2014.
[34] Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large rision-languag models. arXiv preprint 2022:.2402.00253 2024.
11

---

[35] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, pages 26296-26306, 2024.
[36] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36:34892-34916, 2023.
[37] Shi Liu, Kecheng Zheng, and Wei Chen. Paying more attention to image: A training-free method for alleviating hallucination in Ivims. In ECCV, pages 125-140. Springer, 2024.
[38] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepscck-vl: towards real-world vision- vision-language understanding. arXiv preprint arXi::2403.0552. 2024.
[39] Xinyu Lyu, Beitao Chen, Lianli Gao, Hengtao Shen, and Jingkuan Song. Alleviating hallucinations in large languaee models through hallucination optimization. NeurlPS, 37:1228 1722811-120 1-122832, 2024.
[40] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR. 2018.
[41] Alcksander Madry, Alcksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
[42] Shunqi Mao, Chaoyi Zhang. and Weidong Cai. Through the magnifying glass: Adaptive perception magnification for hallucination-fiee vlm decoding, arXiv preprint arXiv:2503. 10183, 2025.
[43] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering henchmark requiring external knowledge. In CVPR, pages 3195-3204, 2019.
[44] Jishnu Mukhoti and Yarin Gal. Evaluating bayesian deep learning methods for semantic segmentation. arXiv preprint arXiv:/811.12700 2018
[45] Xiangyu Qi, Kaixuan Huang. Ashwinee Panda. Peter Henderson, Mengdi Wang, and Prateck Mittal. Visual adversarial examples jailbreak aligned large language models. In AAAI, volume 38, pages 21527-21536, 2024.
[46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748 8763. PMLR, 2021.
[47] Anua Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In EMNLP, pages 4035 4045, 2018.
[48] Alfred P Rovai, Jason D Baker, and Michael K Ponton. Social science research design and statistics: practitioner's 's guide to research methods and IBM SPSS. Watertree Press LLC, 2013.
[49] Christian Schlarmann and Matthias Hein. On the adversarial robustness of multi-moda foundation models. In ICCV, pages 3677-3685, 2023.
[50] Erfan Shayegani, Yue Dong. and Nacl Abu-Ghazall Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models. In ICLR, 2024.
[51] Quan Sun, Yuxin Fang, Ledell Wu. Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint 2rXiv: 15389, 2023.
[52] Ilya Sutskever, Oriol Vinyals, and Quec V Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27, 2014.
[53] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 13/2,6199 2013.
[54] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.091 092880 2023.
[55] Chenxi Wang, Xiang Chen, Ningyu Zhang, Bozhong Tian, Haoming Xu, Shumin Deng, and Huajun Chen. Mllm can see? dynamic correction decoding for hallucination mitigation. arXiv preprint 241.::2410.11777, 2024.
[56] Junyang Wang, Yuhang Wang, Guohai Xu. Jing Zhang. Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang, An 11m- free dulensoimenii benchmark for milms hallucination evaluation. CoRR, 2023.
12

---

[57] Junyang Wang, Yiyang Zhou, Guohai Xu, Pengcheng Shi, Chenlin Zhao, Haiyang Xu, Qinghao Ye, Ming Yan, Ji Zhang, Jihua Zhu, et al. Evaluation and analysis of hallucination in large vision-language models. arXiv preprint arXiv:2308. 15126, 2023.
[58] Wenhui Wang. Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng. Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for vision and vision-language tasks. In CVPR, pages 19175-19186, 2023.
[59] Yubo Wang, Chaohu Liu, Yanqiu Qu, Haoyu Cao, Deqiang Jiang, and Linli Xu. Break the visual perception: Adversarial attacks targeting encoded visual tokens of large lainugla models. In ACM MM, pages 2024.
[60] Chunzhao Xie, Tongxuan Liu, Lei Jiang. Yuting Zeng, Yunheng Shen, Weizhe Huang. Jing Li, Xiaohua Xu, ct al. Tarac: Mitigating hallucination in Ivims via temporal attention real-time accumulative connection. arXiv preprint arXiv:2504.0m 04099, 2025.
[61] Cihang Xie, Jianyu Wang, Zhishuai Zhang, Yuyin Zhou, Lingxi Xie, and Alan Yuille. Adversarial examples for semantic segmentation and object detection. In ICCV, pages 1369-1378, 2017.
[62] An Yang. Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang. Guanting Dong. Haoran Wei, Huan Lin, Jialong Tang. Jialin Wang. Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai. Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Ni, Pei Zhang, Peng Wang. Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu. Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024.
[63] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14 14178, 2023.
[64] Zihao Yue, Liang Zhang, and Qin Jin. Less is more: Mitigating multimodal hallucination from an eos decision perspective. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 1 11766-1 17766-1177 11781, 2024.
[65] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, pages 11975-119999 11986 2023.
[66] Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip Ilievski. Milms know where to look: Training- free perception of small visual details with multimodal llms. arXiv preprint arXiv:5002.17422, 2025.
[67] Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li. Ngai-Man Man Cheung, and Min Lin. On evaluating adversarial robustness of large vision-language models. NewrIPS, 1.:54111-54133 2023.
[68] Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. In ICLR, 2024.
[69] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. ICLR, 2024.
[70] Lanyun Zhu, Deyi Ji. Tianuun Chen, Peng Xu, Jieping Ye, and Jun Liu. Ibd: Alleviating hallucinations in large vision-languae models via image- biased decoding, arXiv prepeint arXiv:2402. 18476, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We introduced our approach to mitigate object hallucination in title, abstract, and introduction. Also, we summarized our contributions explicitly Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed limitation of our work in Appendix Sec
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification: We provide the complete set of assumptions and full proofs for Lemma 3.1 and Theorem 3.2 with appropriate references to the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Ys]]
Justification: We provide code for reproducibility and detailed implementation details in Appendix Sec. C benchmarks and baseline models in Appendix Sec. D along with the
corresponding GitHub link.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers; Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
cases, in which
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide our code implementation in supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.com wttpcc//pppppppeeeeeeeeeeeeeeeee
httaa//gguiiee//eeeuabiisisssss /at////odeSimsssieeeeeeaa ssionooicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provide experimental setting and details in Appendix Sec. C Cad Sec.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide error bars experimental results in Fig. 3 and report
results' statistical significance of Fig. in Sec. 3.1.2 and Sec. 3.2.3
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are of range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We specify in Appendix Sec. C that all main experiments were conducted using NVIDIA A100 GPU with 80GB of memory.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:////wwwrrp.cpppcciileeeeeee
Answer: [Yes]
Justification:
We comply with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Ye]]
Justification: We discuss both the potential positive and negative societal impacts of our work Appendix Sec.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our paper does not involve releasing any models or datasets that pose a high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite all utilized code, benchmark datasets, and models, and provide the corresponding GitHub links in Appendix Sec. D
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, latasets,papersstt wwwpppppppphhaaooe thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
We will provide README file alongside the released code in the supple- mentary materials, which includes usage instructions, details of the benchmark datasets, and descriptions of the models used in our experiments.
Justification:
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMSs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Our core method development in this research does not incorporate large language models as any essential, novel, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurl)) atte:e///eerrrppccc/oooffereee/ /cC////oeeeeeencc//2///LLLL for what should or should not be described.
20

---

A Proofs
A.1 Proof of Lemma 3.1
Lemma 3.1 (Approximate local Gaussianity under small perturbation). Let f (ffo be smooth L-layer neural network parameterized by 0. For an imput a E Nx
x3 define the hidden state at layer t as (1) = fa o...o f1( For perturbed input x with ||c|| S for sufficiently small is > O. define the perturbed hidden state z o..... fi(x + c). Then, under the assumption that the perturbation is small and f E C Z(t) can be locally approximated by Gaussian centered at 20, with a third-order remainder in the log-density.
Proof. Let f = {felim be smooth I--ayer neural network parameterized by 0, and let 2(1) := ft f.... fx( denote the hidden state at layer for a clean input IR Nx3, For a perturbed input t+c, with ||]| I &e for small 0, define the perturbed hidden state ztt) == fi
For the clean and perturbed inputs, define
y" = [(;; 0) = (2(0;; y: f +;;0 8 +el:00)) ((,0)
(A1)
where ( = fr. ..... fe+1, 0(*) are its parameters, and e' is the residual vector at layer / induced by the input perturbation e. The perturbation is chosen maximize the adversarial objective Clly-- - yr or equivalently minimize
- y" 12). under ||e|| S k.
Motivated by this, we approximate the conditional distribution hidden states near (00) using a local energy-based form,
pa(z v²) x exp( - C/ (1))
(A2)
for z in a neighborhood of 2(t), Since twice continuously differentiable, the conditional log- density admits second-order Taylor expansion around 2(())
po (= y²) = loooooo (1) (1") + (= - =00) T v. logpo(= 1 11" 1==0(0) + (2 )0)) H (1) (2 2() R(z),
(A3)
where (11) == V loggo(z> 2=)2)=(()) is the Hessian and R(2) O(1 - =1()
The first-order term vanishes as follows:
V. po(= vn))==ate -0 J -2--JJoo
(((0)=;;0)) -10( -1000)))) (mm2)))
2m2(()) = (A4)
Therefore,
log logpo(z v²) = log lo (2 1² + [z - (1 20) + R(2).
(A5)
The quadratic term coincides with the log-density of a Gaussian centered (1) with covariance (- (-H(0)-1, while the remainder R(2) is of order O[] 23))
Therefore, the perturbed hidden state (t) under small input perturbations can be locally approximated by a Gaussian centered at (00), with approximation error of third order in the log-density.
A.2 Proof of Theorem
Theorem 3.2 (Upper bound of differential entropy increases as hidden state deviation increases under adversarial attack). Let be input image, and let be small adversarial perturbation. Define the perturbed input as := x + e. Let f f= (ft)tz be a smooth L-block transformer that processes a sequence of N imput tokens. Let 2(t) :- oo f1())) IRNxe xd and z := f.....o (X) E IR xd be the hidden states at layer t for the clean and perturbed inputs, respectively. Denote the i-th token representation layer as 200) E Rdd and 7!!0 E RR. If z(1) changes smoothly with small c, then the upper bound of the differential entropy of - increases as I., [[l 2 20"113) increases.
21

---

Proof. Let z be an input imgge and a small perturbation satisfying ||c|00 S k, where & is sufficiently small for first-order Taylor expansion. Define
5
1 (cx+));
(A6)
where f: denotes the hidden state of token at layer t, and f ft o...o fi is assumed to be twice continuously differentiable.
By the multivariate Taylor expansion of f!0 (xr c) around x, we have
z 2, c (c).
(A7)
where n(00 =
az(") oz E Rd D is the Jacobian matrix, and | (0 (c)l = O(|c(]?)
With the assumption of the perturbation upper bound k, the remainder R(d (c) is negligible compared to the linear term. Under this assumption, define the deviation:
AZ!" 2,"0 -x10 s!"e.
(A8)
Let E, : -E(ccT] Then the covariance of AZ!!) is
Az2 == Cov (A 1 = 1 (J.
(A9)
By the local Gaussianity assumption (Lemma 3.1 _ can be approximated multivariate Gaussian. Hence, by the entropy formula for multivariate Gaussians, the differential entropy is
h h(( - log (2nre)) det(>
(A10)
Applying the AM-GM inequality to the eigenvalues of Σ 20(0), we obtain
det(2 22 1/d tr tr(=)1(()) = Bl| |
(A11)
Thus, the entropy is bounded as:
M(2 (1) < 2
log
E1 C,
(A12)
where C' = _ (2Tee is a constant.
Hence, the upper bound of the entropy increases as E E|| 1 proof.
increases, which completes the
A.3 On practicality of the proved upper bound
Assuming that the deviation of hidden states follows a Gaussian distribution, the differential entropy of each token is proportional to the determinant the covariance matrix Daz. However, our empirical analysis reveals that this covariance matrix is highly low-rank. By decomposing the covariance matrix obtained from 2048 adversarial attacks on the visual tokens of 100 images with LLaVA-I.5-7B (28 using PCA, we found that the top 8 components (8/ 18/1024 = 0.8% of the total dimension) account for 94.2% (+004%% of the total variance, with most eigenvalues close to zero. Under such conditions, computing det(EAz (EAz) for entropy estimation becomes numerically unstable, as values underflow to zero, making direct entropy comparison infeasible. contrast, using tr( tr(Eaz) provides a numerically stable alternative that is theoretically well-grounded under anisotropy and preserves token-wise uncertainty ordering. This trace-based measure also aligns with the qualitative uncertainty maps in Fig. 2 further supporting its practical validity.
22

---

B Code
To support reproducibility, we include the implementation of our method the supplementary material. Detailed instructions for running the code and setting up the environment are provided in the accompanying README. md file.
C Implementation Details
As our method is designed to work in conjunction with various LVLMs and existing mitigation methods such as OPERA, VCD, PAI and Devils, we set the value of on individually for each combi- nation, shown in Table A The selected au values performance throughout all experiments in the main paper. As described Section [.1 PGD-based adversarial attacks performed with k = 3 and 200 iterations. For uncertainty estimation, masks M are extracted from layers S = (1,.... 10} of the vision encoder. The masking operation is applied within the self-attention mechanism of the vision encoder, targeting layers 13-17 for L.L.VA-1.5 and Shikra, and layers 9-16 for MiniGPT-4. All experiments in the main paper were conducted on an NVIDIA A100 GPU with 80GB of memory.
used consistently to evaluate hallucination
Table A1: Values of 7th for each model and method combination. We determine on individually for each combination and use the selected value consistently across all evaluations to ensure fair and robust comparisons.
Model
Greedy
OPERA
VCD
PAI
Devils
L.LVA--.5-7B
1.1
1.1
1.8
1.9
LLLVVA-1.5-13B
1.2
1.2
1.1
1.6
1.6
Shikra-7B
1.0
1.0
1.5
1.9
MiniGPT-4
0.0
2.0
0.0
1.1
-0.1
D Experimental Details
D.1 Benchmarks
CHAIR. To evaluate the robustness of image captioning models against object hallucination, we adopt the CHAIR [47 metric (Caption Hallucination Assessment with Image Relevance). This bench- mark quantifies hallucination by comparing generated captions with ground truth object annotations and sentence descriptions in the MSCOCO dataset. Two variants, CHAIR; and CHAIR,, measure hallucination the object and sentence levels, respectively, as shown Eq.
This metric enables systematic comparison of hallucination severity across models and offers insights into the alignment between visual input and generated language beyond standard evaluation metrics. We use the prompt "Please describe this image in detail.".
POPE. To obtain a more reliable and instruction-agneen -ggnosiii assessment of object hallucination in large vision-language models (LVLMs), we adopt the POPE (Polling- Object Probing Evaluation) framework (3T]. Unlike traditional caption-based metrics that are sensitive to prompt phrasing and rely on manual parsing, POPE probes model's visual grounding through binary yes/no questions about object presence. This enables stable and scalable evaluation across both annotated and unannotated datasets. POPE constructs evaluation sets using three sampling strategies: Random, Popular, and Adversarial. Fach strategy targets different source hallucination, allowing us to test whether models tend to hallucinate arbitrary objects, frequently occurring objects, or objects that often co- occur with those actually present in the image. We use the prompt "Is there a/an [object] in the image?".
AMBER. evaluate object hallucination comprehensively in large vasion-language models (LVLMss) we adopt the AMBER benchmark [56]. AMBER assesses hallucinations across both generative and discriminative tasks, focusing on three primary types: existence, attribute, and relation. In the generative setting, it employs metries such CHAIR, Hal, and Cog to measure hallucination
23

---

frequency, object coverage, and cogmitive tendencies. For discriminative tasks, standard binary classi- fication metrics are used, and the AMBER Score integrates CHAIR from the generative setting with the F1 score from the discriminative setting. Notably, we focus exclusively on 'existence" subset to assess object hallucination, which involves generating descriptions of objects that are not present in the input image. We use the prompt "Describe this image." for generative task and "Is there a [object] in this image?" ? for discriminative task.
D.2 Base models
35 a versatile multimodal model de- veloped for visual instruction tuning. LLaVA-1.5 builds upon the original LLaVA 36 architecture by integrating two-layer MLP as vision-language connector, leveraging the vision encoder, and incorporating academic task-oriented VQA data with response formatting prompts. These modifications significantly enhance the model's s capability for both visual reasoning and instruc- tion following, while retaining strong data efficiency. LLa VA-1.5 achieves competitive performance across broad set of multimodal benchmarks using only publicly available data and modest com- putational resources. To investigate the robustness of our method across different model scales, we conducted experiments using both the 7B and 13B versions of LLaVA-1.5. This enabled to evaluate whether our approach maintains performance consistency under varying model capacities. For the experiments, we utilized the official implementation along with the provided code and model weights.
LLaVA-1.5. In experiments, employed LLaVA-1.5
[46]
Shikra. In our experiments, we adopt the Shikra-7B [9] model, a LVLM specifically designed for referential dialogue. Shikra-7B integrates LLLP-ViT-L/B [46] vision encoder with Vicuna-7B language model via a simple alignment layer, allowing end-to-end processing without the need for additional vocabularies, position encoders, detection modules, external plug-ins. A key feature of Shikra is its ability to represent spatial information directly in natural language using numerical coordinates, allowing it to handle both inputs and outputs involving region references seamlessly. This architecture supports a broad range of vision-language tasks, including Visual Question Answering (VQA), image captioning, referring expression comprehension (REC), and PointQA, all within uni- fied framework and without task-specific fine-tuning. Its strong performance across both conventional and lecation-sensitive tasks makes it a compelling choice for measuring object hallucination. For the experiments, we utilized the official implementation along with the provided code and model weights.
MiniGPT-4. In our experiments, we employed MiniGPT-4 (69] as a vision-language model to evaluate effectiveness of our method. MiniGPT-4 combines frozen vision encoder from BLIP- (5I) with Q-Former) and large frozen language model, Vieuna, using a single trainable linear projection layer to align visual features with the input space of the language model. The model is pre-trained on approximately 5 million image-text pairs to establish initial multimodal capabilities. To address issues such as repetitive or fragmented outputs observed after pretraining, a second stage fine-tuning is applied using curated set of 3,500 detailed image- deseription pairs, formatted with a conversational prompt template. This two-stage training strategy improves the fluency and relevance of the model's 's responses, enabling it to handle variety of vision- language tasks more effectively. When applying our methodology to MiniGPT-4, we conducted the adversarial attack on the features prior to their input into the Q-Former. For the experiments, we utilized the official implementation 5 allng with the provided code and model weights.
2 [30] CIAA-CLIPViTTGGA
D.3 Baselines
Greedy. Greedy decoding is one of the most basic decoding strategies for generative language models, where the token with the highest prediction probability is selected at each step. This approach fast and straightforward to implement. Among various decoding strategies for LVLMs, we adopt the naïve and fundamental greedy decoding method as one of our baselines to evaluate the object hallucination mitigation performance of our method.
httper//gtthhb..comaaaiirrrraaat
24

---

Table A2: Runtime comparison between MC dropout and our method using PGD-based adver- sarial attack. When comparing the mean runtime, our method is x5.1 faster. The symbol + denotes the la interval.
Method
MC dropout
Adversarial attack (Ours)
Time (s)
12.4 (#0.12)
2.43 (=0.08)
OPERA. The authors of OPERA 20 identify that object hallucination in LVLMs is closely linked to specific knowledge aggregation patterns within the model's self-attention matrix. It defines tokens that induce such attention patterns as summary tokens and mitigates hallucination by detecting excessive attention toward these tokens and preventing their influence on next-token prediction. Specifically, OPERA extracts local window from the self-attention map, quantifies the degree of aggregation via column-wiss multiplication, and applies a logit penalty during beam search to suppress over-confident candidates. While effective, OPERA relies on beam search, which introduces significant additional computational cost. For comparison and integration with our method, we used the official implementation provided by the authors.
Visual Constrastive Decoding. The authors of Visual Contrastive Decoding (VCD) 45 attribute object hallucination to statistical biases, such as object cooccurrence frequencies in training data, and language priors inherent to large language models. By injecting Gaussian noise into the input image, the LVLM's reliance on visual information is reduced, causing it to lean more heavily on these language priors. To counteract this, VCD introduces both the original image and distorted version v' as input, computes their respective output probability distributions, and then extrapolates contrastive probability distribution that suppresses language-driven biases. For comparison and integration with our method, we use the official implementation 5 When applying our method to VCD, performed uncertain token suppression only the original image V.
The authors of Paying more Attention to Image (PAT) 33 argue that object hallucination arises when visual information is ignored and propose training-free free method enhance the influence of images during inference. Specifically, they manipulate the self-attention matrix to amplify attention toward visual tokens and selectively strengthen particular attention heads to guide the model toward more trustworthy directions. To avoid excessive attention toward the boginning-of-senteen token, they introduce a layer prior that excludes shallow layers from modulation. Additionally, they compare outputs with and without the input image to attenuate language model biases. Since PAI does not modify the vision encoder, our method can be additionally applied. For comparison, we utilized the official implementation
PAI.
(BOS)
Devils in the middle layers. In Devils in the Middle Layers (Devils) [24], the authors find that in large vision-language models (LVLMs), visual information is strongly processed in the middle layers of the language model. They observe that inactive attention can induce hallucinations, and that during such instances, attention heads tend to focus inconsistently on unrelated objects. To address this, the authors propose integrating information across attention heads during inference to encourage focus on more consistent visual regions. They achieve this by reweighting the attention scores to emphasize coherent areas. Since this is an intervention on the LLM component, their methodology is applicable in our setting well. To implement it, we adopted their official codebase
E
Additional Analysis
E.1
Monte Carlo Adversarial attack
In the main paper, we verify the similarity between the uncertainty map U obtained via adversarial attacks and the one derived from the Monte Carlo (MC) dropout using pre-trained vision encoder. To
httpp://giicuu..com/hikkit
httpp://ggithub.cco//aAa--- NLP-
httpp://giitmb..co//LLLL////
25

---

Table A3: Object hallucination benchmark results under varying attack strengths ([eel To investigate the effect of adversarial perturbations on the image encoder, we applied PGD attacks of different magnitudes for 200 iterations to LL
and evaluated performance using the CHAIR benchmark. Adversarial attacks on the image encoder increase the likelihood of hallucinated outputs, with the severity of hallucination correlating positively with the attack strength.
|
CHAIR, +
CHAIR;
Recallt
Precision
F1
47.4
12.2
78.9
76.9
77.9
53.0
16.2
76.9
72.9
74.8
64.0
25.5
63.0
62.4
62.7
5
65.6
25.9
55.9
60.1
57.9
61.6
26.6
50.5
59.6
54.7
further confirm this similarity, provide additional qualitative comparison in Fig- VV Although our method tends to slightly overestimate the uncertainty. consistently identifies high- uncertainty regions that closely align with those highlighted by MC dropout. To assess the computational effi- ciency of our approach, we compare the runtime of uncertainty estimation using Monte Carlo dropout and our adversarial-basee method. Specifically, ILLVA--15-775 The adversarial attack is performed 100 times with A = 3 top perturbations, while the Monte Carlo dropout requires 1,000 forward passes, both executed on single NVIDIA RTX 4090 GPU. The results, presented in Table [22] demonstrate that our method enables significantly more efficient extraction of uncertainty masks, highlighting its practical advantage in identifying visually uncertain tokens.
apply both techniques to the vision encoder from
E.2 Effect of adversarial attacks on LVLM outputs
We conducted PGD-based adversarial attacks on the vision encoder to identify the uncertain visual tokens. To evaluate whether such attacks effectively influence the output of LVLMs, we applied adversarial perturbations with varying magnitudes of and performed both quantitative and qualitative analyses.
As shown in Fig. A2 the responses generated from the attacked images often exhibited hallucinations or failed to produce correct answers. As demonstrated in Table A3 we also observe that higher attack intensities lead to increased severity of hallucinations. These experimental results highlight that the visual features extracted by the vision encoder play a crucial role in LVLMs' performance of
downstream task, emphasizing that enhancing visual perception is critical for reducing hallucination and improving overall reliability.
E.3 Consistency and robustness of uncertainty masks from adversarial attacks
We identify uncertain visual tokens by applying PGD-based adversarial attacks the features of the vision encoder. In our implementation, the attack is initialized from the original image without added noise. To evaluate the consistency and robustness of the resulting uncertainty masks M, also perform attacks with different initial noise seeds, generating diverse adversarial perturbations. From each perturbed image, we extract a mask and compute the mean Intersection over Union (mloU) between the masks M generated from different seeds.
As shown in Table A4 the uncertainty masks M remain highly consistent across different initializa- tions. Qualitative examples in Fig. A3 further demonstrate that the uncertainty maps and masks M maintain stable and coherent structures. These results confirm the reliability of our method in consistently identifying uncertain tokens under varying adversarial conditions.
F
Additional Ablation Studies
Masking Threshold Hyperparameter dth- To construct the binary uncertainty mask M, we introduce threshold hyperparameter oth Its optimal value depends on the characteristics of each model and method combination, and determined through grid search. Table presents an ablation study conducted on the LLaVA-1.5-70 model using six different threshold values. Considering the
26

---

Table A4: Mask consistency measured by mean Intersection over Union (mloU), We applied adversarial attacks to the LLaVA-1.5-7B image encoder on 500 images across five different seeds and measured the mloU verify mask consistency. The results indicate that the masks obtained through adversarial attack are robust and consistent. The threshold was set to 1.1.
Seed pair
(0, 1)
(0, 2)
(0, 3)
(0, 4)
(1, 2)
mloU
0.899 (1.0.034)
0.898 (100.035)
0.898 (100.36))
0.899 (1.0.036)
0.899(10.035)
Seed pair
(1.3)
(1,4)
(2, 3)
(2, 4)
(3, 4)
mloU
0.898 (+0.036)
0.898 (+00.035) 0.897 (+0.036) 0.897 (+0.036) 0.897 (+0.036)
Table A5: Ablation study of the thresholding parameter Tth for generating the uncertainty mask M. We use LLaVA-1.5-7B with greedy decoding and evaluate hallucination performance while varying the threshold dh.
Greedy
0.8
0.9
1.0
1.1
1.2
1.3
C
47.4
27.0
27.0
30.0
29.2
33.6
36.4
C,4
12.2
8.4
8.2
9.0
9.3
9.7
10.3
F1
77.9
76.7
77.7
77.6
78.2
78.0
78.5
trade-offss among C., C₁ and Fl score, we select oth = 1.1 as it yields the best overall performance. Based on this analysis, apply the optimal oth for each configuration in our experiments.
G
A Additonna Quantitative and Qualitative Results
G.1
Computational Cost
Our method identifies uncertain tokens via PGD-based adversarial attacks implemented through backpropagation, which naturally introduces additional computational overhead compared to standard greedy decoding. To quantify this cost, we measure the extra inference time and compare with existing hallucination mitigation methods. As shown in Table A6 while our method does incur some additional overhead, it offers comparable or even lower inference time than several baselines, achieving favorable balance between performance and efficiency.
G.2 Additional quantitative results
Applicability of our method larger model. We assess the scalability and generalizability of our method using the larger LLa LLaVA--155133) model. As shown in Table A7 our method delivers substantial improvements over the greedy decoding baseline, reducing C, by 15.2 and C; by 2.9. also integrates effectively with variety of existing approaches, achieving the best performance when combined with Devils (C, = 20.4. C = 6.0). These results demonstrate that our method generalizes well across model scales and enhances wide range of existing hallucination mitigation strategies.
Table A6: Additional inference time introduced by each method compared to standard greedy decoding. We performed text generation with request image description with max 32 tokens. All experiments were conducted using LLaVA-115-1 on an NVIDIA A100 GPU. We report the mean and standard deviation over 30 samples. Although our method introduces some overhead due to backpropagation from PGD attacks, it remains comparable to or even faster than existing approaches.
Method
Additional inference time (s)
OPERA
9.518110.011
VCD
1.646-10.001
PAI
1.567=0.022
Devils Ours
0.01440.001
2.469+10004
27

---

primary evaluation Table A7: Quantitative results on CHAIR benchmark for LLaVA-1.5-13B, We report object hallucination (C., CJ) for various mitigation methods and their combination with our method. The maximum token length is set to 512. A % denotes the relative improvement in performance.
Greedy
OPERA
VCD
PAI
Devils
Method
Orig.
+Ours
A%
Orig.
+Ours
4%
Orig.
+Ours
A%
Orig.
+Ours
4%
Orig.
+Ours
A%
C.4
45.4
30.2
133.4%
40.2
30.4
124.4%
49.0
35.4
127.8%
38.6
32.4
116.1%
28.2
20.4
126
C
11.2
8.3
125.9%
10.9
8.9
118.2%
13.4
10.3
123.19
9.9
8.4
115.2%
8.7
6.0
131.8%
F1
79.1
78.9
10.2%
78.0
76,9
11.4%
77.3
76.3
11.3%
78.7
79.1
10.3%
78.4
78.0
10.5%
Table A8: Quantitative results of our method on state-of-the-art LVLMs. We apply our approach to two SOTA models, DeepSeek- VL and Qwen2.5-VL, and compare performance against greedy decoding. For DeepSeek-ViL we set du = 1.0, while for Qwen2.5-VV V-V we use oh = 0.0. These results demonstrate that our method is applicable to wide range of LVLMs, including the most
recent architectures.
CHAIR
POPE
Method
C.4
C2 4
F1 f
Rand.
Pop.
Adv.
DeepSeck-Ve VL (Greedy)
25.8
6.6
72.7
88.7
88.0
84.9
+Ours
22.4
5.5
72.6
88.8
88.0
85.1
Qwen2.5-VL (Greedy)
29.6
7.8
76.0
84.2
83.7
83.3
+Ours
28.6
7.0
76.8
84.3
83.8
83.4
Table A9: Additional quantitative results for an alternative adversarial attack on a Q- Former-based LVLM architecture. MiniGPT-4 uses Q-Former to effectively compress image tokens, which confers robustness to image-only perturbations. By jointly perturbing the Q-Former's learnable query vectors together with the image, enable stronger attack and observe additional gains in attack effectiveness.
Method
C.4
C
FI f
Greedy (MiniGPT-44
31.0
11.4
67.3
+Ours (Image only)
29.0
10.6
67.5
+Ours (Image + Query)
27.0
9.3
68.1
Applicability of our method to the state-of-the-art models. In the main paper, we conducted extensive experiments on LLaVA-1.5, Shikra, and MiniGPT, which are commonly used as target models in object hallucination mitigation studies and therefore served benchmarks. To further validate the applicability of our approach, we additionally evaluated state-of- the-art models such DeepSeek- VL (38) and Qwen2.5- VL [4. These models not only demonstrate strong performance, but also involve joint fine-tuming of the vision encoder during vision- languaee alignment training, making them suitable indicators of the scalability of our method. The results A8
presented table A8 confirm that our approach effectively reduces object hallucination even in these latest models.
Alternative attack methods on Q-Former design architecture. We observed that adversarial attacks applied solely the image have limited effectiveness in Q-Former based architectures (e.g., MiniGPT-4). This appears to stem from the robustness introduced by the architectural design that relies learnable queries. To validate this hypothesis, we additionally optimized the input queries during ad adversarial attacks to examine whether our approach provides further advantages. Unlike images, the query vectors are continuous, and thus we imposed noise constraint on the query vector q such that the perturbation scale matches that applied to the image.
|<||o (max (q) min(g))
||s/
(A13)
255
2
where ey the adversarial noise injected query vectors g, is the noise added to the victim image. The results are presented in table A9 which report the outcomes of adversarial attacks jointly applied
28

---

Table A10: Average length of generated text with standard deviation. We report the average length of generated texts across different models and hallucination mitigation methods, with and without our approach. Values are presented as mean standard deviation. Our method slightly reduces output length, which has been linked to lower hallucination rates in LVLMs.
Greedy
OPERA
VCD
PAI
Devils
Model
Orig.
+Ours
Orig.
Orig.
+Ouss
Orig.
Orig.
LLaVA-7B
49111104
426.1005
403
406_118
517.1144
4201122
5141118
487.1120
504.1206
448 1433
LLaVA-13B
495=101
440#114
452=136
402=142
515=108
436:122
510=122
468::115
4062141
3814:12
Shikra-7B
514#110
475=108
370=120
354=109
524=113
487=113
493+195
4274213
383+202
368+205
MiniGPT-4
408+2066
418+-202
301+-335
304-110
167
404+172
284+122
282--130 130
4154-444
3914-389
Table A1l: Effectiveness of our method ap- plied to different decoding baselines. We eval- uate our method on 1ALaAA-155-- using vari- decoding strategies, including greedy decod- ing, beam search, DoLa and VAR. We set the Nheam 5. Across all settings, method con- sistently reduces hallucination metrics (C,, C.) while maintaining or improving Fl score.
Table A12: Comparison of uncertainty estima- fion methods for generating mask M. We eval- uate the effectiveness of our adversarial attack- based uncertainty estimation method against MC dropout on LLaVA-1.5-7B using the CHAIR dataset.
Method
C,+
C
4
F1
f
Greedy
47.4
12.2
77.9
Method
C.4
C +
Fl f
+Ours (w/Adv. attack)
29.2
9.3
78.2
Greedy
47.4
12.2
77.9
+Ours (w/MC dropout)
32.6
10.5
77.8
+Ours
29.2
9.3
78.2
Beam search
47.2
12.7
77.8
+Ours
28.2
8.6
78.5
DoLa
46.0
12.2
78.5
+Ours
30.4
9.5
78.2
VAR
46.8
12.5
77.9
+Ours
29.4
9.1
78.1
to both the image and the Q-Former queries. The evaluation on the CHAIR benchmark demonstrates that our method can achieve further performance improvements when combined with additional architectural considerations. However, for methodological consistency, the main paper focuses only on adversarial perturbations applied to the image.
Length of generated text. highlights that overly long outputs from LVLMS often lead to object hallucinations, the generated content exceeds the model's s visual perception. As shown in Table ATO our method consistently and slightly reduces the length of image descriptions across various models and hallucination mitigation methods. However, in the case of MiniGPT-4, due to its Q-Former architecture, masking uncertain visual tokens within the vision encoder is less effective. As result, the generated text length may occasionally remain unchanged even slightly increase.
Application of our method to other baselines. To validate the generalizability of our method for mitigating object hallucination in LLa LLa 1.5-7B, we apply it to alternative decoding strategies, including beam search decoding [52]. DoLa 12] and VAR 25 using the CHAIR dataset. As shown in Table A1T our method consistently reduces hallucination rates while maintaining or even improving the FI score.
Comparison of uncertainty estimation of visual token: Our Method vs. MC Dropout. Epistemic uncertainty of visual tokens introduced by pre-trained vision encoder can be estimated using MC Dropout. However, this approach often requires intensive computation due to thousands of forward passes. As an efficient alternative, we propose a method that estimates uncertainty of visual tokens using PGD-based adversarial attacks.
perform experiments on I.L.VA-1.5-71 using the CHAIR dataset and compare the uncertainty masks M for visual tokens, generated using Eq between our method and MC Dropout. As shown
29

---

in Table A12 our approach achieves comparable or better performance while being more compu- tationally efficient. These results highlight that our PGD-based uncertainty estimation effectively captures the epistemic uncertainty of the pre-trained vision encoder and reliably identifies uncertain visual tokens.
Regarding the lower performance of MC dropout compared to our method, we conjecture that although MC dropout widely used for uncertainty quantification, it remains only one estimation technique. In contrast, our approach provides a more conservative estimate of uncertainty through an upper bound, which we believe accounts for its superior performance.
G.3 Additional qualitative results
Qualitative examples of binary uncertainty masks M. Fig. binary uncertainty masks M generated for various input images under PGD-based adversarial attacks applicd to the vision encoder of LLaVA-1.5-7B.
presents additional examples of
Qualitative examples of our method on various LVLMs with different mitigation methods. We present additional qualitative examples of our method applied different combinations of LVLMs -AaVA-115-7B and Shikra-7B) and hallucination mitigation techniques, including greedy decoding, OPERA, VCD, PAI, and Devils. Our method integrates well with these approaches and effectively reduces object hallucinations by preventing the generation of non-existent objects. Fig. A54 illustrate qualitative examples on the CHAIR and POPE datasets using LLaVA-1.5-70 and Shikra- across various hallucination mitigation methods.
Qualitative examples of failure cases. Fig. A25 presents qualitative examples of failure cases from our proposed method. Although method consistently mitigates hallucinated responses, it occasionally fails to prevent all hallucinations.
H
Discussion
We statistically demonstrate that epistemic uncertainty within the vision encoder contributes to object hallucination and address this issue through self-attention masking at intermediate layers. To understand how LVLMs change their integration of visual information after applying our method, we measured the entropy of the LLM's s attention distribution over image tokens across all layers and heads. Entropy serves as an indicator of whether the model attends broadly or narrowly, with higher entropy reflecting the use of wider range of visual evidence rather than reliance tokens. Using 500 images, we found that the average entropy of LLaVA increased from 1.5746 in the original model to 1.9717 with method. This increase suggests that approach encourages broader and more balanced attention over reliable visual tokens, enabling the model to integrate visual information more effectively while reducing over-reliance uncertain inputs, consistent with findings from prior work (37]
small subset of
I Broader Impacts
We proposed a method to improve the reliability of Large Vision-Language Models (LVLMs) by identifying and masking uncertain visual tokens in the vision encoder, key source of object hallucination. In contrast to existing approaches that intervene at the language model level, our method operates solely the vision encoder and demonstrates effectiveness across variety of models and settings.
Our method offers significant societal benefits by improving safety and reliability in critical applica- tions such as medical imaging, assistive technologies, and autonomous systems. However, it may also inadvertently suppress valid but ambiguous visual information, which could disproportionately affect underrepresented groups and reinforce existing dataset biases, raising important concerns about potential negative socictal impacts.
30

---

J Limitations
Despite its advantages, our method has several limitations. First, while effectively reduces hallucina- tions, the proposed masking strategy within self-attention process of vision encoder may result in the loss of visual information, leading to slight performance degradation in certain peaception-orientte tasks. Second, the reliance on PGD-based adversarial attacks to estimate uncertainty increases in- ference time. Third, the masking strategy lacks a formal theoretical foundation, instead relying on trace-based approximation of uncertainty as surrogate for the determinant of the covariance matrix. Finally, the method is less effective for models like MiniGPT-4, which utilize a Q-Former abstract visual information before integrating it with language. In these cases, masking at the vision encoder has limited downstream impact. Addressing these issues a promising direction for future research.
31

---

P.Morsan
PMooeann
PMMrvan
Original image
MC Dropout
Map U (Ours)
Original image
MC Dropout
Map U (Ours)
Figure A1: Additional visual comparison of estimated uncertainty from MC dropout and our method. Our uncertainty map U identifies uncertain regions similar to the uncertainty map obtained via MC dropout. MC dropout was applied to the self-attention and ML. residuals of the -L- vision encoder with a dropout rate of p = 0.5. Layers = 5 to j = 8 were used, and the variance of each token was estimated over 1,000 forward passes. For the adversarial attack, we applied 100 iterations of PGD with & = 3. The MC-based uncertainty values were log-scaled for visualization clarity.
32

---

image detail.
image detail.
USER
Original
Aftacked
The image features a dog lying on shelf,, surrounded by various shoes, dog pasitioned in midkle of the shelf, with its head resting on pair of shoes, the shelf is filled with numerous shoes, some which placod close the dog. while around the shelf. The seene creates comfortable atmosphere for dog.
The image features small, fluffy dog sitting pair, possibly-s-sized pair of headphones. The dog positioned the center, and the headphones placed on the dog's head. The dog is wearing pair of headphones, and the dog is a cute. The dog sitting in the background.
LIUM
IVLM
LVLM
scattered and
Please deseribe the image detail.
Please describe the image in detail.
USER
USER
Original
Atlacked
The image leatures two girafles 2oo unclosure. giriffe standing the left side of the enclosure, while the other is on the right side. They are both cating from fooder, which is placod the enclosure. The girafles pasitioned close cach other, creating sense of companionship.
The image features giraffe with a baby giraffe back, standing bathroom The baby girffee being hugged by the adult giraffis creating beartwarming
LVLM
IVLM
middle
IVLM
buthroom equipped sink and mirror, which reflects the giratles' presence in the room.
Please describe the image detail.
Please describe the image in detail.
USER
USER
The image features group of birds perched en tree sitting
The image features a group of five monkeys sitting on brunch. They all facing possibly looking at something off-camera. The munkeys are positioned various heights on the branch,
nine hirds total, closer to the of the branch and others positiened farther down. The birds are of various sizes, indicating diverse group species. scene captures the beauty of nature and the harmony between the birds and their environment.
direction,
INLM
the top and others the bottom. The seene captures the monkeys' natural hubitat and their social behavior.
Figure A2: Qualitative comparison LVLM outputs when using the original image versus the adversarially attacked image input. When conditioned on the attacked image (right column), the model generates descriptions involving non-existent objects or scenes, indicating more severe object hallucination effect compared to the original image (left column). This experiment was conducted on the VA-1. 11.5-BBBB 7577B
model using the CHAIR dataset, with and 200 PGD iterations.
33

---

Original image
Sced 0
Seed
Sced 2
Sced
Sced 4
Mup U
Mask M
Map U
Mask M
Map U
Mask M
Mup U
Mask M
Figure A3: Uncertainty maps and masks M generated from different initial noises using PGD-based adversarial attacks. We qualitatively demonstrate the consistency of the uncertainty maps U and corresponding masks M (oa 1.1), generated using PGD-based adversarial attacks with five different random seeds for initializing noise. Despite variations in the initial noise, the resulting uncertainty maps U and masks M remain highly similar, highlighting the robustness and stability of the attack-bar based uncertainty estimation VA-a.AA115111 vision encoder.
34

---

Input image
Map
Mask M
Input image
Map
Mask
Figure A4: Additional qualitative examples of uncertainty maps U and masks M using PGD- based adversarial attacks. We visualize the uncertainty maps U and the corresponding masks M (with onh = 1.1) generated from adversarially perturbed inputs across various samples. All examples were obtained using the L.LVVA-1.5-BB vision encoder.
35

---

Please describe the image in detail.
The image features bathroom shower, which covered in newopaper pages. The walls of the bothroom are adorned with a unique design, featuring combination newspaper tile. There are several towels in buthroom, with one placed on the sink and another on the floor. Additionally, there are two bottles, one near the sink and the seen
while sink and mirror above The sink situated next to
LLav
other on the floor. cup can
sink, and buok placed
The image features bathroom with sink and mirror. The sink located on the lefl side of the bathroom, while the mirror is positicneed above it The bathroom also has shower, which situated en the right side the room. The walls of the bathroom adorned with unique and interestingg design, featuring collage of newspaper clippings. This artistic choice adds a touch of personalty and character to the space.
ELaVA Ours
Input image
Please deseribe the image in detail.
USER
The image features large white and green bus driving down city street. The bus is pusitionee in the middle the street, and appeans be public transit bus. There are several people walking along the sidewalk, with some of them carrying handbags. In addition the bus, there are two bicycles parked on sidewalk, one closer
LLVA
side of the image the other further to right. A traffic light can be seen on the left side of the street, and a fire hydrant is located near the center of the scene.
The image shows large white has driving down down the road. The bus is in the middle of the street, and it is surrounded by buildings. The bus is also driving past stop, which located the left side of street. bus is focus of the image, and is driving down the road, providing transportation services to the people in the arca.
The hus public transit and is driving
LLaVA
Input image
Ours
Please describe the image in detail.
The image features a red bicycle parked in a room, possibly bedroom. The bicycle is leaning against bed, which covered with blankets and sheets. The room uppeurs to be cluttered, with variour iterns scattered around. There are several sports balls in the room, with one located near the left side the bed, unother in the middle, and third top left corner and the other on the right side of the room, A chair can be seen in the background and cup placed on surface the middle of the
LLas
the right side. Additionally, there two bottles,
the
The image fearures a bicycle parked next to appears to be old- fashioned model. The bed covered with blanket, and there are several sports balls scattered around bed. The sports
bicycle is leaning against hed, and
LLaVA Ours
various sizes are placed different positions bed. The soene gives off casual and relaxed almosphere.
Inpur image
Please descrihe image in detail.
USFR
The image features a small, white, square toilet with a hole in the middle, located on a tiled floor. The toilet positioned in middle the room, surrounded by tiles. The appears to dirty, and is person standing ncxi to the toilet. possibly using it. In the scene, there are also two bottles, one image. Additionally, there is
LLaA
located the top corner the other near top right comer cup placed on the floor, closer to the center of the scene.
The image features a dirty, white toilet with hole the middle, located in a bathroom. The toilet surrounded by tiled walls, giving it somewiaa old and worn appearance. The toilet siluated in the corner of the bathroom, and the tiles are visible on the walls and floor. The overall scene gives off a sense of disrepair and neglect.
LLaVA
Inpus image
Ours
Please describe image in detail.
USER
The image features man sitting couch living room, bolcing Wii remote his hand. He appears to he playing a video game,, possibly on Nintendo Wii console. The couch is positioned against wall, and there chair located near the left side the
the background, there are several books scattered around, possibly indicating reading or relaxation area. dining table can be seen in the room, cup placed on Adcitiionall, there cell phone resting on the lable, and bottle located near the edge of the table.
LLVA
sitting couch, holding Wii remote in his hand, He wearing gray shirt and appears to he enjoying some leisure time. The couch is positioned in a living room, and there are several books seattered around room, situated far away from the couch.
The image features
ETaVA
Some baoks are placed on a shelf. choir also visible the
Inpol image
Figure A5: Additional qualitative results of our method on VA-1.5-. with greedy decoding. On the CHAIR dataset, our method effectively reduces object hallucinations, preventing the model from generating non-existent objects in the image. Hallucinated object words are highlighted in red.
36

---

Please deseribe the image in detail.
USER
The imagee depicts a small bathroom with a while loikt positiened in the eenier of the room. Nest to the toilet, there sink with mirror
Above sink. also peesent in the bothroom. In addition to the main fistures, thare are two cups placed on the floor, one near side of side of the toilet. The buthroom appcars to be clean and well-mainiained.
soup dispenser, towel
OPERA
the other closer
right
can seen
The image depicts a small bathroom with a white toilet and a sink. The toiler pasitioned towards the side the bathroom, while the sink located right side. bathroom appears tiled, giving clean and well-maintaimed appearance, Addicionally, there is toilet bruch placed near the toilet, rendy for
LLVA
Input image
OPERA
Ours
Please describe the imago in detail.
USER
woman sitting on edge appears be enjoying her time in the room. The hed occupies significant portion of the scene, the right side
image
hedroom, She wearing
room, one located the left side of the bed and the other on the right side. potted plant can be seen on the left side of the right the
extending
image.
are couches
LLaV
OPERA
room, adding a touch of greenery to space. Addirionally, bed, possibly belanging to the woman or someone else in the room.
cup placed
bedroom. positioned near center which is covered with a while comforter. The bed is adorned with coloeful quilt, adding vibrani touch room. woman appears relaxed enjoying time bedroem.
image features woman sitting on
LL
Imput image
OPERA
Please describe the image detail.
USER
The image features cozy bedroem with neatly made bed sinuated in the center of the room. The hed is blanket, creating a warm inviting atmouphere. addition to the hed, there is large window in the roem, allowing natural light fill the space. A potted plant is placed near window, adding touch greenery
LLVA
the room. bookshelf filled with numerous books, showcasing variety of reading materials for the occupant. A reading.
OPERA
chair positioned the providing a comfortable seating items in the room include vase, a bottle, and a clock, adding the overall decor and functionality of space.
Imput image
image depicts cazy with a blue comfomer, and there are several books scattered around the room. potted plant is placed near window, adding touch greenary chair in the room. The dresser located on the left side of the bed. A mirrar also present in the room. positioned
scene bed positioned a large window.
covered
LLAVA OPERA Ours
space. addition
bed,
side bod. rel relaxation and reading.
room appears comfortable inviting space
Please descrihe the image in detail.
The image features black standing bod, curiously looking cup placed on the bod. The sppears to snitfing the cup, possibly intrigued by ils contenis. In the room, there dining table few items on bottle can be seen on the table, well as bowl and spoon, Additionally there are two books placed on the table, one closer the loft side and the other near the center. A remotc control also present on the table, slightly to the left the center.
LLVA OPPRRA
inside cup. appears to drinking from the cup, which is placed near the bod. The cat is positionced in the middle of the bed, and the cup is towards
image features
standing
Inpot image
LLMV
-OPRRA
bed. The captures the curiosity and playfulness as
Ours
interacts with the
Figure A6: Additional qualitative results of our method on LLaVA-1.5-7B with OPERA. On the CHAIR dataset, our method effectively reduces object hallucinations, preventing the model from generating con-existent objects in the image. Hallucinated object words highlighted in red.
37

---

Please describe the image detail.
USER
The image features small wooden boot. possibly canoe or sailboat, floating on body of water near dock. The boat positioned towards the center the soenc, surrounded by calm waters, here pier visible nearby, with several benches located around the arca. In addition the beat and benches, there two diming tables and chairs placed gatherings. The secne suggests Icisurely, peacefal sctting, where people can enjoy spending time.
LLaVA
pier. likely for people enjoy their meals
The image shows a small white bost sitting in large body of water. The best appears be floating ncar dock and is surrounded other boots the water. Some of these other boats are positioned closer the shore, while others are further out in the water. The socne captures a screme and peaceful almosphere with the bost calmly resting in the water.
Inpur image
Ours
Please describe the image in detail.
USER
image features a girl wearing a purple scarf cating hot dog with mustard on it. She is biting into hot dog while holding the hun in her hand, taking big hite. There are couple of bortles placed near the odge of the frame, and dining table can seen in background. Another person visible the hackgroundd but they are not the main focus of the image. Overall, the scene captures casual and enjoysble moment of someone savoring a delicious dog.
VCD
image features yourge girl wearing red jacket to take a bite our of. She is holding the hor dog in her hand, which appears to be wrapped a bun and oovered
standing
dog she about
LLaVA
Input image
VCD
girl smiling she gets ready enjoy her meal.
Please describe the image in detail.
commuter train traveling above ground curved hridge elevated railway. The train is long, with several cars visible, catrying possengers through the city. moving under cloudy sky, creating moody atmosphere. Apart the left side of the scene, and stop sign can be elevated milway. overall scene captures essence urban transportation, the continues to provide efficient travel for the city's residents.
image depicts
tmin, there traffie light the right side, indicating road crossing beneath
LLVVA
VCD
on a of its joursey. The train metion, captaring its movement through long stretch of the track. The train appears an urban mansportationn vehicle, interesting sight behold.
moving along track. train traveling across bridge, giving clear
Ingut image
ELaVA
NCD
Please describe the image detail.
USER
The image displays plate filled with delicious sandwich and a fresh salad. The sandwich served toasted bun and festures meat, possibly turkey, choese, creating a mouth-watering combination. The salad consists of red cabbage, adding colorful and crunchy element the dish. There numerous picces broceoli visible the salad, seattered across the plate, making nutritious and appetizing meal. In addition the sandwich. the plate also includes small portion of donut, providing sweet toach to the meal.
LLVVA
The image displays a sandwich and salad on white plate. The sandwich appears to be burger or panini, with slice of cheese in the center. The salad consists of variety of leafy greens and pickle, making for a visually appealing and healthy meal. The dish ready to be enjoyed, offering combination of flavors and textures for delicious dining experience.
Input image
LLaNA NCD
Figure A7: Additional qualitative results of our method on LLaVA-1.5-7B with VCD. On the CHAIR dataset, our method effectively reduces object hallucinations, preventing the model from generating non-existent objects in the image. Hallucinated object words are highlighted in red.
38

---

Please describe the image in detail.
USER
woman standing front of display cupcakes, possibly at bakery or She is smiling and appcars to be enjoying the event. The woman is wearing an apron and surrounded by various cupeakes.
image
scene, some placed on rablle The woman is standing in front of the cupeakes, and she seems to be enjoying the event. The scene room with dining table and chairs nearby.
are sevenal cupcakes
others on
LAVA
The image features waman standing in front of display of cupeakees. She is wearing an agron and bakery. The
standing front display of cupeakes, which are arranged in a tower-like formation. There are several cupeakes on display, with various flavors and toppings. The preparing them. Te seene is set in hakery or similar environment, with a variety of cupeakes and a woman wearing spron.
appears baker
Input image
PAI
slanding
the display, possibly admiring the cupeakes or possibly
Ours
Please describe the image in detail.
The image coptures skateboarder perfomning a trick in the air, with crowd of people watching the impressive stunt. skateboarder the middle scene, with skatehoard sevenal people in the soene, with some standing closer the skatebainder and others further away, woman standing near skateboarder, while people watching performance. onlookers are sprend out across the scene, with some standing closer to the skatehourder and others further away. the background, there dining table, and chair visible the scene, skatehoarder's impressive trick has captured the attentinn of the onlookers, making memorable moment the image.
are
LaVV
Ingut image
The image captures a skateboarder performing trick in the air, with a crowd of people watching the impressive stunt. skatebourder the middle of scene, him, with some standing cleser to the skatcboarder and others further away. There are several people in the scene, some standing closer skatcboarder and others further away. person holding skatcboard, and the skateboarder the air, performing trick. The socne is tilled with excitement and anticipation as the skateboarder showcases skills.
spectators spread out around
LLaVA
PAI
Owns
Please describe the image in detail.
USER
The image features cozy living with television sitting furnished with a couch, chair, and television. The couch is placed in the middle of the room, while the chair situated the left side the room. The television placed the wooden stand, and turned on. In the room, there is also dining table with vase on it, and cup on the rable. The vase placed the table, and the cup simuated nearby. The appears relaxation and entertainment
a wooden stand. The is
PAI
comfortable and inviting space for
image feutures cozy living zoom television, couch, chair. television is placed on wooden stand, and is turned on. The room also contains a chair and couch, creating cortortable and invicing atmosphere.
Imput image
LLaVA
situated nearby. The couch is placed in the room, and the chair is situated close it. The room appears be a comfortable space for relaxation entertainment.
the room,
wooden stand,
PAI
Owns
Please describe the image detail.
USER
The image shows man brown shirt cutting lange sub sandwich cutting hoard in kitchen. He using knife to cut the sandwich. which placed on a cutting board. The man focused on his task. and the kitchen appears be well-equioped with various kitchen items. In the kitchen, there refrigerator, a microwave, a sink, and a knife, A hottle is also visible in the scene. A person is standing nearby, possibly observing the man's actions wuiring for their
LLVV
the sandwich. The kitchen is well-equipped, and the man appears to he the main focus of the scene.
The image shows man standing in kitchen, preparing a sandwich on cutting bourd. He is using knife cut the sandwich, which is placed cutting board. The man is focused on his task, and be is wearing brown shirt. In the kitchen, there are various items, such as bottle of soda, cup, and knife. The cup placed the counter, while the knife located neur the sandwich. The man is surrounded by these items, indicating busy kitchen environment.
Input image
LLAMA
Ours
Figure A8: Additional qualitative results of our method on LLV55--5--BB with PAI. On the CHAIR dataset, our method effectively reduces object hallucinations, preventing the model from generating non-existent objects in the image. Hallucinated object words are highlighted in red.
39

---

Please describe the image in detail.
USER
The image features surrounded by various kitchen utensils, including a microwave, a touster, and a stove. The man is standing in front of microwave oven, which locared the side the kitchen There multiple knives and spoons in the kitchen, with some of them placed on a countertop. The knives are scartered around the kitchen, with them being placed man appears to be enjoying his time in the kitchen, surrounded by various utensils and appliances.
standing a kitchen, posing with smile and holding his hands out. He is
LLVVA
Devils
counteriop and others the microwave. The
man is standing in front of'a wall with pots and on it He is holding his hand out to the side, and be man wearing a gra and is standing front wall with pots and
posing for a picture. on it. The pots and pans arranged in way that makes il easy for the man to with them. The man is also holding a spoon in his hand, which is placed ea the wall.
Impur image
LL Devils
Plense describe the image in detail.
USER
The image shows small kitchen with a white reffigerator and a white oven. The reffigerator is located on the right side of the kitchen, while the oven the left side. The kitchen is equigped with sink, mierowave, and a toaster oven, which is placed on the counter. The kitchen also features toaster oven, a sink, and refrigerator. The refrigerator positioned counter. The kitchen is well-oquipped with various appliances, making a functional and efficient space.
LLaVA Devils
the and the sink located the
The image shows small kitchen with white refrigenator, white stove, and a wooden cabinet, The refrigeratoe white, and the stove white and bluck. The kitchen is small and compaci, wilh white refigerator and wooden cabinet. The reffigerator pasitionedd next to the stove, and the cabinet placod in front of The kitchen is well-organizod, with white rofrigerator and wooden cabinct.
LLaVA
Ingut image
Devils
Owrs
Please deseribe the image detail.
USER
image fentures two cats sitting suitcase, likely on or chair. cats comfortably
resting in the suitcase, which placed table. The cals positionned nexi to each other, with
and other the right side cats scem be onjoying ing their time logether the suitease. The table is surrounded by chairs, with chair on the left side and another on the right side. The cats are positioned in such a way that they are both silling the suileasc, creating cncy and comfortable environncct for them.
on the left
suitease. The suitcase is placed on table, and
LLaVA
Devile
The image features two cats sitting top a suitease, which is placed on chair, The cats are both grey white, they appear sleeping resting comfortally. The suitcase is in color, and cats are pasitioned such that they are almost completely covered by the suitense. The cats are sitting
Ingut image
LLaNA Devils
Owrs
their positioning suggests that they are enjoying a cozy spot.
Please describe the image detail.
USER
The image features a motorcycle parked en the side of road, with beautiful mountain in the background. motorcycle is parked their journey. The motorcycle is parked on the side of the road, and the rider is likely enjoying the scenic view the Mountain National Forest, motorcyele parked on side of the road, the rider is taking a break from their journey.
side of road, the rider likely taking break
LLvV
Devile
The image fentures red motorcyele parked on the side of road, with bemitiful mountain in the background. The molorcydle is parked on the side of the road, and the sumounddnng area is filled with greenery, including trees and grass. The motoreyele is positioned on the side the road, and the mountain can be soen the distance. The scene set a picturesque location, with the motoreycle parked on the side of the road, and the mountain the background.
Input image
LLaVA Devils
+ Ouns
Figure A9: Additional qualitative results of our method on LLaVA-1.5-7B with Devils. On the CHAIR dataset, our method effectively reduces object hallucinations, preventing the model from generating non-existent objects in the image. Hallucinated object words are highlighted in red.
40

---

Is cining image??
there image!
USER
USER
Yes, there is dining table the image, and sitting at while playing video game.
Yes, there is the imuge, and the man
playing video gume
LLVV
LLaVA
No, there is no dining table in the image. The holding wii
No, lhere the The shows man holding a remote and smillng.
LLaVA
smiling.
LLaVi
Input image
backpock image?
motorcycle image?
Yes, there is backpack in the image, and it is located the double
Yes, there is motorcycle in the inage, and is parked the red Souble-dccker bus.
LLaVA
LLAMA
No, there backpack in the image.
image festures red double-dccker bus.
No, motorcyele image.
Inpui image
Ours
Is there handbag in the image?
Is there skatehhard in image?
USER
USER
Yes, there is handbag in the image, and it is placed the ground person sitting the sidewalk.
Yes, there is skateboard in the image, and is keaning against the yellow shack.
No, there is no handbag in the image.
No, there no skateboard the image.
LLaVA Ours
LLaVA + Ours
Inpor image
Is there bottle the image?
cup in image?
Yes, there is bottle in the image, which is locuted
Yes, there is cup in the image, placed on the
LLaVA
LLaVA
No, there bottle the image. The image Jeatures a desk with a laptop, mouse, and a monilor.
No, thare in the image. The
Retures a desk with a laptop. monitor, and
Inpui image
Ours
Ours
Figure A10: Additional qualitative results of method on LLaVA-1.5-7B with greedy decoding. On the POPE dataset, our method correctly identifies objects present in the image. Correct and incorrect answers are highlighted in blue and red, respectively.
41

---

pollied image?
there tie the image??
Yes, there is tie the image, which is hanging the door.
No, there no potted plant in the image
LL OPERA
ELaVA
OPERA
potted plant imazee.
image..
LLAV
LLaVA
Input image
image?
image?
Yes, there is bus in the background of the image.
Yes, there is chair the image.
LLaVA
LLaVA
OPERA
OPERA
image.
image.
LLaVA
LLaWA
Input image
OPERA
Owrss
Ours
there mouse the image!?
Is there cup in image?
No, there the image.
Yes, there cup the desk the
LLw
LLWV
Yes, there is mouse the image.
No, there no cup in image.
Input image
OPERA
OPERA Ours
Ouars
there image?
there bottle in image?
USER
USER
Yes, there is car in image, and the man and woman standing in front of it.
No, there a no bottle in the image.
LLaVA OPERA
LLaVA
OPERA
No, there no car the image. The image
fentures and standing each other.
there bottle in image.
LLaVA
LLAV
Input image
OPERA
OPERA
Figure A11: Additional qualitative results of method LLaVA-1.5-7B with OPERA. On the POPE dataset, our method correctly identifies objects present in the image. Correct and incorrect answers highlighted in blue and red, respectively.
42

---

Is there dining table imnge?
there cup the image?
USER
USSER
Yes, dining tuble image, allong with surihoards, backpacks, and people standing stop.
Yes, there is cup in the image, hold by one of the people getting off the bus.
LLavA
LLAVA VCD
VCD
No, there no dining table in the image. The imaaee features public transmil bus with man standing next to it.
No, cup image.
LLaVA
LLaVA
VCD
Imput imuge
dining image?
image?
Yes, coach image, cat sitting on top of it next the computer keyboard.
Yes, there is dining table in the image with computer keyboard
LLaVA
LLMV
No, there is dining table in the image. The image keyhoard sining next to it.
there couch in image. The image features a computer keyboard and cat.
HaVA
Input image
Ours
Ours
Is there cur in the image?
Is there cup the image?
USER
USER
Yes, there car the image, parked in the background the living room.
Yes, there is cup in the image. is placed top a coffee living room.
LavA
LLVA
image. features living room with flax sureen tv and window.
No, there no cup in image.
LaVA
Inpui image
VCD
VCD
Ours
+ Owrs
backpack image?
handhag image?
Yes, there backpack the image, placod near the tablc where the cake and coffee savved.
Yes, there is handhagg the image, located
table where the cutting cake.
1LaVA
LLaVA
No, there is nn backpack in the image. The
No, handhag image.
LLaVA
table.
Input image
Ours
Ours
Figure A12: Additional qualitative results of our method on LLa with VCD. On the POPE dataset. our method correctly identifies objects present in the image. Correct and incorrect answers highlighted in blue and red, respectively.
43

---

Is there motoreyele the image?
Is there dining table image?
USER
USSER
Yes, there is motoreycle cle parked next to bus the image.
Yes, there is dining table the image.
iPod
LLaVA
LLVV
PAI
there motorcycle the image.
No, there dining table the image.
LLaVA
LLaVA
Imput imuge
dining image?
image?
Yes, there is chair the image, which placod the old-fashioned
Yes, there dining table the
LLaVA
LLaVA
No, dining image.
No,
image.
LaVA
Input imago
Ours
Ours
Is there dining table in image?
there bicycle in the image?
USER
USER
Yes, there dining table the image.
Yes, there bicycle in image
LLVVA
No, there is mo dining table in the image.
No, there no bicycle in the image.
ILVA
LLaVA
Inpur image
PAI
PAI
Ours
+ Owrs
image?
cup image?
Yes, there is car the image, which parked of computer
Yes, there is cup sitting on the desk in the image.
LLaVA
LLaVA
image.
No, cup image.
LLaVA
Input
Ours
Ouurs
Figure A13: Additional qualitative results of our method on LLaVA-1.5-7B with PAI. the POPE dataset, our method correctly identifies objects present in the image. Correct and incorrect answers are highlighted in blue and red, respectively.
44

---

Is there skis image?
there the image?
USER
USSR
Yes, there is skis display in the image, which is located the airport.
Yes, there is car in image, which parked the sirport parking lot.
LLAVA
LLAVA
Devils
Devis
No, there are no skis in the image. The image shows people walking through airport with their lageage.
No,
image.
LLaVA
LLaVA
Devilk
Imput imuge
dining image?
image?
Yes, there is dining table in the image, and sitting on top
Yes, there is couch in the image, and the cat is sitting on top
LLaVA
LLaVA
No, there is dining table in the image. The computer
No, there couch in image. The
Lav
image features keyboard.
features a cat next to computer keybaard.
Input image
Devils
Ours
Owrs
Is there handbag in image?
Is there truck in the image?
USER
USER
Yes, there handbag the image, and the child holding
Yes, there is truek the image, and the child truck.
is sitting
LLVA
handbug image. image shows a child in car seat holding toy.
image. image feuturrs a child in seat, holding toy.
ILVA
LLaVA Devils
Inpur image
Devils
Ours
Owrs
dining image?
Is
image?
Yes, there is dining table in the image, which located bad.
Yes, there is tv the image, which located the right side the bed.
LLaVA
LLaVA
there dining table the The image shows a hed with colorful quilt on it.
No,
image.
LLaVA
Input image
Ours
Ours
Figure A14: Additional qualitative results of our method LLaVA-L15-7B with Devils. On the POPE dataset, our method correctly identifies objects present in the image. Correct and incorrect answers highlighted in blue and red, respectively.
45

---

Please describe the image in detail.
E
USER
The image fearures a man sitring on the floor in front of a minor, taking picture of himself. He is wearing a green shirt and appears be comfortable as he captures the moment. The man is using cell phone take the picture, and there is hundhag nearty on the floor. In the room, couch can be seen in the background, occupying a significant portion of the space. here are also two vases placed at different spots the room, akding decorative elements to the secne.
Shilza
sitting the floor in front of mirror. He taking picture of himself using cell phone, cupturing the reflection in the mirror. The man is wearing green shirt and shorts, which can be seen clearly the mirror. The primary focus is the taking picture of himself through the mirrrr.
The image festures
Shikra
Input image
+ Ours
Please describe image detail.
USER
The image shows a man standing in a grassy field, swinging hasehall hat with baseball glove on the other hand. He uppears he enjoying his time ourdoors, actively enguged game baseball. The field is illuminated by a light in the background, creating a warm and inviting atmosphere for the gume. Around the field, there are few cars parked distance, suggesting that field locsled in public where people can gather and play sports.
Shikaz
The image features young playing frishoe grassy field. He throw ing motion, holding the frisbee with his right hand while his left hand is positionee behind him. There are several other people in the background, possibly enjoying the ouldoors participating the The seene appears be during the evening or night, with the grassy field being illuminated. creating a lively atmosphere.
Shikra
Input image
+ Ours
Please describe the image in detail.
USER
image features woman sitting on chair with looking at cell phone, which she is holding in her other hand. The chair is positioned in way that allows to comfortably check her phone while relaxing. Additionally,
racket hand. the process
few books placed neurby, suggesting that the woman might be in casual and comfortable setting. possibly at home public space with bookshelf. The seane captures moment leisure or relaxation
Shikaz
woman.
The image features a woman sitting in a chair with tennis racket her hand. Sbo appears be wearing glasses and likely dressed in casual oulfit. The woman is holding the teanis racket in front of her, possibly in anticipation of playing or warming up for a game.
Input image
Ours
Please deseribe the image in detail.
USER
The image features acrial view of large truck driving down street. The truck filled with various ilems, inclading cow, trailer full of containers, and numeroou garbage-fille fifllee barrels. The cow can occupying a significant postion of the back of the truck, while the burrels scattered throughout the trock hed. person is visible near the truck, possibly observing the scene involved the transportation process. The image captures the essence of busy street with truck carrying diverse lond.
Shika
The image depicts large truck driving down a street, its cargo compariment filled with various items. seen, likely being transported to farm another location. The truck's
the back the truck, side displays mix of different materials, including two large recls of rope and a crushed can. Additionally, there is blue hamell the truck, which contributes the diverse load carries. The muck appears to be moving al a moderate speed, and its cargo compariment is quite full, showcasing the diverse items carries.
Shikes
Impur image
+ Ours
Please describe the image in detail.
The image features group of men in uniform, working together in navy ship's control room. They are wenring headphhnee sunglasses, focusing on their tasks they operate various communication equipment. hhere are least live individuals the room, with some sitting table and others standing near windows. A laptop placed bandheld radio on his shoulder. The room equipped with TV, which is visible in the background. Additionally, a book can be seen on the tablc, possibly a reference material or guide for the tasks they are performing.
Stilee
front the men, and one
individuals using
The image features group of men in navy uniforms, working logether in control room. They are using beadphones and microphons to communicate with each other, with some of them also wearing sunglasses. There laptop in the diligenlly, likely engaged in their tasks, as they navigate the control room and communicate with one another.
Input image
Shikra + Ours
placed the left side the room. The working
Figure A15: Additional qualitative results of our method on Shikra-7B with greedy decoding. On the CHAIR dataset, our method effectively reduces object hallucinations, preventing the model from generating non- con-existent objects in the image. Hallucinated object words are highlighted in red.
46

---

Please describe the image detaill
USER
image depicts cory scene positioned towards the ieft side of the saitcase, while the other cat is the right side. They appear to be relaxed cornoortable their shared luggage. In addition
grey lying
black
suitcase, there are few other objects in the scene. A laptop can be spotted on the left side of the image, while chair is dining and tabletap
Shiisa
OPERA
right
visible
The image depicts cazy with on the left side the suitease, while the other cat on the right side. They appear to he relaxing and enjoying ench other's compuny in luggage. the background, chair partially visible, situnted behind the suitcase where the cats are sitting.
sitting together inside black suitease. One
Shikra
Input image
Please describe the imaaee detaill.
The features sitting the flour in frunt of mirror, picture of himself' with digital camera. He wearing green shirt and appears to be comfortable us captures the moment with the atmosshere with couch the background the right and two chairs nearby. dog can be seen resting or relaxing on the floor, located near the man's right hand side the lower bower-height perspective of rug wooden floor.
The room has
Shikm
OPERA
floor front mirror. He taking of himself using cell phone, capturing the reflection in the minorr The man wearing green shirt and shorts, giving and relaxed vibe the
The image features
Shiira
Input image
OPERA
Ours
Please describe the detail.
image features
building prominent
clock positioned
the center of the building, making casily visible to passersby. The building's white facades give il modern uppearance. sidc, as well as cle on the left side of the soanc. These additional objects add to the overall
vicinity building, motorcycle
Shikra
stmosphere
positioned near the center of the building, making casily visible to passersby. The building's white color adds cleanliness simplicity seene.
image white building large
Shreen
OPERA
Inpur image
Please deseribe imago detail.
USER
image depicts pencefull scene viewpoint overlooking large river flowing through a valley.
single wooden bench is placed at the edge of the hill, providing comfortable searing area for visitors to scenic surronddngss
backgraund, bridge can be crossing adding to the picturesque setting. Near the bench, hundhag and hackpack are placed, likely belonging enjoying walk
relax
The Reatures weoden bench sinuated the top of bill, overlooking besutiful lendscape that includes a river and bridge. The bench facing the soanic view, offering a picturesque spot to sil and enjoy the view.
Inpux image
OPERA Ours
Please descrihe the image detail.
USER
on sink with mirror ahove it, which accompanied by vanity. On the countertop, there is bottle of wine plazed near sink. with matching red bath mat. The overall color scheme of the bathroom predomisantly white, giving elegant appearance.
image features large
saparute
hathruh shower curtain, bathrom includes
Shikra
OPERA
The image fentures a clean bathroom with a sink and a bathrab. The sink has large mirror above and accompanied
Shikm
OPERA
curtain. hathroom appears spotless well-maintained whole.
Input image
Ours
Figure A16: Additional qualitative results of our method on Shikra-7B with OPERA. On the CHAIR dataset, our method effectively reduces object hallucinations, preventing the model from generating non-existent objects in the image. Hallucinated object words are highlighted in red.
47

---

Please describe the image in detail.
The image depicts scenic view of river flowing throughs valley, with wooden bench situsted grussy hill overlooking the surrounding landsaaee The bench located at pleusant viewpoint, offening a great vantage point of the river and the hills beyond. In scene, there boat visible the river, likely enjoying the calm waters as sails along the valley. Additionally, train can he seen on the tracks, following the path the river, adding another element of interest the picturesque laadscape.
Shilea
VCD
The secoe fentures wooden beneh sitting the edge grassy hill, overlooking scenic view includes river running through valley. The bench facing the water, offering picturesque and relaxing spot sit and lake in sucroundings.
Shikes
Input image
VCD
Please describe the image in detail.
USER
The image depicts a serene ouidoce scene featuring locae pine free growing next to wooden bench, The bench is positioned
elearing beneath the tree, providing resting spot for visitors passers-by. The area seems to be quiel, piciuresque location, possibly overlooking a bcautiful landscape, such as valley town below. There few people scattered around the seene, likely enjoying the tranquility of the lucation. Some are close the bench, while others are closer to the free or slanding at slight distance. A handbag
Shikra
VCD
the ground near the bench, possibly belonging
of the visiturs.
beautiful pine with a wooden bench positioned right underneath it. The tall and slender, reaching lowards the sky. It appears to be a sunny day, casting light on the tree and beach. making
The image
Shikea
close the providing a eocy spot and enjoy the view. The landscape in the background censists of mountain, which can be seen pecring the There also a patch of grass next tree bench.
scene, The bench
Input image
Ours
Plense describe the image detail.
USER
ONE WA
The image captures
with one-way sign pointing the right. A wall with graffiti visible the background, adding an artistic touch to the urban landseape. speny paint can be seen as part the graffiri, highlighting the creativity present the In the foreground, there bench providing place for people to sit and observe the surroundings Several cars are parked ocarby, with two located on the right side of the sozne one closer the the image.
Shikes
fire hydrant also present close
The image features one-way sign positiened next wall. wall. there a graffiti drawing of hund, udding unique decorative touch to the signage. In the background, bench can be seen side of scene.
Input image
VCD
Owrs
Please descrihe the image in detail.
USER
The image feutures black and white sitting wooden dining table, intently inspecting wooden bowl. The bowl has ball of yam or a skein of string in catching the cat's attention. The cat appears to be eating playing with the ball, background, away from the yarn-filled bowl, making the seene look like could be part of playful cat- oriented environment.
closer and closer to it. A second cat
in the
Shikza
The image features black and while cat sitting on a dining table, curiously inspecting a weoden bowl. The is the locus of the scene, occupying significant portion of the trame, The bowl, which is empty, placed near the cat, drawing its attention. The table serves as surface for the cat to explore, as it looks the bowl with interest.
Shikrs
Input image
VCD
Figure A17: Additional qualitative results of our method Shikra-7B with VCD. On the CHAIR dataset, our method effectively reduces object hallucinations, preventing the model from generating non-existent objects in the image. Hallucinated object words are highlighted in red.
48

---

Please deseribe the image detail.
USER
image shows man brown shirt, standing in a kitchen and preparing a sandwich countertop. He the process of elicing large sandwich, which appears be sub sandwich. There are sevenl ttems on the countertop. including a bottle, a cup, a knife, and poir of scissors. microwave also visible the background. The scene depicts man working sandwich the kitchen. suggesting that he might be preparing a meal or snack.
Shikaa
image shows man standing kitchen. preparing sandwich by cutting large sandwich half. Ile appears to be making sandwich with knife, standing at a kitchen counter. The sandwich placed on cutting hoard,
several ingredients visible counter, saggesting variety ingredients to he used in the sundwich. The man seems to be focused on the sandwich, and he is likely in the process preparing
Input image
Durs
Please descrihe the image in detail.
USER
The image shows white bowls filled with creamy contain broccoli soup or chowder, with small portions of broccoli in it. There are also two large bowls placed cutting bourd, adding the scene. In the image,
placed on dining table. The bowls
Shikra
seen resting the bowls, suggesting that the soup or sauce could be served as side dish or a light meal. The presence of fork alsu indicates that the
PAI
in dining where people enjoy their meal.
The image shows two while bowls filled wilh creamy soup containing broccoli and other vegelables. The bowls are placed on weeden dining table, and there is a spoon resting near them. The dining table is set in the display the buwls and the soup.
Shikes
Input image
PM
Owrs
Please descrihe the image in detail.
USER
The image shows white bowl filled with green apples, which are placod white bowl, There are several apples in the bowl, including mix of green and rod apples. The bowl is filled with mix of green apples, with some red apples as well. The green apples are arranged various positions and oricalations, creating an interesting composition. The apples are placed white bowl, which is placod dining table.
Shikea
The image shows close-up of bowl filled with numerous green apples, which placed white bowl. There are various apples in the bowl, with some positionedd the front and others the back. The apples araaged different and creating visually appealing composition. The bowl filled with several apples, some which are stacked on lop a smaller apples, creating a vibrant green color palette.
Input image
Ours
Please describe the image in detail.
USER
The image shows park with display of many colorful umbrellas hanging the air, creating vibrunt and lively atmosphere. These umbrellas are suspended upside down, with their unique positioning creating arristic and visually appealing closer to the ground and others higher up the trees. There are several people in the seene, possibly admiring the display participating it. The inage, providing a place for people to sit and onjoy the view of the colorful umbrellas above them.
The umbrellas spread throughout the park, with
Shikma
also includes beneh located in the middle of
The image shows park with display of many coloriul umbeellas hanging from trees. These umbrellas are arranged in a forest area, creating a visually uppealing and artistic scene. The umbrellas are of various sizes and are spread throughout the forest. The vibrant colors of the umbrellas contrast with the green foliage of the trees the background, adding unique and lively touch the landscape.
Input image
Shitra PAT
+ Ours
Figure A18: Additional qualitative results of our method on Shikra-7B with PAI. On the CHAIR dataset, our method effectively reduces object hallucinations, preventing the model from generating non-existent objects in the image. Hallucinated object words are highlighted in red.
49

---

Please deserihe the image detail.
USER
image features wonden dining table table, there are wine glasses and cups, indicating wine testing event. The table is filled with numerous wine plasses bottles, emphasizing the guests' experience. The table set up as wine tasting event,
bottles placed on Surounding the
Stiken
with the purpose of providing an enjoyable stmosphere for the attaddoss. The table has total of seven wine bottles and six wine glasses placed on showessing variety of wines tasted. The tuble also his and a potted plant docoration neadby, adding touch of greeucty and elegance the sctting.
Devils
The image features wooden dining table with various bottles of wine and wine glasses placed on it. The table set up like with muliiple wine bottles wine glasses arranged neatly. table is surrounded by collection of wine bottles and wine glasses, creating atmosphere of reluxation and enjeyment.
Input image
Shilea
Devils
Ours
Please descrihe the image detail.
USER
silling park bench surrounded large flock of pigsons, several birds eagorly gathering around him. The man is wearing baseball cap, and it scems that be the center attention this svene. In the background, indicating that the scene takes place in an urban setting. There are also few cars and truck parked neurby, adding the overall atmosphere the scene.
image features
Shikra
multiple cars and few truffic lights visible,
Devils
The scene features man sitting on park bench surrounded by a large flock of pigeons, with several The man wearing bsscball cap, and birds are gathered around him, with some birds also the bench next to him. In the image, there are numereus pigeons present, with a mix birds surrounding ing the man. on the bench next to him. The man is likely professional bascball player, and the birds are likely attracted to him as result of their presence the scenc.
Shiken
birds taking liking to
Inpur image
Devils
Ours
birds are eager to get close the man, and some are even perehedd
Please descrihe the image in detail.
USER
The image showeases laptops, one smaller and another larger, sitting white desk in The smaller luptop is placed in front of the larger laptop, while the lurer luptop situated hehind it. The two Inptops connected music or other entertainment purposes. In the background there is clock on the wall, adding decocative touch the room. Four people two of them closer to the foreground and the other two further away. Additionally, there are two cell phones the secnc, locsled closer the larger laptop and the olher near the smaller laptop. clock can be seen in the image. The setup creates an atmosphere of technology and leisure in the room.
keyboard and cell phone, indienting that they might be used for streaming
Shikaa
Devils
be seea various positions throughout the image, with
Input image
The image features laptops placed top of white dask. laptop pasitioned closer the left side of the room, while the other laptop is located towards the right side of the room. The wall behind the laptops a picture banging the wall.
Shikra Devils
Please describe the image in detail.
USER
The image features a heautifully deconated hirthay cake with heach theme. The cake is placeri on table, and has name on it, 'Happy Birthday' written on the front. cake decorated combination of lettering and palm trees, which makes it unique and creative design. The cake displayed on diainn table, and it's available an additional decoration centerpicce for 50th birtsday celebention. The cake placed dining table, and it's a grent addition to the event. The cake covered with blue tablecloth, adding festive touch the overall theme.
Shin Devils
Sumehar Bahany
image features hirthday cake cake is decorated with blue hackground, and the words "Happy Birthday" is written on top.
theme, complete surfboard palm trees. The
Input image
Shitra
Devils
Ours
Figure A19: Additional qualitative results of our method on Shikra-7B with Devils. On the CHAIR dataset, our method effectively reduces object hallucinations, preventing the model from generating non-existent objects in the image. Hallucinated object words are highlighted in red.
50

---

bicycle image?
there traffic image?
USER
USER
Yes, there is traffii light in the image, and motorzycle riding in front of bus.
Yes, there is bleycle in the image.
Shikra
Shikm
No, there is no bicycle in the image. The riding motorcycle front of bus.
No, there mo traffic light in the imagee. The photo man riding motorcycle. green bus, and road.
image
Stikiz
Shikrn
Input image
Is dining image?
there light image?
Yes, there is dining table in the image, and it located the middle
Yes, there is trafffic light visible in the image. stopped traffic light.
Stikm
Shikra
No, there dining table in the imsage. The image features large green and white bus with door open, parking the stroct.
The answer no.
Inpui image
Ours
Is there cup the image?
In there person in the image?
USER
USER
(Ca))
Yes, there bloody knife the and it is in form of fruit with knife in it.
Yes, there is in the who holding pair bloody knives.
No, there knife the image, there visible.
No, there a knife image, person visible.
Shikra Ours
Shikza Owrs
Input image
Is laptop in imags?
cell phone the image?
No, there is no luptop in the image. The image festures large computer setup with monitors on desk.
Yes, there is cell phone the image.
Stiken
Shitaa
Yes, laptop the image, to the desklop computer on the desk.
The answer no.
Shikra
Shikza
Inpui image
Ours
Owrs
Figure A20: Additional qualitative results of our method Shikra-7B with greedy decoding. On the POPE dataset, our method correctly identifies objects present in the image. Correct and incorrect answers highlighted in blue and red, respectively.
51

---

penson image??
dining imange?
Yes, there is dining tuble the image, which up the beach.
The nnswer no.
Shiker
Shiken OPERA
OPERA
Yes, there the image, standing the water the beach.
dining tablee imagee.
OPEIEA
OPPIRA
Input image
image?
cup image?
Yes, there book the bed the
Yes, there is in the image.
Shilrra OPERA
Shikra
OPERA
me.
no.
Shilka
Input image
OPERA
Owrss
Ours
there cup in image?
there chair in image?
Yes, there is bloody kniife the image, and hovening either the top half of apple or severed head.
Yes, there is chair the linage, and has knife
The answerr no.
The answer no.
Input image
OPERA
OPERA
Ouars
Ours
there bed the image?
there sports ball in image?
USER
USER
(No answer)
Yes, there is sports ball in the image.
Shikra
Shikro
OPERA
OPERA
there bed the
The
Shikra
Shrkra
Input image
OPERA
OPERA
Figure A21: Additional qualitative results of our method Shikra-7B with OPERA. On the POPE dataset, our method correctly identifies objects present the image. Correct and incorrect answers highlighted in blue and red, respectively.
52

---

Is there couch in image?
there cup the image?
USER
USEE
Yes, there is couch in the image.
Yes, there is cup in the image.
Shikza
Shikra
VCD
VCD
There are two vases with Dowers the image, The placed weoden lable or shelf.
No, there in the image. There
Shiken
Shikm
VCD
only two flower vases.
Imput imuge
dining image?
traffic light image?
image, sittnng the street next sidewalk,, There also traffic light visible in background.
Yes, there dining table the
Stikm
Shitra
The image does not show dining table.
The image does show traffic light, there is no traffic light the image.
street
city.
Shiken
Inpus image
Ours
Ours
Is there penson the image?
there track the image?
USER
USER
Yes, there person walking ocar the traffic light the inage.
The
The unswer is no.
The answer is yes.
Shikza
Shikka
Inpur image
VCD
VCD
Ours
+ Owrs
image?
image?
Yes, there car the imago, miniature placed near the bed in the room.
(No answer)
Shikm
Shitra
Yes, bed in image, and appears
Shikes
answer
to situated in hedroam.
Ours
Ours
Figure A22: Additional qualitative results of our method on Shikra-7B with VCD. On the POPE dataset, our method correctly identifies objects present in the image. Correct and incorrect answers highlighted in blue and red, respectively.
53

---

is there refrigerator the image?
Is there dining table image?
USER
USSR
Yes, there is dining table the middle of the the image.
Yes, there is refrigerated truck in the image.
Shikra
Stikrn
PAI
No, there diming table the imsgee. The answer no.
The
Shiken
Shikm Ours
Imput imuge
image?
potted plant image?
Yes, there is large group of people entering sirplanc.
Yes, there is large puss the middle of the picture.
Stikm
Shitra
there bus the image. The
No, there potted plant the image. The answer mo.
no.
Input imago
Ours
Ours
Is there laptop in image?
Is there cell phone in the image?
USER
USER
No, there taptop in the image. So the answer no.
Yes, there is keyboard on the desk the image.
Yes, there laptop in the image.
The answer is mo.
Shikea PAI Ours
Shikaa
Inpur image
PAI
+ Owrs
dining table image?
image?
Yes, there bus the image.
The answer is yes.
Shikm
Shitra
answer
anower
Input image
Ours
Ours
Figure A23: Additional qualitative results of our method on Shikra-7B with PAI On the POPE dataset, our method correctly identifies objects present in the image. Correct and incorrect answers highlighted in blue and red, respectively.
54

---

Is there chair in image?
Is there dining table image?
USER
USSER
The answer yes.
Yes, there is dining table the image.
Shikra
Shilkn
Devils
Devils
The does not display dining table. the answer mo0
The
Shiken
Devils
Shikrn Devile
Imput imuge
dining table image?
image?
Yes, there is group of people silling in bost
Yes, there is one.
the
Stikm
Shitra
The image provide dining table. the answer mo.
anewer
Input imago
Devils
Ours
Ours
Is there car the image?
Is there couch in image?
USER
USER
The yes.
The yes.
Shikea
image. So the unswer no.
answer
car
Shikaa
The answer mo.
Inpur image
Devils
Devils
Ours
Owrs
traffic light image?
dining image?
Dech AEAAAA day Triai
Yes, there is dining table with pile of apples the image.
Yes, there large pile apples the image.
Shikm
Shitra
there lot apples front the shop window.
No, there diining table the So the answer no.
Input image
Ours
Ours
Figure A24: Additional qualitative results of our method on Shikra-7B with Devils. On the POPE dataset, our method correctly identifies objects present in the image. Correct and incorrect answers highlighted in blue and red, respectively.
55

---

Please describe the image detail.
USER
image depicts hedmomm with accupying a significunt purtion space. there is television turned on, displaying show. A handhag is placed on the hed, and a remote control is
nearby, various items. The dresser contains several books, while the shelf holds collection of hottles. A cell phone
hoth which
LLaVA
bed, and cup seen placad dresser. room appears cory and functioonal living
The image (catures bedroom with bad television placed
The bed
covered with striped blanket, and there are several books scattered around the room. A hindbag placed bed, remole two bottles in the reom, one located near the bed and the other en the dresser. A chair is also present in suggosts cozy
seen nearby. In oddition
and television, there
Input
the room, pasilioned comfortable living space.
overall atmusphere
Please describe the
detail.
USER
The image features man sitting couch in living room, holding a Wii remote in his hand. He appears he playing video game, passibly Nintendo console. wall, and there is chair located near the left side of the room. In the background, there are several seen in
positioned against
LLVA
scattered around, possibly indicating reading or relexation area. dining room, with a placed it Additionally, there cell phone resting on the table, and bottle is Jocated near the edge of table.
sitting couch. holding Wii remote bis band. He wearing shirt and appears to be enjoying some Icisurc time. The couch positioned living room, there are several books scattered around the Some books are placed shelf, while others the floor. chair is also visible in the room, situated near the couch.
The image fentures
Input image
Figure A25: Failure cases of our method on LLLVA--15-7B with greedy decoding, On the CHAIR dataset, method effectively reduces object hallucinations but fails completely prevent the generation of non-existent objects. Hallucinated object words are highlighted in red.
56


Reviews:
[{"review_id": "XiqnSsw5lp", "reviewer": "Reviewer_RPtF", "summary": "This paper addresses object hallucination in vision-language models (LVLMs), where models generate descriptions containing non-existent objects. The authors link this issue to uncertainty in visual tokens from the vision encoder and propose an efficient solution using adversarial perturbations to identify and mask uncertain tokens. Their method reduces hallucinations by 38% in benchmarks like CHAIR while maintaining output quality, and works 5x faster than traditional uncertainty estimation techniques. The approach is compatible with existing LVLM architectures and requires no additional training.", "questions": "please see my weakness part.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "----------- Strength -----------\n1.\tThe proposed inference-time masking strategy reduces object hallucinations without fine-tuning or additional data, preserving model capabilities while improving output reliability through selective suppression of uncertain visual tokens identified via adversarial perturbations.\n2.\tUnlike conventional approaches that frequently compromise output quality while addressing hallucinations, the proposed method achieves substantial reductions in spurious object generation while consistently maintaining—and in certain cases even enhancing—both the F1 metric performance and the overall descriptive accuracy of generated captions. This dual benefit of hallucination suppression and quality preservation stems from the targeted, uncertainty-aware masking mechanism that selectively filters only the most unreliable visual tokens without disrupting valid visual information flow.\n3.\tThe comprehensive evaluation framework encompasses multiple state-of-the-art LVLM architectures (LLaVA-1.5, Shikra, MiniGPT-4) and rigorously assesses performance across established benchmarks (CHAIR, POPE, AMBER), demonstrating consistent effectiveness in hallucination reduction while maintaining model performance across different architectural paradigms and evaluation metrics. This systematic validation confirms the method's architectural agnosticism and practical viability for real-world deployment scenarios.\n4.\tThe paper excels in clarity, presenting complex technical concepts through well-structured explanations and intuitive visualizations. It maintains a strong balance between mathematical rigor and accessibility, particularly in explaining how adversarial perturbations reveal epistemic uncertainty in visual tokens. The writing effectively guides readers through both theoretical foundations and practical implementations.\n\n----------- Weakness -----------\n1.\tWhile Theorem 3.2 shows a consistent relationship between feature changes caused by adversarial perturbations (‖Δz‖₂²) and the upper limit of visual token uncertainty — effectively identifying unstable, high-uncertainty tokens as key triggers for hallucinations — does this vision-focused approach truly capture all problematic visual tokens? Specifically, could the current method miss certain troublesome tokens because of: (1) language model biases (e.g., frequent object pairings) that don’t depend on visual uncertainty; (2) mismatches between visual and language information that are not caught by looking at visual changes alone; or (3) errors that happen later during decoding, beyond what visual-layer analysis can detect?\n2.\tGiven the observed reduction in efficacy when applied to Q-Former-based architectures like MiniGPT-4, how might the proposed uncertainty-aware masking strategy be adapted to effectively address the bottleneck created by intermediate query-based representation layers, which currently attenuate the impact of vision encoder modifications on final model outputs? Specifically, could jointly optimizing perturbations across both visual and query token spaces, or developing cross-modal uncertainty propagation mechanisms, help bridge this performance gap while maintaining the method's training-free advantages?\n3.\tWhile the proposed method effectively reduces object hallucinations as demonstrated on CHAIR and POPE benchmarks, does it simultaneously maintain robust performance on the comprehensive MME evaluation? Specifically, does the vision token masking approach preserve the model's essential visual perception and reasoning capabilities across MME's diverse tasks, particularly for fine-grained recognition and complex compositional understanding? This verification is crucial to ensure the hallucination mitigation does not compromise the model's fundamental multimodal generation quality.\n4.\tGiven the demonstrated efficacy of the proposed uncertainty-aware masking approach across several LVLM architectures, how might its performance generalize to state-of-the-art models like DeepSeek-VL, particularly considering potential differences in visual token representation distributions, cross-modal alignment mechanisms, and inherent uncertainty patterns that could influence both hallucination suppression effectiveness and core multimodal capabilities? This critical evaluation gap warrants investigation to fully establish the method's applicability across next-generation vision-language models.", "was_revised": true, "final_justification": "I'd like to thank the authors' response to my questions. In the rebuttal, some of my concerns have been addressed. Considering the overall quality, especially the method's performance generalize to state-of-the-art models like DeepSeek-VL, which is also raised by reviewer E6ux, I will keep my rating.", "rebuttal": "We sincerely thank the reviewer for the thoughtful feedback. We appreciate the recognition of our method’s effectiveness in reducing hallucinations without fine-tuning or extra data, while preserving or improving caption quality. We are also grateful for the acknowledgment of our uncertainty-aware masking strategy and the clarity of our presentation. We carefully address your all questions and concerns as follows and will incorporate them into our revision.\n\n---\n\n>*W1. Does this vision-focused approach truly capture all problematic visual tokens?*\n\nOur work assumes that both the vision encoder and language model contribute to object hallucination, and focuses specifically on the vision-side uncertainty—an underexplored but complementary factor.\n\nThe challenges you raise—(1) language priors, (2) vision-language misalignment, and (3) decoding errors—are well-known and addressed by existing methods. Rather than aiming to cover all causes, we reveal new dimension (4) **epistemic uncertainty in the vision encoder**, which we show is strongly correlated with hallucination.\n\nTheorem 3.2 and our empirical results demonstrate that visual tokens sensitive to adversarial perturbations—used as a proxy for uncertainty—are predictive of hallucinated content. Masking these tokens significantly reduces hallucinations without altering the language model.\n\nOur method is training-free, vision-only, and integrates well with language-side approaches like OPERA, PAI, and VCD. This plug-and-play nature makes it a practical and effective complement to broader hallucination mitigation strategies.\n\n---\n\n>*W2. Could jointly optimizing perturbations across both visual and query token spaces, or developing cross-modal uncertainty propagation mechanisms, help bridge this performance gap while maintaining the method's training-free advantages?*\n\nThank you for the insightful suggestion. Following your proposal, we conducted additional experiments where we jointly applied adversarial perturbations to both the input image and the 32 query tokens in MiniGPT-4. Based on the resulting uncertainty, we additionally masked 6 high-uncertainty query tokens in two cross-attention layers of the Q-Former. As shown in Table R1, this adaptation led to further improvements in performance. We appreciate your thoughtful idea and will include this result in the revised version.\n\n**Table R1.** Quantitative results on Q-Former masking with joint adversarial attack. Greedy decoding was applied for the experiments. (Note: Minor differences from the main paper are due to nondeterminism across GPU hardware.)\n|Model|$\\text{CHAIR}_s$↓|$\\text{CHAIR}_s$↓|F1↑|\n|:-|:-:|:-:|:-:|\n|MiniGPT-4|31.0|11.4|67.3|\n|**+Ours**|29.0|10.6|67.5|\n|**+Ours+Q-Former masking**|**27.0**|**9.3**|**68.1**|\n\n---\n\n>*W3. Does it simultaneously maintain robust performance on the MME evaluation, particularly for fine-grained recognition and complex compositional understanding?*\n\nTo address your concern regarding the impact of our visual token masking strategy on the model’s visual perception and reasoning capabilities, we conducted additional evaluations on a representative subset of the MME benchmark, as shown in Table R2. Specifically, we selected one task from each of the four major categories in MME [R1]: Position (coarse-grained recognition), Scene (fine-grained recognition), OCR (text understanding), and two tasks requiring logical reasoning—Numerical Calculation and Code Reasoning. The results demonstrate that our method effectively reduces object hallucination while preserving the essential visual perception and reasoning capabilities of the LVLMs, suggesting that our masking approach does not compromise multimodal generation quality.\n\n**Table R2.** Quantitative results of our method on MME benchmark with LLaVA-7B. \n|Method|Position|Scene|OCR|Numerical|Code|\n|:-|:-:|:-:|:-:|:-:|:-:|\n|LLaVA-7B|123|156|125|50|78|\n|**LLaVA-7B+Ours**|123|151|133|53|83|\n\n*Reference*\n\n[R1] Fu, et al. \"MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models,\" arXiv 2023.\n\n---\n\n>*W4. How might its performance generalize to state-of-the-art models like DeepSeek-VL?*\n\nThank you for the suggestion. As you suggested, we evaluated our method on one of the recent models DeepSeek-VL [R2] and Qwen2.5-VL [R3]. As shown in Table R2 and Table R3, our method remains effective, suggesting that uncertain visual tokens are a persistent issue even in state-of-the-art models.\n\nDespite differences in visual token distributions and cross-modal alignment, epistemic uncertainty in the vision encoder remains intrinsic. This enables our method to generalize well across architectures without retraining or structural changes.\n\n**Table R2.** Quantitative results of DeepSeek-VL and Qwen2.5-VL on CHAIR benchmark.\n|Model|$\\text{CHAIR}_s$↓|$\\text{CHAIR}_s$↓|F1↑|\n|:-|:-:|:-:|:-:|\n|DeepSeek-VL|25.8|6.6|**72.7**\n|**DeepSeek-VL+Ours**|**22.4**|**5.5**|72.6\n|Qwen2.5-VL|29.6|7.8|76.0\n|**Qwen2.5-VL+Ours**|**28.6**|**7.0**|**76.8**\n\n\n**Table R3.** Quantitative results of DeepSeek-VL and Qwen2.5-VL on POPE benchmark.\n|Model|Random↑|Popular↑|Adversarial↑|\n|:-|:-:|:-:|:-:|\n|DeepSeek-VL|88.7|88.0|84.9\n|**DeepSeek-VL+Ours**|**88.8**|**88.0**|**85.1**\n|Qwen2.5-VL|84.2|83.7|83.3\n|**Qwen2.5-VL+Ours**|**84.3**|**83.8**|**83.4**\n\n*Reference*\n\n[R2] Lu, et al. \"Deepseek-vl: towards real-world vision-language understanding,\" arXiv 2024.  \n[R3] Bai, Shuai, et al. \"Qwen2.5-vl technical report,\" arXiv 2025."}, {"review_id": "mV105RJp7q", "reviewer": "Reviewer_o1SC", "summary": "The paper addresses a research question:\n-  LVLMs' object hallucination by epistemic uncertainty.\n\nFirst, the paper discovers the uncertainty of visual tokens in a visual encoder of LVLM. To approximate the uncertainty, the paper generates a perturbed image by adding small noise by using adversarial attacks. After that, the mask is extracted from the layer-wise uncertainty map. Compared to MC dropout, the approximated epistemic uncertainty map is aligned; the computation is faster. The paper investigates how the hallucination positively correlates with uncertainty. Based on the approximated uncertainty, the paper proposes a masking strategy, which is training-free. \n\nThe paper validates the proposed method on both open-generation and VQA-style benchmarks. When applying the proposed method to the prior work, the performances are generally improved. The ablation studies, such as vision encoder layers, and masking strategies, also validate the proposed method.", "questions": "Please see above weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- Both uncertainty and hallucinations are important research for trustworthiness. Connecting this research is interesting and promising.\n- The paper has proposed the approximation method for uncertainty, which is faster than the MC method. \n- The writing is well-structured as follows: Efficient uncertainty approximation -> Analysis of uncertainty and hallucination -> Mitigation using uncertainty map -> Validation of the proposed method.\n\n## Weaknesses\n- Although the proposed approximation is efficient, what is the inference time compared to other techniques, e.g., VCD without the proposed method?\n- As shown in Fig. 7 of the paper and the qualitative results of the appendix, the length of generation is shorter than the method without the proposed method. Thus, including the average length of the generation would be helpful in understanding the proposed method. Additionally, I think that a discussion is needed on this point.\n- In Table A10 of the appendix, using MC dropout is not better than the gradient-based method. As I understand the main paper, MC dropout is more accurate but inefficient. I wonder why the performance of adversarial attacks is better than MC dropout.\n- The discussion with the prior work [1] regarding the proposed uncertainty map would be beneficial.\n[1] Hanjing Wang and Qiang Ji, Epistemic Uncertainty Quantification For Pre-trained Neural Networks, CVPR 2024", "was_revised": true, "final_justification": "The paper and the proposed method are well-motivated. My concerns were addressed during the rebuttal, and the authors also discuss inference time and the different behavior compared to MC dropout. Since the authors plan to revise the paper to include these points and further classification, I will keep my initial positive score.", "rebuttal": "We sincerely thank the reviewer for recognizing the importance of connecting uncertainty and hallucination for enhancing trustworthiness. We appreciate the positive assessment of our efficient uncertainty approximation method as a promising alternative to MC-based approaches, as well as the acknowledgment of the clear and well-structured presentation of our contributions. We carefully address your questions and concerns as follows and will incorporate them into our revision.\n\n---\n\n>*W1. What is the inference time compared to other techniques, e.g., VCD without the proposed method?*\n\nWe report the mean and standard deviation of inference time in Appendix Table A6 and discuss this in lines 702-707. As shown in the table, our method achieves comparable runtime to methods like VCD [R1] and PAI [R2], and significantly outperforms OPERA [R3] in terms of inference efficiency. \n\n*Reference*  \n[R1] Leng, Sicong, et al. \"Mitigating object hallucinations in large vision-language models through visual contrastive decoding,\" CVPR 2024.  \n[R2] Liu, et al. \"Paying more attention to image: A training-free method for alleviating hallucination in lvlms,\" ECCV 2024.  \n[R3] Huang, et al. \"Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation,\" CVPR 2024.\n\n---\n\n>*W2. Including the average length of the generation would be helpful in understanding the proposed method.*\n\nAs you pointed out, our method slightly shortens the generated output. We report the mean and standard deviation of text lengths in Appendix Table A8 and discuss this in lines 715–720. Importantly, the reduction in length is modest and aligns with prior findings [R4–7] that **overly long outputs** can increase hallucination. This suggests our method may help by encouraging more concise and grounded descriptions.  Following your feedback, we will expand the discussion in the main text.\n\n*Reference*  \n[R4] Yue, et al. \"Less is more: Mitigating multimodal hallucination from an eos decision perspective,\" ACL 2024.  \n[R5] Han, et al. \"Skip\\n: A simple method to reduce hallucination in large vision-language models,\" ICLRW 2024.  \n[R6] Li, et al. \"The hidden life of tokens: Reducing hallucination of large vision-language models via visual information steering,\" ICML 2025.  \n[R7] Min, et al. \"Mitigating hallucinations in large vision-language models via summary-guided decoding,\" NAACL 2025.\n\n---\n\n>*W3. Using MC dropout is not better than the gradient-based method.*\n\nThank you for your insightful comment. While MC Dropout is a widely used method for approximating epistemic uncertainty, it is itself an approximation and not necessarily the most accurate in practice. The results in Table A10 reflect this: MC Dropout does not consistently outperform other methods, including ours, particularly under adversarial perturbations.\n\nOur method is theoretically motivated by Theorem 3.2, and Figure 2 empirically demonstrates that the uncertainty it measures behaves similarly to that of MC Dropout. This supports the claim that our method offers a principled and reliable way to quantify epistemic uncertainty, while also being significantly more efficient. We will revise the paper to clarify this point and to emphasize the practical advantages of our approach in mitigating hallucinations under perturbations.\n\n---\n\n>*W4. The discussion with the prior work [1] regarding the proposed uncertainty map would be beneficial.*\n\n>[1] Hanjing Wang and Qiang Ji, Epistemic Uncertainty Quantification For Pre-trained Neural Networks, CVPR 2024\n\nThank you for the valuable suggestion. We will include a discussion comparing our uncertainty map with Wang & Ji (CVPR 2024). While both methods estimate epistemic uncertainty without retraining, their approach focuses on model-level uncertainty via parameter sensitivity for classification, whereas ours targets token-level uncertainty in vision encoders using adversarial perturbations, specifically to mitigate object hallucination in LVLMs. We will highlight this distinction and its relevance in the revised manuscript."}, {"review_id": "frrPxDSJFV", "reviewer": "Reviewer_E6ux", "summary": "This work identifies uncertain visual tokens in the vision encoder as a key cause of object hallucination in large vision-language models  and proposes a strategy to mitigate it, leading to significant improvements on some benchmark evaluations.", "questions": "see Strengths And Weaknesses.", "limitations": "N/A", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n\nThe proposed method demonstrates promising performance on some benchmarks.\n\nWeaknesses:\n\n(1) The overall theoretical part appears to lack sufficient rigor, which may affect the reliability of its conclusions. For example, in Lemma 3.1:\n\n1. What exactly is meant by “small perturbations”?\n2. What does “locally follows” refer to?\n3. Additionally, the hidden states “z” and “$z^*$” are introduced without prior definition or explanation.\n\n(2) More importantly, in the proof of Lemma 3.1 and Theorem 3.2, the use of the “$\\approx$” symbol is concerning. Drawing theoretical conclusions based on an imprecise approximation is not appropriate in a scientific context and raises questions about the soundness of the argument.\n\n(3) In Theorem 3.2, it is meaningless to state \"$\\text { upper bound of the differential entropy of } Z_i^{(t)} \\text { increases as } \\mathbb{E}_\\epsilon\\left[\\left\\|Z_i^{(t)}-z_i^{(t)}\\right\\|_2^2\\right] \\text { increases. }$\" We can say that the upper bound of any value X increase as any other (non-negative) value Y increases, because, for example, we can simply set the upper bound of X to be X+Y\n\n(4) There is no significant performance gap between the proposed method and baseline methods in POPE benchmark.", "was_revised": true, "final_justification": "(1) The new proofs provided by the authors in the rebuttal are still informal, and I gave the details in my latest reply.\n\n(2) Furthermore, the main theoretical claim in Theorem 3.2—that “an upper bound of a quantity increases as another quantity increases”—is, from a theoretical standpoint, meaningless, as I explain in the first review, which greatly undermines the value of the paper.", "rebuttal": "We sincerely thank the reviewer for acknowledging the promising performance of our method on benchmark evaluations. We carefully address your questions and concerns as follows and will incorporate them into our revision.\n\n---\n\n>*W1. The overall theoretical part appears to lack sufficient rigor.*\n\nWe thank the reviewer for the thoughtful and constructive feedback. While the mathematical derivation in Lemma 3.1 is valid, we will clarify the terminology and address the reviewer’s concern. Below, we address each specific point:\n\n- **Notation for hidden states:** To improve clarity, we now define the hidden states explicitly. The clean hidden state at layer $t$ is denoted as $z^{(t)} \\in ℝ^{N \\times d}$, and the corresponding perturbed hidden state as $Z^{(t)} \\in ℝ^{N \\times d}$, for input $x$ and perturbed input $x + \\epsilon$, respectively. Here, $N$ represents the number of input tokens, and $d$ is the dimensionality of each hidden state.\n\n\n\n- **Small perturbations:** Our Lemma 3.1 and Theorem 3.2 are grounded in the widely used principle of linear approximation under small perturbations [R1], providing theoretical support for our method. In our work, we define small perturbations as $\\epsilon \\in \\mathbb{R}^{n \\times 3}$ to the input $x \\in \\mathbb{R}^{n \\times 3}$, where $n$ is the number of image pixels and 3 corresponds to the RGB channels. The perturbation magnitude is constrained by $\\|\\epsilon\\|_\\infty \\leq k$ for a sufficiently small constant $k > 0$, ensuring that the induced change in the hidden representation, $\\|Z^{(t)} - z^{(t)}\\|$, is small enough for higher-order terms in the Taylor expansion to be safely ignored. Importantly, beyond this theoretical support, our empirical findings demonstrate that these small perturbations allow for accurate uncertainty estimation (Fig. 2) and effectively reduce object hallucination, resulting in strong performance on benchmark datasets (Table 1).\n\n\n- **Locally follows a Gaussian distribution:** This statement stems from a first-order Taylor expansion under small perturbations and is consistent with standard Laplace approximation techniques [R2, R3], which are widely used in local analyses of neural networks. Specifically, it means that the distribution of the hidden state $Z^{(t)}$ from perturbed input can be approximated by a Gaussian distribution around the original hidden state $z^{(t)}$. We will clarify this point in the revised version of the paper.\n\n\n*Reference*  \n[R1] Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy. \"Explaining and harnessing adversarial examples,\" ICLR 2015.  \n[R2] Pascanu, Razvan, and Yoshua Bengio. \"Revisiting natural gradient for deep networks,\" arXiv 2013.  \n[R3] Kirkpatrick, et al. \"Overcoming catastrophic forgetting in neural networks,\" PNAS 2017.\n\n### **Clarified Lemma 3.1**\n\n**Lemma 3.1 (Local Gaussianity of hidden states under small input perturbation)**  \nLet $f = \\{f_t\\}_{t=1}^L$ be a smooth $L$-layer neural network parameterized by $\\theta$, and define the hidden state at layer $t$ as\n\n$$\nz^{(t)} := f_t \\circ f_{t-1} \\circ \\cdots \\circ f_1(x)\n$$\n\nfor a input $x \\in R^{N \\times 3}$. Consider a perturbed input  $x + \\epsilon$, where the perturbation $\\epsilon \\in R^{N \\times 3}$ satisfies $\\|\\epsilon\\|_\\infty \\leq k$ for sufficiently small $k > 0$. Define the corresponding hidden state as\n\n$$\nZ^{(t)} := f_t \\circ f_{t-1} \\circ \\cdots \\circ f_1(x + \\epsilon).\n$$\n\nThen, the hidden state $Z^{(t)}$ is approximated by a Gaussian distribution around $z^{(t)}$.\n\n---\n\n>*W2. The use of the \"$\\approx$\" symbol is concerning.*\n\nRegarding the use of the approximation symbol “$\\approx$” in Lemma 3.1 and Theorem 3.2, we clarify that this approximation is both theoretically justified and standard in the context of smooth neural networks [R4, R5]. Additionally, as illustrated in Fig. 2 and similar to the behavior of MC dropout, the empirical results also support the validity of this approximation.\n\nThe approximation results from a first-order Taylor expansion around the input $x$ with small additive noise $\\epsilon$, as shown in Eq. A.2 (Lemma 3.1) and Eq. A.5 (Theorem 3.2). Since $\\epsilon$ is bounded (i.e., $\\|\\epsilon\\|_\\infty \\leq k$ for sufficiently small $k$), the induced residual $\\|Z^{(t)} - z^{(t)}\\|$ remains small, allowing higher-order terms to be safely ignored. This type of local linear approximation is widely used in the analysis of neural networks and their functional behavior [R2, R3] (Laplace approximation).\n\nWe also provide empirical support in Fig. 3, which shows that $\\|Z^{(t)} - z^{(t)}\\|$ is small in the early layers of the image encoder; accordingly, we rely only on these early-layer features for our approximation.\n\nWe will clarify this rationale in the revision to improve transparency and precision.\n\n*Reference*  \n[R2] Pascanu, Razvan, and Yoshua Bengio. \"Revisiting natural gradient for deep networks,\" arXiv 2013.  \n[R3] Kirkpatrick, et al. \"Overcoming catastrophic forgetting in neural networks,\" PNAS 2017.  \n[R4] Balduzzi et al. \"Neural Taylor approximations: Convergence and exploration in rectifier networks.\" PMLR, 2017.  \n[R5] Fu et al. \"Unfolding Taylor's approximations for image restoration.\" NeurIPS 2021.\n\n---\n\n>*W3. In Theorem 3.2, it is meaningless to state \"upper bound of the differential entropy $Z^{(t)}_i$ increases as $𝔼 [|Z^{(t)}_i-z^{(t)}_i|^2_2]$ increases\".*\n\nWe appreciate the reviewer’s feedback and would like to clarify the role of the upper bound in Theorem 3.2.\n\n#### **A. Our upper bound provides a conservative estimate of uncertainty**\n\nAs shown in [R6], conservative uncertainty estimation—i.e., **intentionally overestimating** rather than underestimating epistemic uncertainty—is *theoretically desirable* for multiple applications.\n\n> “Our uncertainties are conservative, ensuring that our algorithm is never more certain than it should be.”  \n> — *Ciosek et al., ICLR 2020* [R6]\n\nOur upper bound follows this principle:  \n- It is **monotonically linked** to adversarial representation deviation (Theorem 3.2),  \n- And ensures that we **never overlook truly uncertain regions**, even if the bound is loose.\n\nThis reflects the key idea behind conservative epistemic uncertainty estimation.\n\n*Reference*  \n[R6] K. Ciosek et al., \"Conservative Uncertainty Estimation by Fitting Prior Networks,\" ICLR 2020.\n\n#### **B. The upper bound is empirically useful**\n\nOur experiments demonstrate that:\n\n- Uncertainty maps derived from adversarial attack closely resembles with ones from Monte-Carlo dropout uncertainty estimations. (Fig. 2, A1)\n- Masking tokens identified via the upper bound leads to consistent reduction in object hallucination. (Fig. 4, 6)\n- Achieving up to **38% CHAIRs reduction** across various models and decoding strategies.(Table 1, 2)\n\nThis shows that the bound is not only theoretically motivated but also reliably useful in practice.\n\n\n#### **C. The bound reflects meaningful structure**\n\nWhile it is true that any upper bound can be trivially increased, Theorem 3.2:\n\n- Is derived from adversarial representation deviation,  \n- Provides a **monotonic and interpretable proxy** for epistemic uncertainty,  \n- And is consistent with the conservative estimation framework supported in [R6].\n\nIn summary, the upper bound in Theorem 3.2 is not only meaningful but also **conservative by design**, making it **theoretically sound**, **practically effective, and empirically validated**. While formally proving the tightness of the bound remains a challenging task, our empirical results strongly suggest its usefulness in practice. We agree that establishing tightness more rigorously is a promising direction for future work.\n\n*Reference*  \n[R6] K. Ciosek et al., \"Conservative Uncertainty Estimation by Fitting Prior Networks,\" ICLR 2020.\n\n---\n\n>*W4. There is no significant performance gap between the proposed method and baseline methods in POPE benchmark.*\n\nWe would like to clarify that the performance improvement on POPE of our method is not small. As shown in Table 1, methods such as OPERA, VCD, PAI, and Devils similarly exhibit performance degradation or marginal improvements under this setting. For instance, under LLaVA-1.5-7B, Devils (CVPR 2025) achieves slight gains of +0.3%p (Rand.), +0.6%p (Pop.), but degrades by -0.7%p on the Adversarial split.\n\nAlthough our method follows a similar trend, it is consistently effective across various models and mitigation strategies in a plug-and-play manner, requiring no architectural changes or retraining. In this context, our method yields **comparable** or **better** improvements: when combined with Devils (CVPR 2025), we observe gains of **+0.4%p** (Rand.), **+0.9%p** (Pop.), and **+1.3%p** (Adv.) on LLaVA-1.5-7B. With VCD (CVPR 2024), we achieve **+1.9%p**, **+0.6%p**, and **+1.4%p** on the respective splits. These results consistently demonstrate the robust effectiveness of our method."}, {"review_id": "bdCfndwnSq", "reviewer": "Reviewer_bSvX", "summary": "### Background\n- LVLMs suffer from object hallucination, that is, they generate descriptions of objects that are not in the input image.\n### Method\n- The authors argue that uncertain visual tokens within the vision encoder in LVLMs are a key factor.\n\nStatistical analysis shows that high epistemic uncertainty in visual tokens => the occurrence of hallucinations.\n\nFurthermore, they show theoretically and empirically that in early visual encoder layers, large representation deviations under small adversarial perturbations => high epistemic uncertainty.\n\n- Based on their findings, the authors propose a new method to alleviate object hallucination for LVLMs.\n\nSpecifically, the method uses adversarial perturbations to identify uncertain visual tokens.\n\nThen, it applies a binary uncertainty mask to attention outputs in visual encoders, reducing the influence of uncertain visual tokens and alleviating object hallucination.\n### Results\n- In experiments, the proposed method can be combined with different prior approaches, significantly reducing object hallucinations in various LVLMs.", "questions": "Please see the Weaknesses section.", "limitations": "Yes. The limitations and potential negative societal impact are discussed in the supplementary materials.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n- The paper proposes a new perspective: uncertain visual tokens from visual encoder can result in object hallucination.\n- The experiments are comprehensive, indicating that the proposed method can improve different LVLMs and prior approaches.\n\n## Weaknesses\n### Major\n\n- A few papers apply adversarial perturbance to images for mitigating hallucination. I appreciate that the authors dig into vision encoders and also show their method is compatible with prior ones, but I am not sure if the proposed method is technically novel or provides new insights. This may be very subjective, so I would like to see opinions from ACs and other reviewers.\n- Recent works [1-2] show that LVLMs tend to believe language priors (blind faith in text) instead of visual information. I think, the proposed method only modifies visual information from vision encoders and cannot mitigate hallucination caused by language priors. For example, when an LVLM is describing a table, it may tend to say there is a chair, because the text training data usually contains a table and a chair together. \n\n[1] VLind-Bench: Measuring Language Priors in Large Vision-Language Models\n\n[2] Words or Vision: Do Vision-Language Models Have Blind Faith in Text?\n\n- The evaluated LVLMs are kind of old. There are many new LVLMs with improved visual information processing modules, like Cambrian-1, Qwen2.5-VL, BLIP-3o and so on. I'm not sure if the proposed method still works on them. Besides, now that new LVLMs are using better vision modules, the benefit from the proposed method may shrink.\n\n- Masking some tokens may lead to information loss. As shown in Figure 7, I notice the LVLMs answers are much shorter when using the proposed method. It seems that, after we delete uncertain tokens, the models tend to describe fewer objects. Does this lead to less informative answers from LVLMs? Will this hurt the performance of models on VQA or reasoning tasks?\n\n### Minor\n- In Section 3.1, the authors propose estimating uncertainty based on adversarial attack, and show that it aligns with MC dropout. But in 3.2.1, the authors start to use MC dropout (lines 175-178) to estimate uncertainty maps. I'm wondering if the proposed uncertainty estimation is not associated with object hallucination.\n\n- The proposed method is not specific to LVLMs but to vision encoders. The authors show that visual encoders may have uncertain tokens, misleading LVLMs to generate object hallucination. This issue may happen in other scenarios when we use foundation visual encoders. For example, if I use CLIP to do image classification (without LLM part), the classification results may also be mislead by the uncertain tokens in the encoder. Thus, I think the \"uncertain token\" issue may be a common problem in encoders, while object hallucination is just an application when we use visual encoders in LVLMs. The authors are recommended to explore other applications of their findings and method in the future.\n\nFor LVLMs, I expect that the authors can explore, how LVLMs handle uncertain information from the encoder and how LVLMs balance between visual information and language prior, which is missing in the paper.\n\n- Line 198, is the indentation a typo?", "was_revised": true, "final_justification": "Thank the authors for the detailed response. \n\n- Novelty. Addressed.\n\n- Hallucinations caused by other reasons. As also noted by Reviewer RPtF, this paper only focuses epistemic uncertainty in the vision encoder, and does not consider other important hallucinations, such as those caused by language priors. But the proposed method is well-justified, effective, and compatible with prior hallucination mitigation approaches. Thus, this point does not diminish the value of the authors' contributions.\n\n- Experiments on newer LVLMs (also noted by Reviewer RPtF). The authors added additional experiments on Qwen2.5-VL and DeepSeek-Vl. Addressed.\n\n- Information loss due to the proposed method (also noted by Reviewer o1SC). The proposed method shortens the generated output of LVLMs, but maintains LLaVA-7B's performance on MME benchmark. The experiment scope is kind of limited, because the authors only validate their method with one model on a relatively simple benchmark. Partly addressed.\n\nMy concerns have been largely addressed, and I raise my score to 4.\n\nThe authors are encouraged to extend their method to handle various hallucinations and may explore uncertainty of vision encoders in other applications.", "rebuttal": "We sincerely appreciate the reviewer’s recognition of our new perspective on uncertain visual tokens as a source of object hallucination. We also thank the reviewer for noting the comprehensiveness of our experiments and the broad applicability of our method across various LVLMs and prior approaches. We carefully address your questions and concerns as follows and will incorporate them into our revision.\n\n---\n\n### Major\n\n>*W1. I am not sure if the proposed method is technically novel or provides new insights.*\n\nThank you for highlighting the strengths of our work, especially our focus on vision encoders and compatibility with prior methods.\n\nWhile previous works [R1-2] have applied adversarial perturbations mainly to reduce hallucinations, our method takes a fundamentally new dimension: it repurposes adversarial perturbations to measure epistemic uncertainty inside the vision encoder. Our method uniquely uses adversarial perturbation to quantify epistemic uncertainty of visual tokens within the vision encoder—enabling principled, visual token-level masking to suppress hallucinations. This introduces a novel perspective by connecting visual representation uncertainty with hallucination mitigation of LVLMs, which has not been explored in prior work.\n\nAs noted by Reviewer o1SC, both uncertainty and hallucinations are important research for trustworthiness, and connecting the two is **interesting and promising**. Reviewer RPtF also emphasized that the performance improvements in our method stem directly from the uncertainty-aware masking mechanism that selectively filters only the most unreliable visual tokens, which **supports its novelty and effectiveness**. We believe these points affirm the broader relevance and impact of our contribution.\n\n*Reference*  \n[R1] Zhang, et al. \"Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMs,\" arXiv 2025.  \n[R2] Wang, et al. \"Mirage in the eyes: Hallucination attack on multi-modal large language models with only attention sink,\" arXiv 2025.\n\n---\n\n>*W2. I think, the proposed method only modifies visual information from vision encoders and cannot mitigate hallucination caused by language priors.*\n\nOur work assumes that both the vision encoder and language model contribute to object hallucination, and focuses specifically on the vision-side uncertainty—an underexplored but complementary factor.\n\nThe cause of object hallucination in LVLM you raised—blind faith in textual information—are indeed well-known and addressed by existing methods. Rather than aiming to cover the cause from language models, we present new dimension **epistemic uncertainty in the vision encoder**, which we show is strongly correlated with hallucination.\n\nTheorem 3.2 and our empirical results (Table 1, 2 and Fig. 4, 6) demonstrate that visual tokens sensitive to adversarial perturbations—used as a proxy for uncertainty—are predictive of hallucinated content. Masking these tokens significantly reduces hallucinations without altering the language model.\n\n\nOur method is training-free, vision-only, and integrates well with language-side approaches like OPERA, PAI, and VCD. This plug-and-play nature makes it a practical and effective complement to broader hallucination mitigation strategies.\n\n---\n\n>*W3. There are many new LVLMs with improved visual information processing modules, like Cambrian-1, Qwen2.5-VL, BLIP-3o and so on. I'm not sure if the proposed method still works on them.*\n\nThank you for the suggestion. As you suggested, we also evaluated our method on one of the recent models DeepSeek-VL [R3] and Qwen2.5-VL [R4]. As shown in Table R2 and Table R3, our method remains effective, suggesting that uncertain visual tokens are a persistent issue even in state-of-the-art models.\n\nDespite differences in visual token distributions and cross-modal alignment, epistemic uncertainty in the vision encoder remains intrinsic. This enables our method to generalize well across architectures without retraining or structural changes.\n\n**Table R2.** Quantitative results of DeepSeek-VL and Qwen2.5-VL on CHAIR benchmark.\n|Model|$\\text{CHAIR}_s$↓|$\\text{CHAIR}_s$↓|F1↑|\n|:-|:-:|:-:|:-:|\n|DeepSeek-VL|25.8|6.6|**72.7**\n|**DeepSeek-VL+Ours**|**22.4**|**5.5**|72.6\n|Qwen2.5-VL|29.6|7.8|76.0\n|**Qwen2.5-VL+Ours**|**28.6**|**7.0**|**76.8**\n\n\n**Table R3.** Quantitative results of DeepSeek-VL and Qwen2.5-VL on POPE benchmark.\n|Model|Random↑|Popular↑|Adversarial↑|\n|:-|:-:|:-:|:-:|\n|DeepSeek-VL|88.7|88.0|84.9\n|**DeepSeek-VL+Ours**|**88.8**|**88.0**|**85.1**\n|Qwen2.5-VL|84.2|83.7|83.3\n|**Qwen2.5-VL+Ours**|**84.3**|**83.8**|**83.4**\n\n*Reference*  \n[R3] Lu, et al. \"Deepseek-vl: towards real-world vision-language understanding,\" arXiv 2024.  \n[R4] Bai, Shuai, et al. \"Qwen2.5-vl technical report,\" arXiv 2025.\n\n---\n\n>*W4. It seems that, after we delete uncertain tokens, the models tend to describe fewer objects. Does this lead to less informative answers from LVLMs? Will this hurt the performance of models on VQA or reasoning tasks?*\n\nThank you for pointing this out. We acknowledge that masking uncertain visual tokens may lead to slightly shorter outputs, as shown in Figure 7 and Table A8. However, this reduction is modest and has been observed to help reduce hallucinations, consistent with prior studies [R5-8] that **associate overly lengthy outputs with increased hallucination risk.** To quantitatively support this, we included the average text lengths and standard deviations in Appendix Table A8 and discussed it in lines 715–720.\n\nIn response to your concern about potential degradation in visual understanding or reasoning (*e.g.*, VQA or reasoning), we conducted further evaluations using the MME benchmark [R9] across key categories including coarse/fine-grained recognition, OCR, and logical reasoning (Numerical Calculation and Code Reasoning), as summarized in Table R2. Our method preserved performance across these tasks, suggesting that while it reduces the influence of uncertain tokens, it maintains essential visual perception and reasoning capabilities.\n\n**Table R2.** Quantitative results of our method on MME benchmark with LLaVA-7B. \n|Method|Position|Scene|OCR|Numerical|Code|\n|:-|:-:|:-:|:-:|:-:|:-:|\n|LLaVA-7B|123|156|125|50|78|\n|**LLaVA-7B+Ours**|123|151|133|53|83|\n\n*Reference*  \n[R5] Yue, et al. \"Less is more: Mitigating multimodal hallucination from an eos decision perspective,\" ACL 2024.  \n[R6] Han, et al. \"Skip\\n: A simple method to reduce hallucination in large vision-language models,\" ICLRW 2024.  \n[R7] Li, et al. \"The hidden life of tokens: Reducing hallucination of large vision-language models via visual information steering,\" ICML 2025.  \n[R8] Min, et al. \"Mitigating hallucinations in large vision-language models via summary-guided decoding,\" NAACL 2025.  \n[R9] Fu, et al. \"MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models,\" arXiv 2023.\n\n---\n\n### Minor\n\n>*W1. In Section 3.1, the authors propose estimating uncertainty based on adversarial attack, and show that it aligns with MC dropout. But in 3.2.1, the authors start to use MC dropout (lines 175-178) to estimate uncertainty maps. I'm wondering if the proposed uncertainty estimation is not associated with object hallucination.*\n\nWe used MC dropout in Section 3.2.1 specifically for analysis purposes because it is a well-established and well-known method for uncertainty quantification. Our goal was to clearly and rigorously demonstrate the association between uncertain visual tokens and object hallucination. To ensure this analysis was based on a widely recognized technique, we opted for MC dropout instead of our method.\n\n---\n\n>*W2. I think the \"uncertain token\" issue may be a common problem in encoders, while object hallucination is just an application when we use visual encoders in LVLMs. The authors are recommended to explore other applications of their findings and method in the future.*\n\nThank you for your insightful suggestion regarding broader applicability. As you pointed out, our method is designed to intervene solely in the vision encoder and thus holds potential for extension beyond LVLMs to other vision tasks that rely on pre-trained encoders. However, in this work, our analysis and experiments are focused specifically on the relationship between uncertain visual tokens and object hallucination in LVLMs. It remains an open question how our method would behave in tasks such as zero-shot image classification with CLIP. We agree that exploring these directions would be a valuable avenue for future research.\n\n---\n\n>*W3. The authors can explore, how LVLMs handle uncertain information from the encoder and how LVLMs balance between visual information and language prior.*\n\nWe appreciate the reviewer’s insightful suggestion. To examine how LLMs handle uncertain visual information, we analyzed the attention maps of LLaVA-1.5-7B over image tokens, before and after applying our method.\n\nWe measured the entropy of the LLM’s attention distribution over image tokens (across all layers and heads) to assess whether the LLM attends broadly or narrowly. A higher entropy implies that the model utilizes a wider range of visual evidence rather than relying on a few visual tokens.\n\nWe evaluated the average entropy of LLM attention over image tokens using 500 images:\n* Entropy of original LLaVA = 1.5746\n* Entropy of LLaVA + Ours = 1.9717\n\nThe **increased entropy** indicates that our method encourages broader and more balanced attention over reliable visual tokens, helping the model better integrate visual information while mitigating over-reliance on uncertain inputs.\n\nWe will include this result in the final version to clarify how our approach influences the LLM's use of visual features.\n\n---\n\n>*W4. Line 198, is the indentation a typo?*\n\nThank you for pointing out the indentation issue at line 198. It is indeed a typo error. We appreciate your careful reading and helpful feedback."}]

Author rebuttal:

Meta-review: This paper initially received two borderline accept, one borderline reject, and one reject. After the rebuttal and reviewer discussions, the scores remained mixed. However, the AC finds that the strengths of the work outweigh the weaknesses and recommends acceptance.

The paper introduces a training-free and efficient method to mitigate object hallucination in LVLMs by masking uncertain visual tokens identified via adversarial perturbations. The approach is computationally efficient, and consistently reduces hallucination rates while preserving output quality. Reviewers appreciated the clarity of writing, systematic experimental validation across multiple LVLM architectures, and the practical viability of the proposed method.

Several concerns were raised. 
- Theoretical rigor: Reviewer E6ux questioned the rigor of the theoretical analysis, particularly Theorem 3.2. While the rebuttal did not fully convince this reviewer, the AC believes the concerns are addressable through revision, as the overall intuition and methodology are addressed in the rebuttal. 
- Scope of hallucination causes: Multiple reviewers (RPtF, bSvX) pointed out that the method only mitigates a certain type of hallucinations arising from visual token uncertainty. This limits the generality of the approach.
- Evaluation: The strategy sometimes shortens generated outputs, raising questions about informativeness, though rebuttal experiments showed preserved performance on MME. Finally, while additional experiments on Qwen2.5-VL and DeepSeek-VL were provided, the scope of evaluation on state-of-the-art models and broader benchmarks remains somewhat limited.

Despite these limitations, the paper makes a clear and practical contribution: it identifies a new perspective on hallucination causes, proposes a lightweight and effective mitigation strategy, and demonstrates its compatibility across architectures. The AC considers the theoretical concerns remediable in revision and finds the empirical results sufficiently compelling.
Also, while Reviewer RPtF rated Borderline Reject, considering the fact that the rebuttal seems to address most of the questions in the review comment from the perspective of this AC and the final justification is incomplete, this AC found that the rating is overly pessimistic.

Therefore, the AC supports acceptance under the condition that the authors update the parts the reviewers commented on, such as the theoretical presentation, expanding evaluations to newer models, with the rebuttal contents in the final version.
Decision: Accept (poster)