Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMs
Authors: Kejia Zhang
Keda TAO
Jiasheng Tang
Huan Wang
Venue: NeurIPS 2025 poster ()
Keywords: Large Vision-Langugae Model
Object Hallucination
Primary area: deep_learning

Abstract: Large vision-language models (LVMs) extend large language models (LLMs) with visual perception capabilities, enabling them to process and interpret visual information. A major challenge compromising their reliability is object hallucination that LVMs may generate plausible but factually inaccurate information. We propose a novel \textit{visual adversarial perturbation (VAP)} method to mitigate this hallucination issue. VAP alleviates LVM hallucination by applying strategically optimized visual noise without altering the base model. Our approach formulates hallucination suppression as an optimization problem, leveraging adversarial strategies to generate beneficial visual perturbations that enhance the model's factual grounding and reduce parametric knowledge bias. Extensive experimental results demonstrate that our method consistently reduces object hallucinations across 8 state-of-the-art LVMs, validating its efficacy across diverse evaluations.

Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMs
Kejia Zhang'
Keda Tao?
Jiasheng Tang Huan Wang2*
Xiiamen University Westlake University
DDAO Academy, Alibaba Group "Hupan Lab
Project Page: https: https://keejazhhagg roobuust github.thh...... io/iii//p/ wttp/poiiinuueeleeeeee
(Original Image]
Frezen
Inference
Zero-Gradient
AdversarialNosee
Large Vision-Language Model
(2024-09, 9,48)
LLaV ov
Intem-yL2 (2024407,
12.35%
(2024-05, 9.58)
+1.0%%
Original Image
Visual Noise
Perturbed Image
Instruct-BLa
toys image?
(2023-06, 7.18)
+3.m
No, there is no bear the Image. The image shows two space suit.
DespSesk-112
Yes, there are two beor tays in the image, flonting in space with planet
(2028-11, 0.18)
2.004
bankground.
We introduce VAP (visual adversarial perturbation), criining-free approach that strategically injects beneficial visual noise to mitigate object hallucination in LVMs without altering the complex base model. Right: Our method consistently improves performance across state-of- the-art LVMs under the POPE hallucination evaluation setting [26].
Figure 1: Left:
Abstract
Large dision-language models (LVMs) extend large language models (LLMs) with visual perception capabilities, enabling them to process and interpret visual infor- mation. A major challenge compromising their reliability is object hallucination that LVMs may generate plausible but factually inaccurate information. We propose a novel visual udversarial perturbation (VAP) method to mitigate this hallucination issue. VAP alleviates LVM hallucination by applying strategically optimized visual noise without altering the base model. Our approach formulates hallucination sup- pression
optimization problem, leveraging adversarial strategies to generate beneficial visual perturbations that enhance the model's ' factual grounding and reduce parametric knowledge bias. Extensive experimental results demonstrate that our method consistently reduces object hallucinations across 8 state-of-ther of-the-art LVMs, validating its efficacy across diverse evaluations.
Introduction
Large vision-language models (LVMs) integrate visual and textual information, providing capabilities for addressing complex cross-modal understanding challenges [43, 5, 38, 22). Despite their remark- able advancements, LVMs often generate plausible yet factually inaccurate outputs, eliciting harmful content such as misinformation or biased representations [26, 33]. Addressing these limitations is critical enhancing the reliability and applicability of LVMs real-world scenarios.
*Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Prior research indicates that hallucinations in LVMs arise from the interaction between biased parametric knowledge and real-world data distributions [2, 15, 11]. This phenomenon is driven by two primary mechanisms. First, the long-tail distribution of training data induces systematic biases in parametric knowledge, resulting in spurious correlations and factual inconsistencies [26, 27]. Second, the extensive parameter spaces of large language models (LLMS) within LVMs amplify these biases, particularly given the LLMs' predominant role in the inference pipeline [23, 30, 39J. This LLM dominance potentially suppresses critical visual signals, increasing hallucination frequency [36, 42, 24]. Consequently, the embedded biased parametric knowledge substantially compromises LVMs' capacity to accurately process real-world data.
Existing solutions mitigate this challenge via two strategies: fine-tuning [27, 54, 4] and decoder optimization [17, 30, 8]. These model-centric interventions adjust LVMs' internal mechanisms through parametric updates or algorithmic refinements [29]. They have achieved substantial success
in reducing hallucinations, laying crucial groundwork for improving LVM reliability.
Unlike prior model-centric approaches, we introduce a paradigm shift in hallucination mitigation that leverages the intrinsic mechanisms of hallucinations. This perspective stems from crucial obser- vation that while hallucinations arise from biased parametric knowledge, they manifest specifically during the processing of real-world visual inputs [16, 2]. This understanding reveals an elegant solu- tion: strategically crafted perturbations to visual inputs can redirect LVMs' decision- makiing processes away from parametric biases without altering the original model's architecture or mechanisms.
This insight motivates visual adversarial perturbation strategy, where adversarial optimization through zero-gradient techniques introduces beneficial visual noise to the original image. This noise guides the model ground its responses in actual visual content rather than relying on parametric knowledge biases. The power of this approach lies in its exploitation of visual inputs as concrete factual anchors, fundamentally different from language prompts that often reinforce existing parametric biases [41, 50J. Notably, our method functions in a fully black-box manner requiring no access or modification the LVM, making it practical and efficient solution.
Building on this foundation, we propose visual adversarial perturbation (VAP), a novel technique de- signed to mitigates ballucinations by applying beneficial adversarial perturbations to visual inputs (as shown in Figure (left)). Adversarial perturbations, traditionally considered as "poison" due to their initial disruption of model decisions, are reformulated to specifically align model responses with visual content and mitigate parametric knowledge bias. By adversarially optimizing visual noise, VAP refines LVM decision-making in data-centric manner, transforming perturbations from factor of degradation into corrective "cure" that effectively mitigates object hallucinations.
We evaluate the effectiveness of VAP using complementary hallucination assessment frameworks: POPE [26] and BEAF [53] for closed VQA evaluation, and CHAIR [36] for open-ended generation tasks. Our extensive experiments across 8 state-of- ofate-of-the-ar (SOTA) LVMs demonstrate that VAP consistently mitigates hallucinations across diverse evaluation settings.
Overall, our contributions are structured as follows:
We propose visual adversarial perturbation, which mitigates object hallucinations in LVMs by injecting beneficial adversarial noise into visual inputs without modifying the model.
We formulate object hallucination mitigation By refining adversarial strategies, beneficial visual noise is generated through zero- -raddett optimization influence model decision-making maaiinn and alleviate hallucinations.
adversarial visual noise optimization.
Extensive experiments across evaluation settings-including text-axis, text- and vision-axes. and open-ended captioning-valinn the efficacy of our method in reducing hallucinations.
2 Related Work
2.1 Large-Vision Language Models
In recent years, the field has witnessed advancements in large vision-language models (LVMs). LVMs have been developed to tackle real-world multimodal challenges such as image captioning and visual question answering [52, 47, 40, 58]. They typically operate through a pipeline comprising a visual encoder, modsl connector, and a large language model (LLM), enabling seamless interaction
2

---

between visual and linguistic features. State-of- -fre---rtt systems leverage extensive datasets and adopt a two-stage training paradigm: pretraining on diverse multimodal corpora [35, 37], followed by fine-tuning with task-specific instructions [28, 32]. This methodology allows LVMs to interpret and respond to complex multimodal inputs with remarkable efficacy [25, 10).
2.2 Hallucination in LVMs
Hallucination refers to the generation of textual responses that deviate from or contradict the actual visual content, leading factual inaccuracies biased information in LVMs [26, 2]. These hallu- cinations primarily arise from intrinsic limitations of LVMs, specifically: (1) the long-tail distribution of training data, which introduces systematic biases into the model's parametric knowledge [57, 54]; and (2) the vast parameter space of LLMs, which dominate the inference process and exacerbate these biases [29, 30]. Due the fundamental role of objects in computer vision and multimodal research, current evaluation frameworks primarily concentrate on object hallucination [36, 57].
Prior work has explored two model-centric strategies mitigate object hallucinations in LVMs: fine-tuning and decoding strategies. These interventions target the underlying parametric knowledge bias that leads hallucinations. Fine-tuning approaches like REVERIE [21] and HalluciDoctor [54] update the parametric knowledge through comprehensive instruction data to suppress hallucinations. Meanwhile, decoding-based methods such PDM [12] and OPERA [17] mitigate hallucinations by intervening in the model's decoding process. In contrast to these model-centric strategies, we approach the challenge from data-centric perspective, proposing a novel adversarial visual perturbation technique that directly mitigates object hallucinations through visual perturbations.
3 Methodology
We propose visual adversarial perturbation (VAP) to mitigate object hallucination in LVMs. VAP formulates an adversarial strategy to align the LVM responses with visual content while reducing the impact of parametric knowledge bias (Section 3.2). These objectives guide the adversarial optimiza- tion process, which generates beneficial visual noise to improve model performance (Section 3.3). overview of our framework is shown in Figure 2.
3.1 Preliminaries
Notations Let fe denote LVM, where a represents the input image, is the query prompt, and w is the model's generated response, such that w fo( c). We define Io the CLIP text encoder converting textual data into semantically meaningful embeddings. For adversarial perturbation, we denote 5 as the perturbation vector and Ls the surrogate adversarial loss guided by strategy = [$1, ... s,]] The perturbed image is defined as ê = x + 6, c is the magnitude of perturbation, and Ω represents the adversarial knowledge utilized during the adversarial optimization process.
Adversarial Perturbation Adversarial perturbation against LVMs typically involves adding imper- ceptible visual noise to influence model decisions [56, 9J, which can significantly alter the model's s output. The optimization of such perturbations can be formulated as:
8 = arg max (15) + 5,
b~),(())
where 5 represents the adversarial perturbation to be optimized, L(s) represents the adversarial objective function under strategy S, and Ω indicates the available adversarial knowledge. The perturbation is bounded within an e-ball B. Specifically, the adversarial perturbation is optimized by computing the gradient follows:
= z av + (,
where a is the step size, and the gradient V, is computed with respect to the vision input x.
3.2 Adversarial Strategies
Our adversarial goal is formulated as two principal objectives: (1) optimizing the semantic alignment between the response and the corresponding visual content of LVMs, and (2) mitigating the negative influence of parametric knowledge bias.
3

---

Original Image
Visual Adversarial Optimization
Adversarial Knowledge
Yes, there an oirplane the image.
Distrrted Image
Original Imago
Large Viston Language Model
LVM
poo
poo
Prompt [c): chere airpiame in the image?
Perturbed Image
Toxt Encoder
Text Encoder
Text Encoder
Null Teat (4)
No, there is no airpiane image.
Text Encodirr
XoM LVM
recviiyy
LVM
Response
Frozen
BlackrAccess
Text Embedding
LVM Inference
Zero Gradient
Noise Generation
Semantic Similrity
Figure 2: Overview of our proposed method. VAP generates visual noise by optimizing three strategies based on adversarial knowledge: (1) aligning responses under prompted and unprompted settings to preserve image-consistent semantics, (2) introducing uncertainty via distorted inputs to expose hallucination bias, and (3) minimizing representational similarity between original and distorted views to suppress parametric priors. Adversarial knowledge refers to structured conditions used to drive the optimization. The resulting perturbation mitigates object hallucinations.
Alignment LVM Response with Grounding Visual Content Hallucinations LVMs manifest the generation of semantically plausible responses but diverge from the actual visual content. To mitigate this, our proposed methodology promotes enhanced alignment between the model's responses and the actual visual content:
L
max {S(fo(z + + 6,,, +, 5,0))}
(~),
where -). signifies the calculation of semantic correlation between the two generated responses. fo(x + 6, c) represents the model's 's output given the perturbed vision input x + 5 with the conditional query prompt e, and fa (x++ 0) signifies the visual semantic description when the prompt is replaced with empty token 0. This loss term Ls, quantifies the semantic alignment between conditionally guided responses and the model's autonomous interpretation of visual content, thereby enhancing response consistency with the underlying visual semantics.
Despite the improvements, the alignment between responses and visual content may still be influenced by parametric knowledge bias, particularly an over-reliance on linguistic priors [4]. Such bias can distort the model's 's interpretation of visual information, leading to hallucinatory patterns. As discussed in Section 1, LVMs often prioritize linguistically anchored priors over visual signals, thereby exacerbating existing biases. Our alignment strategy addresses this by mitigating both misalignment and bias.
Mitigating Parametric Knowledge Bias Visual uncertainty [15, 24, 51] serves critical metric for quantifying parametric knowledge bias. It is quantified by generating a contrastive negative image through the introduction of noise the original image:
p((z)) = NZ;; Vera, (1 - )),
where fo represents the noise scheduling coefficient timestep T, controlling the magnitude of perturbation applied to the original image x.
To further mitigate parametric knowledge bias, we introduce a dual-setting approach that reduces the semantic similarity between LVM responses to original and distorted visual inputs under both conditional (with query prompt) and unconditional 0 (without query prompt) configurations.
In the conditional e setting, our approach minimizes the semantic similarity between the perturbed input x + and the contrastive negative image æ:
Lsp min (S(So( fo ())), (m)
where fo(2.0 0) denotes the LVM's output given the visually uncertain input. Loz promotes more discriminative and context-sensitive responses hetween prompted and unprompted conditions, thereby effectively reducing the model's 's dependency linguistic priors.

---

In the unconditional setting, our methodology minimizes the semantic similarity between responses the perturbed image x + ó and its contrastive negative counterpart E:
Ss min {S(fo(x + 6, 0), fT, 9))
(6)
(x),
where Lay alleviates the propensity to hallucinate, further mitigating the dominant influence of linguistic priors.
The loss terms L. Lry and L,3 collectively regulate LVM responses to ensure consistency with visual content while mitigating parametric knowledge bias in LVMs. We formulate our complete optimization objective as weighted combination of these loss terms:
Cs c, = L51 Ls2
C33
(7)
of
02
03
where of (i E (1,2.3)) are balancing coefficients that modulate the contribution each loss component. This formulation achieves dual objective: L,, ensures strong semantic alignment between model responses and visual content, while Lag and Lag collectively mitigate parametric knowledge bias through consistent interpretation across visual perturbations.
3.3 Visual Adversarial Optimization
To optimize our adversarial objectives Ls, leverage the CLIP text encoder ge ) as surrogate model, capitalizing on its superior discriminative capabilities for textual representation [48]. This approach contrasts with the limited semantic separability in LLM representations:
S(:,)) = (.) go (.),,,
(8)
where .) measures the similarity of the LVM's response under different conditions. Then, we compute the numerical loss Cs( c, f), which enables the optimization of the perturbation d. 6 represents a carefully crafted visual perturbation designed to optimize the strategic objective:
ö V V=(Cs(a, 4))
The final adversarial perturbation generated by adding noise to the input image æ, yielding the visual adversarial perturbed image c:
(10) where a denotes the learning rate of adversarial strategies. The generated perturbed image å exhibits superior optimization characteristics with respect the objective Ls, outperforming the original images æ while meticulously preserving the semantic integrity of vision input.
f t (Css,,
aV
Due the autoregressive nature of LVMs, direct gradient computation is challenging. To address this, we optimize the similarity- optimization. Specifically, we apply zero-order optimization technique [6], which approximates the gradient by evaluating the loss at perturbed inputs and estimating the optimal perturbation direction:
loss using a gradient-free method [56, 34], termed zero-grade grrdden
N
V V=( x N
Σ N
Nas [[Cs(x + B. c, n, v)
(11)
n=
0, 7
where Yn is sampled from distribution P(y), 3 controls the sampling variance, and N denotes the number of queries. The term ~ P(7) ensures perturbation diversity through the property Ely 7] = I. A detailed step-by-step algorithm of VAP is provided Appendix I.
Experiments
To thoroughly assess VAP, we conduct experiments from five perspectives:
Evaluating VAP's effectiveness in mitigating hallucinations across eight LVMs. Fidelity: Ensuring that visual understanding and reasoning capabilities are preserved.
Consistency:
Compatibility: Demonstrating VAP's orthogonality to other methods and complementary benefits.
Efficiency: Reducing computational cost via lightweight solution achieving 1/8x overhead.
Component Analysis: Assessing the contribution of each module through ablation.
5

---

4.1 Experiment Setup
Implementation Details We evaluated our method on 8 SOTA LVMs: LLaVA [28], LLaVA-
Onevision (OV) [25], Instruct-BLIP [10], Intern-VL2 [7], Intern-LL2-MPO [7], Qwen-VL2 [46],
DeepSeek-VLL VL.2 [49], and 6--mmma2 [31]. In our experiments, we set the parameters as
= 1/255, 3 = 8/255, N = 10, and 2. Due to the differences across LVMs, we assigned model-specific balancing coefficients o, (where i E 1,) and T.
Detailed model descriptions, configurations are provided in Appendix A. Additionally, an in-depth analysis of the ablation study and individual components can be found in Appendix C.
Evaluation Benchmark Our evaluation is divided into two main categories: (1) Closed VQA for object hallucination evaluation: Text-axis evaluation POPE [26] and vision-/text-ax. evaluation BEAF [53] settings. (2) Open-ended evaluation: Image caption generation CHAIR [36] setting. (3) non-hallucination evaluation: Factual object recognition and open-ended factual understanding tasks using MME [14] and AMBER [45] (See in Appendix J.1). Further details are provided in Appendix B, and comprehensive examples are presented in Appendix E.
1) POPE: POPE evaluates hallucinations along the text axis by generating VQA pairs through question manipulation. We randomly selected 500 samples from the MS-COCO dataset and generated 9,000 evaluation triplets using POPE's three sampling strategies.
2) BEAF; BEAF evaluates hallucinations along vision/text axes by manipulating scene information and questions for fine-grained analysis. BEAF incorporates change-aware metrics such as TU, IG, SBp. SB,, ID, and FITUID for comprehensive evaluation. BEAF includes 26,064 evaluation triplets.
3) CHAIR: CHAIR evaluates hallucination by generating captions and measuring the proportion of objects mentioned in captions but not present in images. Specifically, we randomly select 1,000 samples from the MS-COCO dataset for evaluation. The assessment uses two metries:
CHAIR, |hallucinated objects|
CHAIRs hallucinated captions)
(12)
|captioned objects|
all captions|
where CHAIR, is calculated at the object level, and CHAIRs calculated the sentence level.
4) AMBER/MME: AMBER and MME serve as comprehensive evaluation benchmarks for multi- modal large language models. They assess various attributes of multimodal capabilities, focusing on both perception and cognition in discriminative and generative tasks.
4.2 Experimental Results
Results on text-axis hallucination evaluation Table presents comparative results under the POPE (Polling-based Object Probing Evaluation) setting". Our experimental methodology includes three sampling strategies: Random, Popular, and Adversarial Sampling for negative object selection, each generating 3,000 evaluation triplets. Across all settings, integrating VAP through visual noise injection consistently improved the performance eight state-of-the-arr LVMs, with the largest gains observed in Intern- Intern-VL2: +2.81% 1% in accuracy and +2.09% in Fl score. Notably, the most significant improvements appear under adversarial sampling (Figure 1-right), indicating that VAP effectively mitigates parametric knowledge bias in LVMs. This is particularly relevant adversarial sampling tends to trigger feehffreguecy hallucinated objects, highlighting the data distribution bias in LVM training and the dominant role of L.LMs.
Results on Vision-/Text-Axis Hallucination Evaluation Table 2 presents comparative results under the BEAF (BEfore-AFter) framework, which enables fine- fine-grained analysis through vision-axis manipulation and change-aware metrics, offering deeper insight than standard accuracy. Applying VAP led to consistent improvements across most metrics for all LVMs.
Notably, TU improved by 2.31% %, SB, by 1.76%, SB,, by 1.04%, and FI by 1.74%. Gains across TU, IG, SB. SB,, ID, and FITUID indicate that VAP mitigates hallucinations under varied scene conditions by promoting genuine object understanding over spurious correlations. The marked TU gains further suggest that VAP's visual perturbations guide models toward more grounded predictions, validating its role in suppressing parametric bias and enhancing visual reasoning [53].
Due space limitations, complete precision and recall results are provided in Appendix C.I.

---

Table 1: Text-axis evaluation comparison under three evaluation settings of POPE the validation set of MSCOCO: Random Sampling (selecting absent objects), Popular Sampling (choosing the most frequent missing objects based dataset-wide occurrence), and Adversarial Sampling (ranking
objects by co-occurrence with ground-truth and selecting the most frequent ones). The values in green indicate the percentage improvements achieved by our proposed method.
Popular
Random
Adversarial
LVM
Vision Input
Ace.t
F11
Acc.1
F11
Acc-1
Flf
Original
85.57
86.19
88.97
89.09
79.80
81.79
LLaVA-v1.5
+VAP
86.67
87.18
90.00
90.07
80.97
82,82
Instroct-BLIP
Original
+VA
84.06 +0.76
83.30
83.67 +0.99
82.85
89.00
88.
-0.87 87
88.12 +0.90
87.18
82.03 +0.70
81.33
81,99 +0.78
81.21
Intern-VL2
Original
+VAP
86.18 +2.07
84.11
81 6.64
84.19 +22.00
86.30 -1.10
85.14
82,60
84.08 +-148
84,81 +2.81
82,00
82.79 +-2.00
80,70
Original
87.51
86.53
88.68
87.58
86.28
85.55
Intern-VI V1P
+ VAP
89.08
88.27
90,20
89.30
88,13
87.55
DeepSeek V1.2
Original +YAP
87.60
86.80
10.50
86.70 0..54
85.86
89.30
88.70
00.00
88.31
87.64
10.67
87.13 00.00
86.47
86.28 10.78
85.55
Qwen-VL2
Original + VAP
89.10 +9.97
88.13
87 7..6
88.65
+-0.07
91.16 -0.56
90.60
90.54 +0.55
89.99
86,27
87.30
87.02
86.02
LLaVA-OV
Original + VAP
$8.30
87.33
89.53
88.51
87.17
86,27
88,93
87.93
89.87
88.83
87.76
86.69
6-iem66-Gemm22
Original -VA
87.96
86, 88
88. 38.96
87.87
86,22
85.32
88.44 +9.48
N7.40
+0..5
89.59
-0.05
88.54
+0.67
86.85
00.69
86.03
+00.7
Table 2: Vision-/text-Axis evaluation comparison under the BEAF Benchmark. Compared to the text-axis hallucination evaluation, BEAF includes the change-aware hallucination metrics: TU, IG, SBp SB,, ID, and Flr Although some metries show slight degradation, the overall performance demonstrates consistent improvement. The values in green indicate the percentage improvements achieved by our proposed method, while the values in red reflect the performance degradation.
BEAF Benchmark
LNM
Vision Input
Acc.
TU1
[GL
SB,l
SB,
ID J
Flyum 1
LLaVA-v1.5
Original
+VAP
80.36 +0.87
79.99
74.35
74.06
10.00
34.83
34,25
-0.58
0.33
0.27
00.00
60.74
60.72
00000
4.66
00.6
5.05
5.42
0.37
50.97
50.31
0.96
Original
81.91
73.55
33.35
0.78
50.73
15.12
5.45
49.30
+VA
82.07 H0.6
73.96
10.41
33,83
00.48
0.48
0.50
50.59
0.34
15.10
0.02
5,30
0.16
49.85
0.88
Intem-VV.2
Original 4 VAP
88, 88.38
88.69
79.72 +0.62
79. 10
66.15
64.12
0.97
1.33
11.58
12.63
21.28
21.89
60.63
6.05
6.20
77.63
76.17
Original +VAP
89.21
82.56
63.24
23.67
12.31
5.23
75,86
89.63
82.72
65.06
0.45
21.91
12,55
4.49
77.40
DeepScck-VL2
Original
+AP
89.39
89.72 10.85
83.12
82.51
10.01
67.04
68.11
1:00
0.44
0.50
0.00
17.37
17.88
0.51
14.06
14.56
0.00
2.98
3.02
0.04
79.27
80.03
0.76
Qwen VL2
Original
+AP
87.96
88.39 -0.45
81.13
10.44
54.78
56.18
1.440
D:28
0.00
32.49
33.68
11.1
11.24
11.03
0021
4.89
4.38
0.51
69,78
70.79
1.01
ILLVA-OV
Original
VAP
90.76
91.07 +0.85
85.01
84.53
10.68
65,80
67.16
1.33
0.12
0:30
10.18
20.81
21.32
-0.11
11.73
12.77
-100
2.46
2.55
00.00
78.56
79.54
0.08
Ovim1.6-Gemma2
Original
YAA
90.91 H9.7
90.12
84,53
83.04
1.40
68.56
66.25
2.31
0.25
0.28
0.00
19.69
19:94
00.55
13.52
11.48
2.04
2.76
2.41
80,54
78,80
1.774
Results on Open-Ended Caption Generation Hallucination Evaluation Table 4 reports our model's performance under the CHAIR (Caption Hallucination Assessment with Image Relevance) setting." Applying optimized VAP to original images yields consistent reductions in object hallucination across diverse query prompts. For example, under the prompt "Generate a short caption of the image," Intern- V1.2 achieves CHAIR, and CHAIR, reductions of 0.68 and 0.90, respectively, with VAP.
These results highlight VAP's effectiveness in open-ended vision-language tasks heyond binary VQA. By mitigating hallucination, VAP improves the semantic alignment between captions and visual content, reduces parametric bias, and enhances the factuality and relevance of generated descriptions.
$ *CHAIR is limited to 80 segmentation categories, which may induce classification bias [26]. We restrict responses 30 characters to focus on prominent objects.
7

---

Table 3: Comparison of VAP improvements on POPE (text-axis) and BEAF (vision/text-axis). Only Accuracy and F1 are shown for compactness. Green numbers indicate performance gains.
LVM
Vision Input
POPE-Popular
RoPE-Random
POPE-Adverarial
BEAF
Acct
FI1
Acc?
F11
Acet
F17
Acct
FI"
LLvV1.11.5
Original
VVAP
86.67 +1.30
85,57
87.18
86.19
40.00
90.00
$8.97
+1.33
89.09
+0.5
79.80
+1.17
81.79
+1.00
79.99
10.37
74,06
+0.20
Instruct-LLIP
Original
+VAP
84,06 10.76
83.30
83.67
82. 85
10.82
89.00
88.13
+0.87
88,12
87.18
10.0
82,03 F2700
81.33
81,99 10.78
81.21
82.07 +0.30
81.91
73.96
73.55
10.41
Original
$4.11
81.64
85.14
82.00
80.70
88.38
79.10
+YAP
86.18
84.19
86.30
84.08
84.81
82,79
88.69
79.72
Original
$7.51
86.53
88.68
87.58
86.28
85.55
89,21
82.56
Intem- VL2-MPO
+VAP
11.57
88.27 11.44
90.20
1155
1.72
1188
87.55 12.00
10.43
82.72 10.15
DeepSeck V1.2
Original
86.80 86.80
88,
87.66
82.51
+VA
87,60
86,70
89.30
88.31
87.13
86.28
89.72
83.12
Qwen-Vi..
Original
+VA
89.10 +0.97
88.13
88.65
87.68
10.57
91.16
90.60
+0.50
90.54 +0.55
89.99
86.27
+1.00
86.02
+3.00
87.96
+0.00
$1.13
+0.41
LLaVA-OV
Original
VAA
88.93
88, 30
87.93
87. 33
89.87
89. 53
88.83
88.3 51
87.76
87. 17
86.69
86.27
91.07
90.76
85.01
84.53
Ovis
Original -AA
88
90.12
83.04
$8.44
87.40
89.59
88.54
86.85
86.03
90.91
84.53
Table 4: Comparison of object hallucination evaluation under the CHAIR setting. I denotes "Generate short caption of the image", and I2 denotes "Provide brief description of the given image". The values in green indicate the percentage improvements achieved by our proposed method.
LVM
Vision Input
CHAIR:
CHAIR:
CHAIR
CHAIR:
Original
3.97
6.60
4.01
6,90
LLaVA-v1.5
+VAP
3,82
6,50
0.330
3.86
9.33
6,50
-0.40
Original
1883
2.90
2.14
3.40
Instruct-BLIP
+VAP
1.71 -2.1
2.70
-0.20
1.96 18.18
3.10 -0.30
Inter-VL22
Original
+VAP
4.22 00.68
4.90
6.60
7,50
00.90
4.65 0044
5.14
8.90 -0.60
9.50
Original
5.53
8.90
6.35
13.40
Intern VL2-MPO
VAP
5,39
8.60
6.17
12,60 -0.80
DeepSeek-VE V1.2
Original
2,00
2.60
1.84
4.50
+VAP
1.94 0.00
2,20 00.40
1.66
0.188
4.30
-0.20
Original
3.27
5.20
3.45
6.20
Qwen-VL.2
+VAP
2.98 -0.0-...
4.80
00.40
3.23 0..22
5,70
-0.50
Original
1.96
3.30
2.71
4.50
LLaVA-OV
VAP
185 -0.11
3,10
2.41
00.30
4.20
-0.30
Ovisl.6-Gemmal
Original VAP
3,90 -0.17
4.07
6.30
6.20
0..10
5,56
5,80
-0.24
14.30
14.50
-0.20
4.3 Analysis and Discussion
Effectiveness of VAP and Gaussian noise on hallucinations Figure 3 compares the impact of VAP and Gaussian noise applied to original images under equal-strength perturbations. Gaussian noise consistently degrades performance across eight models, while VAP preserves or improves This highlights VAP's effectiveness in three aspects: Firstly, VAP introduces beneficial semantic noise, whereas Gaussian noise increases uncertainty and disrupts visual features. Secondly, VAP enhances alignment between model outputs and visual content via its adversarial strategy, reducing hallucinations. Thirdly, unlike Gaussian noise, which merely blurs input, VAP semantically challenges the model to mitigate parametric knowledge bias.
Illustration of the effectiveness on closed VQA and open-ended tasks Figure 4 presents results from examples in closed iision-question-anee answer (VQA) and open-ended image captioning tasks. Panels (a) and (b) demonstrate that the visual noise introduced by our method suppresses object hallucinations in LVMs under scene-change situations without disrupting their normal perceptual capabilities (i.e., the noise does not lead to incorrect decisions). Additionally, Panels (c) and (d) show that our method mitigates object ballucinations in open-ended tasks without reducing the amount
8

---

10
Accuracy 01
Score (hp
023
62
Waemins Mi--
Mark-dy
Plemnta
Worn1D-- Soep
Laradyy
bent A-hooer
Figure 3: Comparison of original images with our VAP and Gaussian noise of equal strength (e 2). We highlight the performance drop caused by Gaussian noise compared to VAP. Experiments were conducted under the POPE adversarial setting, evaluated by Accuracy and FI Score.
(1) wwwlannntteee Seaiin-/tts-aaal ovaluation under original Image
vidon-/tnet-aess evaluation under nake image [-VAP]
implince
benean in maagge?? miet
thercan ansoanee the
ONTr umpiane thessnger
parict ane surnny
parked unway ansent
dudle detiol
open ended evalaation original image
open
image (HAP)
deserptinn thegaemarge
secrppoc
The vvage show smik clisd actng on criinng noar soumn sving set
The vaage stons small whits shi sttigg en puininy swing
on bandy
whic in Noob penily abowc sand and'a aeggraaninnnrrtee slanda searly..
inshgroand slg
theswnn
Figure Examples of the vision-queesionn-asserr (VQA) tasks before and after applying our proposed method to the original images. (a) and (b) demonstrates the suppression of hallucinations in vision-/text-ax evaluations. (c) and (d) shows the reduction of ballucinations in open-ended tasks. Specifically, we use the LLaVA-v1.5 [28] as an example.
of information in the LVMs' responses. These consistent findings highlight the effectiveness of the VAP method. More comprehensive examples can be found in Appendix E. In-depth analyses of generalization are provided in Appendix D)
Computational cost analysis and efficient proxy-based solution We report the computational cost of VAP optimization and present a more efficient approach. Our innovative proxy-based strategy leverages smaller-scale models to generate adversarial perturbations, which are then effectively transferred to larger models. As illustrated in Table 5, our approach reduces generation time by up eightfold while maintaining comparable accuracy. Notably, VAP generated by the Intern-VL2-- 1B model and applied to the Intern- VL2-8B model achieves an accuracy of 84.07%, compared to 84.81 % with self- generaeddd VAP, with only a minor increase in runtime (+41ms vs. +298ms). This demonstrates that our proxy solution efficiently introduces beneficial noise that is generalizable across models, sustaining inference latency and enabling scalable deployment across large iision-languae models, thus enhancing overall system efficiency.
Conclusion
This paper presents visual adversarial perturbation (VAP), an innovative data-centric training- free method reduce object hallucinations in large languael models (LVMs) by introducing imperceptible noise to visual inputs. Unlike model-centre approaches requiring complex modi- fications, VAP strategically applies beneficial noise to visual data, grounding model responses in actual content and reducing reliance biased parametric knowledge. Extensive evaluations on the POPF, BEAF, CHAIR, AMBER, and MMH benchmarks show that VAP significantly decreases object hallucinations across various settings, enhancing LVM reliability.

---

Table 5: Computational cost and efficiency analysis of proxy-based VAP generation. The table presents the performance and runtime evaluation of Intern- [7] and Qwen-VL.2-77 [46] under different vision input strategies. The proxy-base based approach substantially reduces computational overhead while preserving strong hallucination suppression performance.
LVM
Vision Inpul
Pruxy Model
Accuracy($) 1
Runtime (A100 per time)
Computational Cost
Original
82.00
160ms
Miemn-VL--BB
-VAP
Intem- Intem-VL2-8B
VL2-8B
84.81 (+-2.81)
+298ms
1x
VAP-Proxy
Intem- --.2-V1.2-1B
84.07 (+2.07)
+39ms
1/8x
Original
86.27
133ms
Qwen-V1.22 VL.27B
+ -VAP
Qwen-VL22 VL.2-B
87.30 (+1.08)
+245ms
1x
VAP-Proxy
Qwen- V1.2-2B
86.87 (+00600)
+48ms
1/5x
Our findings highlight the effectiveness of visual adversarial perturbations as a novel "poison as cure" strategy, uniquely demonstrated here. A key contribution is the consistent mitigation of model ballucinations in a black-box setting through noise addition, without compromising image understanding. Although VAP introduces computational overhead, propose proxy-based approach for efficient noise generation, maintaining performance while reducing costs to one-eighth. This work underscores VAP's potential transformative approach in enhancing LVM accuracy and reliability, paving the way for future research in data-centric model improvement.
Acknowledgment
This paper is supported by Young Scientists Fund of the National Natural Science Foundation of China (No. 62506305), Zhejiang Leading Innovative and Entrepreneur Team Introduction Program (No. 2024R01007), Key Research and Development Program of Zhejiang Province (No. 2025C01026), Scientific Research Project of Westlake University (No. WU2025WF003, Chinese Association for Artificial Intelligence (CAAI) & Ant Group Research Fund AGI Track (No. 2025CAAI-ANT-13) and the Special Support Talents Program of "Xi Hu Ming Zhu Program" in Hangzhou. We thank Xinjun Lin for the aesthetic insights provided in this paper.
References
[1] H. Bai, S. Jian, T. Liang, Y. Yin. and H. Wang. Ressvd: Residual compensated svd for large language model compression. arXiv preprint 2rxhv:2505.20172 2025.
[2] Z. Bai, P. Wang. T. Xiao, T. He, Z. Han, Z. Zhang, and M. Z. Shou. Hallucination of multimodal large language models: A survey, arXiv preprint arXiv:2404.189) 2024.
[3] A. F. Biten, L. Gómez, and D. Karatzas. Let there be clock on the beach: Reducing object hallucination in image captioning. In WACV, 2022.
[4] C. Chen, M. Liu, C. Jing, Zhou, F. Rao, H. Chen, B. Zhang, and C. Shen. PerturboLLaVA: Reducing multimodal hallucinations with perturbative visual training. In ICLR, 2025.
[5] L. Chen, J. Li, X. Dong, P. Zhang, C. He, J. Wang, F. Zhao, and D. Lin. Sharegpt4v: Improving large multi-modal models with better captions. In ECCV, 2024.
[6] P.-Y. Chen, H. Zhang, Y. Sharma, J. Yi, and C.-J. Hsieh. Zoo: Zeroth order optimization based black-bor attacks deep neural networks without training substitute models. Proceedings of the 10th ACM workshop on artificial intelligence and security, 2017.
[7] Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q Zhang, X. Zhu, Lu, B. Li, Luo, T. Lu, Y. Qiao, and Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, 2024.
[8] Z. Chen, Zhao, Luo, H. Yao, Li, and Zhou. Hale: Object hallucination reduction via adaptive focal-or focal-contrast decoding. In ICML, 2024.
10

---

[9] X. Cui, A. Aparcedo, Y. K. Jang, and S.-N. Lim. On the robustness of large multimodal models against image adversarial attacks. In CVPR, 2024.
[10] W. Dai, J. Li, D. LI, A. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. NeurIPS, 2023.
[11] G. Deletang, A. Ruoss, P.-A. Duquenne, E. Catt, T. Genewein, C. Mattern, J. Grau-Moya, L. K. Wenliang, M. Aitchison, L. Orseau, et al. Language modeling is compression. In ICLR, 2024.
[12] A. Favero, L. Zancato, M. Trager, S. Choudhary, P. Perera, A. Achille, A. Swaminathan, and
S. Soatto. Multi-modaa hallucination control by visual information grounding. In CVPR, 2024.
[13] S. Feng, K. Tuo, S. Wang, L. Kong, J. Zhu, and H. Wang. Rewardmap: Tackling sparse rewards in fine-graines visual reasoning via multi-stage reinforcement learning. arXiv preprint arXiv:2510.02240, 2025.
[14] C. Fu, Y-F. Zhang, S. Yin, B. Li, X. Fang, S. Zhao, H. Duan, X. Sun, Z. Liu, L. Wang, et al. Mme-survey: A comprehensive survey on evaluation of multimodal Ilms. arXiv preprint arXiv:2411.15290 2024.
[15] T. Guan, Liu, X. Wu, R. Xian, Z. Li, X. Liu, X. Wang, Chen, F. Huang, Y. Yacoob, D. Manocha, and T. Zhou Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. CVPR, 2024.
[16] A. Gunjal, J. Yin, and E. Bas. Detecting and preventing hallucinations in large vision language models. In AAAI, 2024.
[17] Q. Huang, X. Dong, P. Zhang, B. Wang, C. He, J. Wang, D. Lin, W. Zhang, and N. Yu. Opera: Alleviating hallucination in multi-modie large language models via over-trust penalty and cnnsspectionaallcattin In CVPR, 2024.
[18] Q. Huang, X. Dong, P. Zhang, B. Wang, C. He, J. Wang. D. Lin, W. Zhang, and N. Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and vetrospection-allociiie allocation. In CVPR, 2024.
[19] Y. Jiang, X. Yan, G.-P. Ji, K. Fu, M. Sun, H. Xiong, D.-P. Fan, and F. S. Khan. Effectiveness assessment of recent large vision-language models. Visual Intelligence, 2(1):17, 2024.
[20] Y. Jin, J. Li, T. Gu, Y. Liu, B. Zhao, J. Lai, Z. Gan, Y. Wang, C. Wang. X. Tan, et al. Efficient multimodal large language models: A survey. Visual Intelligence, 3(1):27, 2025.
[21] M. Kim, M. Kim, J. Bae, S. Choi, S. Kim, and B. Chang. Exploiting semantic reconstruction to mitigate hallucinations in vision-language models. In ECCV, 2024.
[22] K. Kuckreja, M. S. Danish, M. Naseer, A. Das, S. Khan, and F. S. Khan. Geochat: Grounded large vision-language model for remote sensing. In CVPR, 2024.
[23] H. Laurençon, Tronchon, M. Cord, and V. Sanh. What matters when building languaelanguage models? arXiv preprint PrXiv:2405.00224. 2024.
[24] Leng, H. Zhang, G. Chen, X. Li, S. Lu, C. Miao, and Bing. Mitigating object hallucinations in large vision- laiguale models through visual contrastive decoding. In CVPR, 2024.
[25] B. Li, Y. Zhang, D. Guo, Zhang, F. Li, H. Zhang, K. Zhang, P. Zhang, Y. Li, Z. Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024.
[26] Y. Li, Y. Du, K. Zhou, J. Wang, X. Zhao, and J.-R. Wen. Evaluating object hallucination in large vision-language models. In EMNLP, 2023.
[27] F. Liu, K. Lin, L. Li, J. Wang, Y. Yacoob, and L. Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In ICLR, 2023.
[28] H. Liu, C. Li, Q. Wu, and Y. J. Lec. Visual instruction tuning. In NeurIPS, 2023.

---

[29] H. Liu, W. Xue. Y. Chen, D. Chen, X. Zhao, K. Wang, L. Hou, R. Li, and W. Peng. A survey on hallucination in large lsionnlanguage models. arXiv preprint arXiv:2402.00255, 2024.
[30] S. Liu, K. Zheng, and W. Chen. Paying more attention to image: A training-free method for alleviating hallucination in Ivims. In ECCV, 2024.
[31] S. Lu, Y. Li, Q.-G. Chen, Z. Xu, W. Luo, K. Zhang, and H.-J. Ye. Ovis: Structural embedding alignment for multimodal large language model. arXiv preprint arXiv:2405.20797 2024.
[32] G. Luo, Y. Zhou, T. Ren, S. Chen, X. Sun, and R. Ji. Cheap and quick: Efficient vision-language instruction tuning for large language models. In NeurIPS, 2023.
[33] Menon, I. P. Chandratreya, and C. Vondrick. Task bias contrastive rision-language models. IJCV, 132(6))2026-20)) 2040,, 2024.
[34] Y. Nesterov and V. Spokoiny. Random gradient-free minimization of convex functions. Founda- tions of Computational Mathematics, 17(2):527-566, 2017.
[35] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[36] A. Rohrbach, L. A. Hendricks, K. Burns, T. Darrell, and K. Saenko. Object hallucination in image captioning. In EMNLP, 2018.
[37] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes,
A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, and J. Jitsev. In NeurlPS, 2022.
[38] K. Shao, K. Tao, C. Qin, H. You, Y. Sui, and H. Wang. Holitom: Holistic token merging for fast video large language models. arXiv preprint arXiv:2505.21334, 2025.
[39] K. Shao, K. Tao, K. Zhang, S. Feng, M. Cai, Y. Shang, H. You, C. Qin, Y. Sui, and H. Wang. When tokens talk too much: A survey of multimodal long-context token compression across images, videos, and audios. arXiv preprint M.XXiv2507..009900 2025.
[40] Y. Shi, Y. Gao, Y. Lai, H. Wang, J. Feng, L. He, J. Wan, C. Chen, Z. Yu, and X. Cao. Shield: An evaluation benchmark for face spoofing and forgery detection with multimodal large language models. Visual Intelligence, 3(1):9, 2025.
[41] A. Shtedritski, C. Rupprecht, and A. Vedaldi. What does clip know about red circle? visual prompt engineering for vims. In ICCV, 2023.
[42] K. Tao, C. Qin, H. You, Sui, and H. Wang. Dycoke: Dynamic compression of tokens for fast video large language models. In CVPR, 2025.
[43] T. Thrush, R. Jiang. M. Bartolo, A. Singh, A. Williams, Kiela, and C. Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In CVPR, 2022.
[44] K. Tuo and H. Wang. Sparsessm: Efficient selective structured state space models can be pruned in one-shot. arXiv preprint wrXiv:2506.0961 2025.
[45] J. Wang, Y. Wang, G. Xu, J. Zhang, Y. Gu, H. Jia, J. Wang, H. Xu, M. Yan, J. Zhang, et al. Amber: An l1m-free multi-dimennional benchmark for milms hallucination evaluation. arXiv preprint 2xXiv:3311.07397 2023.
[46] P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, et al. Qwen2-vl: Enhancing vision-languae model's perception of the world any resolution. arXiv preprint 2xXi:24499.12191 2024.
[47] W. Wang, Z. Chen, X. Chen, J. Wu, X. Zhu, G. Zeng, Luo, T. Lu, J. Zhou, Y. Qiao, et al. VisionlIm: Large language model is also an open-ended decoder for vision-centric tasks. In NewrIPS, 2024.
12

---

[48] A. Wu, Y. Yang, X. Luo, Y. Yang. C. Wang, L. Hu, X. Dai, D. Chen, C. Luo. L. Qiu, et al. Llm2clip: Powerful language model unlock richer visual representation. NeurIPS Workshop. 2024.
[49] Z. Wu, X. Chen, Z. Pan, X. Liu, Liu, D. Dai, H. Gao, Y. Ma, C. Wu, B. Wang, et al. Deepseek- v12: Mixture-of-experts vision- langugge models for advanced multimodal understanding. arXiv preprint 2030022241..0000.,, 2024.
[50] J. Xiao, A. Yao, Y. Li, and T.-S. Chua. Can trust your answer? visually grounded video question answering. In CVPR, 2024.
[51] Z. Xiong, Z. Zhang, Z. Chen, S. Chen, X. Li, G. Sun, J. Yang, and J. Li. Novel object synthesis via adaptive text-image harmony. NeurIPS, 2024.
[52] Xu, W. Shao, K. Zhang, P. Gao, Liu, M. Lei, Meng, S. Huang, Qiao, and Luo. Lylm-ehub: A comprehensive evaluation benchmark for large vision-language models. IEEE TPAMI, pages 1-18, 2024.
[53] M. Ye-Bin, N. Hycon- Woo, W. Choi, and T.-H. Oh. Beaf: Observing before-after changes to evaluate hallucination in vision-language models. In ECCV, 2024.
[54] Q. Yu, Li, L. Wei, L. Pang, W. Ye, B. Qin, S. Tang, Q. Tian, and Y. Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. In CVPR, 2024.
[55] H. Zhang, W. Zhang, H. Qu, and J. Liu. Enhancing human-centered dynamic scene understand- ing via multiple Ilms collaborated reasoning. Visual Intelligence, 3(1)3, 2025.
[56] Y. Zhao, Pang, C. Du, X. Yang, C. LI, N.-M. M. Cheung, and M. Lin. On evaluating adversarial robustness of large vasion-language models. In NeurIPS, 2023.
[57] Y. Zhou, C. Cui, J. Yoon, L. Zhang, Z. Deng, C. Finn, M. Bansal, and H. Yao. Analyzing and mitigating object hallucination in large vision-language models. In ICLR, 2024.
[58] Y. Zuo, Q. Zheng, M. Wu, X. Jiang, R. J. Wang, Y. Zhang, G. Mai, L. V. Wang. J. Zou, et al. 4kagent: agentic any image 4k super-resolution. arXiv preprint ar arXiv:2507. 00000007705,.... 2025.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: The abstract and introduction clearly state the motivation, method, and contri- butions, which are consistently supported by the rest of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations of the proposed approach are discussed in the supplementary material, along with preliminary directions for addressing them.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer:
[NA]
Justification:
The paper does not include formal theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide detailed descriptions of the datasets, evaluation metrics, model configurations, and training procedures necessary to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
15

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: An anonymous code link provided in the abstract with sufficient details for reproduction.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
public/guides/Cone
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ ctttpp..cc/ubbiceec g/guides.....eeeeerrr /eeddddee//oombbiiiicc CooeeSubmis ssi iesionPolicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section 4.1 for detailed experiment settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: While the paper includes comprehensive evaluation across multiple bench- marks, error bars not reported due the high computational cost of repeated runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The paper reports the hardware specifications used for training, and additional details on computational cost provided in the supplementary material.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics ttttpp://neurppi httpppplli/tthaaiiiiiiieeeae
Answer: [Yes]
Justification:
The research complies with the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts in Section 5.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper does not release any new high-risk models or datasets that would require special safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets, models, and code used in the paper properly cited, and their licenses and terms of use are respected as per their original sources.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper introduces new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve human subjects or crowdsourcing experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The study does not involve human participants and therefore does require IRB approval.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The paper does not use LLMs as part the core methodology.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:/ //teur/is.....ee cc/Confereecess22255/// 222L//LM) for what should or should be described.
20

---

A More Details of Experiment Setup
A.1 More Details about Baseline LVMs
In this study, we comprehensively selecte eight state-of-the-an large vision-language models (LVMs) carefully selected to validate the effectiveness of our proposed method. As illustrated in Table 6, our chosen models span critical developments from September 2023 to December 2024, encompassing parameter ranges from 7.IB to 16. and integrating advanced language models like Vicuna, Qwen2, and Gemma2 with sophisticated vision encoders such as CLIP, SigLIP, and custom vision transformers. Our model selection strategy focuses on capturing the latest architectural innovations addressing hallucination challenges in vision- vision-language understanding. By examining models from leading research initiatives including LLaVA, Instruct- Bestruct-BLI,, Intern- VL, DeepSeek, Ovis, LLaVA-OV and Qwen, aim to provide comprehensive hallucination evaluations of current multimodal AI.
Table 6: Detailed information of large vision- vision-languuee language models used in this paper.
LVM
# Parameters
Language Model
Vision Model
Released Date
1.. [28]
7.1B
Vicuna-7B
CLIP ViT-L/14
2023-09
Instruct-BL.n [10]
7.9B
Vicuna- 7B
VIT-G
2023-09
Intern-VLL [7]
8.1B
InternL.M2.5-)
Intern InterViT--3000 ViT-300M
2024-07
Intern-LL2-MPO VL2-MPO [7]
Intern-
8.1B
AnternL.M2.5-75
Intern ViT-300M
2024-11
DeepSeck-V1.2 [49]
16.1B
DeepSeckMoEE-164
SigLIP-400M
2024-12
Owen-VLL [46]
8.3B
Qwen2-78 7B
ViT-Qwen
2024-08
LLaVA-OV [25]
8.0B
Qwen2-7B BBe
SigLIP-400M
2024-08
Ovisl 6- 6--mmma2 [31]
9.4B
Gemma2-918
SigI.IP-400M
2024-11
A.2 More Details about Implementation Details
We conducted experiments across eight state-of- satte-of-the-eeee
the-thttttt art vision-language models: LLa Instruct-BLIP. Intern-VLL Intern- VL2-MPO, DeepSeek-Vir VL2, Qwen- LLaVA-OV, and Ovis1.6. Gemma2. The experiments were performed using NVIDIA RTX 4090 (24GB), A6000 (48GB), and A100 (80GB) GPUs. For the adversarial parameters, we set = 1 1/255, = 8/255, N = 10, and 2 unless otherwise noted. Model-specific balance parameters detailed in Table 7. We employ ViT-L/14 as our default CLIP text encoder (gx) unless otherwise specified.
Table 7: Detailed specifications of large vision- language models used in this paper.
LVM
1/022
VT/22
V1/0²²
T
LLaVA-v1.5 [28]
1.0
1.0
1.0
500
Instruct-BLIP [10]
1.0
1.0
1.0
500
Intern-VIL [7]
1.0
0.5
0.5
200
Intern- Intern-VL2-MPPO [7]
1.0
0.5
0.5
800
DeepSeek- VL2 [49]
1.0
10
1.0
100
Qwen-VL2 [46]
1.0
0.5
0.5
500
LLaVA-OV [25]
0.1
1.0
0.1
200
6-esm.6 [31]
1.0
1.0
1.0
500
More Details of Evaluation Benchmark
B.1 POPE Evaluation
POPE (Polling-basee Object Probing Evaluation) [26] is a simple yet effective framework for assessing object hallucinations LVMs. POPE formulates the evaluation of object hallucinations as a series of binary (yes/no) classification tasks. By sampling hallucinated objects, POPE constructs triplets of the form:
(x, c, w(yo),
(13)
21

---

The number of correctly identified objects that are present in the where z represents the queried image. is the query prompt template, and Wigt), is the ground-truth answer to the query. The triplets generated by POPE include those with a "yes" response based on ground-truth objects and "no" responses obtained by sampling from negative objects. There are three strategies for negative sampling:
Randomly samples objects that do not exist in the image. Popular Sampling: Selects the top-k most frequent objects in the image dataset that are absent from the current image.
Random Sampling:
Adversarial Sampling: Ranks all objects based on their co-occurrence frequencies with the ground-truth objects and selects the top-k frequent ones that do not exist in the image.
POPE employs the following evaluation metrics to measure performance:
TP + TN
Accuracy
(14)
TP + TN FP + FN
TP
Precision
(15)
TP FP TP
Recall TP + FN'
(16)
Precision Recall
F1 Score 2
(17)
Precision Recall
In the above equations:
TP (True Positives):
image.
TN (True Negatives): The number of correctly identified objects that are absent from the image.
FP (False Positives):
The number of objects incorrectly identified present in the image.
FN (False Negatives): The number of objects that are present in the image but were not identified by the model.
These metrics provide a comprehensive evaluation of the model's ability to accurately identify the presence or absence of objects, thereby quantifying the extent of hallucinations in LVMs.
B.2 BEAF Evaluation
BEAF (BEfore and AFter) [53] extends the evaluation framework beyond the text-axis hallucination assessment of POPE by simultaneously considering both text- and vision-axes. Additionally, BEAF introduces change-aware metrics, enabling a more granular evaluation of object hallucinations. Similar to POPE, BEAF employs binary classification tasks using triplets; however, it accounts for more complex perceptual changes within the dataset.
Dataset Definition BEAF utilizes dataset G composed of tuples:
G [(Xo, Xm, C, W. W... E
(18)
where X, denotes the original image. Xm represents the change-aware manipulate image. C is the question. W. and Wm, are the corresponding answers for the original and manipulated images, respectively. E (True, False) indicates whether the question pertains to object that has been removed in the manipulated image.
Filter Function To facilitate the extraction of specific subsets from G based on input conditions, BEAF defines a filter function:
Filter(b,, bm by) = fh IsCorrectWW.) = bo, IsCorrect(Wn.) = bm. E = b,, h G}. (19) where h (X,, Xm, C, W., W.. F). Here, b,, bms and b, are boolean values (True, False) that specify the desired correctness and relation flags for filtering.
22

---

Evaluation Metrics Based on the Filter function, BEAF defines the following fine-graine percep- tual change metries:
Fileer(True, True, True)]
TU [iitee(TTuee V False, True V False, True)]
x 100,
(20)
Filter(False, False, True)]
IG (True V False, True V False, True)
x 100,
(21)
Filter(True, False, True)]
SB, Filter(True V False, True V False, True)
100,
(22)
[Filter(False, True, True)]
SB,, Filter(TTue V False, True V False, True)]
100,
(23)
ID Fileer False, False) + |Filter(aalse, True, False)
100,
(24)
Filter(True False, True False, False)]
x TU
FITUD = 1+ (100 - ID)
(25)
where TU represents True Understanding, IG denotes Ignorance, SB refers Stubbornness, and ID signifies Indecision. These metrics provide a more nuanced evaluation of the model's capacity to recognize and adapt to perceptual changes across textual and visual contexts, offering a comprehensive assessment of hallucinations in LVMs.
C
More Details of Experiment Results
C.1 Evaluation of Text-Axis and Vision-/Text-Axis Hallucinations
Table 8 presents the performance evaluation of Precision (Prec.) and Recall under the POPE and BEAF experimental settings. The results demonstrate that our method achieves effective improvements in both text-axis and vision-/text-axi hallucination evaluations. While a slight decrease in Recall is observed in some cases, the overall performance exhibits significant enhancement. Notably, the decline in Recall is minimal, whereas the improvement in Precision more pronounced, further validating the effectiveness of our approach.
Table 8: Comparison of text-axis evaluation across three POPE evaluation settings: Random Sampling, Popular Sampling, and Adversarial Sampling the MSCOCO validation set. Additionally, vision- and text-axis evaluations are conducted under the BEAF benchmark. The values highlighted in green represent the percentage improvements achieved by proposed method, whereas the values in red indicate performance degradation.
LVM
Vision Input
POPE-Popular
HOPE-Randon
POPe-Adversaril
BEAF
Prec.7
Recall
Prec.
Recall
Prec.1
Recall
Prec.7
Recall?
LLLVA-115
Original
VVP
82.87
83,95 11.06
90.67
90.09
10.55
89.47
88.13
1.34
90.07
90.67
10000
75.27
74.45
90.73
92.04
11.51
61.77
62.32 10.56
92.43
92.13 0:30
Instruct-BLIIP
Originul
85.15
85.78 +0.65
81.67
8D.67
+1.00
95.70
94.83
4007
80.67
+1.00
82.21
+0.29
81.33
82.42
+1.00
67.47
67.00
+0.47
81.52
+0.31
Inter--VL.2
Original
VAP
97.41 13.50
62
74.13 12.25
21.90
98.07
97.40
40.07
73.58
71.71
1.87
94,50
92.50
+2.00
71.64
73.66
+2.00
87.40
88.76
1.30
72.35
72.24
+0.00
Intern Intem-VLL2-MP VL22-PO
Originul
93.70
80.39
95.39
80.95
90.55
81.08
82.46
82.67
+VAP
94,11
83.12
96.48
83.12
91.62
83,83
83.52 +1.06
82.73
DeepSeek
Original
VAP
93.52 11.00
80.80
80.13
96.70
97.34
40.64
80.81
80.13
0..5
91.06
92.39
80.93
84.11
85.12 +1.00
80.90
81.21
Qwen-VLL VL2
Originul
+VAP
91.15
92.34 +1.30
84.47
85.26
40.76
96.28
+1.13
84.47
+0.13
87.21
+3.05
84.87
+0.30
78.62
+1.41
83.14-000
83.81
LLaVA.OV
Original
VAP
95.20
96.97 +3.37
80.81
80.67
40.34
99.00
98.0 065
40.94
80.56
80.67
0.11
92.72
93.54
+3.82
$1.13
80.67
+0..00
87.58
88.17 +0.50
82.06 +0.37
81 66
Le-mmaa
Original
79.72
97.87
79.65
91.19
80.16
86.17
80:95
VAP
96.74
79.70
98.44
80.45
91.69
$1.03
86.92
82.27
23

---

C.2 Parameter Sensitive Analysis
Table 9 presents the parameter sensitivity analysis of the adversarial strategies loss function, as the parameters used in our approach vary across different models due to their distinct characteristics. The results indicate that parameter choices significantly impact performance metrics, including Accuracy (Acc.), Precision (Prec.), Recall (Rec.), and F1 -core (F1). Notably, the selection of vI/01, v1/02. and 1/a involves trade-off process, where optimizing one metric may lead to compromises in others. Interestingly, certain parameters yield competitive performance even when set to zero, suggesting potential redundancy in specific configurations. This trade-off underscores the necessity of carefully balancing parameter choices to achieve optimal overall performance.
Table 9: Parameter analysis of the Intern-V1.2 [7] under varying settings of Ø1, 02 and Ø3. The model parameters were fixed as vI/ai = 1.0, v1/02 0.5, and v1/03- = 0.5 without changing the values of v, 22, and Ø3. Performance comparison under the POPE Random evaluation setting, which involves randomly sampling objects that do not exist in the image. We randomly selected 1000
images from the MS-COCO dataset for this evaluation.
V1/ai
V1/02
Vl/ag
Value
Acc. 1
Prec.
Rec.
F1
Acc. 1
Prec.
Rec.
F1
Acc. 1
Prec. f
Rec.
F1
0.0
87.20
95.72
77.24
85.49
86.82
95.65
76.38
84.94
87.54
94.95
78.47
85 85.93
0.1
86.77
95.61
76.22
84.82
87.75
96.52
77.62
86.04
86.82
95.65
76.38
84.94
0.25
86.73
94.78
76.76
84.82
87.83
95.76
78.47
86.25
87.45
94.87
78.16
85.71
0.5
87.45
95.68
77.62
85.71
88.09
96.55
78.32
86.48
87.79
95.72
78.32
86.15
0.75
87.24
94.95
77.93
85.60
87.83
94.95
79.02
86.25
87.58
95.79
78.08
86.03
1.0
87.92
95.80
78.62
86.36
87.50
95.72
77.77
85.82
87.58
95.79
78.08
86.03
C.3
Impact of visual adversarial perturbation and uncertainty
Figure show how model performance varies with different perturbation strengths (c) and distortion levels (T). We observe that performance initially improves with moderate perturbations, peaking before declining as perturbations grow stronger. When e 2 16 or when 7 leads to full Gaussian noise, performance drops below the VoPVAP baseline. This indicates that (1) VAP effectively mitigates hallucinations by reducing semantic similarity between responses to original and distorted views under both conditional (c) and unconditional (0) settings, and (2) excessive perturbation harms visual feature extraction, undermining the model's ability to quantify parametric knowledge bias and ultimately degrading performance.
Porfer Performannce PMOMCCC Metrics (%)
Perfurmanc Natrica 8
12
602 Distorted Neise Stop
935
Perturbation Strength Lx]
Figure 5: Performance of Intern-VLA [7] under varying perturbation and distortion levels under POPE
setting. The model is tested with varying perturbations applied to the original and distorted images.
C.4 Ablation Study
Table 10 explore the effects of various combinations of loss functions (L., Loz L.g) on the performance of the Intern- VL. model under the POPE evaluation setting. The results, as presented in Table 10, indicate that the simultaneous application of all three loss functions yields the highest accuracy and Fl score, achieving 84.81 and 72%79, respectively. This suggests synergistic effect when combining these losses, enhancing the model's ability to generalize effectively. Notably, the combination of Ls, and L.,, also shows a significant improvement over using any single loss function, highlighting the importance of multi-faceted optimization strategies.

---

Table 10: Impact Different Loss Combinations on Model Performance: Ablation Study of Intern- VL2 Using the POPE Evaluation Setting.
L
Lea
Leg
Acc.
FIf
82.00
80.70
83.07
81.55
82.41
81.10
82.36
81.04
84.12
82.19
84.05
82.08
82.66
81.23
84.81
82.79
D Generalization of VAP
The high computational cost of optimizing adversarial strategies poses a significant challenge [20, 44, 1, 19]. practical approach mitigate this challenge leverage smaller-scale models as proxies to generate visual perturbations. Table 11 demonstrates the strong generalization capability of VAP, where perturbations generated by smaller models effectively enhance the performance of larger counterparts. Specifically, applying perturbations from the Intern- VL2- model to Intem-VL2-8B results 1.78% improvement in Fl score, while substantially reducing inference costs-requirin only 1/ of the A100 computation time per sample compared to Intern-VL2-8B. A similar pattern is observed in the Qwen- VL2 series, where proxy-generate noise also leads to consistent performance improvements in larger-scale models. Although the performance gains from proxy- proxy-based pertur- bations are slightly lower than those from target model-generatel noise, they provide an effective balance between computational efficiency and performance enhancement. These findings underscore the potential of VAP scaling hallucination suppression across models different sizes, offering scalable and resource-efficien solution for real-world applications.
Table 11: Generalization performance of VAP across different models. The table compares the results obtained from the original images (left value) and the perturbed images generated using source models under the VAP setting (right value). Experiments are conducted Intern-Vi VL2 and Qwen-VIe WLe models, with the best results highlighted in bold. The inference cost reduction, shown in the last row, is measured relative using the original target models.
Source: Intern-VI.. VL.-1BB
Source: Qwen-VL. VL.2-BB
Metric
Intern- ML2ern-VL2-IB
-L22-4
Intern- VL.2-BB
Qwen-VL2-2B
Qwen-VL.2-BB
Accuracy
69/88/832
81 81.55/82.56
$2.00/84.07
.4.47785.42
86.27/86.8
Precision
89.72/92.13
65.65/87.21
87.40/90.97
83.98/84.85
87.21/88.03
Recall
70.94/72.34
75. 75.05/75.90
72.24/75.50
84, 14.04/85.20
84.87/85.33
Score
79.23/81.04
80.00/81.16
80.70/82.52
84.01/85.05
86. 86.02/86.66
Inference Cost Reduction
1/3x
1/8x
1x
1/5x
25

---

E
Additional Illustration of Hallucination Evaluation
Figure presents comprehensive hallucination evaluation examples from eight state-of-the-an art LVMs, demonstrating the effectiveness of our proposed method across diverse model types. While different models exhibit varying response behaviors, our approach consistently mitigates hallucinations across all cases. Notably, in models such as Intern Bieern-LL-MM- VL2-MPO and Ovis 1aas1.66Gemmm 6-Gemma2, our method not only corrects erroneous responses but also facilitates the generation of more factually accurate reasoning. Moreover, our observations reveal that certain models exhibit fixed template-like responses to queries, such as L.LVVA-O which provides binary responses devoid of visual context. This characteristic underscores the challenges in improving performance for such models, their outputs of this nature pose difficulties in adversarial optimization scenarios. These results substantiate the effectiveness of the introduced visual noise VAP in alleviating hallucinations during the inference process, helping LVMs to achieve more reliable and content-aware predictions by reducing their reliance on spurious correlations and enhancing their focus on visually grounded evidence.
Original Image
A chere image?
Image?
Jolee cwwsain rhe ircgge?
Hic chere the image
No mhre rog
(a) InsPruct-BLIP
(b) LLaVA-OV
Original Image
Origiall trage
Kcheve chal' lmage?
Neleew aa the inage??
keage?
Yas than a the image. bouter ouar the tabk with the segebobiia
Ail, shore chilt image The snage festives sable pol oegantioons vickshey twestocs porsecs servets.
You hondllog everg image, pensel's ward.
hasslay the
(c) LL.
(d) Qwen-VIL
Organd
Ferturool Image
Original mage
Renurrdd Irage
there eeettrryytee Image?
Jum rastoryyeee the tmage?
Horco daysle in isage?
o'er duceckis themmgge?
matorgylic 3o. aeswer
restrepcle hoseeer thoe is bust leage.
NL dheee S.s blyyle the image
Fiss terca bitycle itage
(c) Intern-VI.2
(f) DeepSeck-Vil VL.2
hasshop the inage?
2
shore o handbigg truge??
Is choeorcto is reage?
S tharccctov in evage?
mie stww an noedhey tragee bakg caried by seoos attealt rest a hobtte auyboad thar utter band The hanahay a bust and dboalder
No. Ie the image thee pencc
thi more it moll the ircage to porslly slabte in hurigisues Baniball Chell wnrt The ofetee oxrr dis tzéle whire die piczo l plassa
bokdnn a aaffbord wict
alanang
Noy me louge doer sisw cloil retlngg with soitle chatoo the foome
carying bussees. Thore
are soisic kanabag dctr cariee
(g) Intern- VL2-MPO
(h) 1 1.vi1.6-Geemi Gemma2
Figure 6: Illustrative examples from the POPE hallucination evaluation across eight large vision- language models: (a) Instruct-BI.IP, (b) LLaVA-OV, (c) L.LVA--vI.5, (d) Qwen-VI.2, (e) Intern- VL2, (f) DeepSeek- VL.2, (g) -eeen-V..-MMP. and (h) Ovis1.6-Gemmn representative comparisons between original images and perturbed images enhanced with VAP, highlighting the differences in model responses.
The figure presents
26

---

Orthogonality and Complementaritt with Existing Methods
Unlike conventional model-centric approaches, our proposed method introduces novel paradigm for hallucination mitigation by exploiting the very mechanisms responsible for hallucinations to suppress them. This strategy offers fresh perspective aligning parametric knowledge with visual evidence in large vision-language models (LVMs).
To verify the orthogonality and compatibility of VAP with existing methods, integrate it with both OPERA [18], a recent state-of-the-ar suppression approach, and VCD, another competitive baseline. As shown in Table 12, across four strong LVMs and three evaluation settings (POPE, BEAF, CHAIR), VAP consistently provides complementary gains. For example, on LLaVA-v1.5, VAP + OPERA reduces 5 from 6.90 (Regular) to 6.10, while VAP VCD achieves an lower 5.80 with improved FITUID- Similar compounded benefits are observed on Intern- VL2, where TU rises from 64.12 (Regular) 66.78 (VAP + VCD). Although margins vary across models (e.g., smaller gains DeepSeek-VLL VL.2), the consistent trend demonstrates that VAP operates along an orthogonal axis and integrates effectively with both prior works.
In summary, VAP is metholologically orthogonal to existing strategies, intervening at the visual input level rather than architectural loss modifications, and delivers non-redundant improvements when combined with strong baselines such as OPERA and VCD. This establishes practical path for compounded effectivvness in future hallucination suppression systems.
Table 12: Comparison of hallucination suppression performance across four LVMs (LLaVA-v1.5 VL2, Intern- VL2, DeepSeek-VDD VL2) under three evaluation settings: POPE, BEAF, and CHAIR.
POPE
BEAF
CHAIR
LVM
Method
Acc.
F1
TU 1
Flyum
CHAIR, 4 CHAIRS
Regular
79.80
81.79
34.25
50.31
4.01
6.90
VCD
81.26
83.12
34.62
50.85
3.91
6.20
OPERA
80.32
81.92
34.51
50.48
3.95
6.70
LLaVA-v1.5
VAP
80.97
82.82
34.83
50.97
3.86
6.10
VAP VCD
$2.35
83.54
35.21
51.40
3.62
5,80
VAP OPERA
81.45
83.40
35.22
51.43
3.72
6.10
Regular
86.27
54.78
69.78
6.20
VCD
87.60
87.25
56.12
71.05
3.18
6.00
OPERA
86.68
86.42
55.34
70.18
6.00
Qwen-VL.2
VAP
87.30
87.02
56.18
70.79
3.23
5.70
VAP VCD
87.55
87.18
56.40
70.91
3.11
5,50
VAP OPERA
87.40
87.12
56.32
70.89
3.21
5.60
Regudar
82.00
80.70
64.12
76.17
5.14
9.50
VCD
84.32
82.30
65.88
77.43
4.72
9.00
OPERA
83.12
81.54
64.93
76.75
4.94
9,20
Intern-VL22
VAP
84.81
82.79
66.15
77.63
VAP + VCD
85.60
66.78
78.34
8.50
VAP + OPERA
85.09
66.35
77.78
Regudar VCD
86.47
85.55
67.04
79.27
1.84
4,50
86.65
85.72
67.25
79.43
1.80
4.40
OPERA
86.73
85.84
67.47
79.57
1.77
4.40
DeepSeck-VEL VL2
VAP
87.13
86.28
68.11
80.03
1.66
4.30
VAP + VCD
87.18
86.32
68.18
80.08
1.65
4.20
VAP + OPERA
87.20
86.35
68.22
80.11
1.64
4.20
G Dynamics under Visual Uncertainty
To further understand how hallucinations evolve under degraded vision, we progressively injected Gaussian noise (T) into the inputs of Intern-VIL and tracked two key indicators: S2 (prompt-driven hallucination) and S3 (prior-driven hallucination).
As shown Table 13, without VAP the S2/S3 values remain relatively static, confirming that baseline models lack an uncertainty-awarl mechanism to self-correct hallucinations. By contrast, with VAP the largest gains appear moderate noise levels (T 200), where input degradation sufficient to trigger hallucinations but still informative for grounding.
27

---

At low noise levels (T < 100), the model is already well grounded and improvements are minor, while high noise levels (T 2 500) the input becomes too corrupted. leading diminishing returns. These results demonstrate that VAP leverages uncertainty to suppress prompt- and prior-driven hallucinations, and effective realistic scenarios where vision is degraded but not lost.
Table 13: Dynamics of hallucination suppression under varying levels of visual uncertainty (Intern- VL2). VAP achieves maximal suppression at moderate noise (T' 200), confirming its ability to exploit uncertainty for robust grounding.
S2
S3
Noise T
w/o.1 4
w/ 4
AS2
w/o 4
w/ 4
AS3
0.75
N/A
N/A
0.79
N/A
N/A
100
0.73
0.64
0.09
0.76
0.69
0.07
200
0.70
0.60
0.10
0.74
0.66
0.08
300
0.68
0.59
0.09
0.72
0.65
0.07
500
0.65
0.57
0.08
0.70
0.64
0.06
700
0.61
0.53
0.08
0.65
0.60
0.05
999
0.45
0.40
0.05
0.50
0.47
0.03
H Experimental Evaluation of Perturbation Perceptibility
To confirm that VAP introduces minimal visual distortion, we evaluate the perceptual similarity between original and perturbed images on 500 BEAF image-instruction pairs. Specifically, we measure LPIPS and SSIM, two widely used perceptual similarity metries.
We consider four representative LVMs (LLaVA-v1.5, Qwen-VL2. Intern- VLL, and DeepSeek VL2). For each image, we compute perceptual distances between the original image and: (a) its VAP- perturbed version, and (b) ftaussaannooiee version of the same magnitude.
The results reveal three key observations. First, VAP perturbations are visually negligible: all models achieve LPIPS 0.05 and SSIM 0.95, which aligns with standard perceptual quality thresholds. Second, VAP consistently yields lower LPIPS and higher SSIM than Gaussian noise, demonstrating superior perceptual fidelity. Finally, this confirms that VAP introduces only minimal distortion, thereby preserving visual utility and maintaining trust for real-world deployment.
Table 14: Perceptual similarity between original and perturbed images, measured by LPIPS (J) and
SSIM (t). VAP perturbations remain visually negligible and consistently outperform Gaussian noise.
VAP Perturbation
Gaussian Noise
Model
LPIPS +
SSIM f
LPPPS + SSIM
LLaVA-v1.5
0.037
0.965
0.081
0.902
Qwen- Qw2
0.041
0.962
0.086
0.897
Intern- VL2
0.039
0.967
0.079
0.906
DeepSeek- VL2
0.035
0.969
0.077
0.911
I Algorithm Details of VAP
Algorithm 1 outlines the procedure of our visual adversarial perturbation (VAP) method. VAP mitigates object hallucinations LVMs by optimizing input perturbations that align model predictions more closely with visual evidence while reducing parametric bias. To handle the autoregressive nature of LVMs, we adopt a zeroth-order optimization strategy: sampling N perturbations and approximating the gradient of the adversarial loss without accessing internal model parameters. The final perturbation is projected onto a bounded constraint IB(e) before being applied, yielding perturbed inputs that effectively suppress hallucinations while preserving model usability.
28

---

Algorithm 1 Visual Adversarial Perturbation (VAP)
Adversarial Knowledge: Image Query LVM fa. Null text 0, CLIP Text encoder 9v- Noise magnitude e, Distorted timestep T, Noise scheduling μi, step size a. Zero-Gradien Setting: Number of queries N, Sampling variance B, Sampling noise
Adversarial Setting:
I: Generate a distorted image:
I M( (1 PT)1).
(26)
2: Compute initial responses:
(0) = c)( (0) = fo( fo(0, 0).
(27)
3: Compute initial adversarial loss:
( max ge (rl (T
g )r
(28)
(()) min Ie (ri Ie (r:),,)
(29)
( min go (r2 gr
(30)
4: Compute overall initial loss:
10)
(0)
5
Lis
(31)
2
0g
5: for each zero-grane prro-gradient optimization step E {1...., N do
6: Sample perturbation:
Pi P(t), s.t. Ely T 7
(32)
7: Compute perturbed responses:
1i [n] fo(x 3 In c),
(33)
22
(n) folw + 3 mn 0).
(34)
8: Compute adversarial losses:
((n) max gv (1(20) (r( (1))
(35)
((n) min (r (r( gv (ra),
(36)
52
((n) min (r (r) g (rz).
(37)
9: Compute overall adversarial loss:
((n) (n)
c(u)
<Sn)
(38)
of
end for
11: Estimate perturbation direction via zeroth-order optimization:
Σ (c
(n)
c/0 ).
(39)
N
Project perturbation onto 8 +- Proj
(z) (6).
13: Return response under VAP:
W(VAP) fo( aa
(40)
29

---

Discussion
J.1
Validation of Factual Comprehension
Our primary goal is to demonstrate that VAP does not impair the ability of models to comprehend factual content in images [14, 13, 55, 40J. Below, present quantitative evaluations substantiate this claim.
In Table 15, provide evidence that VAP sustains and enhances model performance in factual object recognition and open-ended factual understanding tasks:
(I) Non-Hallucination Task Evaluation (MME [14]):
We evaluated four LVMs using the MME benchmark, which includes tasks such existence detection, code reasoning, numerical calculations, and scene understanding. The results show that VAP maintains, and sometimes improves, accuracy in these factual and reasoning tasks. This confirms that VAP does not degrade performance on genuine questions,
(2)
Hallucination Grounding (AMBER [45]):
To assess generalization, we used the AMBER benchmark, which covers hallucinations in existence, attributes, and generative tasks. Our findings indicate that VAP enhances multi melti-dimeensonal visual grounding, further supporting its effectiveness without compromising factual understanding.
These evaluations collectively demonstrate that VAP enhances robustness while preserving the model's ' core perceptual and reasoning capabilities.
Table 15: Evaluation VAP MME and AMBER Benchmarks: Results show that VAP significantly enhances the models' abilities to accurately perceive, reason accurately, and ground visual content, confirming its effectiveness in reducing hallucinations while maintaining factual accuracy.
MME (Pereeption and Reasoning)
MME Totalt
AMBER (Hallucination Analysis)
LVM
Vision Input
Exist.
Code?
Calt
Scenet
Scoret
Covert
Hal-Rate,
Cogl
Original
93
40
982
51.7
4.2
LLaVA-v1.5
+VAP
95
43
1010
54.6
29.9
3.6
Original
95
78
73
81
1127
71.7
57.3
5.7
Qwen-VL2
+ VAP
98
80
75
84
1169
72.8
54.1
4,9
Original +AA
90
60
1114
73.7
68.8
8,4
Intern V1.2
93
63
1146
75.2
65.8
7.5
Original
95
45
1024
48.2
9.5
0.4
DeepSeck-VL VL2
+ VAP
98
48
1061
49.1
0.3
J.2
Understanding the Effectiveness of VAP
The consistent performance improvements across different LVMs and evaluation frameworks raise an important question: why does VAP effectively mitigate hallucinations? Our analysis reveals key mechanisms underlying VAP's effectiveness:
Balancing Visual and Language Signals The success of VAP be primarily attributed its abil- ity to rebalance the interaction between visual and language processing in LVMs. This is evidenced by both the significant reduction affirmative responses and performance improvements vision-/text- axis hallucination assessments (Table 2). The BEAF evaluation framework particularly demonstrates how VAP effectively interrupts the model's default reliance on parametric knowledge. The carefully calibrated perturbations strengthen visual signals during the inference process, compelling the model to ground its responses more firmly in visual evidence rather than language priors.
Adaptive Adversarial Noise Generation The effectiveness of VAP is further enhanced by its adaptive noise generation mechanism. Unlike traditional adversarial perturbations that aim to maxi- mally disrupt model predictions, VAP generates "beneficial noise" through zero groogiradeent optimization that aligns response with grounding vision input and mitigates parametric knowledge bias. This selective enhancement is validated across multiple evaluation dimensions: (1) Closed VQA format
30

---

evaluations through both text-axis (POPE) and vision-ttext-axis (BEAF) settings, and (2) Open-ender task evaluation through image caption generation (CHAIR). The consistent improvements across these diverse evaluation settings demonstrate VAP's ability to enhance visual understanding while maintaining task performance.
Architecture-Agnstt Enhancement Our experiments across different model architectures reveal that VAP's effectiveness is not tied to specific architectural choices. This architecture-agnostie nature can be explained by VAP's operation at the input level: it modifies the visual input distribution to better align with the model's learned visual-semantic mappings, regardless of the specific implemen- tation details. This explanation is supported by the consistent performance improvements observed across models with varying architectures, ranging from pure transformer basedd models to hybrid architectures across all three evaluation frameworks (POPE, BEAF, and CHAIR).
The combination of these mechanisms creates a powerful technique for hallucination mitigation:
The rebalancing of visual-language interaction enhances visual perception while reducing spurious correlations stemming from biased language priors.
The adaptive adversarial visual noise generation employs strategic optimization influence LVM decision processes, ensuring that perturbations enhance rather than compromise visual understanding.
VAP operates in completely black-box manner requiring access modification the LVM, establishing as broadly applicable solution across different model architectures.
31


Reviews:
[{"review_id": "bNGj8eGWFG", "reviewer": "Reviewer_PVr3", "summary": "The paper introduces an optimization-based method, leveraging adversarial strategies to generate beneficial visual perturbations that enhance the model’s factual grounding and reduce parametric knowledge bias. Extensive experiments on 8 LVLMs demonstrate the effectiveness of the proposed method.", "questions": "As we can see from Figure 2, the optimization process requires three times of LVLM inference, and I think the process should be at least three times of the original LVLM inference. However, in Table 4, Original + VAP increases the latency from 160ms to (160+298)ms. Why is the extra time less than 2 times? Can you specify what the A100 run time in Table 4 refers to? Is it time for inferencing from one image-query pair?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "[Strengths]\n1. The experiment on 8 LVLMs is really comprehensive and impressive. This demonstrates the generalizable property of the proposed method.\n\n[Weaknesses]\n1. The experiments only compare VAP with the original LVLM, while comparing with other training-free hallucination mitigation methods could be helpful. For example, VCD[1] applies noise to perturb the visual input and mitigate hallucination in a training-free way. \n2. I'm unsure about the efficiency of the method.  The method requires another LVLM to generate perturbation, and this could lead to extra GPU memory cost. And I also have a question about the comparison of latency. Please refer to [Questions].\n\n[1] VCD: Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding", "was_revised": true, "final_justification": "The authors' response addresses my concern regarding the efficiency, hence I would raise my score.", "rebuttal": "Response to Reviewer `PVr3`:\n\nThank you for your thoughtful and constructive feedback. Your recognition of the **generality** across 8 LVMs and the **comprehensiveness** of our evaluation helps reinforce the practical value of our approach. In response to your comments, we highlight several key strengths of VAP:\n\n- **Fully black-box and training-free**, requiring no access to model weights, gradients, or decoding processes;\n- **Deployment-friendly**, as it operates at inference time without any architectural modification;\n- **Consistently effective**, with generalization across models and datasets, independent of prompt or task.\n\nTo clarify and address your concerns, we performed the following analyses:\n\n- We now include **direct comparisons** with recent training-free baselines such as **VCD** and **OPERA**, highlighting both performance and design distinctions;\n- We clarify the **efficiency and latency** reported in Table 4, including implementation details such as **caching, batched inference**, and the role of a **proxy-based variant** for fast deployment.\n\n> We address each of your points in detail below:\n\n---\n\n`Q1:` Lack of comparison to other training-free baselines.  \n`A1:` We clarify our comparison to recent training-free hallucination mitigation methods as follows:\n\n**(1) Conceptual distinction**:  \n  VAP is **fully black-box**, requiring no access to model weights, logits, or decoding logic, and applies to any off-the-shelf LVM. In contrast, VCD [1] and OPERA [2] are **white-box** methods that rely on modifying the decoder or accessing internal representations.\n\n**(2) Capability comparison**:  \nBelow is a concise comparison highlighting the unique advantages of VAP:\n\n| Method | Black-box | No Training | No Model Access | Plug-and-Play | Intervention Stage |\n|---|:---:|:---:|:---:|:---:|---|\n| **VAP (Ours)** | ✓ | ✓ | ✓ | ✓ | Input (pre-decoding) |\n| VCD [1] | ✗ | ✓ | ✗ | ✗ | Decoder (distribution) |\n| OPERA [2] | ✗ | ✓ | ✗ | ✗ | Decoder (distribution) |\n\n**(3) Comparison with OPERA**:  \n  Table 11 shows that VAP consistently outperforms OPERA across 4 LVMs. Further combining both methods leads to stronger hallucination reduction.\n\n**(4) Comparison with VCD**:  \n  - Experiment Setup: We evaluate VAP and VCD on LLaVA-v1.5 and Intern-VL2. \n  - Observation: VAP achieves **comparable or better** hallucination mitigation, and combining VAP with VCD yields the **best overall performance**, highlighting their **complementarity**.\n\n|Model|Method|Acc.↑|F1↑|TU↑|F1_TUID↑|CHAIR_I↓|CHAIR_S↓|\n|---|---|---|---|---|---|---|---|\n|LLaVA-v1.5|Regular|79.80|81.79|34.25|50.31|4.01|6.90|\n| |VCD|81.26|83.12|34.62|50.85|3.91|6.20|\n| |VAP|80.97|82.82|34.83|50.97|3.86|6.10|\n| |VAP + VCD|**82.35**|**83.54**|**35.21**|**51.40**|**3.62**|**5.80**|\n|Intern-VL2|Regular|82.00|80.70|64.12|76.17|5.14|9.50|\n| |VCD|84.32|82.30|65.88|77.43|4.72|9.00|\n| |VAP|84.81|82.79|66.15|77.63|4.65|8.90|\n| |VAP + VCD|**85.60**|**83.41**|**66.78**|**78.34**|**4.41**|**8.50**|\n\nVAP offers a unique combination of **strong performance, full black-box compatibility, and real-world deployability**, which distinguishes it from existing approaches.\n\n[1] *VCD: Mitigating Object Hallucinations in LLMs via Visual Contrastive Decoding*, CVPR 2024.  \n[2] *OPERA: Alleviating Hallucination via Over-Trust Penalty and Retrospection-Allocation*, CVPR 2024.\n\n---\n\n`Q2:` Efficiency and latency concern.  \n`A2:` We clarify the efficiency measurement and optimization strategies as follows:\n\n**(1) Clarification of latency in Table 4**:  \n  The reported A100 runtime refers to the *end-to-end wall-clock latency* for a *single image–query pair* with batch size 1. It includes image preprocessing, tokenization, inference, and decoding. This is a practical measurement, not simply the sum of multiple forward passes.\n\n**(2) Why VAP does not incur full 3× cost**:  \n  While VAP involves three optimization steps:\n  - The image–text pair is encoded *once*, and the resulting `input_ids` are reused.\n  - Perturbed images are processed in *batched mode* (e.g., `image.repeat(...)`), enabling fast parallel inference on GPU.\n  As a result, the actual wall-clock overhead is less than 2×, as shown empirically.\n\n**(3) Efficient variant with proxy model**:  \n  We also introduce a *lightweight proxy-model* variant of VAP. As shown in Table 4 and Appendix D, it reduces the optimization overhead by up to 8× with minimal performance drop, making VAP suitable for real-time or resource-constrained deployment."}, {"review_id": "LJnQpb6EBD", "reviewer": "Reviewer_e6Ap", "summary": "This paper introduces Visual Adversarial Perturbation (VAP), a training-free, black-box method that mitigates object hallucinations in Large Vision-Language Models (LVMs) by injecting optimized \"beneficial\" visual noise into the input image. The core idea is to guide the model to ground its responses in actual visual content rather than relying on biased parametric knowledge, without altering the base model. The methodology formulates this as an optimization problem guided by a composite loss function with three objectives. Because direct gradient computation is challenging in LVMs, VAP employs a zero-gradient optimization technique. The final optimized noise is added to the original image to create a perturbed version that is used for inference. The authors validate their method across eight state-of-the-art LVMs using a comprehensive suite of benchmarks (POPE, BEAF, CHAIR), demonstrating consistent improvements in reducing hallucinations.", "questions": "1.  Could you please add more related works and elaborate on the novelty of VAP in direct comparison to decoding-stage methods like VCD and bias correction techniques like VisualGPTScore, which also leverage distorted images to mitigate hallucinations?\n2.  Given the significant hyperparameter sensitivity shown in Table 6, could you provide a more systematic methodology or a set of guiding principles for tuning these parameters for a new LVM or a new task/dataset? Without this, the method's practical applicability seems limited.\n3.  Could you justify the design choice of relying on unconditional generation ($f_{\\theta}(x, \\emptyset)$) within your loss functions, given the known instability of instruction-tuned LVMs in this setting? How does your method handle the significant variance in the loss objective introduced by different instructions $c$? Would it be better with $S(f_{\\theta}(x+\\delta,c), f_{\\theta}(\\bar{x},c))$ in $\\mathcal{L}_{s_2}$?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1.  The paper addresses object hallucination, which is a critical and widely recognized challenge that hinders the reliability and trustworthiness of modern LVMs. The \"poison as cure\" paradigm, which reframes adversarial noise as a corrective tool rather than an attack vector, is a novel contribution. The data-centric, training-free nature of the method makes it an attractive alternative to model-centric interventions.\n2.  The empirical evaluation is extensive and a clear strength of the paper. Testing on eight different LVMs and evaluating across multiple, diverse hallucination benchmarks (including text-axis, vision-axis, and open-ended tasks) provides strong evidence for the method's general effectiveness.\n\nWeaknesses:\n1.  The paper's positioning is weakened by an incomplete comparison to related work. Several existing methods use a similar setup of contrasting outputs from original and noisy/distorted images to mitigate hallucinations or language bias. For example, VCD [1] contrasts output distributions from original and distorted visual inputs, which is a very similar conceptual setting. VisualGPTScore [2] uses Gaussian noise to estimate and correct for language priors, directly tackling the \"parametric knowledge bias\" that VAP also targets. Earlier works like PixMix [3] have also explored noisy images as a form of data augmentation to improve model robustness. A thorough comparison and differentiation from these methods are crucial for contextualizing the unique contribution of VAP.\n2.  The method's effectiveness is heavily dependent on a set of hyperparameters that must be manually tuned for each specific LVM. As shown in Table 6, the optimal parameters vary across models, suggesting a costly and expertise-driven tuning process is required for any new model. Furthermore, the loss formulation, which contrasts prompted and unprompted responses, implies that the optimal hyperparameters may also be dependent on the specific dataset or the type of instruction used. The parameter analysis in Table 8 does not reveal a clear or monotonic relationship, reinforcing the idea that tuning is a complex trade-off without a simple heuristic. This makes the method appear hard to deploy reliably.\n3.  The method heavily relies on generating responses from a null prompt ($\\emptyset$). For a given image, the unprompted response $f_{\\theta}(x+\\delta, \\emptyset)$ is fixed, but the prompted response $f_{\\theta}(x+\\delta, c)$ will change dramatically with the instruction $c$. This means the optimization target is highly variable, which could lead to unpredictable behavior and further complicates the hyperparameter tuning challenge mentioned above. Besides, instruction-tuned LVMs are optimized to follow explicit prompts, and their behavior in an unconditional setting can be unpredictable, unstable, or produce low-quality text. Basing a core part of the optimization on such an unstable signal is a significant methodological concern.\n4.  The logic for some loss terms is unclear. For instance, in $\\mathcal{L}_{s_2}$, the method contrasts a prompted response with an unprompted response from a *negative* (distorted) image. A more symmetric and perhaps more stable approach would be to contrast two prompted responses, i.e., $S(f_{\\theta}(x+\\delta,c), f_{\\theta}(\\bar{x},c))$. The paper does not justify its specific design choice over more intuitive alternatives.\n\n\n\n### **References**\n[1] Leng, Sicong, et al. \"Mitigating object hallucinations in large vision-language models through visual contrastive decoding.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n[2] Lin, Zhiqiu, et al. \"Revisiting the Role of Language Priors in Vision-Language Models.\" International Conference on Machine Learning. PMLR, 2024.\n[3] Hendrycks, Dan, et al. \"Pixmix: Dreamlike pictures comprehensively improve safety measures.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.", "was_revised": true, "final_justification": "The authors have addressed my concerns regarding insufficient comparisons, hyperparameter settings, and methodological design choices. This is a valuable paper on mitigating object hallucination in VLMs and is likely to attract interest within the community.", "rebuttal": "Response to Reviewer `e6Ap`: Thank you so much for your constructive and insightful comments! We address your concerns point by point as follows:\n\n---\n`Q1:` Clarify novelty compared to VCD, VisualGPTScore, and PixMix.  \n`A1:`  Below is a concise comparison highlighting the unique advantages of VAP over related methods:\n\n| Method | Black-box | No Training | No Model Access | Plug-and-Play | Intervention Stage |\n|---|:---:|:---:|:---:|:---:|---|\n| **VAP (Ours)** | ✓ | ✓ | ✓ | ✓ | Input (pre-decoding) |\n| VCD [1] | ✗ | ✓ | ✗ | ✗ | Decoder (distribution) |\n| VisualGPTScore [2] | ✗ | ✓ | ✗ | ✗ | Output (post-decoding) |\n| PixMix [3] | N/A | ✗ | ✓ | ✗ | Training (data-level) |\n\n- **VAP (Ours)**:  \n  A **training-free**, **fully black-box** optimization method that requires **no access to model internals, gradients, or decoding steps**. VAP directly modifies the input image through targeted perturbation, enabling hallucination mitigation during inference without altering the model or its outputs.\n\n- **VCD [1]**:  \n  Operates in a **white-box** setting by injecting noise and modifying the **decoding distribution**. In contrast, VAP applies perturbation externally and does not depend on decoder access or distributional rewrites.\n\n- **VisualGPTScore [2]**:  \n  Estimates language priors using Gaussian noise and applies **post-hoc reweighting** of generated outputs. Unlike VAP, it requires generation scores and operates **after** response generation, whereas VAP intervenes **before decoding**, actively suppressing hallucinations.\n\n- **PixMix [3]**:  \n  A **training-time** data augmentation method that mixes stylized noise patterns. VAP, by contrast, is applied **at inference time**, requires **no fine-tuning**, and performs **per-instance optimization** guided by vision–language alignment.\n\nWe will include these clarifications in the revised related work section to better contextualize the contribution of VAP.\n\n[1] *Mitigating Object Hallucinations in LLMs via Visual Contrastive Decoding*, CVPR 2024.  \n[2] *VisualGPTScore: Visio-linguistic Reasoning with Multimodal Generative Pre-training Scores*, arXiv 2023.  \n[3] *PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures*, CVPR 2022.\n\n---\n\n`Q2:` Hyperparameter tuning appears model-dependent and costly.  \n`A2:` Below is a clarification of VAP’s hyperparameter efficiency and deployment practicality:\n\n**(1) Most hyperparameters are globally fixed:**  \n  Among the 8 hyperparameters in VAP, $( \\alpha, \\beta, N, \\epsilon) $ are **fixed across all models and datasets**. These require no tuning, significantly reducing setup overhead.\n\n**(2) Remaining parameters are configured once per model:**  \n  The remaining four parameters—**$(\\sigma_1, \\sigma_2, \\sigma_3, T)$**—are **tuned once per model** and reused across datasets and tasks. No re-tuning is needed when switching between instruction types or benchmarks.\n\n**(3) The parameter space is small and shared across models:**  \n  Each $\\sigma$ value is selected from a small discrete set {0.1, 0.5, 1.0}. In practice, several models (e.g., LLaVA-v1.5, InstructBLIP, DeepSeek-VL2, Ovis1.6-Gemma2) use the **same configuration**, supporting VAP’s generalizability and low tuning burden.\n\n**(4) Comparison with other training-free methods:**  \n  While training-free, **VCD [4] requires tuning temperature (e.g., temperature $T$) per model–dataset pair**, making it less scalable. In contrast, VAP avoids such dataset-specific coupling and is better suited for **black-box** deployment.\n\n[4] *Mitigating Object Hallucinations in LLMs via Visual Contrastive Decoding*, CVPR 2024.\n\n---\n\n`Q3:` Concerns about using unconditional generation in the loss function.  \n`A3:` Below is a clarification of our design choice and its empirical justification:\n\n**(1) Instruction-conditioned hallucinations require adaptive perturbation:**  \n  Hallucination severity varies with the instruction $c$, due to differing language priors [5]. VAP addresses this by generating perturbations specific to each $(x, c)$ pair. This variability is by design—it enables VAP to suppress *instruction-specific hallucinations* and generalize across diverse task formats.\n\n**(2) Optimization remains stable across prompts and datasets:**  \n  Despite prompt variability, all VAP hyperparameters are tuned *only once per model* and remain fixed across datasets and instruction types. This reflects the *stability and robustness* of the optimization process.\n\n**(3) Unprompted generation provides a reliable visual anchor:**  \n  When $c = \\emptyset$, the model outputs a vision-only caption [6], serving as a grounded baseline unaffected by instruction priors. This is a key anchor for semantic consistency. Empirically, unprompted generations exhibit lower hallucination rates:\n\n  | Model | Input | CHAIR-I ↓ | CHAIR-S ↓ |\n  |---|---|---|---|\n  | LLaVA-v1.5  | Prompt | 4.0 | 6.9 |\n  | | ∅ | **2.2** | **3.7** |\n  | Qwen-VL2 | Prompt  | 3.5 | 6.2 |\n  | | ∅ | **2.1** | **4.2** |\n\n  These results confirm that $\\emptyset$ serves as a *stable and hallucination-light anchor*, justifying its use in the VAP loss.\n\n[5] *PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training*, ICLR 2025.  \n[6] *Multi-modal Hallucination Control by Visual Information Grounding*, CVPR 2024.\n\n---\n\n`Q4:` Justification for contrasting prompted vs. unprompted responses in $\\mathcal{L}_{s_2}$.  \n`A4:` We clarify the rationale and support it with empirical evidence:\n\n**(1) Motivation:**  \n  - Instruction-tuned LVMs often overfit to language priors $P(y|c)$ [7]. \n  - This compromises visual grounding, especially in hallucination-prone prompts.  \n  - We contrast $P(y|x+\\delta, c)$ with the unprompted baseline $P(y|x)$ via $\\emptyset$ to suppress language bias and restore grounding in visual content.\n\n**(2) Supervision stability:**  \n  Compared to using two prompted responses, the unprompted generation ($c = \\emptyset$) provides a more stable and unbiased anchor. It reflects the vision-only interpretation of LVMs, free from instruction bias.\n\n**(3) Empirical justification:**  \n  Replacing $\\emptyset$ with a second prompted response in $\\mathcal{L}_{s_2}$ degrades both hallucination metrics and performance:\n\n  | Model        | Ref.       | Acc.↑   | F1↑    | CHAIR-I↓ | CHAIR-S↓ |\n  |--------------|------------|---------|--------|-----------|-----------|\n  | LLaVA-v1.5   | $c$        | 80.16   | 82.05  | 3.94      | 6.70      |\n  |              | $\\emptyset$ | **80.97** | **82.82** | **3.82**    | **6.50**    |\n  | Qwen-VL2     | $c$        | 86.48   | 86.33  | 3.35      | 5.90      |\n  |              | $\\emptyset$ | **87.30** | **87.02** | **3.23**    | **5.70**    |\n\n  These results confirm that contrasting with $\\emptyset$ improves both factual accuracy and hallucination robustness.\n\n[7] *Revisiting the Role of Language Priors in Vision-Language Models*, ICML 2024.\n\n---\n\n`Q5:` Guiding principles for hyperparameter tuning.  \n`A5:` VAP is designed with tuning efficiency and cross-task generalizability. Our tuning methodology is summarized below:\n\n**(1) Dataset-agnostic configuration:**  \n  All hyperparameters are selected once per model and reused across all datasets and tasks. No task-specific tuning is needed.\n\n**(2) Globally fixed adversarial parameters:**  \n  Four parameters $(N, \\alpha, \\beta, \\epsilon)$ are kept constant across all experiments. These follow standard interpretations (e.g., perturbation strength) and require no additional tuning.\n\n**(3) Lightweight model-specific tuning:**  \n  Only four parameters $(\\sigma_1, \\sigma_2, \\sigma_3, T)$ are tuned per model using a compact search space:  \n  - $\\sigma_i \\in \\{0.1, 0.5, 1.0\\}$  \n  - $T \\in \\{100, 200, 500, 800\\}$  \n  A validation set of just 300 image–instruction pairs is sufficient for reliable selection.\n\n**(4) Practical deployment:**  \n  This tuning process is efficient, reproducible, and requires minimal manual effort. We will add this strategy to the final version for better clarity and guidance in real-world use."}, {"review_id": "HZgjJLu01l", "reviewer": "Reviewer_c4Gd", "summary": "This paper introduces Visual Adversarial Perturbation (VAP), a training‑free, black‑box method designed to mitigate object hallucinations in large vision‑language models (LVMs). Rather than modifying model parameters, VAP strategically injects optimized visual noise into inputs, guiding the model to rely on actual visual content instead of biased parametric knowledge. The authors formulate hallucination suppression as an adversarial optimization problem, using zero‑gradient techniques over CLIP-based surrogate encoders to generate “beneficial” perturbations. Extensive experiments on eight state-of-the-art LVMs under three evaluation frameworks POPE, BEAF, and CHAIR demonstrate consistent reductions in hallucination metrics without degrading clean‑input performance", "questions": "- Table 10 shows that proxy-generated VAP (e.g., Intern-VL2-1B generating for Intern-VL2-8B) leads to slightly lower performance than self-generated VAP. While the efficiency gain is clear, could the authors discuss potential strategies to further bridge this performance gap, perhaps by exploring different proxy model architectures or more sophisticated knowledge transfer techniques?\n- While I understand that the scope of this work is limited to mitigating hallucinations, given that the authors have used adversarial noise to reduce hallucinations, have the authors explored the effect of robustness given a harmful adversarial image?", "limitations": "Please refer to the weaknesses.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n- The concept of \"visual noise as a cure\" for LVM hallucinations is very innovative. This data-centric, training-free, and black-box approach with three distinct loss terms significantly differentiates it from existing model-centric methods.\n- The paper presents a high-quality and extensive evaluation across eight diverse state-of-the-art LVMs and major hallucination and non-hallucination benchmarks. VAP consistently demonstrates significant reductions in object hallucinations across all tested models and settings without degrading factual comprehension.\n- VAP's black-box nature means it can be applied to any LVM without requiring internal modifications or retraining, and the introduction of an efficient proxy-based solution to generate VAP addresses the potential computational overhead.\n- The paper is well-written and clear, with a logical explanation of the VAP methodology and effective illustrative figures. The ablation study and parameter sensitivity analysis provide valuable insights into the method's robustness and the contribution of its components.\n\n**Weakness**\n- While Section 3.2 mentions the motivation behind each loss term and Table 9 shows the impact of different loss combinations, the paper lacks experimental justification for why the specific three-loss formulation is optimal or necessary.\n- Model-specific hyperparameter tuning ($\\sigma_{1}$, $\\sigma_{2}$, $\\sigma_{3}$, $T$) suggests the method may not be as universal as claimed.\n- While performance improvements are consistent, the magnitude of gains (~1–2% F1 in most cases) may be viewed as incremental, especially on already strong baselines (e.g., Intern‑VL2‑MPO), which may not justify the additional computational cost despite the proxy solution.", "was_revised": true, "final_justification": "The authors have addressed my initial concerns satisfactorily.\nThe paper is technically sound, and the contributions are clear. I believe the overall impact and scope remain somewhat limited. I also agree with reviewer NLrm that the method introduces additional complexity with multiple losses and hyperparameters, which yields only incremental performance improvements. Therefore, I am maintaining my rating", "rebuttal": "Response to Reviewer `c4Gd`: Thank you so much for your constructive and thorough feedback! We address your concerns as follows:\n\n---\n\n`Q1:` Justification for the specific three-loss formulation.  \n`A1:` We clarify the rationale and empirical support for our loss design as follows:\n\n**(1) Functional decomposition**  \nEach loss term addresses a different hallucination factor:\n\n| Loss Term       | Input Setting         | Goal                                 |\n|----------------|-----------------------|--------------------------------------|\n| $\\mathcal{L}_{s_1}$ | $(x{+}\\delta, c)$ vs. $(x{+}\\delta, c)$   | Enforce response consistency under perturbation (visual grounding) |\n| $\\mathcal{L}_{s_2}$ | $(x{+}\\delta, c)$ vs. $(\\bar{x}, ∅)$ | Contrast prompted vs. unprompted outputs (reduce prompt bias) |\n| $\\mathcal{L}_{s_3}$ | $(x{+}\\delta, ∅)$ vs. $(\\bar{x}, ∅)$ | Contrast uncertain responses (suppress parametric priors) |\n\n- $\\bar{x}$: strongly corrupted (negative) image  \n- $c$: user prompt; $∅$: no prompt  \n- $S(\\cdot,\\cdot)$: semantic similarity (e.g., cosine)\n\n**(2) Empirical justification**  \nAs shown in **Table 9**, incremental ablations demonstrate that:\n- Each component improves performance.\n- The **full loss consistently outperforms** partial variants across all metrics and benchmarks.\n\n**(3) No assumption of global optimality**  \nWhile empirically strong, we **do not claim** this loss formulation is globally optimal. Rather, it serves as a **generalizable and practical baseline** that may inspire further refinements in future work.\n\n---\n\n`Q2:` Concern about model-specific hyperparameter tuning $(\\sigma_1,\\sigma_2,\\sigma_3,T)$  \n`A2:` We address the concern about model-specific tuning as follows:\n\n| Aspect | Our Strategy |\n|----|-------|\n| **Granularity** | Tuned **once per model**, reused across all tasks and datasets |\n| **Search Space** | Small grid: $\\sigma_i \\in \\{0.1, 0.5, 1.0\\}$, $T \\in \\{100, 200, 500, 800\\}$ |\n| **Data Usage** | Only **300 validation samples** per model |\n| **Overhead** | Minimal; no prompt- or task-specific tuning |\n| **Black-box Compatible**  | Requires **no gradients or model internals** |\n| **Effectiveness** | Produces **consistent gains** across 8 diverse LVMs |\n\n> These practical settings ensure VAP remains efficient, generalizable, and well-suited to black-box deployment.\n\n---\n\n`Q3:` Concern about marginal performance gains (~1–2%) on strong baselines.  \n`A3:` We respond to the concern on gain magnitude as follows:\n\n**(1) Meaningful gains under strong, saturated baselines**:  \n  Intern‑VL2‑MPO and similar models already exhibit high performance, leaving limited headroom. Gains of 1–2% F1 in such settings are non-trivial—especially without model access or training.\n\n**(2) Training-free and black-box**:  \n  VAP operates with no access to model weights, gradients, or decoding steps. Within this constrained setting, few existing methods remain applicable, making even modest gains valuable.\n\n**(3) Consistent cross-model improvement**:  \n  VAP consistently enhances 8 diverse LVMs across multiple benchmarks. This robustness underscores its practical utility, even if individual gains appear incremental.\n\nWe will highlight this context more explicitly in the final version.\n\n---\n\n`Q4:` Can the performance gap from proxy-based VAP be reduced?  \n`A4:` Yes, our results suggest that the proxy gap can be effectively mitigated:\n\n**(1) Complementary to other black-box methods**:  \n  VAP is orthogonal to approaches like VCD [1]. Combining the two preserves VAP's efficiency while improving accuracy.\n\n**(2) Empirical support**:  \n  As shown below, proxy-based VAP (Intern-VL2-1B) achieves nearly the same performance as self-generated VAP (Intern-VL2-8B) when used with VCD:\n\n  | Vision Input | Proxy Model | Acc. ↑ | F1 ↑ |\n  |---|---|---|----|\n  | Original | — | 82.00  | 80.07 |\n  | + VAP | Intern-VL2-1B     | 84.07  | 82.52 |\n  | + VAP | Intern-VL2-8B     | 84.81  | 82.79 |\n  | + VAP + VCD | Intern-VL2-1B     | **84.95** | **83.03** |\n\n**(3) Future directions**:  \n  In future work, we plan to explore improved proxy selection (e.g., architecture-aligned models) and lightweight distillation to further bridge the proxy–target gap without sacrificing efficiency.\n\n[1] *Mitigating Object Hallucinations in LLMs via Visual Contrastive Decoding*, CVPR 2024.\n\n---\n\n`Q5:` Does adversarial noise used in VAP also improve robustness against harmful perturbations?  \n`A5:` We conducted robustness evaluations following [2] using standard adversarial attacks. Results are summarized below:\n\n| Model | Input | Acc.↑ | F1↑ | TU↑ | F1_TUID↑ | CHAIR_I↓ | CHAIR_S↓ |\n|---|---|---|---|---|---|---|---|\n| LLaVA‑v1.5 | Clean | 79.80 | 81.79 | 34.25 | 50.31 | 4.01 | 6.90 |\n| | Adversarial | 68.40 | 71.86 | 27.93 | 41.24 | 5.47 | 8.10 |\n| | VAP | **80.97** | **82.82** | **34.82** | **50.97** | **3.86** | **6.50** |\n| Intern‑VL2 | Clean | 82.00 | 80.70 | 64.12 | 76.17 | 5.14 | 9.50 |\n| | Adversarial | 70.83 | 69.45 | 55.23 | 66.08 | 8.32 | 11.35 |\n| | VAP | **84.08** | **82.97** | **66.15** | **77.63** | **4.65** | **8.90** |\n\n- Harmful adversarial perturbations degrade both performance and grounding, as expected.  \n- VAP does *not* rely on adversariality; its effect comes from *semantically guided optimization*.  \n\nThese results confirm that VAP’s improvements stem from targeted input adaptation rather than robustness enhancement.\n\n[2] *On Evaluating Adversarial Robustness of Large Vision-Language Models*, NeurIPS 2023."}, {"review_id": "sSrodf9Gkl", "reviewer": "Reviewer_NLrm", "summary": "This paper proposes Visual Adversarial Perturbation (VAP), a method to mitigate object hallucination in LVMs. The authors attribute hallucination to two factors: the long-tail distribution of training data and the extensive parameter space of LVMs. VAP introduces adversarial perturbations generated via three key strategies: (1) Enhancing semantic alignment between model responses with and without conditional guidance, (2) Mitigating knowledge bias by aligning perturbed images (with guidance) to noise images (without guidance). The authors demonstrate VAP’s effectiveness in reducing hallucinations while preserving output fidelity across eight SOTA LVMs, evaluated on five metrics.", "questions": "- In contrast to adversarial attack and poisoning, this method does not require imperceptibility of noise, so constraining epsilon is not necessary in this setting. Is there any comment on it?", "limitations": "- The paper does not compare VAP with other hallucination mitigation methods, making it difficult to assess its relative advantages.\n- The performance of VAP heavily depends on 8 hyperparameters (σ₁, σ₂, σ₃, T, alpha, beta, N, and ε), which hinders in real-world applications.\n- The proposed method requires additional computations for visual noise generation, which slows down inference speed by nearly x2.\n- The claim that hallucinations stem from long-tail data and large parameter spaces lacks empirical or theoretical support. The paper does not explain how these factors directly cause hallucinations or how VAP’s strategies counteract them. For instance, Strategy S1 (alignment of LVM responses with visual grounding) minimizes the influence of conditional text, which may inadvertently amplify biases in the original image rather than mitigate them.\n- The concept of visual uncertainty is introduced without clarifying its connection to parametric knowledge bias.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- The paper is well written and easy to understand. The figures and descriptions of their method are clear.\n- There are comprehensive evaluations, where the method is tested on 5 metrics and 8 models.\n\nWeaknesses\n\n- Please refer to the limitation and question section.", "was_revised": true, "final_justification": "I initially raised concerns regarding the soundness of the proposed method, and I appreciate that these have been satisfactorily addressed in the rebuttal. Accordingly, I have increased my score from 2 to 3. However, I remain hesitant to give an above-threshold score. The method introduces multiple losses and hyperparameters, which contribute to only incremental performance gains. This added complexity may hinder its practical applicability, particularly in real-world settings where extensive hyperparameter tuning can be a significant limitation. \n\nWhile I would not object to the paper being accepted—similar to other works of this nature that have been accepted—I personally lean toward rejection.", "rebuttal": "Response to Reviewer `NLrm`:\n\nThank you for your thoughtful and constructive review. We are glad that you found the paper **clear** and appreciated the **comprehensive evaluation across 8 LVMs and 5 metrics**. Building on your observations, we emphasize the following strengths of our method:\n\n- **Black-box and training-free**: VAP works without accessing model weights, gradients, or internal logits, making it suitable for closed-source APIs and real-world applications.\n- **Plug-and-play**: Applied directly at inference time, with no modification to model architecture or need for retraining.\n- **Task- and dataset-agnostic**: Once configured per model, VAP generalizes across benchmarks and prompt types.\n- **Consistently effective**: Reduces hallucinations while preserving factual grounding across diverse LVMs.\n\nTo clarify your concerns and resolve misunderstandings, we conducted the following efforts:\n\n- Explained the rationale behind constraining $\\epsilon$ (semantic stability, not imperceptibility).\n- Compared VAP with **OPERA** and **VCD**, showing competitive or better results.\n- Clarified the empirical basis of our design and the role of **visual uncertainty**.\n- Refined the interpretation of **Strategy S1** to highlight its goal of reducing prompt overreliance.\n\nWe sincerely thank you for your feedback, which has helped improve the clarity and depth of our work!\n\n> We address each of your points in detail below:\n\n---\n\n`Q1:` Why constrain $\\epsilon$ if imperceptibility is not required?  \n`A1:` We constrain $\\epsilon$ not for imperceptibility, but to ensure:\n\n- **Preserve semantic integrity**: Large perturbations may distort key visual features and degrade response quality.\n- **Ensure reliable bias mitigation**: As shown in Figure 5, excessive $\\epsilon$ disrupts feature extraction, reducing performance.\n- **Maintain stability**: A bounded $\\epsilon$ improves consistency across prompts and supports generalizable use in multi-turn settings.\n\n---\n\n`Q2:` Lack of comparison to other hallucination mitigation methods.  \n`A2:` We provide direct comparisons to two recent training-free methods:\n\n**(1) OPERA [1]**:  \n  As shown in Table 11, VAP outperforms OPERA across 4 LVMs on F1_TUID and CHAIR scores. Combining VAP with OPERA further reduces hallucination, showing complementarity.\n\n**(2) VCD [2]**:  \n  On LLaVA-v1.5 and Intern-VL2, VAP achieves comparable or better results than VCD. Their combination yields the strongest performance:\n\n|Model|Method|Acc.↑|F1↑|TU↑|F1_TUID↑|CHAIR_I↓|CHAIR_S↓|\n|---|---|---|---|---|---|---|---|\n|LLaVA-v1.5|VCD|81.86|83.12|34.62|50.85|3.91|6.30|\n| |VAP|80.97|82.82|34.83|50.97|3.86|6.10|\n| |VAP+VCD|**82.35**|**83.54**|**35.21**|**51.40**|**3.62**|**5.80**|\n|Intern-VL2|VCD|84.32|82.30|65.88|77.43|4.72|9.00|\n| |VAP|84.81|82.79|66.15|77.63|4.65|8.90|\n| |VAP+VCD|**85.60**|**83.41**|**66.78**|**78.34**|**4.41**|**8.50**|\n\n**(3) Design difference**:  \nVCD requires decoder access (white-box); VAP is fully black-box and more generalizable.\n\n[1] *OPERA: Alleviating Hallucination via Over-Trust Penalty and Retrospection-Allocation*, CVPR 2024.  \n[2] *VCD: Mitigating Object Hallucinations in LLMs via Visual Contrastive Decoding*, CVPR 2024.  \n\n---\n`Q3:` The method relies on 8 hyperparameters, which may hinder real-world deployment.  \n`A3:` We address this through a lightweight and structured configuration strategy:\n\n**(1) Four parameters are globally fixed**:  \n$(\\alpha, \\beta, N, \\epsilon)$ are shared across all models and datasets—no tuning required.\n\n**(2) The remaining four are model-specific but dataset-agnostic**:  \n$(\\sigma_1, \\sigma_2, \\sigma_3, T)$ are selected once per model and reused across tasks.\n\n**(3) Compact search space**:  \n$\\sigma_i \\in \\{0.1, 0.5, 1.0\\}$ and $T \\in \\{100, 200, 500, 800\\}$. Several models share identical configurations.\n\n**(4) More scalable than prior methods**:  \nVCD [3] requires tuning per model–dataset pair. VAP avoids this, enabling easier deployment.\n\n[3] *Mitigating Object Hallucinations in LLMs via Visual Contrastive Decoding*, CVPR 2024.\n\n---\n\n`Q4:` VAP increases the inference cost due to additional computations.  \n`A4:` To address the added cost, we introduce a fast proxy-based variant:\n\n**(1) Proxy-based variant significantly reduces cost**:  \n  A lightweight **proxy-VAP** variant achieves **up to 8× faster inference** while preserving strong hallucination reduction (Table 4, Appendix D).\n\n**(2) Zero training, zero gradient, broad generalization**:  \n  VAP requires no model access or fine-tuning, and generalizes across datasets without reconfiguration—offsetting the one-time cost with wide applicability.\n\n---\n\n`Q5:` Lack of theoretical support for the hallucination cause and mitigation mechanism.  \n`A5:` Our explanation is empirical, not theoretical, and grounded in observed LVM behavior:\n\n**(1) Empirical observation, not formal theory**:  \n  The link between hallucinations and long-tail data or parametric priors is based on engineering intuition and patterns documented in prior work [4-7], rather than formal derivation.\n\n**(2) Evidence from literature**:  \n  Instruction-tuned LVMs often default to language priors—especially on underrepresented (long-tail) concepts—when visual evidence is weak, leading to plausible but ungrounded outputs [5,7].\n\n**(3) Clarification of Strategy S1**:  \n  S1 aligns outputs from the prompted and unprompted image, reinforcing grounding in the actual visual input—not removing the condition—but reducing overreliance on the prompt alone.\n\nWe will clarify that these claims are empirically motivated and supported by prior studies.\n\n[4] *Mitigating hallucination in large multi-modal models via robust instruction tuning*, ICLR 2023.  \n[5] *Evaluating object hallucination in large vision-language models*, EMNLP 2023.  \n[6] *What matters when building vision-language models?*, NeurIPS 2024.  \n[7] *Revisiting the Role of Language Priors in Vision-Language Models*, ICML 2024.\n\n---\n\n`Q6:` Clarifying the role of visual uncertainty in diagnosing parametric knowledge bias.  \n`A6:` Visual uncertainty plays a diagnostic role in revealing the extent to which LVMs rely on internal parametric knowledge:\n\n**(1) What is visual uncertainty?**  \n  We define it by gradually degrading the image using Gaussian noise (Eq. 4). Increasing the noise scale $T$ simulates reduced visual information, allowing us to measure the model’s shifting reliance from visual evidence to parametric priors.\n\n**(2) How it reveals parametric bias**:  \n  When visual information deteriorates, models increasingly rely on $P(y|c)$ instead of $P(y|x,c)$. This transition exposes the influence of internal biases, which is exactly what VAP aims to mitigate.\n\n**(3) Empirical evidence**:  \n  Without applying VAP, we evaluate LLaVA-v1.5 and Qwen-VL2 on BEAF under increasing $T$. Both accuracy and F1 degrade sharply as noise increases, confirming that hallucinations intensify under visual uncertainty:\n\n| LLaVA-v1.5 | T=0 | 100 | 200 | 300 | 400 | 500 | 700 | 999 |\n|------------|-----|-----|-----|-----|-----|-----|-----|-----|\n| Acc. ↑     |79.99|78.42|76.31|74.05|71.84|69.88|59.46|51.04|\n| F1 ↑       |74.06|72.55|70.33|68.02|65.44|62.97|47.25|31.48|\n\n| Qwen-VL2   | T=0 | 100 | 200 | 300 | 400 | 500 | 700 | 999 |\n|------------|-----|-----|-----|-----|-----|-----|-----|-----|\n| Acc. ↑     |87.96|86.18|84.09|81.50|79.31|76.83|65.92|57.08|\n| F1 ↑       |81.13|79.41|77.38|74.94|72.13|69.47|53.24|35.21|\n\nThese results confirm that visual uncertainty directly correlates with increased hallucination, validating its role in revealing and mitigating parametric knowledge bias."}, {"review_id": "RS7WAwW5Qe", "reviewer": "Reviewer_997H", "summary": "This paper addresses the persistent issue of object hallucinations in large vision-language models (LVMs). Rather than altering model parameters or decoding mechanisms, the authors propose a training-free, black-box solution called Visual Adversarial Perturbation (VAP).\nVAP works by injecting strategically optimized, imperceptible noise into visual inputs. This adversarial noise is not harmful but beneficial. It shifts the model's attention away from biased parametric knowledge (often over-learned from language) and instead grounds its responses in the actual visual content.", "questions": "1. Can the authors quantify how perceptible the adversarial noise is to human observers? Have any perceptual similarity metrics (e.g., LPIPS, SSIM) or user studies been conducted to ensure that utility or trust is not compromised?\n\n2. While the method is effective for object hallucinations, have the authors tested whether VAP can reduce attribute hallucinations (e.g., color, count, action)?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This paper presents a well-executed approach to mitigating object hallucination in vision-language models. The method is training-free, black-box compatible, and extensively validated across multiple benchmarks and models, demonstrating strong empirical quality. \n\nHowever, the paper lacks theoretical clarity on why the perturbations consistently align model responses with visual content, and it does not adequately address the perceptual or downstream effects of added noise. While the writing is generally clear, the optimization strategy and loss formulation could benefit from more intuitive explanations.", "was_revised": false, "final_justification": "", "rebuttal": "Response to Reviewer `997H`: We sincerely thank you for the thoughtful and insightful feedback! Your suggestions prompted us to evaluate **perceptual fidelity** and **attribute understanding**, confirming that VAP **preserves factual grounding** without harming visual understanding. These additions further strengthen the **clarity** and **empirical support** of our claims.\n\nBelow, we address each point in detail:\n\n---\n\n`Q1:` Experimental evaluation of perturbation perceptibility\n`A1:` We evaluate the perceptual similarity between original and perturbed images using LPIPS and SSIM on 500 *BEAF* image–instruction pairs.\n\n**(1) Setup:**  \n  We evaluate four representative LVMs: **LLaVA-v1.5, Qwen-VL2, Intern-VL2**, and **DeepSeek-VL2**.  \n  For each image, we compute LPIPS and SSIM between the original image and:  \n  - (a) its **VAP-perturbed** version.\n  - (b) a **Gaussian-noised** version with the same $L_2$ magnitude.\n\n**(2) Results:**\n\n| Model | LPIPS (VAP) ↓ | SSIM (VAP) ↑ | LPIPS (Gaussian) ↓ | SSIM (Gaussian) ↑ |\n|---|---|---|---|---|\n| LLaVA-v1.5 | 0.037 | 0.965 | 0.081 | 0.902 |\n| Qwen-VL2   | 0.041 | 0.962 | 0.086 | 0.897 |\n| Intern-VL2 | 0.039 | 0.967 | 0.079 | 0.906 |\n| DeepSeek-VL2 | 0.035 | 0.969 | 0.077 | 0.911 |\n\n**(3) Observations**  \n  - **VAP perturbations are visually negligible**: All LPIPS < 0.05 and SSIM > 0.95, aligning with standard perceptual thresholds.\n  - **Better than Gaussian noise**: VAP consistently yields **lower LPIPS and higher SSIM**, indicating stronger perceptual fidelity.\n  - **Conclusion**: VAP introduces minimal visual distortion, preserving utility and trust for real-world deployment.\n\n---\n\n`Q2:` Can VAP reduce attribute hallucinations (e.g., color, count, action)?  \n`A2:` **Yes.** VAP is designed to address object hallucinations, but it also proves effective in reducing *fine-grained hallucinations* (e.g., color, count, position). Results show that VAP **consistently improves attribute-level grounding without harming factual understanding**.\n\n**(1) Benchmark Results (MME & AMBER)**\nWe report extended results in the appendix (Table 12), including performance on **MME (Perception + Reasoning)** and **AMBER (Hallucination Analysis)**. VAP improves both factual understanding and hallucination robustness across four LVMs:\n\n| Model        | Input   | MME↑ | Exist.↑ | Code↑ | Cal↑ | Scene↑ | Coverage↑ | Halluc.↓ | Cog.↓ |\n|--------------|---------|------|---------|-------|------|--------|-----------|-----------|--------|\n| LLaVA-v1.5   | Orig.   | 982  | 93      | 50    | 40   | 83     | 51.7          | 35.4      | 4.2       |\n|              | +VAP    | 1010 | 95      | 55    | 43   | 86     | 54.6          | 29.9      | 3.6       |\n| Qwen-VL2     | Orig.   | 1127 | 95      | 78    | 73   | 81     | 71.7          | 57.3      | 5.7       |\n|              | +VAP    | 1169 | 98      | 80    | 75   | 84     | 72.8          | 54.1      | 4.9       |\n| Intern-VL2   | Orig.   | 1114 | 90      | 75    | 60   | 83     | 73.7          | 68.8      | 8.4       |\n|              | +VAP    | 1146 | 93      | 80    | 63   | 87     | 75.2          | 65.8      | 7.5       |\n| DeepSeek-VL2 | Orig.   | 1024 | 95      | 40    | 45   | 78     | 48.2          | 9.5       | 0.4       |\n|              | +VAP    | 1061 | 98      | 45    | 48   | 81     | 49.1          | 9.0       | 0.3       |\n\n→ These results confirm that VAP enhances **factual perception** and **reduces hallucination** across diverse LVMs.\n\n**(2) More Attribute-Level Performance** \nPer your request, we provide **more granular evaluations** on attribute-related categories using **MME** and **MMHal**[1]. VAP improves grounding in color, count, position, and comparison categories:\n\n\n**MME Benchmark**\n\n| Model        | Input     | Color↑ | Count↑ | Position↑ | Artwork↑ |\n|--------------|-----------|--------|--------|-----------|----------|\n| LLaVA-v1.5   | Original  | 78.3   | 75.0   | 70.0      | 61.3     |\n|              | +VAP      | 81.6   | 80.0   | 76.7      | 63.0     |\n| Qwen-VL2     | Original  | 86.7   | 70.0   | 78.3      | 75.8     |\n|              | +VAP      | 90.0   | 76.7   | 81.7      | 78.0     |\n| Intern-VL2   | Original  | 90.0   | 70.0   | 73.3      | 83.8     |\n|              | +VAP      | 93.3   | 73.3   | 76.7      | 84.5     |\n\n**MMHal Benchmark**\n\n| Model        | Input     | Attribute↑ | Object↑ | Comparison↑ | Counting↑ |\n|--------------|-----------|------------|---------|--------------|------------|\n| LLaVA-v1.5   | Original  | 2.83       | 1.17    | 1.34         | 2.00       |\n|              | +VAP      | 3.00       | 1.41    | 1.83         | 2.33       |\n| Qwen-VL2     | Original  | 2.17       | 2.83    | 2.50         | 3.17       |\n|              | +VAP      | 2.83       | 3.17    | 3.00         | 3.50       |\n| Intern-VL2   | Original  | 2.83       | 2.42    | 2.83         | 2.34       |\n|              | +VAP      | 3.00       | 2.67    | 3.34         | 2.67       |\n\n### (3) Conclusion  \nVAP mitigates **object-level hallucinations** and consistently reduces **attribute-related errors**, while preserving the **factual reasoning capabilities**.\n\n[1] *Aligning Large Multimodal Models with Factually Augmented RLHF*, ACL 2024."}]

Author rebuttal:

Meta-review: This paper proposes a black-box, training-free method that introduces optimized visual noise to reduce object hallucinations in LVMs. Reviewers found the problem important and the idea of poison as cure innovative. A key strength lies in the extensive experiments across eight state-of-the-art models and multiple benchmarks, showing consistent improvements. Reviewers also appreciated the plug-and-play nature of the method, its general applicability to closed-source APIs, and the careful empirical comparisons added in the rebuttal, including with VCD and OPERA. The rebuttal further clarified concerns about efficiency, perceptual fidelity, and hyperparameter tuning, providing evidence that the method is broadly applicable and not overly burdensome to deploy.

That said, some reservations remain. Multiple reviewers pointed out that the method relies on several loss terms and hyperparameters, raising questions about robustness and deployment practicality. While improvements are consistent, their magnitude is often incremental, especially on strong baselines, and may not fully justify the added complexity. Concerns about theoretical grounding, the necessity of all losses, and reliance on hyperparameter tuning were partially addressed but not fully resolved. On balance, the work presents a technically sound contribution with clear empirical value, though its impact is somewhat limited.
Decision: Accept (poster)