Title: Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Vision-Langugae Model

Object Hallucination
Abstract: Large vision-language models (LVMs) extend large language models (LLMs) with visual perception capabilities, enabling them to process and interpret visual information. A major challenge compromising their reliability is object hallucination that LVMs may generate plausible but factually inaccurate information. We propose a novel \textit{visual adversarial perturbation (VAP)} method to mitigate this hallucination issue. VAP alleviates LVM hallucination by applying strategically optimized visual noise without altering the base model. Our approach formulates hallucination suppression as an optimization problem, leveraging adversarial strategies to generate beneficial visual perturbations that enhance the model's factual grounding and reduce parametric knowledge bias. Extensive experimental results demonstrate that our method consistently reduces object hallucinations across 8 state-of-the-art LVMs, validating its efficacy across diverse evaluations.


Peer reviews:
comments: SUMMARY:
This paper addresses the persistent issue of object hallucinations in large vision-language models (LVMs). Rather than altering model parameters or decoding mechanisms, the authors propose a training-free, black-box solution called Visual Adversarial Perturbation (VAP).
VAP works by injecting strategically optimized, imperceptible noise into visual inputs. This adversarial noise is not harmful but beneficial. It shifts the model's attention away from biased parametric knowledge (often over-learned from language) and instead grounds its responses in the actual visual content.

STRENGTHS AND WEAKNESSES:
This paper presents a well-executed approach to mitigating object hallucination in vision-language models. The method is training-free, black-box compatible, and extensively validated across multiple benchmarks and models, demonstrating strong empirical quality. 

However, the paper lacks theoretical clarity on why the perturbations consistently align model responses with visual content, and it does not adequately address the perceptual or downstream effects of added noise. While the writing is generally clear, the optimization strategy and loss formulation could benefit from more intuitive explanations.

QUESTIONS:
1. Can the authors quantify how perceptible the adversarial noise is to human observers? Have any perceptual similarity metrics (e.g., LPIPS, SSIM) or user studies been conducted to ensure that utility or trust is not compromised?

2. While the method is effective for object hallucinations, have the authors tested whether VAP can reduce attribute hallucinations (e.g., color, count, action)?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Visual Adversarial Perturbation (VAP), a method to mitigate object hallucination in LVMs. The authors attribute hallucination to two factors: the long-tail distribution of training data and the extensive parameter space of LVMs. VAP introduces adversarial perturbations generated via three key strategies: (1) Enhancing semantic alignment between model responses with and without conditional guidance, (2) Mitigating knowledge bias by aligning perturbed images (with guidance) to noise images (without guidance). The authors demonstrate VAP’s effectiveness in reducing hallucinations while preserving output fidelity across eight SOTA LVMs, evaluated on five metrics.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The paper is well written and easy to understand. The figures and descriptions of their method are clear.\n- There are comprehensive evaluations, where the method is tested on 5 metrics and 8 models.\n\nWeaknesses\n\n- Please refer to the limitation and question section.\n\nQUESTIONS:\n- In contrast to adversarial attack and poisoning, this method does not require imperceptibility of noise, so constraining epsilon is not necessary in this setting. Is there any comment on it?\n\nLIMITATIONS:\n- The paper does not compare VAP with other hallucination mitigation methods, making it difficult to assess its relative advantages.\n- The performance of VAP heavily depends on 8 hyperparameters (σ₁, σ₂, σ₃, T, alpha, beta, N, and ε), which hinders in real-world applications.\n- The proposed method requires additional computations for visual noise generation, which slows down inference speed by nearly x2.\n- The claim that hallucinations stem from long-tail data and large parameter spaces lacks empirical or theoretical support. The paper does not explain how these factors directly cause hallucinations or how VAP’s strategies counteract them. For instance, Strategy S1 (alignment of LVM responses with visual grounding) minimizes the influence of conditional text, which may inadvertently amplify biases in the original image rather than mitigate them.\n- The concept of visual uncertainty is introduced without clarifying its connection to parametric knowledge bias.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Visual Adversarial Perturbation (VAP), a training‑free, black‑box method designed to mitigate object hallucinations in large vision‑language models (LVMs). Rather than modifying model parameters, VAP strategically injects optimized visual noise into inputs, guiding the model to rely on actual visual content instead of biased parametric knowledge. The authors formulate hallucination suppression as an adversarial optimization problem, using zero‑gradient techniques over CLIP-based surrogate encoders to generate “beneficial” perturbations. Extensive experiments on eight state-of-the-art LVMs under three evaluation frameworks POPE, BEAF, and CHAIR demonstrate consistent reductions in hallucination metrics without degrading clean‑input performance\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The concept of \"visual noise as a cure\" for LVM hallucinations is very innovative. This data-centric, training-free, and black-box approach with three distinct loss terms significantly differentiates it from existing model-centric methods.\n- The paper presents a high-quality and extensive evaluation across eight diverse state-of-the-art LVMs and major hallucination and non-hallucination benchmarks. VAP consistently demonstrates significant reductions in object hallucinations across all tested models and settings without degrading factual comprehension.\n- VAP's black-box nature means it can be applied to any LVM without requiring internal modifications or retraining, and the introduction of an efficient proxy-based solution to generate VAP addresses the potential computational overhead.\n- The paper is well-written and clear, with a logical explanation of the VAP methodology and effective illustrative figures. The ablation study and parameter sensitivity analysis provide valuable insights into the method's robustness and the contribution of its components.\n\n**Weakness**\n- While Section 3.2 mentions the motivation behind each loss term and Table 9 shows the impact of different loss combinations, the paper lacks experimental justification for why the specific three-loss formulation is optimal or necessary.\n- Model-specific hyperparameter tuning ($\\sigma_{1}$, $\\sigma_{2}$, $\\sigma_{3}$, $T$) suggests the method may not be as universal as claimed.\n- While performance improvements are consistent, the magnitude of gains (~1–2% F1 in most cases) may be viewed as incremental, especially on already strong baselines (e.g., Intern‑VL2‑MPO), which may not justify the additional computational cost despite the proxy solution.\n\nQUESTIONS:\n- Table 10 shows that proxy-generated VAP (e.g., Intern-VL2-1B generating for Intern-VL2-8B) leads to slightly lower performance than self-generated VAP. While the efficiency gain is clear, could the authors discuss potential strategies to further bridge this performance gap, perhaps by exploring different proxy model architectures or more sophisticated knowledge transfer techniques?\n- While I understand that the scope of this work is limited to mitigating hallucinations, given that the authors have used adversarial noise to reduce hallucinations, have the authors explored the effect of robustness given a harmful adversarial image?\n\nLIMITATIONS:\nPlease refer to the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Visual Adversarial Perturbation (VAP), a training-free, black-box method that mitigates object hallucinations in Large Vision-Language Models (LVMs) by injecting optimized \"beneficial\" visual noise into the input image. The core idea is to guide the model to ground its responses in actual visual content rather than relying on biased parametric knowledge, without altering the base model. The methodology formulates this as an optimization problem guided by a composite loss function with three objectives. Because direct gradient computation is challenging in LVMs, VAP employs a zero-gradient optimization technique. The final optimized noise is added to the original image to create a perturbed version that is used for inference. The authors validate their method across eight state-of-the-art LVMs using a comprehensive suite of benchmarks (POPE, BEAF, CHAIR), demonstrating consistent improvements in reducing hallucinations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.  The paper addresses object hallucination, which is a critical and widely recognized challenge that hinders the reliability and trustworthiness of modern LVMs. The \"poison as cure\" paradigm, which reframes adversarial noise as a corrective tool rather than an attack vector, is a novel contribution. The data-centric, training-free nature of the method makes it an attractive alternative to model-centric interventions.\n2.  The empirical evaluation is extensive and a clear strength of the paper. Testing on eight different LVMs and evaluating across multiple, diverse hallucination benchmarks (including text-axis, vision-axis, and open-ended tasks) provides strong evidence for the method's general effectiveness.\n\nWeaknesses:\n1.  The paper's positioning is weakened by an incomplete comparison to related work. Several existing methods use a similar setup of contrasting outputs from original and noisy/distorted images to mitigate hallucinations or language bias. For example, VCD [1] contrasts output distributions from original and distorted visual inputs, which is a very similar conceptual setting. VisualGPTScore [2] uses Gaussian noise to estimate and correct for language priors, directly tackling the \"parametric knowledge bias\" that VAP also targets. Earlier works like PixMix [3] have also explored noisy images as a form of data augmentation to improve model robustness. A thorough comparison and differentiation from these methods are crucial for contextualizing the unique contribution of VAP.\n2.  The method's effectiveness is heavily dependent on a set of hyperparameters that must be manually tuned for each specific LVM. As shown in Table 6, the optimal parameters vary across models, suggesting a costly and expertise-driven tuning process is required for any new model. Furthermore, the loss formulation, which contrasts prompted and unprompted responses, implies that the optimal hyperparameters may also be dependent on the specific dataset or the type of instruction used. The parameter analysis in Table 8 does not reveal a clear or monotonic relationship, reinforcing the idea that tuning is a complex trade-off without a simple heuristic. This makes the method appear hard to deploy reliably.\n3.  The method heavily relies on generating responses from a null prompt ($\\emptyset$). For a given image, the unprompted response $f_{\\theta}(x+\\delta, \\emptyset)$ is fixed, but the prompted response $f_{\\theta}(x+\\delta, c)$ will change dramatically with the instruction $c$. This means the optimization target is highly variable, which could lead to unpredictable behavior and further complicates the hyperparameter tuning challenge mentioned above. Besides, instruction-tuned LVMs are optimized to follow explicit prompts, and their behavior in an unconditional setting can be unpredictable, unstable, or produce low-quality text. Basing a core part of the optimization on such an unstable signal is a significant methodological concern.\n4.  The logic for some loss terms is unclear. For instance, in $\\mathcal{L}_{s_2}$, the method contrasts a prompted response with an unprompted response from a *negative* (distorted) image. A more symmetric and perhaps more stable approach would be to contrast two prompted responses, i.e., $S(f_{\\theta}(x+\\delta,c), f_{\\theta}(\\bar{x},c))$. The paper does not justify its specific design choice over more intuitive alternatives.\n\n\n\n### **References**\n[1] Leng, Sicong, et al. \"Mitigating object hallucinations in large vision-language models through visual contrastive decoding.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n[2] Lin, Zhiqiu, et al. \"Revisiting the Role of Language Priors in Vision-Language Models.\" International Conference on Machine Learning. PMLR, 2024.\n[3] Hendrycks, Dan, et al. \"Pixmix: Dreamlike pictures comprehensively improve safety measures.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\nQUESTIONS:\n1.  Could you please add more related works and elaborate on the novelty of VAP in direct comparison to decoding-stage methods like VCD and bias correction techniques like VisualGPTScore, which also leverage distorted images to mitigate hallucinations?\n2.  Given the significant hyperparameter sensitivity shown in Table 6, could you provide a more systematic methodology or a set of guiding principles for tuning these parameters for a new LVM or a new task/dataset? Without this, the method's practical applicability seems limited.\n3.  Could you justify the design choice of relying on unconditional generation ($f_{\\theta}(x, \\emptyset)$) within your loss functions, given the known instability of instruction-tuned LVMs in this setting? How does your method handle the significant variance in the loss objective introduced by different instructions $c$? Would it be better with $S(f_{\\theta}(x+\\delta,c), f_{\\theta}(\\bar{x},c))$ in $\\mathcal{L}_{s_2}$?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-12"}

comments: SUMMARY:
The paper introduces an optimization-based method, leveraging adversarial strategies to generate beneficial visual perturbations that enhance the model’s factual grounding and reduce parametric knowledge bias. Extensive experiments on 8 LVLMs demonstrate the effectiveness of the proposed method.

STRENGTHS AND WEAKNESSES:
[Strengths]
1. The experiment on 8 LVLMs is really comprehensive and impressive. This demonstrates the generalizable property of the proposed method.

[Weaknesses]
1. The experiments only compare VAP with the original LVLM, while comparing with other training-free hallucination mitigation methods could be helpful. For example, VCD[1] applies noise to perturb the visual input and mitigate hallucination in a training-free way. 
2. I'm unsure about the efficiency of the method.  The method requires another LVLM to generate perturbation, and this could lead to extra GPU memory cost. And I also have a question about the comparison of latency. Please refer to [Questions].

[1] VCD: Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding

QUESTIONS:
As we can see from Figure 2, the optimization process requires three times of LVLM inference, and I think the process should be at least three times of the original LVLM inference. However, in Table 4, Original + VAP increases the latency from 160ms to (160+298)ms. Why is the extra time less than 2 times? Can you specify what the A100 run time in Table 4 refers to? Is it time for inferencing from one image-query pair?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-16