Title: The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?
Venue: NeurIPS 2025
Accepted: yes
Keywords: Object Hallucination

Multimodal Large Language Model
Abstract: Contrastive decoding strategies are widely used to reduce object hallucinations in multimodal large language models (MLLMs). These methods work by constructing contrastive samples to induce hallucinations and then suppressing them in the output distribution. However, this paper demonstrates that such approaches fail to effectively mitigate the hallucination problem. The performance improvements observed on POPE Benchmark are largely driven by two misleading factors: (1) crude, unidirectional adjustments to the model’s output distribution and (2) the adaptive plausibility constraint, which reduces the sampling strategy to greedy search. To further illustrate these issues, we introduce a series of spurious improvement methods and evaluate their performance against contrastive decoding techniques. Experimental results reveal that the observed performance gains in contrastive decoding are entirely unrelated to its intended goal of mitigating hallucinations. Our findings challenge common assumptions about the effectiveness of contrastive decoding strategies and pave the way for developing genuinely effective solutions to hallucinations in MLLMs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper critically challenges the perceived effectiveness of contrastive decoding strategies for mitigating object hallucinations in MLLMs. This paper claims that the gained performance on the POPE benchmark are highly misleading and do not represent genuine hallucination mitigation. It attribute these \"spurious improvements\" to two primary factors: (1) a unidirectional shift of the model's output distribution towards \"yes\" response, and (2) the adaptive plausibility constraint, which effectively degrades sampling decoding to greedy search. To substantiate their claims, the paper introduces and evaluates \"spurious improvement methods\" that achieve comparable performance to contrastive decoding, decisively demonstrating that the reported gains are unrelated to actual hallucination mitigation.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. The paper critically challenges the contrastive decoding paradigm in MLLM hallucination mitigation, providing evidence that the observed performance gains are deceptive. This critique helps explore the actual causes of alleviating hallucinations for the community.\n2. The authors include extensive analyses covering ablation studies on both hallucination and general benchmarks between tricky designs and original contrastive decoding from the perspective of unidirectional shift and adaptive plausibility constraint.\n\n### Weaknesses\n1. The paper criticizes contrastive decoding, but it mostly focuses on older methods. It could be better to analyze on newer methods [1]. [1] Park, Yeji, et al. \"Convis: Contrastive decoding with hallucination visualization for mitigating hallucinations in multimodal large language models.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 6. 2025.\n2. The paper's definition and verification of object hallucination are insufficient, and it does not provide explanations or experimental results for whether more complex relational or attribute hallucinations are similarly unresolved by contrastive decoding.\n3. Despite using LLaVA-v1.5 and QwenVL-7B, the experimental scope for this training-free method could be expanded to include more recent, larger, and architecturally diverse MLLMs (e.g., Qwen-2.5-VL, LLaVA-Next) to ensure broader applicability of its conclusions.\n\nQUESTIONS:\nPlease refer to the weakness below.\n\nLIMITATIONS:\nAs the paper does not introduce a new method, there is no discussion of methodological limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper critically examines popular methods for reducing object hallucinations in multimodal language models, including common approaches like Visual Contrastive Decoding (VCD), Instruction Contrastive Decoding (ICD), and Self-Introspective Decoding (SID). The authors argue that although these methods appear to boost performance on the POPE benchmark, these improvements are misleading. The real cause of the better scores is not actual reduction in hallucinations, but rather two side effects: (1) pushing the model’s answers toward “Yes” to balance the data, and (2) using a constraint that effectively turns sampling into greedy search, which helps on simple yes/no tasks. The authors demonstrate this by introducing their own artificial tricks (like always encouraging “Yes” or tweaking the output layer) that achieve similar “improvements” without addressing hallucination at all. The key takeaway is that current contrastive decoding techniques don’t really solve the problem, even if the numbers look better.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n\n- The paper tackles a real and important issue: how to fairly evaluate progress on hallucination in multimodal models, which has real-world safety stakes.\n\n- The experimental setup is well thought out and includes multiple models, benchmarks, and clever “control” tricks to highlight the central flaw, underscoring a potentially vital claim for the community: current contrastive decoding techniques don’t really solve the problem, even if the numbers look better.\n\n- The analysis of why these methods get better numbers (by unbalancing answers or changing the search strategy) is clear and persuasive.\n\n\n## Weaknesses:\n\n- The paper suffers from some notation and presentation problems that hinder readability. For example, a symbol like p_cd is introduced around Equation (2) without clear definition (presumably referring to a contrastive decoding probability distribution), which may confuse readers. Similarly, some results are hard to interpret due to labeling issues, e.g., Table 6 contains a “Sample*” notation that is never explained, and the annotations in Figure 3 are unclear, making it difficult to follow the figure’s message. These issues suggest the exposition could be more polished and self-contained.\n\n- The main conclusion, “we confirm that while contrastive decoding enhances performance, it ultimately fails to mitigate hallucinations”, remains ambiguous. Could the authors clarify the intended implication and its broader significance?\n\nQUESTIONS:\nIn lines 170–174, the observation that MLLMs tend to generate “NO” is discussed. Is this tendency specific to the POPE benchmark, or does it generalize to other datasets and evaluation protocols? If this is not a general phenomenon, how do the conclusions regarding contrastive decoding’s tendency to produce more “YES” answers remain valid?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors show that contrastive decoding methods that show improvement on the POPE dataset are actually due to spurious reasons, namely 1. the imbalance of yes's to no's, and 2. the collapse to greedy search due to the the adaptive plausibility constraint.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The paper notes a significant limitation in the evaluations of VLLMs on the POPE dataset. This is an important finding for past and future works that cite the POPE dataset. \n\nWeaknesses:\n1. It's unclear that showing a limitation of the POPE dataset is sufficient for acceptance as a conference paper. The authors should further provide background on the impact of the POPE dataset in the broader research field of hallucination detection. As it stands, the paper is not strong enough in my opinion as as a standalone work.\n2. The authors should evaluate other hallucination benchmarks that move beyond binary QA (i.e., free-text, caption-based outputs). One experiment that would strengthen the paper would be to demonstrate this effect on non-binary QA benchmarks as well. Class imbalance is a problem with all binary QA datasets (not unique to POPE), and there are metrics for addressing this.\n\nQUESTIONS:\n1. Do these findings generalize beyond the POPE dataset? What about datasets like TruthfulQA or FactScore that have non-binary outputs?\n2. The model evaluations are quite limited -- the paper only evaluates three models in 7B and 13B. Are larger models beyond 13B still susceptible to this type of bias? It would be useful to show a trend -- does this class imbalance issue worsen or improve with increased model size?\n3. Please discuss metrics to address shortcomings in binary accuracy evaluation, like per-class precision/recall and F1. Performance should be reported in terms of these metrics as well for clarity.\n4. Please include examples illustrating your finding, i.e., questions that were previously true negatives that got flipped to false positives due to this bias. \n5. The discussion should include proposals for clear alternatives and solutions for this issue.\n6. Please clarify the COCO and GQA subsets of the POPE dataset; the authors seem to assume reader knowledge of the dataset which may not always be the case.\n7. Equation 3 has p_0 on the left; should this be p_\\theta?\n8. Tables 5 and 6 use Sample† and Sample* interchangeably; please keep one notation.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper questions the effectiveness of contrastive decoding in reducing object hallucinations in multimodal LLMs. It shows that the observed improvements on the POPE benchmark are primarily due to biased output adjustments and the adaptive plausibility constraint rather than genuine mitigation of hallucinations.

STRENGTHS AND WEAKNESSES:
Strengths:

* This paper provides a thorough analysis of contrastive decoding, identifying its key limitations through empirical and theoretical evidence.
* This paper introduces well-designed baselines that isolate and reveal the misleading factors behind observed performance gains.
* This paper conducts comprehensive experiments across models and tasks, strengthening the validity of its conclusions.
* This paper offers practical insights into evaluation practices and future directions in hallucination mitigation research.

Weaknesses:
* This paper relies heavily on the POPE benchmark, raising concerns about generalizability to more complex or real-world scenarios.
* This paper does not explore or compare with other hallucination mitigation techniques beyond contrastive decoding.
* This paper assumes a consistent response bias pattern (“Yes”/“No”) across datasets, which may not always hold in broader contexts.

QUESTIONS:
* How can future benchmarks be better designed to measure hallucination accurately?
* Can contrastive decoding be restructured to correct both false positives and false negatives?
* How would these findings extend to open-ended generation tasks beyond binary QA?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03