The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?
Authors: Hao Yin
Guangzong Si
Zilei Wang
Venue: NeurIPS 2025 poster ()
Keywords: Object Hallucination
Multimodal Large Language Model
Primary area: deep_learning
TL;DR: Contrastive decoding fails to mitigate object hallucinations in MLLMs—apparent improvements stem from misleading factors rather than genuine effectiveness.

Abstract: Contrastive decoding strategies are widely used to reduce object hallucinations in multimodal large language models (MLLMs). These methods work by constructing contrastive samples to induce hallucinations and then suppressing them in the output distribution. However, this paper demonstrates that such approaches fail to effectively mitigate the hallucination problem. The performance improvements observed on POPE Benchmark are largely driven by two misleading factors: (1) crude, unidirectional adjustments to the model’s output distribution and (2) the adaptive plausibility constraint, which reduces the sampling strategy to greedy search. To further illustrate these issues, we introduce a series of spurious improvement methods and evaluate their performance against contrastive decoding techniques. Experimental results reveal that the observed performance gains in contrastive decoding are entirely unrelated to its intended goal of mitigating hallucinations. Our findings challenge common assumptions about the effectiveness of contrastive decoding strategies and pave the way for developing genuinely effective solutions to hallucinations in MLLMs.

The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?
Hao Yin' Guangzong Sit2
Zilei Wang University Science and Technology of China
Eastern Institute of Technology, Ningbo [rangzonngs//mmal.. @mail. ustc. edu. cn, zlwang@ustc. edu.
(yinhnavi, guangzongsi
Abstract
Contrastive decoding strategies are widely used to reduce object hallucinations in multimodal large language models (MILMs)). These methods work by construct- ing contrastive samples to induce hallucinations and then suppressing them in the output distribution. However, this paper demonstrates that such approaches fail effectively mitigate the hallucination problem. The performance improvements observed on POPE Benchmark are largely driven by two misleading factors: (I) crude, unidirectional adjustments to the model's output distribution and (2) the adaptive plausibility constraint, which reduces the sampling strategy to greedy search. To further illustrate these issues, introduce series of spurious im- provement methods and evaluate their performance against contrastive decoding techniques. Experimental results reveal that the observed performance gains in contrastive decoding are entirely unrelated to its intended goal of mitigating hal- lucinations. Our findings challenge common assumptions about the effectiveness contrastive decoding strategies and pave the way for developing genuinely ef- fective solutions to hallucinations in MLLMs. The source code available https:// http//gttttt.t.////e/ wetthu..com/uset. www./ssttccin///eeeeccoocec cetpn/dddeeeettkkkkk
1 Introduction
The hallucination problem 2 in multimodal large language models (MLLMs) [5 refers the generation of outputs that factually incorrect misaligned with the input data. This issue arises from challenges in aligning diverse data modalities, such as text and images, which amplify reasoning errors. Such hallucinations
have serious consequences in critical domains, including autonomous driving (8] 9 [0 [T] (e.g.. false object detection leading to accidents) and healthcare (c.g., incorrect diagnostic interpretations).
12 13
15 16 are widely recognized as an effective approach to address- ing object hallucination in generative models. As illustrated in Figure [ these methods construct contrastive samples designed to induce hallucinations. then suppress the corresponding output dis- tributions, ensuring closer alignment between model outputs and visual inputs. Representative approaches within this framework include Visual Contrastive Decoding (VCD) (17), Instruction Contrastive Decoding (ICD) (18), and leff-Introspecttvee Decoding (SID) [19]. Their training- nature and purported ability to address hallucinations have made them highly regarded in the field.
Contrastive decoding methods 14
Although methods like VCD have demonstrated remarkable performance improvements the POPE benchmark M. we reveal in Section 4 that these results are highly misleading. In reality, these
"Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

methods fail to effectively address model hallucination. The observed performance gains on the POPE benchmark are primarily driven by two factors:
Misleading Nature of Performance Improvement
R1: A umidirectional adjustment of the output distribution, which simply biases the model towards producing more "Yes" outputs, leading to a balanced distribution on certain datasets.
R2: The adaptive constraints in these methods degrade the sampling decoding strategy into an approximation of greedy search, resulting in deceptively improved performance.
To expose the misleading nature of the improvement in the first scenario, we implemented two forced distribution adjustment algorithms in Section [5.1 to show that the apparent gains of contrastive decoding on the POPE Benchmark are not genuine. The methods are follows: (1) Prompt-Basee Adjustment, where we added prompt to the instruction, such as "Whenever possible, please select Yes." " to bias outputs toward "Yes": and (2) Output Layer Modification, where we altered the output layer to favor "Yes" when the probabilities for "Yes" and "No" were similar. Although neither method mitigates hallucinations, both achieved performance gains comparable to those of contrastive decoding, confirming that these improvements do not represent genuine solution to the problem.
To highlight the misleading nature of the performance improvement in the second scenario, incorporated the adaptive plausibility constraint into the standard sampling strategy and compared its predictions with those from contrastive decoding Section 5.2 The experimental results reveal that, despite having no theoretical connection to hallucination mitigation, the adaptive plausibility constraint accounts for nearly all the performance gains attributed contrastive decoding. This finding underscores that the contrastive decoding methods, in essence, fail to mitigate hallucinations.
Overall, this paper makes the following three contributions:
We identified that the performance improvement of contrastive decoding methods stems from its unidirectional and blunt adjustment of the output distribution, which coincidentally balances the distribution on certain datasets.
We discovered that another key factor driving the performance gains of contrastive decoding methods is their adaptive plausibility constraints, which streamline the sampling strategy into an approximation greedy search.
We developed a series of spurious improvement methods and evaluated their performance against contrastive decoding methods. Our findings convincingly show that contrastive decoding methods do not alleviate hallucinations in any meaningful way.
2 Related Work
Multimodal Large Language Models. The evolution of MLLMs 20 21] has progressed from BERT-
based decoders [22 (23 advanced LLM architectures 24 25 enabling more effective multimodal relationship modeling 226] 27). Models such as BLIP-2 28 and MiniGPT-4 22 employ Q-Former mechanisms to enhance the alignment between visual and textual inputs, facilitating more precise B0 extends this framework by integrating task-specific
cross-modal interactions. InstructBLIP instructions, improving the model's ability to interpret context- senssttieeeeeivveeeeeeeee visual semantics. Meanwhile, LLaVA 31 32 and Qwen-VI. LL [3 adopt simpler linear projection methods that streamline alignment, leading to superior performance in vision-language tasks. Despite these advancements, hallucination remains a persistent challenge that warrants further investigation.
are widely recognized as effective in addressing object hallucination in generative models. Visual Contrastive Decoding (VCD) 17 addresses object hallucination by comparing output distributions generated from standard visual inputs and distorted visual inputs. This approach reduces the model's dependence on linguistic priors within integrated LLMs and minimizes the impact of statistical biases MLLM pretraining corpus. Instruction Contrastive Decoding (ICD) [1], in contrast, focuses on the role of instruction perturbations in amplifying hallucinations. By examining the differences in output distributions between standard and perturbed instructions, ICD detects hallucination-pree content and mitigates its impact effectively. Building upon these two hallucination mitigation methods, numerous approaches, including Adaptive Focal- Coal--Cnt Decoding (HALC) (37), Self Seff-Innrospeectie Decoding (SID) [19].
Contrastive Decoding Strategies. Contrastive decoding 34
35
36

---

and Visual Layer Fusion Contrastive Decoding (VaLiD) [38 have been developed based on similar principles. Although these methods have demonstrated substantial performance improvements on the POPE Benchmark, we will show that these improvements are, in fact, entirely unrelated to the original objective of hallucination mitigation.
Preliminary
This section outlines the main components and operational workflows of three leading hallucination mitigation methods: VCD, ICD, and SID, as illustrated in Figure [ decoding strategies reduce hallucinations and improve consistency with the visual input. The POPE Benchmark is also discussed as one of the most important metrics for evaluating their performance. Further technical details of VCD, ICD, and SID can be found in Section
All three adopt contrastive
Vanilla Decoding. We consider a MLLM parametrized by 0. The model takes as input a textual query and visual input where provides contextual visual information to assist the model in generating a relevant response y to the textual query. The response y sampled auto-regressively from the probability distribution conditioned on the query x and the visual context v. Mathematically, this can be formulated as:
U/ po (ye Vece) exp (logity (ye v,,,,
(1)
where y/ denotes the token at time step I, and yer represents the sequence of generated tokens up to the time step (1
Decading
Contrastive Decoding
Contrantive Decedingg
Precly Mits
Intruspectivve Decodlng
Hornd Dmallag
Origial Vinal Txpat
Our tutrudo
Origial Vanal
Tokenkes
Vidao Umender
Telontave
Telenbore
Largn Largueg Mobi
Lange Languag Mashd
Lange Langany Muchl
Vieilla Liglt
Centratin Lagin
The werkiiir Memia
Blazhin Inlaxe
Eukabday lanshik ladeo
Bistabing cablicas Juica
Using contrastive decoding to correct prediction outcomes
Logis
Lagis
Espectéso: podal hullecinalisss
Oytained Leglt
Expedation: Cameetngg modd's ballacinatan culpas.
Black
Block
Blark
Groes
Realh:: Mamrly whbrittgg
cuputiin "Na".
Vellan
Bealit: Mandy
ontguting POPE
Bilae
like
Blee
Optimized Logils aa) Vanilla Logils a Contrastive Hallucination Logilss
An illustration of hallucination mitigation methods: Visual Contrastive Decoding, Instruc- tion Contrastive Decoding, and Ielf-Introspecctvee Decoding. The hallucination induction module shifts outputs toward negative responses, while the contrastive decoding module shifts them toward positive responses, rather than achieving their intended effects.
Figure 1:
Visual Contrastive Decoding. VCD is a corrective strategy aimed reducing hallucinations in MLLMs. Given a textual query and its corresponding image, the model produces two output distributions: one from the original image and another from perturbed variant, typically generated by applying controlled modifications such as Gaussian noise. By evaluating the divergence between these distributions, VCD constructs contrastive output distribution that improves the reliability and factual accuracy of the model's 's response.
Instruction Contrastive Decoding. ICD addresses hallucinations by leveraging the observation that instruction perturbations, particularly those involving negative prefixes, increase uncertainty in multimodal alignment. ICD operates by first amplifying the probabilities of hallucinated concepts,
3

---

then systematically detaching these from the original ouiput distribution. This contrastive process reduces the model's vulnerability object hallucinations.
Self-Introspective Decoding. SID extends the ideas of VCD and ICD by addressing a key limitation: perturbing the full input can inject too much noise, making it harder to produce useful hallucinations. To resolve this, SID adaptively prunes the input by keeping only small set of image tokens with low attention scores after the early decoder layers (see Figure [ far right). This focused adjustment encourages vision-and-tex hallucinations to emerge during decoding. These hallucinated elements are then separated from the original output distribution to improve the model's reliability.
Adaptive Plausibility Constraint. One key challenge inherent in the three aforementioned methods is the risk of indiscriminate penalization across the entire output space, which can unintentionally sup- press valid predictions and, paradoxically, favor the generation of implausible tokens. To mitigate this, all three methods incorporate adaptive plausibility constraint. This constraint dynamically adjusts penalization based on confidence scores derived from the model's output distribution, conditioned on the original visual input v. Formally, the constraint is defined as:
Vuead (V<1) y/ V pa (ye U,Z, B maxxp (w Pod (ye v, 0 if ye & Vhead (y<+)
2. Vee)
(2)
Here, V represents the output vocabulary of the multimodal large language model (MLLM), and /3 is hyperparameter controlling the truncation threshold of the next-token distribution. higher value of B results in more aggressive truncation, thereby retaining only the most probable tokens.
Polling-based Object Probing Evaluation. POPE 39) offers a robust framework for evaluating object hallucinations in multimodal large language models (MLLMs). Departing from caption-base methods, it frames hallucination detection as a binary task using direct Yes-or-No questions (e.g.. "Is there chair in the image?"), enabling clearer interpretation. The benchmark ensures balanced distribution of "Yes" and "No" samples (50% each). Contrastive decoding- based mitigation strategies have demonstrated their effectiveness primarily through improved performance on POPE, reinforcing their credibility within the research community.
4 Misleading Performance Improvement
In this section, we highlight two misleading factors contributing to the performance improvement of contrastive decoding methods on the POPE Benchmark: (1) Unidirectional output adjustment skews the model towards generating more "Yes" outputs, leading to a balanced distribution in certain datasets. (2) The adaptive plausibility constraint degrades sampling decoding strategy into greedy search, resulting in deceptively improved outcomes.
Table I:
Performance of various contrastive de- coding methods subsets of POPE Benchmark.
Table 2: Output distribution generated from con- trastive inputs in contrastive decoding methods.
Dataset
COCO Random
GQA Adversarial
Dataset
COCO Random
GQA Adversarial
Method
Ace %
Yes
Ace
Yes
Method
Acc
Yes %
Acc
Yes %
Greedy
87.1
39.2
80.9
54.0
Greedy
87.1
39.2
80.9
54.0
VCD
88.6
46.4
78.0
63.3
VCD-C
76.7
28.2
71.5
41.3
SID
87.9
42.3
79.9
57.8
SID-C
79.0
23.6
74.2
43.1
4.1
Unidirectional Output Adjustment
In this subsection, we illustrate how contrastive decoding algorithms can deceptively enhance the performance of MLLMs on the POPE Benchmark by applying targeted, unidirectional modifications to the output distribution. We begin by evaluating the performance of various contrastive decoding methods on the MSCOCO [40] and GQA [41] datasets, analyzing both accuracy and the distribution of the model's responses. For this study, we selected LLaVA-v1.5-77 as the foundational MLLM, using a greedy search decoding strategy.
As shown in Table both VCD and SID significantly skewed the model's 's output distribution toward "Yes" across all subsets. On RSSOOCO-Randon subset, where the original output distribution was

---

skewed toward "No," VCD and SID corrected this imbalance, resulting in more balanced distribution and improved accuracy. Conversely, for -QA-Adversriial subset, where the output distribution was already biased toward "Yes," these methods intensified the skew, ultimately reducing accuracy.
We further illustrate how model outputs change after applying contrastive decoding methods, pro- viding a clearer understanding of their performance improvements. As shown in Figure 2 the method primarily alters predictions from "No" to "Yes," significantly outpacing the reverse, On the ASCOCCO-Radde dataset, where the output distribution is initially skewed toward "No," this adjustment converts many false negatives into true positives, thereby improving accuracy. Conversely, on the GQA- Adve-saraa dataset, which is biased toward "Yes," these modifications lead to the mis- classification of numerous true negatives false positives, resulting in a performance decline. For more details on prediction shifts, please refer to Section
TN
TN
TN
TN
TP
TP
110
74
TP
36
1
103
FP
FP
201
FN
Acaanacy
10.4%
1.t
AcuuaaT
7000
234
87.8%
B81
Amarayy
70.8%
1.04
Yes Propuctios
05.8%
121
Tas Propoctias
63.3%
Tel Propurtix
42.3%
3.19
Yo Propartion
37.8%
3,81
Figure 2: Changes in the distribution of predictions after applying contrastive decoding methods.
To understand why contrastive decoding methods consistently increase the likelihood of "Yes" response, we analyzed the output distribution generated from contrastive samples, as shown in Table In Section 3 proposed that the primary function of contrastive samples is induce hallucinations, allowing contrastive decoding to subsequently filter out these hallucinated elements from the output distribution. However, the results in Table 2 reveal that this objective was entirely unmet. Most outputs derived from contrastive samples were incorrect, not due to successfully induced hallucinations, but because the model overwhelmingly favored "No" responses. This severe bias in the output distribution led to a significant decline accuracy. A more detailed explanation of why outputs generated from contrastive inputs biased toward "No" is provided in Section C
604
(a) LLaVA-v1.5-7B Greedy
(b) LLaVA-v1.5-113B Greedy
(c) QwenVL-CChat-78 Greedy
MRC
Pale
(d) LLaVA-v1.5-7B Sample
LL. VAL. Sample
(f) QwenVL-CCht-771 Sample
Figure 3: Skewness in the Raw Output Distribution of MILMs across Different Datasets
5

---

Based on the above discussion, the practical performance of contrastive decoding methods is illus- trated in the lower section of Figure [ The output distribution derived from the contrastive inputs is heavily biased toward "No." However, the contrastive decoding method suppresses this content in the original output, thereby unilaterally increasing the model's likelihood of answering "Yes." Whether the model's s performance on the dataset improves depends heavily on whether this increased "Yes" frequency leads to a more balanced output distribution. However, as shown Figure 3 the output distribution of MLLMs tends to be biased toward "No" in most data subsets. Therefore, contrastive decoding methods still manage to achieve a strong overall performance on the POPE Benchmark.
4.2 Sampling Decoding Degradation
In this subsection, we will illustrate how contrastive decoding methods misleadingly enhance model performance by degrading sampling-basee decoding strategies into greedy search through the adaptive plausibility constraint.
Notably, the POPE Benchmark, which requires models to answer "Yes" or "No." functions as a binary classification task. Consequently, greedy search the most suitable decoding strategy, rendering sampling- based methods unjustifiable. As shown in the upper-left corner of Figure + experimental results further confirm that greedy search significantly outperforms direct sampling. In fact, greedy search outperforms direct sampling on the vast majority of tasks 42) Additional comparisons of decoding strategies and their effects on prediction outcomes are provided in Section [ However, many contrastive decoding methods report performance improvements using sampling strategies, necessitating a closer examination of these claims.
We revisit the adaptive plausibility constraint introduced in Section 3 and formally defined in Equation This constraint ensures that when the model exhibits high confidence its outputs corresponding to the original input, the candidate pool is refined to retain only high-probability tokens. By incorporating this mechanism into contrastive decoding methods, it aims mitigate adverse effects by preventing the generation of implausible tokens, while safeguarding the coherence and quality of the generated content.
In its original design, the constraint was intended as a complement to contrastive decoding strategies, with no explicit connection mitigating hallucinations. Consequently. it was assumed have no significant effect when applied independently. However, our findings challenge this assumption: under sampling strategy, the constraint emerges as pivotal contributor to performance gains.
86
Greedy
Sample
84
Question: Is there cell phone in the image?
80
Answer: No.
78
Random
Popular
Adversarial
Standard Sampling Strategy
Adaptive Plausibility Constraint
Logits
Yes
Probe
Logits
Probs
Yos
Figure 4: Why does the adaptive plausibility constraint alone result in improvements?
Aroviidpr a detailed explanation of why the adaptive plausibility constraint alone significantly improves performance when using the sampling strategy. For the question "Is there a cell phone in the image?". I.LaVA-v1.5-7B model generates the correct output distribution: Yes: 8.8% and No: 91.2%. Using greedy search strategy, the model consistently produces the correct answer, "No."

---

However, when employing a sampling strategy. there is an 8.8% chance that the model generates the incorrect answer, "Yes."
When the adaptive plausibility constraint is applied, many candidate options are eliminated by setting their logits negative infinity for failing to satisfy the condition:
po (ye U, v, yet) 2 3 max po (w 2, Vet)
w
Among the excluded candidates is the option "Yes." Consequently, the sampling strategy reduces to greedy search, ensuring 100% probability of correctly answering "No."
Consequently, the adaptive plausibility constraint greatly limits the pool of candidate options, trans- forming the sampling strategy into a predominantly greedy search. As demonstrated in Figure 4 MILMS exhibit markedly superior performance on the POPE Benchmark under greedy strategy, underscoring the constraint's pivotal contribution to performance gains.
4.3 Insights
In Section 4.1 we show that when using greedy search as the decoding strategy. contrastive decoding methods modify the model's predictions in unidirectional manner, shifting the output distribution toward Ves. As result, performance improvements primarily depend on whether the model's original output distribution
biased toward No.
In Section 4.2 we demonstrate that when direct sampling used as the decoding strategy, the adaptive plausibility constraint effectively reduces it to greedy search, serving as key driver of the observed performance gains.
These findings suggest that the reported improvements from contrastive decoding may be misleading. Specifically, the gains observed on the POPE Benchmark could falsely imply effective hallucination mitigation when, in reality, they stem from unrelated factors.
5 Spurious Improvement Methods
In this section, we propose series of spurious improvement methods based on the two fundamental reasons for performance improvement discussed in Section 4.3 Although these methods are entirely unrelated to hallucination mitigation, they yield experimental results comparable to contrastive decoding techniques. This evidence suggests that while contrastive decoding enhances performance on POPE Benchmark, it does not address hallucinations.
Prompt Prompt-Based A djustment
Output Layer Modification
User Instructions
NO
Predicted probebilities for YES ND are close.
YES
Is there a black banana in the picture?
NO
LLM
Aditional Prompt
Finel prediction is consistently adjusted YES
Answer Yes if possible.
Figure 5: Illustration of Prompt-Based Adjustment and Output Layer Modification Algorithms.
5.1 Forced Distribution Adjustment
For the first misleading factor in performance improvement, which involves modifying model predictions in single direction to bias the output distribution toward "Yes," we introduce two pseudo- performance enhancement methods: Prompt-Based Adjustment and Output Layer Modification. The implementation of these algorithms is detailed in Figure 5

---

Prompt-Based Adjustment modifies the input side of the model by appending an additional prompt, "Answer Yes if possible." after the user's instruction. This extra input biases the model's output distribution toward "Yes." Output Layer Modification refers to adjustments made at the output stage of the model. After generating its initial prediction, the model evaluates the probabilities of "Yes" and "No." If their difference is small, i.e.,
Pe (Yes) v, po v,,)]
(4)
the prediction is forcibly set "Yey." Here, controls how close the probabilities must be to trigger this modification. This adjustment increases the likelihood of the model predicting "Yes."
Experimental Settings. We selected QwenVL-7B. LLaVA-v1.5-1 1.5A-v and LLaVA-v15-13B as the backbone MLLMs. For decoding, we employed a greedy search strategy. Regarding the critical POPE benchmark, our experiments were conducted on COCO dataset, where the raw output distribution of MLLMs tends be biased toward "No." " As result, all contrastive decoding methods exhibited notable improvements on this dataset. In addition, we conducted experiments on several other benchmarks that have not been evaluated by many contrastive decoding methods, including MME CHAIR 44) NoCaps 45 and LLa Va-VAncBhhh [31, covering both discriminative and generative tasks.
Table 3: Performance of Prompt-Basee Adjustment (PBA) and Output Layer Modification (OLM).
LLaVA-v1.5-7B
LLaVA-v1.5-13B
QwenVL-Chat-77
Category
Method
Accuracy
Yes (%)
Accuracy
Yes (%)
Accuracy
Yes (%
Greedy
87.1 10.0
39.2 10.0
86.7 10.0
38.7 +0.0
85.9 t0.0
37.8 10.0
VCD
88.6 11.5
46.4 17.2
89.2 12.5
44.4 15.7
87.7 f1.8
40.6 12.8
Random
SID
87.9 10.8
42.4 f3.2
87.2 10.5
42.5 13.8
86.5 10.6
39.9 12.1
PBA
87.6 10.5
40.2 11.0
90.2 13.5
45.7 17.0
87.3 1..
41.5 41.5
OLM
89.6 12.5
15.0
90.0 13.3
48.8 110.1
88.2 12.3
43.8 16.0
Greedy
85.8 10.0
40.4 10.0
86.0 10.0
39.4 10.0
85.6 10.0
38.2 10.0
VCD
86.2 10.4
48.8 18.4
87.3 11.3
46.3 16.9
87.1 11.5
41.2 13.0
Popular
SID
85.1 J0.7
14.7
85.1 10.9
44.6 15.2
10.3
39.8 11.6
PBA
86.2 t0.4
41.6 11.2
88.4 12.4
47.5 18.1
86.8 11.2
42.3 14.1
OLM
87.3 11.5
46.5 16.1
88.6 12.6
50.2 110.8
87.4 11.8
44.8 16.6
Greedy
83,61000
42.6 10.0
84.3 10.0
41.0 10.0
84,0 10.0
39.7 10.0
VCD
81.9 11.7
53.1 110.5
83.8 J0.5
49.7 18.7
84.5 10.5
43.7 14.0
Adversarial
SID
82.3 L1.3
47.9 15.3
82.9 .1.4
46.9 15.9
83.2 J0.8
42.5 12.8
PBA
83.7 10.1
44.0 11.4
84.5 10.2
51.3 110.3
84.1 10.1
15.5
OLM
83.6 10.0
50.1 17.5
83.9 J0.4
$4.9 113.9
84.8 10.8
48.4 18.7
Results and Analysis on POPE. The experimental results, presented in Table 3 show that PBA achicves an even greater performance improvement than SID, while OLM surpasses VCD. Prediction accuracy increases as the output distribution approaches balance. However, it worth noting that on the Adversarial subset, when the distribution shifts beyond balance and biases toward "Yes." accuracy begins to decline, aligning with the conclusion in Section 4.3 Although PBA and OLM are not designed for hallucination mitigation, they produce results similar to contrastive decoding methods, suggesting that contrastive decoding does not effectively address hallucinations. For the experimental results on the AOKVQA and GQA datasets, please refer to Section
Table 4: Performance of contrastive decoding methods other benchmarks using greedy search.
Method
MME-P f
MME-C1
CHAIR-S +
CHAIR-I
Nocaps
LLaVA-Bas VA-Bench
Greedy
1475.1 100.0
349.6 100.0
21.6 _0.0
7.2 _0.0
83.8 10.0
65.7 ^0.0
VCD
1389,9 J85.2
294.6
J55.0
25.4 13.8
8.1 10.9
81.2 J2.6
64.6 J1.1
SID
1396.4 178.7
304.1 J45.5
24.9 13.3
7.5 10.3
81.9 J1.9
64.9 J0.8
Results and Analysis on Other Benchmarks. shown in Table 4 for the discriminative task MME, contrastive decoding methods did not yield any performance gains, the model's output

---

did not exhibit an initial distributional bias. In fact, it led to decline in performance. Similarly, in generative tasks such as CHAIR, Nocaps, and LLaVA-Bennee contrastive decoding offered no improvements. This clearly demonstrates that contrastive decoding can only enhance performance discriminative tasks where the initial output distribution is skewed towards "No." However, such improvements do not fundamentally address the issue of object hallucination.
5.2 Standalone Application of the Constraint.
The second misleading factor contributing to performance improvement is that the adaptive plausibility constraint degrades the sampling strategy into a greedy search strategy.
To investigate this, plan apply the adaptive plausibility constraint in isolation while using sampling as the decoding strategy. This will demonstrate the significant performance gains that occur when the constraint forces the sampling strategy to behave like greedy search. When the adaptive plausibility constraint is applied independently, the model's output distribution can be defined as:
Yt po (Us U, Vct) exp (logit, (Ve U, 2, vcr)) Us Viead (J<t)
(5)
Experimental Settings. We utilize QwenVL-7B, LLaVA-vI.5-711 and LLaVA-v1.5-133 foundational MILMs, employing a sampling decoding strategy. Regarding the critical POPE benchmark, our experiments are conducted on the GQA dataset, where the original output distribution of MILMSs relatively balanced. Consequently, the modification introduced by contrastive decoding methods, which shifts the output distribution towards "Yes," does not introduce positive bias. However, since the adaptive plausibility constraint converts the sampling strategy into a greedy search, the model's performance still improves.
our
Table 5: Influence of Independent Application of the Adaptive Plausibility Constraint on Model Performance. Sample refers to the sampling strategy that applies the constraint independently.
LLaVA-v1.5-7B
LLaVA-v1.5-138
Qwen WW-nat--hhat-7B
Category
Method
Accuracy
Yes (%)
Accuracy
Yes (%)
Accuracy
Yes (%)
sample
83.8 10.0
10.0
84.6 10.0
45.9 10.0
81.5 10.0
41.1 10.0
VCD
86.6 12.8
52.5
f6.9
86.7 12.1
49.5 13.6
83.8 t2.3
44.0 12.9
Random
ICD
85.2 11.4
47.0 11.
85.8 11.2
44.9 41.0
82.5 11.0
42.0 10.9
SID
84.9 f1.1
49.1 13.5
86.0 1..4
49.8 13.9
82.9 11.4
43.5 12.4
sample
85.4 1.6
45.1
$0.5
86.1 11.5
45.3 40066
83.0 $1.5
41.8 10.7
sample
10.0
10.0
80.6 10.0
49.9 10.0
76.8 10.0
46.1 10.0
VCD
78.7
59.4
17.3
17.3
82.9 12.3
$2.4
13.3
Popular
ICD
78.1 10.8
54.0
11.9
81.5 10.9
49.3 J0066
77.5 10.7
47.2
SID
78.1
53.7 11.6
82.5 11.9
53.3 13.4
77.9 11.1
41.9
sample*
11.3
52.0 40.1
81.8 11.2
49.6 J0.3
78.1 11.3
46.8 10.7
sample
75.1 10.0
$0.0
78.2 10.0
53.2 10.0
76.4 10.0
45.5 40.0
VCD
11.4
62.5 18.4
80.3 12.1
$7.0 13.8
78.6 12.2
49.2
Adversarial
ICD
75,8
54.2 10.1
79.2 +1.0
52.8 10.4
76,8 10.4
46.0 10.5
SID
76.3
57.5 13.4
78.7 10.5
$7.5 14.3
77.2 10.8
47.5 12.0
sample
76.3 112
54.2
10.1
79.5 11.
53.1 J0.1
77.9 11.5
46.2 10.7
Results and Analysis on POPE. The experimental results in Table 5 show that when MLLMs adopt sampling as the decoding strategy, applying the adaptive plausibility constraint alone yields an approximate 2.5% performance improvement, effectively validating the conclusion in Section Notably, since the adaptive plausibility constraint is entirely unrelated hallucination mitigation yet achieves performance on par with various contrastive decoding methods, this strongly suggests that contrastive decoding methods do not actually mitigate hallucinations.. For the experimental results the 4OKVQA and COCO datasets, please refer to Section
Results and Analysis on Other Benchmarks. As shown in Table 6 across all other benchmarks, employing the adaptive plausibility constraint alone under direct sampling yields the most significant performance gains. This suggests that the main reason behind the performance improvements observed with contrastive decoding lies in the fact that the constraint reduces direct sampling to greedy search. Crucially, this mechanism is unrelated the original goal of mitigating hallucination.

---

Table 6: Performance of contrastive decoding methods on other benchmarks using direct sampling.
Method
MME-P1 1
MME-C1
CHATR-S +
CHAIR-I +
Nocaps
LLa
Sample
1282.1 70.0
286.7 10.0
23.2 10.0
7.7 10.0
81.4 10.0
64.3 10.0
Sample
1392.4 1110.3
302.6 115.9
22.4 10.8
7A [0.3
83.1 f1.7
65.3 1.0
VCD
1361.7 179.6
278.6 18.1
25.8 12.6
8.3 10.6
80.9 [0.5
64.2 [0.1
SID
1364.6 182.5
284.1 12.6
25.2 12.0
7.9
81.2 10.2
64.5 10,2
Discussion on Hallucination Mitigation
Based on the insights from Sections and 5
we propose some new criteria for evaluating object
hallucination mitigation in MLLMs.
Impact of Decoding Strategies. When evaluating on POPE, it is essential account for the substantial influence of different decoding strategies on model performance. Notably, greedy search consistently outperforms sampling-bais -baeed methods such direct sampling and nucleus sampling. If hallucination mitigation method involves modifications to the sampling module, careful consideration must be given whether these changes affect the core properties of the decoding strategy.
Avoiding Unidirectional Modification. When evaluating hallucination mitigation methods, is essential assess whether they alter responses unidirectionally. Given the skewed output distribution of MLLMs across multiple datasets, a method that merely rebalances responses may create the illusion of improved performance. However, such adjustments do not genuinely mitigate hallucinations.
Balancing Correction and Preservation. An effective hallucination mitigation method must strike balance: it should correct incorrect answers while preserving correct ones. However, as shown in Figure some flawed approaches, despite fixing many errors, also introduce unnecessary modifications to originally correct responses. This behavior resembles mere answer editing rather than genuine hallucination mitigation. To enhance evaluation rigor, future studies should explicitly report instances where correct responses mistakenly altered, providing clearer measure of method's true effectiveness.
Conclusion
This study demonstrates that the performance improvements of contrastive decoding on the POPE benchmark largely stem from two misleading factors: (1) unidirectional shift in the model's output distribution, which biases it toward generating "Yes" responses, artificially balancing the distribution in certain datasets, and (2) the adaptive plausibility constraint, which reduces sampling decoding to greedy search. By comparing experimental results from spurious improvement methods and contrastive decoding, we confirm that while contrastive decoding enhances performance, it ultimately fails to mitigate hallucinations.
Impact Statement
The broader impact of this work includes fostering transparent and accountable AI systems, particularly in applications where misinformation can have serious ethical and societal consequences, such as healthcare, legal reasoning, and scientific discovery. Our analysis underscores the importance of critical evaluation in AI research to prevent the deployment of methods that may not work as intended. While our work does introduce risks, it serves as cautionary study that helps guide future research toward more robust and responsible AI development.
Acknowledgements
This work supported by National Natural Science Foundation China under Grant 62176246. This work is also supported by Anhui Province Key Research and Development Plan (202304a05020045) Anhui Province Natural Science Foundation (2208085UD17) and National Natural Science Foundation of China under Grant 62406098.
10

---

References
I Li, Y. Y. Du, K. Zhou, et al. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023.
[2] Liu, F., K. Lin, I.. Li, et al. Mitigating hallucination in large multi-modal models via robust instruction tuning. arXiv preprint arXiv:2306. 14565, 2023.
[3] Lovenia, H., W. Dai, S. Cahyawijaya, et al. Negative object presenee evaluation (nope) to measure object ballucination in vision-language models. arXiv preprint arXiv:2310.05338, 2023.
[4] Gunjal, A., J. Yin, E. Bas. Detecting and preventing hallucinations in large vision language models. arXiv preprint 23X1v:2308.06394 2023.
[5] Chen, Z., W. Wang, et al. How far are we gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404. 160.22444..88,,,,, 2024.
[6] Ye. Q.. H. Xu, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. 14178, 2023.
[7] Chen, K., Z. Zhang, et al. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint ar3iv:2.006. 195 2023.
[8] Mai, J. Chen, B. Li, et al. Llm as a robotic brain: Unifying egocentric memory and control. arXiv preprint 2rXh:23304003344 2023.
191 Liu, H., Y. Zhu, K. Kato, et al. Llm-based human-robot collaboration framework for manipulation tasks. arXiv preprint arXiv:2308. 149.772..449, 2023.
[10] Chen, I.,, O. Sinavski, J. Hünermann, et al. Driving with Ilms: Fusing object-level vector modality for explainable autonomous driving. arXiv preprint arXiv:2310.01957. 2023.
[11] Wu, Z., Z. Wang. X. Xu, et al. Embodied task planning with large language models. arXiv preprint arXiv:2307.01844 2023.
[12] Wang, S., Z. Zhao, X. Ouyang, et al. Chatcad: Interactive computer-aided diagnosis on medical image using large language models. arXiv preprint arXiv:2302.07255 2023.
[13] Hu, M., S. Pan, Li, et al. Advancing medical imaging with language models: A journey from n-grams to chatgpt. arXiv preprint (xXtv:330.004220 2023.
[14] Wu, Y., Y. Zhao, S. Zhao, et al. Overcoming language priors visual question answering via distinguishing superficially similar instances. arXiv preprint arXiv:2209, .05599, 2022.
[15] Gupta, V. Z. Li, A. Kortylewski, et al. Swapmix: Diagnosing and regularizing the over-reliance on visual context in visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5078-5088. 2022.
[16] Niu, Y., K. Tang, H. Zhang, et al. Counterficctaal vqa: A cause-effect look at language bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12700-12710 2021.
[17] Leng, S., H. Zhang, G. Chen, et al. Mitigating object hallucinations in large rision-languag models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13872 13882. 2024.
[18] Wang, X., Pan, L. Ding, et al. Mitigating hallucinations in large vision-language models with instruction contrastive decoding, arXiv preprint arXiv: 18715, 2024.
[19] Huo, F., W. Xu, 7. Zhang, et al. Self-introospective decoding: Alleviating hallucinations for large vision- language models. arXiv preprint 2408::2408.00032.. 2024.
[20] Chen, Z., W. Wang, H. Tian, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 67(12):220101, 2024.
[21] Chen, Z. Wu, W. Wang, et al. Internvl: Scaling up vision foundation models and aligning for generic vinual-linguisti tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185-241 241988 2024.
[22] Li. X., X. Yin, C. Li, et al. Oscar: Object-semantics aligned pre-training for vision- language tasks. In European conference computer vision, pages 121-137. Springer, 2020.

---

[23] Li. J., R. Selvaraju, A. Goumare, et al. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705, 2021.
[24] Touvron, H. T. Lavril, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2330..3339 13971, 2023.
[25] Touvron, H., L. Martin, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
[26] Chiang, W.-L., 2. Li, Z. Lin, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, See htips://vicuna. Imsys. org (accessed 14 April 2023), 2(3)66, 2023.
[27] Bai, J., S. Bai, et al. Qwen technical report. arXiv preprint arXiv:2.009. /6609, 2023.
[28] Li, J., D. Li, S. Savarese, et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. International conference on muchine learning, pages 19730-19742. PMLR, 2023.
[29] Zhu, D., J. Chen, X. Shen, Minigpt-4: Enhancing vision- laniuage understanding with advanced large language models. arXiv preprint arXiv:2304, 10592, 2023.
[30] Dai, W., J. Li, D. Lá, al. Instructhlip: Towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems, 36:49250-49261 2023.
[31] Liu, H., Li, Q. Wu, et al. Visual instruction tuning. Advances in neural information processing systems, 36:34892 34916, 2023.
[32] Liu, H., C. Li, Y. Li, et al. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages (6296-26306, 2024.
[33] Bai, J., S. Bai, al. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966,, 2023.
[34] Yan, H., L. Liu, X. Feng, et al. Overcoming language priors with self- coff-contrastive learning for visual question answering. Multimedia Tools and Applications, 82(11):16343 16358, 2023.
[35] Zhibo, R., W. Huizhen, Z. Muhua, et al. Overcoming language priors with counterfactual inference for visual question answering. In Proceedings of the 22nd Chinese National Conference Computational Linguistics, pages 600-610. 2023.
[36] Han, Y., L. Nie, J. Yin, et al. Visual perturbation-aware collaborative learning for overcoming the language prior problem. arXiv preprint wxXiv:2207.11800 2022.
[37] Chen, Z., Z. Zhao, H. Luo, et al. Hale: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.00 004255 2024.
[38] Wang, J., Y. Gao, J. Sang. Valid: Mitigating the hallucination of large vision language models by visual layer fusion contrastive decoding. arXiv preprint arXiv:2411.15833 2024.
[39] Schwenk, D., A. Khandelwal, C. Clark, et al. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pages 146-162. Springer, 2022.
[40] Lin, T.-Y,, M. Maire, S. Belongie, et al. Microsoft coco: Common objects in context. conference on computer vision, pages 740-755. Springer, 2014.
European
[41] Hudson, A., C. D. Manning. Gqa: new dataset real-world visual reasoning and compositional ques- tion answering. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, 2019.
pages
[42] Li, X. L., A. Holtzman, D. Fried, et al. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint arXiv:2210.15099 2022.
[43] Yin. S., C. Fu, Zhao, et al. A survey on multimodal large language models. National Science Review, LI(22): 2024.
[44] Rohrbach, A., L. Hendricks, Burss, et al. Object hallucination in image captioning. arXiv preprint arXiv:1809 02156, 2018.
[45] Agrawal, H., K. Desai, Y. Wang, al. Nocaps: Novel object captioning scale. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8948-8988 8957. 2019.
12

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
All contribution statements are as accurate as possible, with no overstatements.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
be expected to generalize other settings.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [NA]
Justification: The main contribution this work is to show that previously proposed methods do not work claimed, thereby helping redirect research efforts away from an unproductive path. As the paper does not introduce new method, not applicable to discuss methodological limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer:
[NA]
Justification:
The paper does not involve any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: All datasets used in this work are publicly available. We have included the complete source code in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
14

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All datasets used in this work are publicly available. We have included the complete source code in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We provide detailed descriptions of the experimental setup for each experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Greedy decoding the primary strategy in most experiments, yielding consistent outcomes across repeated runs, thus rendering statistical significance analysis unnecessary. For experiments involving sampling-based decoding, results are averaged over five runs and demonstrate statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experiments were conducted on a server equipped with one A800 GPU, Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We did violate any part the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed the societal impacts of the work in Section 7
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
16

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have ensured that all existing assets used in this work are properly cited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
17

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
All new assets introduced the paper are well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
18

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttpp/////euups..... wwwparippsscccoooeeeeeee cc/Conferencess///////e for what should or should be described.
19

---

A Contrastive Decoding for Hallucinations
This section details the components and workflows of three mainstream hallucination mitigation methods: VCD, ICD, and SID. These techniques employ contrastive decoding strategies to reduce hallucinatory content, ensuring outputs align more closely with the visual input.
A.1 Visual Contrastive Decoding
Visual Contrastive Decoding (VCD) acts as corrective mechanism, reducing hallucinations by contrasting with distributions derived from distorted visual inputs. Specifically, for a given textual query and visual input v, the model generates two distinct output distributions: one conditioned on the original v, and the other on distorted version v'. The distorted input v' is derived by applying predefined perturbations (e.g., a Gaussian noise mask) to e. Subsequently, a new contrastive probability distribution computed by leveraging the differences between these distributions. This contrastive distribution, denoted as Pued, is defined as:
Poed (y (0,0,,)) x²) softmax logity (y 0, x)) t (logit, (y) v, logit (y 0', 22) (6)
A challenge in this process is avoiding indiscriminate penalization of the entire output space, this could unfairly suppress valid predictions while encouraging the generation of implausible tokens. To address this, VCD integrates adjusts penalization based on the confidence levels inferred from the output distribution conditioned on the original visual input The constraint is defined as follows:
adaptive plausibility constraint. This constraint dynamically
Vhead (y-c)) { yt V pg (3/ U. U, Mece) 2 max (w v.2.
Pood (ye V. 1/,,
if yt & Vhead (yece)
where V denotes the output vocabulary of MLLMs, and / is hyperparameter that controls the truncation of the next-token distribution. Larger values of enforce more aggressive truncation, retaining only the tokens with the highest probabilities.
By integrating the contrastive adjustment with the adaptive plausibility constraint, the complete formulation is expressed as follows:
ye softmax [1+ a) logita (ve u, v,e) llogit (ze V1, vet)
(8)
subject to Yt E (Bec).
A.2 Instruction Contrastive Decoding
Based on findings that instruction disturbances with negative prefixes significantly amplify hallu- cinations by increasing multimodal alignment uncertainty, Instruction Contrastive Decoding (ICD) mitigates ballucinations by initially emphasizing the probabilities of hallucinated concepts and sub- sequently detaching these from the original probability distribution. Accordingly, the contrastive distribution, Picd, can be defined as:
Pied (y) v, v, x') softmax
logito (v/ v, À logit, (y v,a')]
A larger À imposes stronger penalty on the decisions made by MLLMs under disturbances. Here, x' represents perturbed instructions involving negative prefixes. Additionally, ICD integrates adaptive plausibility constraint from VCD to prevent the unjust suppression of valid predictions.
A.3 Self-Introspective Decoding
Building on VCD and ICD, SID recognizes that directly perturbing the entire original input introduces excessive uncertainty and noise, hindering the induction of the desired hallucination effect. To address this, as shown on the far right of Figure [ SID adjusts the model architecture by retaining only small subset of image tokens with low attention scores after the early decoder layers. This
20

---

adaptive mechanism enhances the generation of vision Aasion-and-ee and-text association hallucinations during auto- regressive decoding. Subsequently, SID isolates these hallucinations from the original probability distribution, leading to the definition of the contrastive distribution Psid as:
logito (v v, + a (logit, (y v, x) logito (y/ v, æ)) ]
Psid (y c, x) softmax
(10)
Here, 0' represents the MLLM with structural modifications introduced by SID. Additionally, SID incorporates the adaptive plausibility constraint.
B Additional Research on Changes in Prediction
After applying Visual Contrastive Decoding, the prediction shifts of the LLaVA-v1.5-7B model are shown in Figure 6 It is evident that across all datasets, the number of samples transitioning from Positive Negative is significantly smaller than those shifting from Negative Positive. This indicates that the model's output distribution is biased towards Yes.
COCO-Random
cOcO-Popular
COCO-adverserial
AOKVOA-Random
AOKVOA-Popular
AOKVQAaavverseiial
GQA-Random
GQA-Popular
GQA-adversorial
Figure 6: Changes in the distribution of model predictions across all datasets after applying Visual Contrastive Decoding.
C The True Impact of Contrastive Perturbations on Prediction Outcomes
We begin by presenting our conclusion: image perturbations in visual contrastive decoding can cause the model to randomly "overlook" certain elements of the image due to their stochastic nature.
21

---

Consequently, the model's 's response is based on only partial view of the input. This phenomenon is illustrated with concrete example, as shown in Figure 7
Question: How many uncut fruits are in the image?
Time Steps 0
Time Steps = 100
Time Steps = 300
Time Steps = 500
Answer: There four uncut Fruits in the image.
There uncut Fruits in the image.
There uncut fruits in the image.
uncut fruits appear in the image.
Figure 7: Visualization of the Actual Impact of Contrastive Perturbations.
With the original visual input, the model exhibited hallucination, responding: "There are four uncut fruits in the image." After introducing diffusion noise and setting the diffusion time steps to 100, 300, and 500 respectively, the LLaVA-v1.55 5--B1 model produced the following responses: "There are 3 / 2 O uncut fruits in the image," correspondingly. These results demonstrate that the presence of noise causes the model to lose access to certain localized visual details, leading it to respond based on the remaining visible features.
This also helps explain why, in POPE benchmark, contrastive examples often cause the model to answer "No". not due to external bias, but because it genuinely fails to detect the relevant object. However, this effect should not be mistaken for hallucination. fact, contrastive samples do not induce hallucinations. For example, when visual contrastive decoding is applied to this case, the model correctly outputs: "There are four uncut fruits the image."
This brings back to the foundational assumption behind contrastive decoding methods for mitigating hallucinations. These approaches are grounded the belief that object hallucinations in MLLMS primarily stem from overly dominant linguistic priors. and thus attempt to suppress such priors through contrastive mechanisms. However, practice, hallucinations often arise because MILMs fail to correctly ground specific semantic concepts in the visual input, for instance, the concept of "uncut" in the previous example. As such, the failure visual contrastive decoding can be attributed to fundamental flaw in its underlying assumption.
D Performance across different decoding strategies.
Table 7 presents the impact various decoding strategies on the prediction performance of LLaVA- v1.5-7B evaluated on the POPE Benchmark. It is evident that the greedy strategy yields significantly better results compared to the sampling strategy.
Table 7: Performance the MSCOCO dataset across different decoding strategies.
Model
Decoding
Random
Popular
Adversarial
Greedy Sample
87.1 0.0
85.8 |. 0.0
83.6 J 0.0
LLaVA
83.6 3.5
82.4 3.4
80.2 3,4
Greedy Sample
86.0 | 0.0
85.6 J 0.0
84.0 | 0.0
VW
85.2 0.8
84.2 | 1.4
82.3 | 1.7
22

---

E Further Experiments on Forced Distribution Adjustment
Tables and present the performance of Prompt-Based Adjustment (PBA) and Output Layer Modification (OLM) on the VOKVQA and GQA datasets. On both datasets, PBA and OLM maintain performance levels comparable contrastive decoding methods. However, after surpassing the balanced distribution, accuracy declines rather than improving.
Table 8: Performance of Prompt- Based Adjustment (PBA) and Output Layer Modification (OLM)
AOKVQA dataset
LLaVA-v1.5-7B
L.aVA-v1.5-13BB
Category
Decoding
Accuracy
F1-Score
Yes( %
Accuracy
F1-Score
Yes( %
Greedy
88.6
88.1
45.1
88.8
88.1
44.1
VCD
86.8
87.0
52.0
87.7
87.4
47.8
Random
SID
88.6
88.5
48.7
87.6
87.4
48.5
PBA
89.0
88.5
45.9
88.7
89.0
52.8
OLM
89.0
89.1
51.3
87.3
87.9
55.9
Greedy
85.2
85.0
48.5
86.7
86.2
46.2
VCD
82.6
83.6
56.2
85.5
85.5
50.0
Popular
84.1
85.3
$1.0
PBA
84.8
84.8
50.1
83.6
84.8
57.9
OLM
82.6
83.8
57.7
83.6
85.1
59.5
Greedy
78.8
79.8
54,9
80.3
80.8
52.6
VCD
75.5
78.4
63.6
79.4
80.7
56.4
Adversarial
SID
77.8
79.7
59.1
79.2
80.6
57.4
PBA
78.3
79.6
74.7
78.3
OLM
75.0
78.3
65.3
74.8
78.7
68.3
Table 9: Performance of Prompt-Base Based Adjustment (PBA) and Output Layer Modification (OLM)
GQA dataset
LLa VAv1.--.
LLaVA-v1.5-13B
Category
Decoding
Accuracy
F1-Score
Yes(%)
Accuracy
F1-Score
Yes( %
Greedy
89.4
88.9
45.5
89.5
88.9
45.1
VCD
88.0
88.4
53.6
88.1
88.0
Random
SID
88.8
88.7
49.1
88.9
88.8
49.1
PBA
89.4
89.0
47.0
88.8
89.2
53.9
OLM
89.4
89.7
52.9
87.1
87.9
56.9
Greedy
84.0
84.2
50.8
86.4
86.1
48.1
VCD
82.5
83,9
59.1
85.7
85.5
52.1
Popular
SID
82.9
55.0
85.2
53.2
PBA
83.2
83.7
53.1
84.8
84.8
61.9
OLM
79.8
82.0
62.5
80.8
83.1
63.1
Greedy
80.9
81.7
53.9
82.2
82.6
$2.3
VCD
78.0
80.6
63.3
81.2
82.4
56.8
Adversarial
SID
79.9
81.3
57.8
81.1
82.4
57.6
PBA
80.5
81.5
55,9
81.1
82.3
67.3
OLM
76.4
79.6
65.9
75.9
79.6
68.3
23

---

Further Experiments on Standalone Application of the Constraint
Tables 10 and present the performance of the adaptive plausibility constraint on the AOKVQA and COCO datasets. When applied independently, the adaptive plausibility constraint consistently improves performance by 1.5% to 2% across both datasets.
Table 10: Impact of Adaptive Plausibility Constraint (Applied Independently) on AOKVQA Dataset
LLaVA-v1.5-7B
LLaVA-v1.5-133
Category
Decoding
Accuracy
F1-Score
Yes(
Accuracy
F1-Score
Yes( %
sample
84.6
83.9
45.4
84.7
83.9
45.3
VCD
85.9
86.1
51.7
86.5
48.4
Random
ICD
86.5
85.8
45.3
86.3
85.5
44.5
SID
86.8
86.6
48.8
85.8
85.5
48.3
sample
86.5
45.3
85.8
44.8
sample
80.3
80.2
49.7
81.8
81.5
48.2
VCD
81.3
56.2
84.2
51.0
Popular
ICD
82.2
82.1
49.6
83.4
82.9
47.4
SID
82.9
83.3
52.7
82.9
83.1
51.2
sample
82,2
82.1
83.2
47.8
sample
74.8
55.9
77.9
54.0
VCD
74.8
77.6
62.8
78.4
79.7
56.4
Adversarial
ICD
76.1
77.5
56.4
77.5
78.3
53.5
SID
76.8
78.6
58.4
77.9
79.4
57.4
sample'
76.1
77.5
56.4
77.9
78.7
54.0
Table 11: Impact of Adaptive Plausibility Constraint (Applied Independently) on COCO Dataset
LLa VA-v1.555
LLaVA-11
Category
Decoding
Accuracy
F1-Score
Yes (65
Accuracy
F1-Score
Yes( %
sample
83.6
81.8
39.8
83.9
82.3
40.8
VCD
87.8
87.3
46.4
87.8
87.1
44.6
Random
ICD
85.4
83.7
39.8
85.5
83.9
40.3
SID
86.6
85.4
41.9
86.4
85.3
42.7
sample
85.4
83.7
39.8
85.5
84.0
40.6
sample
82.4
80.7
41.0
83.0
81.5
41.7
VCD
85.8
85.5
48.4
86.2
85.7
46.1
Popular
ICD
84.1
82.5
41.1
84.6
83.1
41.2
SID
83.6
82.7
44.9
84.4
83.5
44.8
sample*
84.1
82.5
41.1
84.6
83.1
41.6
sample
80.2
78.7
43.2
81.2
79.9
43.5
VCD
81.1
81.7
52.9
83.0
82.9
49.3
Adversarial
ICD
81.8
80.5
43.3
82.6
81.3
43.2
80.9
80.4
47.5
81.8
81.3
sample'
81.8
80.5
43.3
82.7
81.5
43.4
24


Reviews:
[{"review_id": "UWKtW3ASPJ", "reviewer": "Reviewer_q7te", "summary": "This paper questions the effectiveness of contrastive decoding in reducing object hallucinations in multimodal LLMs. It shows that the observed improvements on the POPE benchmark are primarily due to biased output adjustments and the adaptive plausibility constraint rather than genuine mitigation of hallucinations.", "questions": "* How can future benchmarks be better designed to measure hallucination accurately?\n* Can contrastive decoding be restructured to correct both false positives and false negatives?\n* How would these findings extend to open-ended generation tasks beyond binary QA?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n* This paper provides a thorough analysis of contrastive decoding, identifying its key limitations through empirical and theoretical evidence.\n* This paper introduces well-designed baselines that isolate and reveal the misleading factors behind observed performance gains.\n* This paper conducts comprehensive experiments across models and tasks, strengthening the validity of its conclusions.\n* This paper offers practical insights into evaluation practices and future directions in hallucination mitigation research.\n\nWeaknesses:\n* This paper relies heavily on the POPE benchmark, raising concerns about generalizability to more complex or real-world scenarios.\n* This paper does not explore or compare with other hallucination mitigation techniques beyond contrastive decoding.\n* This paper assumes a consistent response bias pattern (“Yes”/“No”) across datasets, which may not always hold in broader contexts.", "was_revised": false, "final_justification": "", "rebuttal": "# *Initial Acknowledgment*\n\n**We are grateful to the reviewers for their valuable and encouraging feedback. The insightful suggestions and comments helped us identify areas for clarification and further improvement. In the following, we address each of the reviewers’ concerns point by point.**\n\n---\n\n# *W1: This paper relies heavily on the POPE benchmark, raising concerns about generalizability to more complex or real-world scenarios.*\n\n**We appreciate your concern. Nevertheless, our study extends beyond the scope of POPE.** As shown in Tables 4 and 6 of the paper, we have extended our analysis to more realistic scenarios, encompassing a broader range of tasks, including discriminative tasks, captioning tasks, and open-ended QA tasks**. Specifically:\n\n- Discriminative tasks: POPE, MME\n- Captioning tasks: CHAIR, NoCaps\n- Open-ended QA tasks: LLaVA-Bench\n\nThus, we have in fact demonstrated the ineffectiveness of contrastive decoding-based hallucination mitigation methods across a diverse set of tasks.\n\nThat said, as you rightly pointed out, we did devote significant space in our paper to analyzing the performance of hallucination mitigation methods specifically on the POPE benchmark. This is because the papers proposing methods such as VCD primarily used POPE to validate their effectiveness. Our aim is to demonstrate the ineffectiveness of these methods in a point-by-point manner, thereby strengthening the persuasiveness of our paper.\n\n---\n\n# *W2: This paper does not explore or compare with other hallucination mitigation techniques beyond contrastive decoding.*\n\nIndeed, you are correct. As you have noted, the core objective of our paper is to verify the ineffectiveness of hallucination mitigation methods designed around contrastive decoding, as reflected in our title: \"Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs.\"\n\n**Hallucination mitigation is a broad and evolving research field, and we do not intend to suggest that all mitigation methods are ineffective, that would clearly be an inaccurate conclusion**. For instance, approaches based on reinforcement learning fine-tuning have shown promising effectiveness to date.\n\n**However, it is important to emphasize that demonstrating the ineffectiveness of contrastive decoding alone in mitigating hallucinations holds substantial value for the research community.** Hallucination mitigation methods based on contrastive decoding have become a widely adopted and influential line of work. Over the past year, more than 30 papers following this paradigm have been published at top-tier conferences, including CVPR, ICLR, NeurIPS, AAAI, and ICML. By revealing the fundamental limitations of this research paradigm, our findings can help the community avoid unnecessary expenditure of resources and effort, thereby redirecting attention toward more promising directions.\n\n---\n\n# *W3: This paper assumes a consistent response bias pattern (“Yes”/“No”) across datasets, which may not always hold in broader contexts.*\n\n**We believe there may be a misunderstanding. We do not assume that every dataset exhibits a consistent bias pattern.** In fact, aside from POPE, the other evaluation datasets used in our paper—MME, CHAIR, NoCaps ,and LLaVA-Bench—do not display any clear or systematic bias of this kind.\n\n**However, on datasets that do not exhibit such bias pattern, applying hallucination mitigation methods like VCD leads to a clear drop in prediction accuracy (see Table 4 in our paper).** This further supports our conclusion: the accuracy improvements observed with VCD on the POPE dataset are incidental and primarily driven by the unique bias patterns present in MLLM outputs on POPE. When evaluated across a broader range of tasks and datasets, **the experimental results provide even stronger evidence that methods like VCD are, in fact, ineffective.**\n\n---\n\n# *Q1: How can future benchmarks be better designed to measure hallucination accurately?*\n\nThank you for your excellent question. When it comes to object hallucination, the primary focus of our paper, we believe that a well-designed hallucination benchmark should satisfy two key criteria: **question clarity** and **response complexity**.\n\n- **question clarity:** The textual output of MLLMs tends to be more diverse and flexible compared to that of LLMs, primarily because images inherently contain far more information than text. As a result, any textual description of an image inevitably involves a degree of abstraction and selective emphasis. For example, a caption might omit a small cup located in the upper-left corner of the image. However, if the model is explicitly asked whether there is a cup in the upper-left corner, it may respond “yes.” This illustrates why captioning tasks are often unsuitable for evaluating hallucinations in MLLMs: the omission of certain objects may reflect linguistic conventions or summarization strategies rather than true hallucinations. **Therefore, we argue that object hallucination evaluation in MLLMs requires explicit and well-defined questions.**\n\n- **Response Complexity**: Datasets like POPE are indeed designed with the goal of asking clearly defined questions. However, their binary (yes/no) format results in minimal information in the response, which makes it difficult to assess whether the model truly understands the concept of the object, has accurately located it within the image, or has simply guessed correctly by chance. This ambiguity complicates the evaluation of hallucinations. Therefore, we believe that a series of short, content-rich questions would be a more effective testing approach. For instance: \"What is the object in the center of the image?\", \"What color is it?\", or \"What object is to its right?\" **Compared to binary classification tasks, such short-form Q&A prompts elicit more detailed answers, making it easier to identify hallucinations in object understanding.**\n\n---\n\n# *Q2: Can contrastive decoding be restructured to correct both false positives and false negatives?*\n\nFirst, we can confidently state that current hallucination mitigation methods based on the principle of contrastive decoding are incapable of simultaneously correcting both false positives and false negatives. **Looking ahead, is it possible to redesign contrastive decoding to address both issues at once? We remain skeptical.**\n\n**The core issue lies in a key assumption underlying contrastive decoding-based hallucination mitigation: that object hallucinations arise primarily from overly strong language priors**. As a result, most contrastive decoding methods are designed to weaken the visual input in order to expose these language priors, thereby suppressing them during generation.\n\nHowever, our experiments challenge this assumption in two ways:\n\n- When the visual input is weakened, the model does not exhibit the expected hallucinations. Instead, it generates outputs based solely on the limited visual features that remain (**see Appendix C for details**).\n- The outputs of the contrastive samples, which are intended to reflect language priors, often do not align with the hallucinated predictions. This suggests that multimodal object hallucination may not necessarily stem from strong language priors.\n\nTaken together, these findings suggest that we are still far from developing a contrastive decoding strategy capable of simultaneously correcting both types of errors.\n\n---\n\n# *Q3: How would these findings extend to open-ended generation tasks beyond binary QA?*\n\n**Indeed, our paper extends the analysis to open-ended generation tasks.** Although the flexible nature of outputs in such tasks makes it difficult to perform the kind of direct quantitative evaluation used on POPE, **we provide a qualitative analysis in Appendix C**. Through several open-ended QA examples, we show that the model’s handling of contrastive samples is largely driven by partial visual inputs, rather than the generation of the hypothesized hallucinations.\n\n**Furthermore, as shown in Tables 4 and 6 of our paper, we evaluate the performance of methods like VCD on captioning tasks (CHAIR, NoCaps) and open-ended QA tasks (LLaVA-Bench)**. We find that on these datasets, which do not exhibit clear output biases, **these methods actually lead to a decline in prediction accuracy.**\n\nTo summarize our experimental findings:\n\n- On the POPE dataset, methods like VCD lead to improved prediction accuracy. **However, our analysis shows that this improvement stems from shifts in the output distribution, rather than genuine mitigation of hallucinations.** Notably, the original papers proposing these methods primarily reported results on POPE, which has led many to believe that such methods are effective.\n\n- **On other datasets, these methods even result in decreased prediction accuracy**, which further demonstrates that contrastive decoding strategies fail to effectively mitigate hallucinations.\n\nAs the above results demonstrate, hallucination mitigation methods based on the contrastive decoding paradigm are **ineffective across all types of hallucination evaluation tasks.**\n\n---\n\n# *Final Acknowledgment*\n\n**We sincerely thank the reviewers once again for their time, effort, and valuable feedback. We hope our responses have resolved the concerns raised, and we are happy to provide additional clarification if needed. We truly appreciate your thoughtful review, and we wish you all the best in your future research endeavors.**"}, {"review_id": "L7ihTbjMYU", "reviewer": "Reviewer_z19D", "summary": "The authors show that contrastive decoding methods that show improvement on the POPE dataset are actually due to spurious reasons, namely 1. the imbalance of yes's to no's, and 2. the collapse to greedy search due to the the adaptive plausibility constraint.", "questions": "1. Do these findings generalize beyond the POPE dataset? What about datasets like TruthfulQA or FactScore that have non-binary outputs?\n2. The model evaluations are quite limited -- the paper only evaluates three models in 7B and 13B. Are larger models beyond 13B still susceptible to this type of bias? It would be useful to show a trend -- does this class imbalance issue worsen or improve with increased model size?\n3. Please discuss metrics to address shortcomings in binary accuracy evaluation, like per-class precision/recall and F1. Performance should be reported in terms of these metrics as well for clarity.\n4. Please include examples illustrating your finding, i.e., questions that were previously true negatives that got flipped to false positives due to this bias. \n5. The discussion should include proposals for clear alternatives and solutions for this issue.\n6. Please clarify the COCO and GQA subsets of the POPE dataset; the authors seem to assume reader knowledge of the dataset which may not always be the case.\n7. Equation 3 has p_0 on the left; should this be p_\\theta?\n8. Tables 5 and 6 use Sample† and Sample* interchangeably; please keep one notation.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: The paper notes a significant limitation in the evaluations of VLLMs on the POPE dataset. This is an important finding for past and future works that cite the POPE dataset. \n\nWeaknesses:\n1. It's unclear that showing a limitation of the POPE dataset is sufficient for acceptance as a conference paper. The authors should further provide background on the impact of the POPE dataset in the broader research field of hallucination detection. As it stands, the paper is not strong enough in my opinion as as a standalone work.\n2. The authors should evaluate other hallucination benchmarks that move beyond binary QA (i.e., free-text, caption-based outputs). One experiment that would strengthen the paper would be to demonstrate this effect on non-binary QA benchmarks as well. Class imbalance is a problem with all binary QA datasets (not unique to POPE), and there are metrics for addressing this.", "was_revised": true, "final_justification": "The authors provide a clear problem with sufficient evidence to back up their claims. I have updated the score based on their provided feedback.", "rebuttal": "# *Initial Acknowledgment*\n\n**We are grateful to the reviewers for their valuable and encouraging feedback. The insightful suggestions and comments helped us identify areas for clarification and further improvement. In the following, we address each of the reviewers’ concerns point by point.**\n\n---\n\n# *W1: It's unclear that showing a limitation of the POPE dataset is sufficient for acceptance as a conference paper.*\n\n**We believe there may be a significant misunderstanding here. In fact, the main focus of our paper is not to highlight a limitation of  POPE dataset, but rather to demonstrate that hallucination mitigation methods based on the idea of contrastive decoding are fundamentally ineffective**, despite the fact that over a hundred such methods have been proposed in the past year. **Please allow me to clarify the central arguments of our paper in detail below.**\n\n**First, allow us to briefly introduce the current state of research on hallucination mitigation methods based on the concept of contrastive decoding.** Over the past year, more than 30 studies on contrastive decoding-based hallucination mitigation have been published at top-tier conferences such as CVPR, NeurIPS, and ICLR. These works generally follow a similar research paradigm: they propose a novel hallucination mitigation strategy grounded in contrastive decoding, evaluate its performance on benchmarks such as POPE and MME, and report improved prediction accuracy—thereby demonstrating the effectiveness of their proposed method.\n\n**The purpose of this paper is to demonstrate that this class of methods (such as VCD, ICD, SID, and many others of a similar kind) is fundamentally ineffective.**\n\n**We first demonstrate that the accuracy improvements reported by methods such as VCD on POPE and MME are in fact illusory.** In Section 4 of the paper, we analyze how the apparent performance gains on POPE and MME primarily stem from two misleading factors: \n- crude, unidirectional adjustments to the model’s output distribution\n- the adaptive plausibility constraint, which effectively reduces the sampling strategy to a greedy search. \n \nFurthermore, in Section 5, we introduce two dummy methods, PBA and OLM, and show that they can easily achieve similar improvements on POPE and MME. This further supports our claim that the observed accuracy gains of methods like VCD do not result from a genuine reduction in hallucination.\n\n**In addition, we broaden the scope of our experiments to cover a wider range of tasks, and find that methods such as VCD actually lead to a degradation in performance.** Specifically, we evaluate these methods on image captioning tasks (CHAIR, NoCaps) and an open-ended question answering task (LLaVA-Bench), as reported in Table 4 and Table 6. Our results indicate that in open-ended generation scenarios, these methods not only fail to help, but can even reduce accuracy.\n\n**Building on the above two points, we provide a detailed demonstration that methods such as VCD are entirely ineffective at mitigating hallucinations.** Nevertheless, research in this area continues to proliferate, with numerous researchers investing significant time and effort into a fundamentally flawed direction. With this paper, we aim to correct the prevailing, yet misguided, research paradigm—and potentially the research trajectory itself—**in order to help the community avoid unnecessary expenditure of resources and effort.**\n\n---\n\n# *W2&Q1: The authors should evaluate other hallucination benchmarks that move beyond binary QA. What about datasets like TruthfulQA or FactScore that have non-binary outputs?*\n\n**Importantly, our experiments are not limited to discriminative tasks such as POPE. As shown in Tables 4 and 6 of the paper, we have extended our evaluation to include open-ended generation tasks as well.** These tasks span a diverse set of benchmarks:\n\n- Discriminative tasks: POPE, MME\n- Captioning tasks: CHAIR, NoCaps\n- Open-ended QA task: LLaVA-Bench\n\n**Additionally, Appendix C presents a qualitative analysis demonstrating that the core mechanism of contrastive example processing is prediction based on partial image inputs**, thereby illustrating how methods such as VCD impose coarse modifications on output distributions in open-ended generative tasks.\n\n**We appreciate your interest in the TruthfulQA and FactScore datasets—both are excellent benchmarks. However, they are not suitable for evaluating methods such as VCD, ICD, and SID,** as these approaches require image inputs. Since TruthfulQA and FactScore are designed for purely text-based tasks, they cannot accommodate image-conditioned methods like VCD.\n\n---\n\n# *Q2: Are larger models beyond 13B still susceptible to this type of bias?*\n\n**We are somewhat puzzled, as the forced modification of the output distribution originates from the nature of methods like VCD themselves, and is independent of the underlying model size, as analyzed in Section 4 of our paper.** Are you suggesting that such methods might not affect the output distribution when applied to larger-scale models? In any case, we have demonstrated the biased output distribution (YES%) on the LLaVA-NeXT-34B model, as shown in the table below.\n\n| **Method** | **Random**   | **Popular**  | **Adversarial** |\n|:----------:|:----------:|:---------:|:-----------:|\n| Greedy | 41.4 ↑0.0 | 42.5 ↑0.0 | 45.0 ↑0.0   |\n| VCD | 48.5 ↑7.1| 51.3 ↑8.8 |   55.2 ↑10.2   |\n\nBy comparing Table 3 in the paper with the table above, we observe that larger-scale models yield improved initial prediction accuracy, partially mitigating the initial output bias. **However, methods such as VCD continue to enforce output distribution modifications (with a bias toward \"Yes\") with consistent intensity, showing no reduction in the degree of adjustment.**\n\n---\n\n# *Q3: Please discuss metrics to address shortcomings in binary accuracy evaluation, like per-class precision/recall and F1.*\n\nThank you for the suggestion. Due to space constraints, we only report precision, recall, and F1-score on the POPE-COCO-Random dataset in the table below. Full results on the complete POPE dataset will be added in the appendix.\n\n| **Method** | **Precision**       | **Recall**          | **F1-Score**        |\n|:----------:|:------------------:|:-------------------:|:-------------------:|\n| Greedy     | 97.3% ↑0.0%    | 76.2% ↑0.0%      | 85.5%  ↑0.0%   |\n| VCD        | 91.5% ↓5.8%       | 85.0% ↑8.8%        | 88.1% ↑2.6%     |\n\nThe table shows a notable drop in precision and a clear increase in recall. This is consistent with the fact that the model tends to predict \"NO\" more often on COCO-Random, while methods like VCD shift the output distribution toward \"YES.\" \n\n---\n\n# *Q4: Please include examples illustrating your finding, i.e., questions that were previously true negatives that got flipped to false positives due to this bias.*\n\nThank you for your suggestion. As external links are not permitted in the rebuttal, we can only provide the textual portion of the example (with the image input omitted), as shown below. The complete example, including the image, will be included in the appendix of the revised paper.\n\n- {\"question_id\": 16, \"prompt\": \"Is there a motorcycle in the image?\", \"vanilla-text\": \"No\", \"vcd-text\": \"Yes\", \"image\": \"COCO_val2014_000000429109.jpg\"}\n- {\"question_id\": 82, \"prompt\": \"Is there a bowl in the image?\", vanilla-text\": \"No\", \"vcd-text\": \"Yes\", \"image\": \"COCO_val2014_000000569839.jpg\"}\n\n---\n\n# *Q5: The discussion should include proposals for clear alternatives and solutions for this issue.*\n\nHallucination mitigation is a broad and active research area. **This paper solely aims to demonstrate that hallucination mitigation methods based on contrastive decoding are ineffective.** However, there exist many other effective approaches to address hallucination. For example:\n\n- Reinforcement learning-based fine-tuning strategies for hallucination mitigation. [1]\n\n- Test-time intervention techniques designed around latent knowledge space. [2]\n\n---\n\n# *Q6: Please clarify the COCO and GQA subsets of the POPE dataset.*\n\n**Thank you for your valuable suggestion. We would like to briefly provide some background information.** POPE originally refers to a polling-based query method proposed by the authors. In the original POPE paper, this method was applied to images from three datasets—COCO, AOKVQA [3], and GQA [4]—which together constitute the widely recognized POPE benchmark. Accordingly, COCO, AOKVQA, and GQA refer specifically to the image sources utilized. A detailed introduction to POPE will be included in the appendix.\n\n---\n\n# *Q7&Q8: Equation 3 has $p_0$ on the left; should this be $p_\\theta$? Tables 5 and 6 use Sample† and Sample\\* interchangeably; please keep one notation.*\n\nYes, you are absolutely right. In Equation (3), it should indeed be $p_\\theta$, and the term \"Sample†\" should be used consistently in both Table 5 and Table 6. Thank you for pointing out these issues. We apologize for our oversight and will make the necessary corrections.\n\n---\n\n# *Final Acknowledgment*\n\n**We sincerely thank the reviewers once again for their time, effort, and valuable feedback. We hope our responses have resolved the concerns raised, and we are happy to provide additional clarification if needed. We truly appreciate your thoughtful review, and we wish you all the best in your future research endeavors.**\n\n---\n\n[1] Yang, Zhihe, et al. \"Mitigating hallucinations in large vision-language models via dpo: On-policy data hold the key.\" CVPR 2025.  \n[2] Liu, Sheng, et al. \"Reducing hallucinations in vision-language models via latent space steering.\" arXiv preprint arXiv:2410.15778 (2024).  \n[3] Schwenk, Dustin, et al. \"A-okvqa: A benchmark for visual question answering using world knowledge.\" ECCV 2022.  \n[4] Hudson, Drew A., and Christopher D. Manning. \"Gqa: A new dataset for real-world visual reasoning and compositional question answering.\" CVPR 2019."}, {"review_id": "lv4tGDeNzu", "reviewer": "Reviewer_RnVz", "summary": "This paper critically examines popular methods for reducing object hallucinations in multimodal language models, including common approaches like Visual Contrastive Decoding (VCD), Instruction Contrastive Decoding (ICD), and Self-Introspective Decoding (SID). The authors argue that although these methods appear to boost performance on the POPE benchmark, these improvements are misleading. The real cause of the better scores is not actual reduction in hallucinations, but rather two side effects: (1) pushing the model’s answers toward “Yes” to balance the data, and (2) using a constraint that effectively turns sampling into greedy search, which helps on simple yes/no tasks. The authors demonstrate this by introducing their own artificial tricks (like always encouraging “Yes” or tweaking the output layer) that achieve similar “improvements” without addressing hallucination at all. The key takeaway is that current contrastive decoding techniques don’t really solve the problem, even if the numbers look better.", "questions": "In lines 170–174, the observation that MLLMs tend to generate “NO” is discussed. Is this tendency specific to the POPE benchmark, or does it generalize to other datasets and evaluation protocols? If this is not a general phenomenon, how do the conclusions regarding contrastive decoding’s tendency to produce more “YES” answers remain valid?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths:\n\n- The paper tackles a real and important issue: how to fairly evaluate progress on hallucination in multimodal models, which has real-world safety stakes.\n\n- The experimental setup is well thought out and includes multiple models, benchmarks, and clever “control” tricks to highlight the central flaw, underscoring a potentially vital claim for the community: current contrastive decoding techniques don’t really solve the problem, even if the numbers look better.\n\n- The analysis of why these methods get better numbers (by unbalancing answers or changing the search strategy) is clear and persuasive.\n\n\n## Weaknesses:\n\n- The paper suffers from some notation and presentation problems that hinder readability. For example, a symbol like p_cd is introduced around Equation (2) without clear definition (presumably referring to a contrastive decoding probability distribution), which may confuse readers. Similarly, some results are hard to interpret due to labeling issues, e.g., Table 6 contains a “Sample*” notation that is never explained, and the annotations in Figure 3 are unclear, making it difficult to follow the figure’s message. These issues suggest the exposition could be more polished and self-contained.\n\n- The main conclusion, “we confirm that while contrastive decoding enhances performance, it ultimately fails to mitigate hallucinations”, remains ambiguous. Could the authors clarify the intended implication and its broader significance?", "was_revised": true, "final_justification": "I read the rebuttal from the authors and the discussion among other reviewers before the final justification.", "rebuttal": "# *Initial Acknowledgment*\n\n**We are grateful to the reviewers for their valuable and encouraging feedback. The insightful suggestions and comments helped us identify areas for clarification and further improvement. In the following, we address each of the reviewers’ concerns point by point.**\n\n---\n\n# *W1: The paper suffers from some notation and presentation problems that hinder readability.*\n\nThank you for pointing out these issues. We sincerely apologize for the lack of clarity in some of the markup definitions and expressions, which may have affected readability. Below, we clarify each of the ambiguous parts one by one.\n\n## *1. A symbol like $p_{cd}$ is introduced around Equation (2) without clear definition, which may confuse readers.*\n\nWe apologize for the lack of clarity in our explanation. Your understanding of $p_{cd}$ is absolutely correct, it indeed refers to the probability distribution generated by the contrastive decoding strategy. We will promptly add a clear definition of $p_{cd}$ in the manuscript as soon as we are able to modify the PDF.\n\n## *2. Table 6 contains a \"Sample\\*\" notation that is never explained.*\n\nThank you for pointing out this issue. We sincerely apologize for our use of an incorrect superscript symbol in this instance. The correct notation should be *Sample†*, consistent with the definition provided in Table 5, which refers to the sampling strategy that applies the adaptive plausibility constraint independently. We deeply regret any confusion this oversight may have caused.\n\n## *3. The annotations in Figure 3 are unclear, making it difficult to follow the figure’s message.*\n\n**Here, we provide a more detailed explanation of the information presented in the four subfigures of Figure 3.**\n\n- The top-left subfigure uses accuracy as the unit on the vertical axis, illustrating that using greedy decoding as the strategy yields significantly higher prediction accuracy compared to direct sampling.\n\n- The top-right subfigure presents a specific test example, featuring the question “Is there a cellphone in the image?” and the ground-truth answer “No.”\n\nWe suspect that the main source of confusion may stem from the two subfigures in the lower half of Figure 3. **These subfigures are intended to illustrate why using direct sampling as the decoding strategy can lead to an unfair improvement in model prediction accuracy when the Adaptive Plausibility Constraint is applied.**\n\n- As shown in the bottom-left subfigure, when using the direct sampling strategy under normal conditions, the model predicts the correct answer “No” with a probability of 91.2%, but still produces the incorrect answer “Yes” with a probability of 8.8%. These values can be found in the bar chart labeled \"Probs.\"\n\nAccording to Equations (2) and (3), the Adaptive Plausibility Constraint penalizes candidates with relatively low predicted probabilities. This is achieved by setting their logits to negative infinity, as illustrated in the two bar charts labeled “logits,” which causes their softmax probabilities to become zero.\n\n- As shown in the bottom-right subfigure, after applying the Adaptive Plausibility Constraint, the logits value for the incorrect answer “Yes” is set to negative infinity, causing its predicted probability to drop to zero. As a result, even though the decoding strategy remains direct sampling, the probability of selecting the correct answer “No” increases from 91.2% to 100% (see the bar charts labeled “Probs” for comparison). As demonstrated in our conclusion, this is equivalent to a degeneration of the direct sampling strategy into greedy search.\n\n**Thank you for pointing out the lack of clarity in the presentation of this figure. We will revise the figure to include clearer and more explicit annotations, and we will also expand the accompanying textual explanation to ensure the figure is easier to interpret.**\n\n---\n\n# *W2: The main conclusion remains ambiguous. Could the authors clarify the intended implication and its broader significance?*\n\n## *1. Clarification of Ambiguities in Our Conclusion*\n\nHere, “enhance performance” refers specifically to the improvement in prediction accuracy achieved by methods such as VCD on discriminative datasets like POPE and MME.\n\n**However, this performance gain is illusory**. As shown in Section 5 of our paper, even **two dummy approaches**, Forced Distribution Adjustment and Standalone Application of the Adaptive Constraint, **are able to achieve similar numerical improvements** in prediction accuracy on POPE and MME, comparable to those of methods such as VCD. Furthermore, we demonstrate that on other datasets, including CHAIR, NoCaps, and LLaVA-Bench, methods like VCD can even lead to a decline in prediction accuracy.\n\nThis clearly indicates that the numerical performance gains achieved by methods such as VCD primarily result from incidental output distribution adjustments and potentially unfair decoding strategies, as analyzed in Section 4, rather than from genuinely mitigating hallucinations.\n\n## *2. The Broader Significance of Our Conclusion*\n\nTo highlight the significance of this conclusion, **we would first like to provide a brief overview of the current research landscape on hallucination mitigation methods based on the idea of contrastive decoding.**\n\n**Over the past year, more than 30 papers on hallucination mitigation methods based on contrastive decoding have been published at top-tier conferences such as CVPR, ECCV, NeurIPS, and ICLR.** These works generally follow a similar research paradigm: a novel contrastive decoding-based method is proposed, evaluated on benchmark datasets such as POPE and MME, and reported to achieve higher prediction accuracy, thereby claiming superiority in hallucination mitigation.\n\n**However, as we have analyzed and demonstrated in this paper, these proposed methods do not truly mitigate hallucinations.** Instead, their reported improvements stem from coincidental adjustments to output distributions on limited datasets and the potential use of inconsistent decoding strategies, leading to misleading performance gains.\n\n**The number of papers in this area continues to grow, with many researchers investing significant time and effort into what may ultimately be a misguided direction.** Through this work, we aim to correct the current flawed research paradigm, and possibly the underlying research trajectory itself, in order to help the community avoid unnecessary expenditure of resources and effort.\n\n---\n\n# *Q1:  In lines 170–174, the observation that MLLMs tend to generate “NO” is discussed. Is this tendency specific to the POPE benchmark, or does it generalize to other datasets and evaluation protocols?* \n\nWe address this question in two parts:\n- For discriminative tasks (e.g., binary Yes/No questions), the same trend persists not only on the POPE dataset, but also across other datasets.\n- For open-ended generation tasks, the trend may manifest in different forms, but the core conclusion, that contrastive decoding fails to mitigate hallucinations, consistently holds.\n\n## *1. Discriminative Tasks (Binary Yes/No Answers)*\n\n**Across all discriminative tasks, applying methods such as VCD consistently leads to an output distribution that is biased toward the answer “YES.”** To further support this point, we present the change in the proportion of “YES” responses on the MME dataset. As shown in the table below, the application of VCD significantly increases the frequency of “YES” answers.\n\n|   **Method**  | **MME-Count** | **MME-Color** | **MME-Existence** | **MME-Position** |\n|:----:|:----:|:----:|:----:|:----:|\n| Vanilla | 61.67% | 56.67% | 50.00% | 66.67%  |\n| VCD  | 71.67% | 63.33% | 53.33%  | 73.33%  |\n| Difference  | +10.00% | +6.66%  | +3.33% | +6.66% |\n\n## *2. Open-Ended Generation Tasks (CHAIR, NoCaps, LLaVA-Bench, etc.)*\n\n**In our paper, we extend the observation of output bias, specifically, the tendency to generate “YES”, to open-ended generation tasks.** Unlike binary tasks where responses are limited to “YES” or “NO,” open-ended tasks allow for more flexible and diverse outputs. As a result, the shift in output distribution cannot be characterized merely by an increased frequency of “YES” responses.\n**In Appendix C,** we provide a qualitative analysis of several examples from open-ended question answering tasks, revealing a more fundamental impact of contrastive decoding methods:\n\n**The contrastive examples do not expose or guide the model to hallucinations; instead, they force the model to make predictions based on incomplete visual inputs.** As a result, the model may randomly penalize highly relevant candidate answers, depending on how the incomplete input is interpreted, which offers no real benefit to predictive accuracy. In some cases, this may even lead to decreased accuracy when the contrastive output happens to align with the correct answer.\n\n**This conclusion is also supported by our experimental results.** As shown in Table 4, applying VCD to captioning tasks such as CHAIR and NoCaps, as well as to open-ended QA tasks like LLaVA-Bench, leads to a decrease in prediction accuracy.\n\n**In summary, the phenomena we reveal in this paper span multiple tasks and datasets.** We chose to focus our in-depth analysis on the POPE dataset because methods such as VCD have been primarily validated on this benchmark. By doing so, we aim to provide a point-by-point response to these prior works, thereby making our argument more direct and comprehensive.\n\n---\n\n# *Final Acknowledgment*\n\n**We sincerely thank the reviewers once again for their time, effort, and valuable feedback. We hope our responses have resolved the concerns raised, and we are happy to provide additional clarification if needed. We truly appreciate your thoughtful review, and we wish you all the best in your future research endeavors.**"}, {"review_id": "mm4F6maeMc", "reviewer": "Reviewer_bPyU", "summary": "This paper critically challenges the perceived effectiveness of contrastive decoding strategies for mitigating object hallucinations in MLLMs. This paper claims that the gained performance on the POPE benchmark are highly misleading and do not represent genuine hallucination mitigation. It attribute these \"spurious improvements\" to two primary factors: (1) a unidirectional shift of the model's output distribution towards \"yes\" response, and (2) the adaptive plausibility constraint, which effectively degrades sampling decoding to greedy search. To substantiate their claims, the paper introduces and evaluates \"spurious improvement methods\" that achieve comparable performance to contrastive decoding, decisively demonstrating that the reported gains are unrelated to actual hallucination mitigation.", "questions": "Please refer to the weakness below.", "limitations": "As the paper does not introduce a new method, there is no discussion of methodological limitations.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. The paper critically challenges the contrastive decoding paradigm in MLLM hallucination mitigation, providing evidence that the observed performance gains are deceptive. This critique helps explore the actual causes of alleviating hallucinations for the community.\n2. The authors include extensive analyses covering ablation studies on both hallucination and general benchmarks between tricky designs and original contrastive decoding from the perspective of unidirectional shift and adaptive plausibility constraint.\n\n### Weaknesses\n1. The paper criticizes contrastive decoding, but it mostly focuses on older methods. It could be better to analyze on newer methods [1]. [1] Park, Yeji, et al. \"Convis: Contrastive decoding with hallucination visualization for mitigating hallucinations in multimodal large language models.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 6. 2025.\n2. The paper's definition and verification of object hallucination are insufficient, and it does not provide explanations or experimental results for whether more complex relational or attribute hallucinations are similarly unresolved by contrastive decoding.\n3. Despite using LLaVA-v1.5 and QwenVL-7B, the experimental scope for this training-free method could be expanded to include more recent, larger, and architecturally diverse MLLMs (e.g., Qwen-2.5-VL, LLaVA-Next) to ensure broader applicability of its conclusions.", "was_revised": true, "final_justification": "My comments are properly addressed. The authors show significant efforts on the newly added experimentations. As a result, I increase my rating. But I hope that the modifications, such as details of more complex hallucinations, added in rebuttal can be included in the revised version, which are important for the completeness of limitation discussion.", "rebuttal": "# *Initial Acknowledgment*\n\n**We are grateful to the reviewers for their valuable and encouraging feedback. The insightful suggestions and comments helped us identify areas for clarification and further improvement. In the following, we address each of the reviewers’ concerns point by point.**\n\n---\n\n# *W1: The paper criticizes contrastive decoding, but it mostly focuses on older methods. It could be better to analyze on newer methods [1].* \n\n**In fact, Self-Introspective Decoding (SID) presented in our paper was published at ICLR 2025 and, like Convis, is a contemporaneous work and among the most recent approaches.** However, we understand your interest in more recent comparative decoding methods. **Therefore, we present experimental results of Convis [1] here.**\n\n**We report the performance of Convis on the MSCOCO dataset from the POPE benchmark in the tables below.** Convis shows an output distribution similar to VCD, heavily biased toward YES. This bias enhances performance on the random and popular subsets but reduces it on the adversarial subset. Despite this, our two dummy methods, PBA and OLM, achieve prediction accuracy gains comparable to those of Convis.\n\n***Table: Changes in Output Distribution (Yes%) Induced by Convis***\n| **Method**      | **Random**   | **Popular**   | **Adversarial** |\n|:--------------:|:------------:|:-------------:|:----------------:|\n| Greedy   | 39.2 ↑0.0    | 40.4 ↑0.0     | 42.6 ↑0.0        |\n| VCD   | 46.4 ↑7.2    | 48.8 ↑8.4     | 53.1 ↑10.5       |\n| Convis   | 43.4 ↑4.2    | 45.7 ↑5.3     | 48.8 ↑6.2        |\n| PBA    | 40.2 ↑1.0    | 41.6 ↑1.2     | 44.0 ↑1.4        |\n| OLM   | 44.2 ↑5.0    | 46.5 ↑6.1     | 50.1 ↑7.5        |\n\n***Table: Prediction Accuracy of Convis***\n| **Method** | **Random**| **Popular**| **Adversarial** |\n|:--------:|:--------:|:---------:|:--------:|\n| Greedy  | 87.1 ↑0.0    | 85.8 ↑0.0     | 83.6 ↑0.0   |\n| VCD   | 88.6 ↑1.5    | 86.2 ↑0.4     | 81.9 ↓1.7   |\n| Convis   | 88.1 ↑1.0    | 85.9 ↑0.1     | 82.1 ↓1.5  |\n| PBA    | 87.6 ↑0.5    | 86.2 ↑0.4     | 83.7 ↑0.1  |\n| OLM   | 89.6 ↑2.5    | 87.3 ↑1.5     | 83.6 ↑0.0  |\n\n**The following table presents the performance of Convis on other datasets, including MME, CHAIR, NoCaps, and LLaVA-Bench.**  It can be observed that, similar to methods like VCD, ICD, and SID, the prediction accuracy decreases after applying Convis.\n\n| **Method** | **MME-P↑** | **MME-C↑** | **CHAIR-S↓** | **CHAIR-I↓** | **Nocaps↑** | **LLaVA-Bench↑** |\n|:----------:|:---------:|:---------:|:-----------:|:-----------:|:----------:|:----------------:|\n| Greedy | **1475.1**    | **349.6**     | **21.6**    | **7.2**     | **83.8**     | **65.7**       |\n| Convis    | 1343.6    | 306.3     | 24.1        | 8.7     | 81.7       | 64.9     |\n\n\nThe experimental results above demonstrate that, like VCD, ICD, and SID, Convis does not effectively mitigate hallucination.\n\n---\n\n# *W2: The paper's definition and verification of object hallucination are insufficient, and it does not provide explanations or experimental results for whether more complex relational or attribute hallucinations are similarly unresolved by contrastive decoding.*\n\n## *1. Definition of Object Hallucination*\n\nWe appreciate your attention to the definition of object hallucination. **However, our definition is consistent with that adopted in the original papers of methods such as VCD.** Specifically, the definition of object hallucination in the VCD paper is as follows:\n\n> Object hallucination, in this context, refers to the phenomenon where LVLMs generate textual content that is semantically coherent but inconsistent with the ground-truth objects present in the given image.\n\nSince methods such as VCD claim to address this type of object hallucination, we aim to provide a point-by-point response in our paper to demonstrate that they, in fact, fail to resolve such hallucinations.\n\n## *2. Whether relational or attribute hallucinations also remain unresolved?*\n\n**In fact, our experiments on the MME and LLaVA-Bench datasets, as reported in Tables 4 and 6 of the paper, include evaluations of both relational and attribute hallucinations.** Due to space constraints, we only presented the overall scores for these datasets and omitted the detailed results for individual subsets. However, several of these subsets are specifically designed to assess relational and attribute hallucinations. We present the corresponding subset-level results below.\n\n| **Method**  | **MME-Existence** | **MME-Count** | **MME-Position** | **MME-Color** | **LLaVA Bench-Complex** |\n|:--------:|:-------:|:-------:|:-------:|:------:|:------:|\n| Greedy  | **190.0**  | **145.0**  | **113.3**   | **170.0**   | **81.2**  |\n| VCD   | 180.0  | 135.0   | 96.7    | 150.0   | 79.4   |\n\nAs shown, on subsets such as Color and Position from the MME dataset and the Complex subset from LLaVA-Bench, methods like VCD lead to a noticeable drop in prediction accuracy. This clearly indicates that such methods fail to mitigate relational and attribute hallucinations.\n\n---\n\n# *W3: Despite using LLaVA-v1.5 and QwenVL-7B, the experimental scope for this training-free method could be expanded to include more recent, larger, and architecturally diverse MLLMs.*\n\nWe appreciate your interest in exploring more recent MLLMs. **In accordance with your suggestion, we have conducted additional experiments using the LLaVA-NeXT model.** The tables below present VCD's performance on the COCO dataset within the POPE Benchmark, based on the LLaVA-NeXT-13B model. As observed, the results closely mirror those obtained with LLaVA-v1.5-7B. VCD continues to impose a bias in the model’s output distribution toward answering YES, resulting in improved accuracy on the Random and Popular subsets, but reduced accuracy on the Adversarial subset. Notably, the two dummy methods introduced in the paper, PBA and OLM, continue to yield comparable performance gains with ease.\n\n***Table: Impact of VCD on Output Distribution (Yes%) in LLaVA-NeXT-13B***\n| **Method** | **Random**   | **Popular**  | **Adversarial** |\n|:------------:|:------------:|:------------:|:-------------:|\n| Greedy | 40.0 ↑0.0    | 41.6 ↑0.0    | 44.3 ↑0.0      |\n| VCD | 47.4 ↑7.4    | 50.0 ↑8.4    | 54.6 ↑10.3     |\n| PBA | 41.8 ↑1.8    | 42.5 ↑0.9    | 44.6 ↑0.3      |\n| OLM | 45.6 ↑5.6    | 47.3 ↑5.7    | 51.1 ↑6.8      |\n\n***Table: Prediction Accuracy of VCD with LLaVA-NeXT-13B***\n| **Method** | **Random**   | **Popular**  | **Adversarial** |\n|:-----------:|:------------:|:------------:|:-------------:|\n| Greedy | 88.4 ↑0.0      | 87.3 ↑0.0   | 84.8 ↑0.0     |\n| VCD    | 89.6 ↑1.2    | 87.5 ↑0.2    | 82.9 ↓1.9     |\n| PBA   | 89.1 ↑0.7    | 88.0 ↑0.7    | 84.7 ↓0.1    |\n| OLM   | 90.2 ↑1.8    | 87.8 ↑0.5    | 84.1 ↓0.7     |\n\nBelow are the experimental results of VCD based on the LLaVA-NeXT-13B model across several datasets, including MME, NoCaps, and LLaVA-Bench. Consistent with the findings on LLaVA-v1.5-7B, the application of the VCD method leads to a noticeable drop in accuracy.\n\n| **Method** | **MME-P↑** | **MME-C↑** | **CHAIR-S↓** | **CHAIR-I↓** | **Nocaps↑** | **LLaVA-Bench↑** |\n|:----------:|:---------:|:---------:|:-----------:|:-----------:|:----------:|:---------------:|\n| Greedy | **1575.0** | **332.0** | **18.6**    | **6.8**     | **97.4**   | **87.1**         |\n| VCD        | 1488.3     | 302.1     | 22.7        | 7.9         | 93.8       | 85.2             |\n\n**Finally, we would like to clarify why our experiments are limited to LLaVA-v1.5 and QwenVL-v1.5.** The original papers proposing methods such as VCD primarily conducted their experiments using these two models. Since the aim of our study is to demonstrate the ineffectiveness of these methods, **we intentionally adopted the exact same experimental settings to ensure a fair and direct comparison.** This approach strengthens the validity and persuasiveness of our conclusions. For this reason, we did not extend our experiments to newer models.\n\n---\n\n# *Final Acknowledgment*\n\n**We sincerely thank the reviewers once again for their time, effort, and valuable feedback. We hope our responses have resolved the concerns raised, and we are happy to provide additional clarification if needed. We truly appreciate your thoughtful review, and we wish you all the best in your future research endeavors.**\n\n---\n\n[1] Park, Yeji, et al. \"Convis: Contrastive decoding with hallucination visualization for mitigating hallucinations in multimodal large language models.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 6. 2025."}]

Author rebuttal:

Meta-review: This paper provides a careful and insightful analysis of contrastive decoding for hallucination mitigation in VLLMs. It identifies key limitations through both empirical and theoretical evidence, supported by well-designed baselines that isolate misleading factors behind observed performance gains. The experiments are comprehensive, spanning multiple models and tasks, and the findings offer practical insights into fair evaluation practices and future directions for hallucination research.

The work highlights an important flaw in the evaluation of VLLMs on the POPE dataset and convincingly shows that apparent improvements from contrastive decoding often arise from unbalanced answers or altered search strategies rather than genuine mitigation. This critique is clear, well-argued, and of strong relevance to the community, especially given the real-world safety implications.

Overall, this is a well-executed, timely, and impactful contribution. Most reviewers recommend acceptance.
Decision: Accept (poster)