Decoupling Contrastive Decoding: Robust Hallucination Mitigation in Multimodal Large Language Models
Authors: Wei Chen
Xin Yan
Bin Wen
Fan Yang
Tingting Gao
Di ZHANG
Long Chen
Venue: NeurIPS 2025 poster ()
Keywords: Multi-modal Large Language Model
RLHF
Direct Preference Optimization
Contrastive Decoding
Primary area: deep_learning

Abstract: Although multimodal large language models (MLLMs) exhibit remarkable reasoning capabilities on complex multimodal understanding tasks, they still suffer from the notorious 'hallucination' issue: generating outputs misaligned with obvious visual or factual evidence. Currently, training-based solutions, like direct preference optimization (DPO), leverage paired preference data to suppress hallucinations. However, they risk sacrificing general reasoning capabilities due to the likelihood displacement. Meanwhile, training-free solutions, like contrastive decoding, achieve this goal by subtracting the estimated hallucination pattern from a distorted input. Yet, these handcrafted perturbations (e.g., add noise to images) may poorly capture authentic hallucination patterns. To avoid these weaknesses of existing methods, and realize ``robust'' hallucination mitigation (\ie, maintaining general reasoning performance), we propose a novel framework: Decoupling Contrastive Decoding (DCD). Specifically, DCD decouples the learning of positive and negative samples in preference datasets, and trains separate positive and negative image projections within the MLLM. The negative projection implicitly models real hallucination patterns, which enables vision-aware negative images in the contrastive decoding inference stage. Our DCD alleviates likelihood displacement by avoiding pairwise optimization and generalizes robustly without handcrafted degradation. Extensive ablations across hallucination benchmarks and general reasoning tasks demonstrate the effectiveness of DCD, \ie, it matches DPO’s hallucination suppression while preserving general capabilities and outperforms the handcrafted contrastive decoding methods.

Decoupling Contrastive Decoding: Robust Hallucination Mitigation in Multimodal Large Language Models
Wei Chen', Xin Yan", Bin Wen Fan Yang". Tingting Gao Di Zhang', Long Chen
HKUST, University of Waterloo, Kuaishou Technology wchendb@connect. susteectt.ut.hht ust.hk longchendust.hi
Abstract
Although multimodal large language models (MLLMs) exhibit remarkable rea- soning capabilities on complex multimodal understanding tasks, they still suffer from the notorious "hallucination" issue: generating outputs misaligned with ob- vious visual or factual evidence. Currently, training-based solutions, like direct preference optimization (DPO), leverage paired preference data to suppress hal- lucinations. However, they risk sacrificing general reasoning capabilities due the likelihood displacement. Mcanwhile, training- -freinng-rreeeee solutions, like contrastive decoding, achieve this goal by subtracting the estimated hallucination pattern from a distorted input. Yet, these handcrafted perturbations (e.g., add noise to images) may poorly capture authentic hallucination patterns. To avoid these weaknesses of existing methods, and realize "robust" hallucination mitigation (i.e., maintain- ing general reasoning performance), we propose a novel framework: Decoupling Contrastive Decoding (DCD). Specifically, DCD decouples the learning of posi- tive and negative samples preference datasets, and trains separate positive and negative image projections within the MLLM. The negative projection implicitly models real hallucination patterns, which enables vision-aware aware negative images in the contrastive decoding inference stage. Our DCD alleviates likelihood dis- placement by avoiding pairwise optimization and generalizes robustly without handcrafted degradation. Extensive ablations across hallucination benchmarks and general reasoning tasks demonstrate the effectiveness of DCD, i.e., matches DPO's hallucination suppression while preserving general capabilities and out- performs the handcrafted contrastive decoding methods. Code is available in https: bettps//gittubb com Caam/HUUST-LLe Http://KUSS-Loorroo//////oo/
1 Introduction
Today's multimodal large language models (MLLMs) 2 5 have demonstrated remarkable general reasoning capabilities by integrating visual and textual understanding, facilitating applications such as medical image analysis (6, 7 and multimodal search engines [8]. Despite their versatility,
critical limitation persists: MILMs may generate outputs that contradict obvious factual evidence or
misrepresent visual inputs, known as the hallucination problem 9 10 11 12] For instance, models may describe objects absent from an image (e.g., claiming "dog" in cat-only scene) or fabricate implausible relationships (e.g., asserting "a person riding a bi bicycle" when only bicycle is present). Such hallucinations erode users trust and hinder deployment in high-stakes domains like healthcare [6] autonomous driving [33
"Correponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

DPO
Ours
VCD
Ours
Aegatter
Sampleer
Projectoo
MLLM -+
Training Steps
Training Steps
(a)
(b)
Figure I: Comparison between existing hallucination mitigation methods and DCD. (a) Training- based method (e.g., DPO (4)): DPO directly optimizes the likelihood gap between positive (correct) and negative (hallucinatory) responses using preference datasets. However, maximizing this gap (y" vs. y can inadvertently lower the probability both responses, causing likelihood displacement and potential degradation of general reasoning capabilities. Here, v, yt. and y denote images, questions, positive responses, and negative responses, respectively; 0 represents model parameters, and a is the contrastive decoding coefficient. (b) Training-fie free method (c.g.. VCD (IS)) vs. DCD; Traditional contrastive decoding (VCD) reduces hallucinations by comparing model outputs from original (v and artificially distorted (v c.g., noise-added) visual inputs at inference time.
To mitigate this hallucination issue, recent training-bused approaches 116 17
18
99 20 draw
inspiration from reinforcement learning from human feedback (RLHF) 21 a finetune paradigm
that aligns models with human preferences. These RLHF methods typically involve two stages: 1) Hallucination Preference Dataset Construction. Recent efforts [16] 17 18] 19] 20 22 collect paired positive-negative samples form the preference dataset, where positive responses are the correct answers and negative responses are the hallucinatory answers. These "high-quality" negative samples are often collected from goner--geerratd hallucinatory outputs, ensuring alignment with the real ballucination observed in MLLMs. 2) Preference Optimization Training. Direct preference optimization (DPO) [4] is the most prevalent and well-explored approach train MILMs with preference datasets. It bypasses complex reinforcement learning pipelines by directly maximizing the likelihood gap between positive and negative responses. While DPO demonstrates efficacy in hallucination mitigation, this paired-sample optimization process risks inducing likelihood displacement problem [23]: By maximizing the gap between positive and negative answers, DPO may inad inateettntly lower the probabilities of both responses (as shown in Figure [a)). It potentially sacrifices the model's general reasoning capabilities and leads to performance degradation in open- ended tasks.
28 resort to contrastive decoding 30 to alleviate hallucination. They hold the assumption that MLLM is easier to have the hallucination issue with distorted inputs. For example, image perturbations disrupt semantic coherence and amplify hallucinatory tendencies. By transferring the log-likelihood differences of model outputs with that of distorted images, contrastive decoding methods force MILM focus more on images details (cf. Figure (b). However, existing perturbation strategies are handcrafted and artificial, such as adding noise to images [5].. Therefore, these artificial contrastive distributions may not reflect the authentic ballucinations produced by MLLMs, they are vision-and-ter agnostic and can introduce uncertainty noise in the decoding process 27 which is not robust in complex tasks.
In parallel, training-free methods [ 24 25 26
In this paper, we aim to avoid these weaknesses of existing methods, and realize a more robust hallu- cination mitigation. By "robust", we hope the method can not only significantly reduce hallucination cases, but also preserve general capabilities challenging reasoning tasks. To this end, we propose a novel framework: Decoupling Contrastive Decoding (DCD). Specifically, DCD has two designs: 1) Decoupling Learning. We decouple pairwise positive-negative samples learning of preference dataset into separate learning to alleviate likelihood displacement. 2) Vision-awart Negative Image. We learn negative image projector from negative samples, to replace the vision-and agnostic image perturbations in contrastive decoding.
In the training phrase, we utilize positive and negative samples to separately train a positive image projection and negative image projection in MILMs. By decoupling the learning of positive and negative samples, our approach not only circumvents the likelihood displacement problem inherent to DPO but also generalizes robustly across diverse domains. In the inference stage, we adopt the negative image projection to project original image features into "negative" image features in contrastive decoding. Unlike synthetic perturbations which may distort legitimate contextual relationships instead of specifically suppressing hallucinatory features, model-generater negative

---

samples in preference datasets accurately capture real hallucination distributions. In this way. our learnable negative image projection which trained on negative samples implicitly models hallucination patterns in contrastive decoding. Our method ensures that hallucination suppression is guided by real hallucination patterns rather than handcrafted perturbations, thereby preserving the model's is ability to generate coherent and creative outputs in open-ended scenarios.
To validate the effectiveness of the proposed DCD, we conduct extensive experiments across multiple benchmarks, including hallucination--peciiie benchmarks 31 33 34
and general multimodal rea- soning tasks 335 36 37 38 Our DCD achieves comparable hallucination suppression performance
to DPO while maintaining or even improving accuracy on general benchmarks, whereas DPO incurs noticeable performance degradation general ability benchmarks. Compared contrastive decoding methods, DCD demonstrates superior generalization, outperforming it across all benchmarks.
Moreover, thanks the decoupled learning design, our method even can learn from negative samples solely (i.e., only train a negative image projection). When fine-tuning a projector solely on negative (hallucinatory) responses from the preference dataset, we observe significant hallucination mitigation, whereas training on the positive responses yields marginal improvement. This phenomenon suggests that the model has already internalized sufficient knowledge about positive responses in the supervised fine-tuning phase, and the following RLHF phase provides limited gains. In contrast, we are the first reveal that: Explicitly learning from negative samples equips the model with discriminative awareness of hallucination patterns. which complements its existing knowledge. Looking forward, we hope our observations will pave the way for new advancements hallucination mitigation and more general MILMM alignment.
Conclusively, our contributions
follows:
1) Decoupled Learning for Robust Alignment. We propose Decoupled Contrastive Decoding (DCD), the first framework to separate positive/negative sample optimization from preference datasets in MLLM training. It alleviates the likelihood displacement problem in DPO [1].
preserving general capabilities while mitigating hallucinations.
2) Vision-Aware Hallucination Suppression. We introduce a learnable negative image projector trained on real hallucinatory samples. Unlike handcrafted perturbations (e.g.. VCD [5)), ), this projector generates distortions grounded in actual MLLM errors, enabling precise suppression of hallucinations.
3) Paradigm Shift in Preference Learning. We reveal that negative samples alone suffice for hallucination mitigation, challenging the prevailing that explicit modeling of errors (not just positive alignment) is critical for robustness.
parodigm-showing
4) Comprehensive Experiments. Extensive ablations and results demonstrate that our method achieves competitive performance with training- based methods (e.g., DPO 14] on halluci- nation benchmarks while maintaining general ability.
2 Related Work
Multimodal Large Language Model (MLLM). MLLMs have witnessed remarkable advancements have shaped the paradigm of current MLLMs' architecture: vision encoder 43 44 to process visual input, an LLM 45 46 to reason and generate text, and
these days. Previous arts 339
40
41
42
to bridge the gap between the visual and textual representations. The training for MILMs typically involves two main stages: pre-training and post-training. The provides the model with a strong foundation of general knowledge. The post-training aligmment stage consists of two phases: supervised fine-tuning (SFT) [49] and reinforcement learning from human feedback (RLHF) 14. 50) 51 52). This process refines the model's task task-specific performance and encourages alignment with human preferences. Building upon this foundation, current research continuously pushes the boundaries of their capabilities [53] [4]
cross-modal projector 40
47
48
large-scale pre-training stage [49
55 [6 5 (7). Meanwhile, some research investigates alternative architectures that could shape the
future of MLLMs, such as Omni [58 59 60 61).
MoE 62 63 64), Encoder-Free [65 66 67], and
Any-to-Any [68 69 70 (
Hallucination Preference Alignment. To reduce hallucinations and align the model with human values, prior efforts are made via instruction tuning (2] or reinforcement learning from human feedback (RLHF)
14
50
51 52) Some preliminary efforts extend such preference alignment
3

---

Forward
Backwand
Optional
paired proference data
Images
Images
Encoder
Plo
Projection (vr)
Negative (P
Poenive g.(?)
Language Model
Language Model
Training Stage
DPO
Ours
Projection (
Ingus Texts In rio varge is
Input Tests: dee imape Ahene
Vision Encodar
Language Model 0
I Language Model
Addd Noive
Anvengrie Trucor
Masegede Tracue
Negative 2 (v)
Tracer
Toace Srud Co
Trenu
Piople
Zaccli
Paplee
(1+ial (rggt logita
(1++a)) (ott))llgta(ylppp Collo
Inference Stage
VCD
Ours
Figure 2: Comparison of DCD with DPO [14] and VCD 11 in the training and inference stages. (a) Training stage: DPO jointly optimizes positive-negative responses, risking likelihood displacement. Our method (DCD) separately learns positive and negative image projections avoid this issue. (b) Inference stage: VCD uses artificial noise negative inputs, whereas DCD leverages learned negative visual features that reflect authentic hallucination patterns, enhancing effective hallucination suppression.
techniques to Multimodal Large Language Models (MLLMs) 22 20 RLHF-V [66 collected fine-grained preference dataset with annotated correctional human feedback. In contrast, BPO [88] utilized an automatic method to construct preference datasets, by distorting the image inputs of of MILMs to obtaín biased responses. Similarly, RLAIF-V [17] and VL.Fedbback [9] obtain large-scale human-level preference annotations through MLLMs. These preference datasets offer promising foundation for mitigating hallucination and bias. Our approach leverages these datasets for positive and negative projection learning.
Contrastive Decoding. Contrastive Decoding was introduced by et.al. [30 to mitigate LLMs' undesirable outputs during text generation. As hallucinations are more common in the "amateur" model, they can be constrained by maximizing the likelihhdood difference between an "expert" and an "amateur". Existing methods extend this technique to MI.LMss to combat hallucinations through various debiasing strategies. Text-debiasing methods generate positive logits by amplifying image attention [72), negative text-biased logits via image manipulations, such as noisy images [I5], no images 773. edited images (29), and downsampling (29]. Image-debiasing methods generate negative mmage-bassed logits via disturbance instructions 74 or select from the differences between field-of-vied view pairs [24]. Unlike these approaches, our method leverages preference datasets to train separate positive and negative projections which provides robust contrastive signal, unbiased by text or image manipulations.
3 Preliminary
Direct Preference Optimization (DPO). DPO [14] alignment framework that directly optimizes an MILM to adhere to human preferences. Given a preference dataset DD {(x, v, Mes 31) of prompts æ, images v, positive responses yue, and negative responses yı, DPO leverages pairwise loss to align the model Ty with human feedback. The core objective function can be formulated as:
To(yu v) Tee x
To To r)) Tee((yn |x, v) )}
(D -B( log
log
-log
(1)
where Tee is a reference model (i.e., initial SFT model), 3 is a hyperparameter constant, and o (v(v)).))
denotes the sigmoid function. The term log mer(ylar,)) represents the probbabbyyy difference between the optimized model and the reference model, effectively acting an implicit reward signal.

---

By maximizing the likelihood of positive responses over negative ones under this reparameterization, DPO circumvents reward modeling while maintaining stable optimization.
Likelihood Displacement (33 identifies a critical limitation in DPO's optimization mechanism. This occurs because DPO's pairwise loss only maximizes the relative likelihood gap between preference pairs (yw, 3t) while allowing an arbitrary distortion of absolute probabilities for other responses. Consequently, the model may experience degraded performance on non-preference tasks the reference model previously handled well.
Visual Contrastive Decoding (VCD). MILMs process visual inputs and textual queries to generate responses y through auto-regressive decoding. The token probability distribution at each time step is:
(uelo, x., Met)) ix exp (Iogit,(unll, (<<t))
where V<r denotes the generated token sequence prior to time step t. Despite their capabilities, MLLMs frequently exhibit object hallucinations: generating textual descriptions that contradict visual evidence. Visual Contrastive Decoding (VCD) [15] is a training-free method designed to mitigate object hallucinations in MILMs.
In VCD, the model processes both the original visual input e and distorted version v', which is gener- ated by introducing controlled noise to By comparing the output distributions Po(Uelo, x. V-ce) and po(ye /o, yer). VCD adjusts the decoding process to suppress tokens that are likely hallucinations. The adjusted probability distribution Pvea (y/1; [v, v', x)) is computed as:
(Vel e', Ir, Meee) = softmax [[1 + a) llggit (Ve (u, Vei)) a logits 2, ver)l, (3)
where a is a hyperparameter controlling the influence of the distorted input. However, these artificial contrastive distributions may not accurately reflect the real hallucinations generated by MLLMs, as they are vision-and-text agnostiie and can introduce uncertainty in the decoding process.
4 Decoupling Contrastive Decoding
As shown in Figure 2 our method decouples the learning of positive and negative responses through three key components: (1) Negative Samples Learning, which trains a learnable hallucination projection to model error patterns; (2) Positive Samples Learning, which preserves the model's fidelity to ground-truth responses; and (3) Contrastive Decoding, which suppresses hallucinations by contrasting original and learned negative representations.
4.1 Motivation
To address the likelihood displacement problem inherent in DPO's joint optimization of positive and negative responses, we propose Decoupling Contrastive Decoding (DCD, Algorithm [ to decouple their learning processes-sepaneee while explicitly suppressing hallucinatory patterns from negative ones. Drawing inspiration from VCD's contrastive suppression mechanism, we hypothesize that hallucination mitigation can be achieved by contrasting the original visual context against learnable negative projection that encodes plausible hallucinatory deviations, rather than relying handcrafted perturbations. Unlike VCD's static noise-based distortions, which may misalign with authentic hallucination distributions, our learnable projection dynamically adapts to capture domain-agnostic hallucination features during training. By decoupling positive and negative learning, our approach circumvents the collateral suppression of non- non-preference responses while preserving the model's general reasoning capabilities.
enhancing the model's fidelity to positive samples
4.2 Negative Samples Learning
We train a hallucination-awarre negative image projection (v) to encode visual features that correlate with hallucinatory patterns. Given negative (hallucinated) response y1 paired with image we optimize 9s to maximize the likelihood of generating y/ when using the negative visual embedding ō (v):
Lugg - ()) log To(yaaz, 0t),
(4)
where 0 the parameter of the MILMM. This forces Js to learn transformations of e that align with the error distribution in yr effectively mapping "hallucination paalucinatioonprrmedd" embedding space.
5

---

Algorithm 1: Decoupling Contrastive Decoding
Input:
MILLM TTg, textual input x; image v, positive response You negative response yt, suppression strength a
Output: Generated response based on and
Initialize G6 and Go identically
while training do
Compute negative embedding: गt go(v)
Update Ie by minimizing Lneg (1 log (M ūt)
Compute positive embedding: UN (v)
Update De by minimizing fpux = Ra log (owll,, kw)
end
while inference do
Initialize V/ BOS,
while * EOS do
Compute positive logit,,. logit, (z, tw yct)
Compute negative logit; logit (Vel 2, 01, Vet)
Compute contrastive logit (1 a:) logit,. logit;
V1 arg max softmax (logit)
f = t +
end
end
4.3 Positive Samples Learning
To preserve factual alignment, we concurrently train the original image projection Io (v) using positive samples (I, U, Dw
Epos - (z,,,,)) log me (y (z, Uw).
T. = gv()).
(5)
Crucially, 9v and Io are initialized identically but updated independently, allowing the model to maintain dedicated pathway for faithful visual grounding while Je specializes in hallucination patterns. The language model parameters 0 remain shared across both objectives.
4.4 Inference Stage
During inference, suppress hallucinations by contrasting token likelihoods conditioned the positive (fw) and negative (6r) embeddings:
logit,, w logit (ye xx Omr Metr)
(6)
logit, logit, (Ve Ir, 01, Mete)
(7)
logit- (1 a)) logit,, a logit,
(8)
where modulates the suppression strength. Unlike VCD's static noise perturbations. (v) is dynamically adapted to the input image v, ensuring hallucination suppression aligns with contextually plausible hallucinations rather than arbitrary distortions.
5
Experiments
5.1 Experiment Setup
Hallucination Preference Datasets. We evaluated approach four widely-used hallucination preference datasets: RLHF-V [16] (human-annotanl visual preferences), BPO 18] (data-augmented synthetic preference pairs), RLAIF-V [17 (Al-annotated preferences), and VL.eeaback [19] (dense visual faithfulness annotations). For VLFeedback, we threshold responses using Visual Faithfulness scores (above four were considered positive, and those below two were considered negative), while others provide explicit preference pairs. Our method leverages both positive and negative samples to learn disentangled projections, with ablation studies on negative-only training.

---

General Performance
Hallucination
MMHal
Average"
SEED
MathVistal
MMStar
MMMU
MM-Vet
Hallusion
Score
Rate
LLavA-
58.57
27.9
30.20
34.6
23.7
1.79
0.70
39.22
35.69
VCD 15
56.98
27.0
31.33
33.1
24.4
1.64
0.72
39.01
35,30
Fine-tuned on RLHF
[16]
DPO [14
57.37
28.5
33.30
33.6
24.4
1.97
0.65
38.07
35.87
SimPO 75
58.00
28.1
33.40
33.0
26.7
1.95
0.69
36.70
35.98
Ours (Neg. Only)
58.60
27.8-o.7
33.00_...0
34.7+1.1
25.1
1.80
1.80-a.17
0.70
40.38
2:33
36,59 40.81
Ours (Pos. & Neg.)
58.55
28.0-o.6
34.53
34.5
25.0 1.7
-0.20
0.69 +0.04
40.48
22.41
36.84....8
Fine-tuned BPO
18
DPO [14]
54.48
26.6
33,00
35.6
29.7
1.61
0.64
37.85
36.21
SimPO 16
$7.07
27.6
32.47
34,3
27.3
1.24
0.80
39.53
36.58
Ours (Neg. Only)
48.60.1.53
28.3..07
33.20
34.4-
29.4
2.00
30.39
0.66..0.02
40.17
+0.64
4.76
Ours (Pos. Neg.)
58.61 11.54
27.9 vo.a
34.47 (1.47
34.1
29.5
40.65
0.60
-0.0
39.54..o.oz
37.35 00.77
Fine-tuned RLAIF-
DPO [4]
$7.43
26.8
33.13
34.9
25.5
1.90
0.66
35.96
35.62
SimPO 65
$7.89
27.8
32.80
33.2
27.1
1.67
0.71
36.80
36.24
Ours (Neg. Only)
58.57....6
28.7
33.07
33.07-0.06
34.3-0.6
25.6 1.5
1.70-
+0.00
39,85
43.00
36.68 40.44
Ours (Pos. & Neg.)
58.56....77
28.4..0.6
34.53
34.0....9
25.5 1.6
1.86
0.69.
40.0s
39.43..22
22.63
36.73.0044
Fine-tuned on VLFeedback [19]
DPO [4]
56.87
26.9
32,27
33.0
26.6
2,18
0.68
31.55
34.53
SimPO 66
58.24
28.0
31.47
32.7
27.0
1.74
0.75
30.28
34,98
Ours (Neg. Only)
58.62 .o.as
20.
33.2
44.4
26.1
o.33
0.69
+0.01
39.75 4..20
36.60, .1.02
Ours (Pos, & Neg.)
58.59
.0.35
28.1.12
34.61
42.54
34.1,1.1
27.3
1.80
0.70..0.2 39.96...44
37.11 w0.18
Performance comparison on general and hallucination benchmarks. "Neg. Only" means only trained on negative samples of preference datasets, "Pos. & Neg." is trained in both positive and negative samples, + indicates lower is better, and, ddnotees that the values of MMHal are not counted the average score. i For those benchmarks which need GPT to evaluate, we utilized GPT-4o 24-05-13.
Table 1:
Evaluation Benchmarks. We evaluated our proposed method's ability to mitigate hallu-
Random
Popular
Adversarial
Acc
F1
Acc
F-
Acc
Fi
cination and maintain general performance across diverse tasks. Hallucination Bench- marks: We used MM-Vet [34] (open-ended VQA), MMHal (32 (hallucination severity scoring), HallusionBench (33) (adversarial vi- sual contradictions), and POPE 31 (object existence verification) to assess the hallucina- tion. General Benchmarks: We selected SEED- Bench (36] (multimodal understanding). MM- Star 38 (complex VQA), and MMMU 37 (multi-discipline university-level problems) for general performance evaluation. These bench- marks provide comprehensive coverage of tasks for MLLMs. We also evaluated our method on Math Vista [35] to assess the performance on mathematical visual reasoning. We reported accuracy for most benchmarks. For MMHal, we reported the average score and hallucination rate. For POPE, we report accuracy and FI- score across all three sampling settings (random, popular, and adversarial).
L.LVA-1.5
86.70
85.23
84.73 83.63
83.53
82.22
VCD 15
87.73 87.16
85.38
85.06
80.88
81.33
Fine- nonedd RLHFV
[16]
DPO 14
78.77 73.31
78.57
73.12
77.80
72.41
SimPO 75
76.33 69.02
76.07
68.78
75.73
68.48
Ours (Neg. Only)
87.07 85,51
85,83
84.35
83,47
82.18
Ours (Pos. & Neg.)
86.97
85.39
85.77
84.26
83.47
82.16
Fine-tuned on BPO [8]
DPO 14
85.87
84.14
84.47
82.84
82.67
81.29
SimPO 65
86.27 84.59
85.37
83.75
82.73
81.35
Ours (Neg. Only)
87.80 86.60
86.25
85.11
83.67
82.84
Ours (Pos. & Neg.)
87.67
86.45
86.20
85.08
83.73
82.87
Fine-tumed on RLAIF-V
(1x]
DPO 14
86.50 85.01
85:40
83.99
82.20
81.14
SimPO 75
84.20 81.48
83.53
80.85
82.27
79.68
Ours (Neg. Only)
88.83 87.95
86.13
85.45
83.27
82.94
Ours (Pos. & Neg.)
88.70 87.77
86.03
85.30
83.23
82.85
Fine siee-ttoedd V..eedbback
119
DPO 14
74.03 64.93
73.87
64.78
73.57
64.52
SimPO 75
78.43 72.64
78.33
72.55
77.76
72.03
Ours (Neg. Only)
87.03
85.48
85.87
84.38
83.43
82.15
Ours (Pos. & Neg.)
87.27 85.69
85.72
84.45
83.53
82.24
Table 2: Performance comparison on POPE [3] which is about existing problems (i.e., "Yes"!"No" hallucination questions). "Neg. Only" means only trained negative samples of preference datasets, "Pos. & Neg." is trained in both positive and nega- tive samples.
Implementation Details. We conduct our ex- periments on LLaVA 1.5-7B M. training only the image projection layer while keeping all other parameters frozen. For training, we use the above four ballucination-related preference datasets: RLHF-V V [66 is trained for epochs, while the remaining datasets are trained for I epoch each on NVIDIA A100 80GB. Hyperparameters for contrastive decoding follow the configuration recommended VCD [15,, ensuring consistency with this baseline approach. For the DPO baseline, we follow the training setting of BPO [18].
7

---

POPE
POPE
SEED
MM-Vet Hallusion
SEED
MM- Vet Hallusion
Acc
F1
Ace
F1
LLaVA-I.3 LI
58.57
23.7
39.22
84.73
83.63
LLaVA-1.5 0
58.57
23.7
39.22
84.73
83.63
Add Noise
56.98
24.4
39.01
85.67
84.16
Random
58.34
26.1
39.49
86.10
84,93
Other image
57.39
25.1
37.01
86.13
84,97
Pre-train
58.50
26.4
39.74
84.83
83.74
Nega. Projection
58,60
29.4
40.17
86.25 85.11
SFT
58.60
29.4
40.17
86,25 85.11
Table 3: Ablation study of the type of negative
Table 4: Ablation study of types to initialize weight for negative image projection. "Random" means randomly initialing the projection weights, "Pre-train" denotes utilizing the model's pre-train stage projection weights to initial, and "SFT" is using the model's ss supervised- syaperviedd-nnttninng stage pro- jection weights to initial. This experiment is trained on BPO [8]. For POPE [I], we report the results of the adversarial set here.
image embedding used to contrastive decoding. "Add Noise" is adding noise to the image to get negative image embedding which is adopted by VCD [15], "Other image" means randomly sam- pling another image negative image embedding, and "Nega Projection" is our method trained on
BPO [8] which utilizes a negative image projec- tion to get negative image embedding. We present the adversarial set results for POPE (3]
5.2 Quantitative Results
Table 1 and Table 2 demonstrate DCD's effectiveness across hallucination and general reasoning benchmarks:
Hallucination Suppression. Our approach outperforms DPO 14 and VCD [5] on POPE (Table 2 improving F1 score over DPO across dataset variants. Notably, adversarial POPE accuracy reaches 83.73% (vs. DPO's 82.67%) )),, indicating robustness to challenging distractors. On open-ended hallucination metrics (Table D. we achieve comparable performance or outperform DPO on MM- Vet and reduce MMHal hallucination rates, validating our method's 's capacity to suppress hallucinations without over-constraining free-form responses.
Crucially, our method avoids DPO's performance degradation in general reasoning tasks. On MMStar and Math Vista (Table J. we surpass DPO while maintaining SEED-Bench accuracy within 0.1% % of the original LLaVA-1.5. This contrasts with DPO's 1 1.2-4.1 % drops on BEED-Bench, confirming that likelihood displacement undermines DPO's s generalizability. DCD even enhances Math Vista performance by 0.6-1.9 %, suggesting that hallucination suppression improves numerical reasoning by reducing spurious correlations.
General Capability Preservation.
Comparison to VCD. While VCD marginally improves POPE accuracy, it degrades performance on complex benchmarks like Math Vista (-0.9 %) and open-end benchmarks like Hallusion Bench (-0.2 %). Our method outperforms VCD across all metrics, demonstrating that learned negative embeddings better capture authentic hallucination patterns than static noise perturbations.
5.3 Ablation Studies
To better understand the effectiveness of our method, we conduct comprehensive ablation experiments analyzing key design choices. All experiments use the same base model and training configuration for fair comparison.
Types of Negative Image Embedding. We first investigate different strategies for obtaining negative image embeddings contrastive decoding. As shown Table the naive noise injection approach (adding 500-step noise to original images in VCD (15)) improves performance on POPE [31 (a binary hallucination benchmark contains "Yes" " or "No" question) but degrades general multimodal under- standing ability SEED-Bench (36). Randomly using other images negatives partially preserve general capabilities while further boosting POPE performance, but introduces significant performance drops HallusionBench B3), which contains adversarial visual contradictions. Our learnable negative projection approach achieves the best balance it substantially improves performance on hallucination benchmarks (MM- Vet [34], HallusionBench, and, POPE) while maintaining SEED- Bench performance. This demonstrates that explicitly learning hallucination patterns outperforms heuristic-based negative sampling.
Negative Projection Initialization. Table 4 compares initialization strategies for the negative image projection module. Initializing with supervised fine-tuning stage weights yields significantly
8

---

Question: Resed image, which country Fry lightel lovel of owareness?
Question: Does the shirt have differens color than the kite?
LLaVA 1.5 According to the image.
LLLVVA L.;: No. the the kite both have the same color
Tina Fay has the highest level tooal owareness the Unitedd States.
VCD: image,
highest level of total awarenees the Unitted Kingdom, with a soual of 34%
the shint and the kite have the some color.
white, making
white and
DPO Austzala
DPO: No Dases the gift shin lavee any cslurful elements" You 2,0 2. 1.5 To S () é 1.7
Ours: Auore ding image Tim Fey has the highest level of total mesernnes Asstmilia
Ours: Yes, the shitt dhe gitl weating is white, while kite green, Nut, and red.
Question: How many suspension bridges are there in the image? Three
visile the option's leter Bom given chaiiess directly.
Question:
Question:
sexture
Two
image? B.
the groundf Muddy Sundy Rocky D. Smooth Answer with the option's leterr foouthe given cluices dineclly.
One Four
with optict's the choices directly.
LL
VCD:
VCD.A
DPO:8 8
DPO:A
DPO:A
Ours: C
Ours: B
Ours:
Figure 3: Comparison of visualization samples among VCD negatives solely on BPO 18
15
DPO
14 and our method (trained
better results than random initialization or using pre-trained stage weights. We attribute this to better alignment with the hallucination patterns observed in MLLMs after instruction tuning. The pre-trained stage weights, while containing general visual knowledge, lack specific signals about common hallucination errors made by supervised fine-tuned models.
Positive and Negative Learning. We con- ducted an ablation experiment to further assess the effectiveness of positive and negative sam- ples in preference datasets. As shown in Ta- ble[5] learning solely from positive samples does not result in significant performance improve- ments. In contrast, learning solely from negative samples leads to greater performance enhance- ments on hallucination benchmarks such as MM- Vet (34, HallusionBench (33] and POPE [31. Thanks to our approach of decoupling positive and negative sample learning, all of our learn- ing methods ("Positive", "Negative", and "Posi & Nega") do not experience performance degradation on the general ability benchmark SEED- Bench [36]. We conclude that in preference datasets, the most benefit is derived from negative samples. This is because the model has already encountered many positive samples during the supervised fine- mun-tuning stage, but has not been exposed to negative samples during this stage.
POPE
SEED
MM-Vet Hallusion
Acc
LLa W
58.57
23.7
39.22
84.73 83.63
Positive
58.64
24.3
39.43
85.73 84.18
Negative
58.60
29.4
40.17
86.25 85.11
Pos. & Neg.
58.61
29.5
39.54
86.20 85.08
Table 5: Ablation study of positive and negative samples learning. "Postive" means only learn from positive samples, "Negative" denotes only learn from negative samples, and "Pos. & Nega." " is trained in both positive and negative samples. This experiment is trained on BPO (18). For POPE [31, we report the results of the adversarial set here.
5.4 Qualitative Analysis
Case Study. In the 1st row of Figure VCD fails to address the hallucination issue in the table scene, whereas both DPO and our method succeed. However, on the right side, DPO provides an incorrect answer and responds oddly by self-questioning and using another language (e.g.. Japanese here) due to the likelihood displacement. In the 2nd row (samples from SEED-Bench), VCD and DPO incorrectly answered general ability questions that the baseline model (LLaVA-11.5 7B) originally answered correctly, while our method can preserve baseline model's original capability.
Hallucination Generated by Negative Images. As illustrated in the first row of Figure adding noise to an image sometimes fails to induce hallucinations in the model. Using such noisy images as negative examples in contrastive decoding may decrease the probability of arriving at the correct answer, leading to reduced performance. Our learnable negative image projection triggers likely hallucinations in the original image (e.g., in the bottom left image of Figure "motorcycle" and

---

"people"). ). This approach generates potential hallucinations based on the original image and helps
mitigate them through contrastive decoding.
Question: Explain the elements of image with thoruugh attemion detail.
Question: What is the fruit?
VCD: The festures red and black parked in grassy field. surounded by a other vehicles. tractor main fooms of suene, with its vimtgge design and classic appearance. additio the background, including a truck and car. The truck located un the right side of the tractor, while the car positioned further back the left side. field appears be gatheriing place for various vehicles machinar,, pussibly for an event exhibiiion.
Noisy Image
there several other vehicles
VCD: The fruit covered plastic wrap.
The image featurres langu, red tracter parked in field. The surounded variety car, and a motoreycle. The field is filled with gress, and there are several people scattered throughout
Ours:
Original Image
vehicles, including
Ours: The fruit covered layer of chocolate.
whom shining
standing near the vehicles. brightly, ureating a warm and inviling atmospbere. The image cuptures leisure and relaxation, the people and vehicles enjoy
sky blue,
Figure 4: Model response generated by using negative image embeddings as inputs for positive image embeddings. For "VCD", utilize noisy images as image inputs and for "Ours", we utilize negative image projection to project image inputs.
Conclusion
We introduce a novel method mitigate hallucinations in MLLMs by decoupling the learning of positive and negative outputs through positive and negative image projections. This approach dynamically models authentic hallucination patterns, effectively suppressing contradictions without compromising general reasoning capabilities. Unlike training- based methods (e.g.. DPO) which suffer from the likelihood displacement issue, training-free methods (e.g., VCD) which rely on static perturbations, DCD optimizes vision-awar aware negative image features in contrastive decoding, This enables competitive hallucination reduction while maintaining performance in open-ended tasks. Our experiments demonstrate that focusing negative (hallucinatory) samples significantly enhances the model's discriminative awareness, complementing the knowledge gained from supervised fine-tuning. This work advances the deployment of trustworthy MLLMs in high-stakes scenarios by striking balance between accuracy and creativity.
Acknowledgment
This work was supported by the National Natural Science Foundation of China Young Scholar Fund (62402408) and the Hong Kong SAR RGC Early Career Scheme (26208924).
References
[1] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision Pattern Recognition, pages 26296-26306. 2024.
[2] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengehuan Zhang, Raghuraman Krish- namoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as unified interface for vision-language multi-task learning, arXiv preprint urXiv:2310.09.
2023.
[3] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl. vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966. 2023.
[4] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI- Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The lama 3 herd of models. arXiv preprint
axXiv:2407.21783 2024.
[5] Zhe Chen, Weiyun Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open- source multimodal models with model, data, and test-time scaling, arXiv preprint arXiv:2412.05277 2024.
10

---

[6] Chunyuan Li, Cliff Wong. Sheng Zhang, Naoto Usuyama. Haotian Liu, Jianwei Yang, Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large aanguag-andd-viionn assistant for biomedicine in one day. Advances in Neural Information Prucessing Systems, 36, 2024.
[7] Fenglin Liu, Tingting Zhu, Xian Wu, Bang Yang, Chenyu You, Chenyang Wang, Lei Lu, Zhangdaihong Liu, Yefeng Zheng. Xu Sun, et al. A medical multimodal large language model for future pandemics. NPJ Digital Medicine, 6( ())222 2023.
[8] Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Wiki-llava: Hicrarchical retrieval -aggeetee generation for multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1818-1826, 2024.
[9] Zechen Bai, Pichao Wang. Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404. 18930, 2024.
[10] Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang. Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint ar arXiv: 2402.0020,, 2024.
[11] Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang. Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages I 12944-12953, 2024.
[12] Yiyang Zhou. Chenhang Cui, Jachong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. arXiv 2023.
preprint arXiv:2310.007
[13] Can Cui. Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang. Kuci-Da Liao, et al. A survey on multimodal large language models for autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 958-979, 2024.
[14] Rafael Rafailov, Archit Sharma, Fric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems. 36, 2024.
[15] Sicong Leng. Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision- languuge models through visual contrastive decoding. In Proceedings of the /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13872- 13882, 2024.
[16] Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tap Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior aligmment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13807-13816, 2024.
[17] Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Sat- Chua, et al. Rlaif-v: Aligning milmss through open-soure ai feedback for super gy trustworthiness. arXiv preprint arXiv:2405. 17220, 2024.
[18] Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang, Runtao Liu, Rui Pan, and Tong Zhang. Strengthening multimodal large language model with bootstrapped preference optimization. In European Conference on Computer Vision, pages 382-398. Springer, 2024.
[19] Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang. Liang Chen, Yazheng Yang. Benyou Wang. Lingpeng Kong, and Qi Liu. Vlfeedback: A large-scale ai feedback dataset for large vision-language models alignment. arXiv preprint arXiv:2410.004211 2024.
[20] Yiyang Zhou Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402. 11411, 2024.
[21] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almcida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
[22] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint NDv:2309.14525, 2023.
[23] Noam Razin, Sadhika Malladi, Adithya Bhaskar, Dangi Chen, Sanjeev Arora, and Boris Hanin. Un- intentional unalignment: Likelihood displacement in direct preference optimization. arXiv preprint arXiv:2410.088-7 2024.
[24] Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li. and Jiawei Zhou. Hale: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.0m 00425, 2024.
11

---

[25] Shi Liu, Kecheng Zheng, and Wei Chen. Paying more attention to image: A training-free method for alleviating hallucination Ivlms. In European Conference on Computer Vision, pages 125-140. Springer. 2024.
[26] Yeji Park, Deokyeong Lee, Junsuk Choe, and Buru Chang. Convis: Contrastive decoding with halluci- nation visualization for mitigating hallucinations in multimodal large language models. arXiv preprint arXiv:2408. 13906, 2024.
[27] Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang, Zhicheng Chen, and Peilin Zhao. Self- introspective decoding: Alleviating hallucinations for large vision-language models. arXiv preprint Sati:2408.020322 2024.
[28] Junzhe Chen, Tianshu Zhang. Shiyu Huang, Yuwei Niu, Linfeng Zhang, Lijie Wen, and Xuming Hu. Ict: Image-object cross-level trusted intervention for mitigating object hallucination in large vasion-language models. arXiv preprint arXiv:241 1.5266,, 2024.
[29] Yi-Lum Lee, Yi-Hsuan Tsai, and Wei-Chen Chiu. Delve into visual contrastive decoding for hallucination mitigation of large vision-language models. arXiv preprint arXiv:2412. 0677755, 2024.
[30] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Fisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation optimization. arXiv preprint arXiv:2210. 15097 2022.
[31] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023.
[32] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li. Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang. Yiming Yang. et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309. 14525, 2023.
[33] Tianrui Guan, Fuxiao Liu. Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, ct al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14375-1438) 2024.
[34] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.
2023.
[35] Pan La,, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang. Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255 2023.
[36] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal Ilms with generative comprehension. arXiv preprint arXiv:3307.16125. 2023.
[37] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Partern Recognition, pages 9556-9567, 2024.
[38] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang. Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-languaa models? arXiv preprint 2:X3::2403200330 2024.
[39] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-344 -4916, 2023.
[40] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-iman pre- pe-training for unified languag understanding and generation. In International conference on machine learning. pages 12888-12900. PMLR, 2022.
[41] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision- models with instruction tuning. 2023.
[42] Lin Li, Jun Xiao, Guikun Chen, Jian Shao, Yueting Zhuang, and Long Chen. Zero-shot visual relation detection via composite visual cues from large language models. Advances in Neural Information Processing Systems, 36:50105 16, 2023.
[43] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- trainingng In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975- 1 1986. 2023.
12

---

[44] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PmLR. 2021.
[45] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Erie Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 19222.19975 13971, 2023.
[46] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint vrXiv:2307.09288. 2023.
[47] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716 23736, 2022.
[48] Jiaming Lei, Lin Li, Chunping Wang. Jun Xiao, and Long Chen. Seeing beyond classes: Zero-shot grounded situation recognition via language explainer. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 1602-1611, 2024.
[49] Alce Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. Improving language understanding by generative pre-training. 2018.
[50] Nisan Stiennon, Long Ouyang. Jeftrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedhack. Advances in Neural Information Procesing Systems, 33:3008-3021, 2020.
[51] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint MXXi::220400866 2022.
[52] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 06347, 2017.
[53] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang. Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang. Wei Ding, Zheren Fu, Yiheng Xu, Jiaho Ye, Xi Zhang, Tianbao Xie, 7esen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025.
[54] Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. Cogvlm2: Visual language models for image and video understanding. arXiv preprint arXiv:2408. 16500, 2024.
[55] Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better understanding rision-language models: insights and future directions. Workshop on Responsibly Building the Next Generation of Multimodal Foundational Models, 2024.
[56] Haotian Liu. Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning. ocr. and world knowledge, January 2024.
[57] Lin Li, Guikun Chen, Hanrong Shi, Jun Xiao, and Long Chen. A survey on multimodal benchmarks: In the era of large ai models. arXiv preprint arXiv:2409. 18.9./8// 18142, 2024.
[58] Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Shaoqi Dong, Xiong Wang, Di Yin, Long Ma, et al. Vita: Towards open-source interactive omni multimodal 1mm. arXiv preprint arXiv:2408.08 05211,05111110121 2024.
[59] Yadong Li, Jun Liu, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Guosheng Dong, Da Pan, et al. Baichuan-omni-l.5 technical report. arXiv preprint arXiv:2501. 15368, 2025.
[60] Hanrong Ye, De-An Huang, Yao Lu, Zhiding Yu, Wei Ping, Andrew Tao, Jan Kautz, Song Han, Dan Xu, Pavlo Molchanov, et al. X-vila: Cross-modality alignment for large language model. arXiv preprint ar0iv:2405. 19335, 2024.
[61] Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang. Xin Yan, Yilun Du Yining Hong, and Chuang Gan. 3d-vla: A 3d rision-language-actionn generative world model. arXiv preprint arXiv:2403.n0 00963, 2024.
[62] Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseck-v12: dixture-of-expers vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10388 2024.
[63] Bin Lin, Zhenyu Tang. Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang. Munan Ning et al. Moe-llava: Mixture of experts for large vision language models. arXiv preprint arX1v:2401.15947, 2024.
13

---

[64] Dongxu Li. Yudong Liu, Haoning Wu, Yue Wang. Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang. Bei Chen, and Junnan Li. Aria: An open multimodal native mixture-of-experts model. arXiv preprint
urXiv:2410.05993, 2024.
[65] Haiwen Diao, Xiaotong Li. Yufeng Cui, Yueze Wang, Haoge Deng, Ting Pan, Wenxuan Wang, Huchuan Lu, and Xinlong Wang. Evev2: Improved baselines for encoder-free vision-language models, 2025.
[66]
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşırlar. Introducing our multimodal models, 2023.
[67] Yangyi Chen, Xingyao Wang, Hao Peng, and Heng A single transformer for scalable vision- laiguale modeling. arXiv preprint Wxiv:2407.06433 2024.
[68] Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint
arXiv:2405.098/6 2024.
[69] Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang. Zhen Li, Qiying Yu, al. Emu3: Next-token prediction is all you need. arXiv preprint wrXiv:2409.1880 2024.
[70] Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one modti-modal model. arXiv preprint arXiv:2408. 11039, 2024.
[71] Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pre pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint 2501. 17811, 2025.
[72] Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. lbd: Alleviating hallucinations in large vision-languaa language models via image- biased decoding, arXiv preprint arXiv:2402. 2024.
[73] Alessandro Favero, Luca Zancato. Matthew Trager. Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Multi-modal hallucination control by visual information grounding. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 14303-14312, 2024.
[74] Xintong Wang, Jingheng Pan, Liang Ding, and Chris Biemann. Mitigating hallucinations large vision- language models with instruction contrastive decoding. arXiv preprint 1rXiv:2403.18715 2024.
[75] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Advances Neural Information Processing Systems. 1121224911122235 2024.
[76] Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu. Hao Wang. Dianbo Sui. Yunhang Shen, Ke Li. Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. Science China Information Sciences, 67(12)2220055 2024.
14

---

Appendix
A
Social Impacts
Our work addresses the critical challenge of hallucination in multimodal large language models (MLLMs), with profound implications for the safe deployment of AI systems across socially sensitive domains. By mitigating factual contradictions and visual misrepresentations, our method enhances model reliability high-stakes applications such as medical diagnostics and autonomous driving. Beyond safety-critical scenarios, our approach fosters trust in Al-assisted decision-making tools for education, legal documentation, and content moderation by ensuring outputs align with observable evidence. This reliability is particularly crucial for combating misinformation in era of Al- generated content proliferation. Additionally, our findings on the importance of negative sample learning offer insights for developing more efficient alignment frameworks, potentially democratizing access to robust MLLMs for resource-constrained institutions.
B
Limitations
While our method achieves robust hallucination mitigation, two key limitations warrant consideration. First, the contrastive decoding framework inherently doubles computational overhead during inference due to parallel processing of original and eggative-projectee image features. Second, negative image projection relies on the quality and diversity of hallucination patterns in preference datasets. While current datasets predominantly cover common object hallucinations (e.g., spurious object mentions), they may underrepresent complex multimodal hallucinations involving spatial reasoning causal relationships. Future work could explore adaptive weighting mechanisms to handle such edge cases.
C Additional Experiments
Results on Qwen 2.5 VI. 3B. As shown in Table AT and Table A2, on lightweight yet strong backbone, DCD preserves or slightly improves general capability while consistently suppressing hallucinations. Averaged over settings, our method surpasses DPO in three of four preference regimes, with "Pos.&Neg." typically the most stable variant. Overall, these appendix results support DCD's supervision- and model-agnostic transferability: learned negative esspeiia with joint positive-negative
authentic hallucination patterns and extend cleanly to compact backbones, pointing to straightforward scaling on Qwen variants and broader preference signals.
D Theoretical Foundation
1. Key Proposition: A Sufficient Condition for Eliminating Likelihood Displacement
From Puirwise to Decoupled Optimization. DPO maximizes a paired log gap:
(f) - (e, w- [log a (8 [log mo(y+ + ar. log To ax, [r,,i]]]=
which guarantees the gap widens but is agnostic to the absolute values of log no (y+ x. v) and log Toly" Ir, v). As result, both likelihoods can drift downward likelihood displacement effect that degrades general reasoning (see Figure [ a) DPO in the paper).
Our Decoupled Objective. Instead, we minimize two independent cross-entropies:
min E (( l- log Te (y* x. (v))),
min E E
11 log No (y x. (v)))
and combine the results only at inference time:
logit. = (1 + a:) logit - a logit
As shown in Figure 1(a) Ours, this ensures that log m | xr, v) increases while log (y ar, v) suppressed. This construction provides a lower-bound guarantee on general reasoning performance, consistent with our empirical results (Table I] SEED-Bench).
15

---

2. Necessity and Sufficiency of the Negative Projector
Theoretical View. We interpret the negative image projection J4 learning adversarial negative distribution 4o (0) the vision-feature space V, which maximizes
KI. (p(u v) p(u 2, (o(()))
equivalent to maximizing the InfoNCE lower bound. Thus, learning only the negative projector implicitly provides gradient-shaped penalty during inference, which explains why our Neg-only training consistently reduces hallucination.
Empirical Evidence. As shown in Table I the Neg-only variant nearly matches Pos Neg hallucination benchmarks, and clearly outprrorrs Pos-only. To our knowledge, ours is the first work to show that negative-only learning can suffice.
General Performance
Hallucination
MMHal
Average"
SEED
MathVistal
MMStar
MMMU
MM-Vet
Hallusion
Scure
Rate J
Qwen 2.5 VL
66.67
61.0
53.27
44.4
32.5
2.01
0.69
50.68
54.75
VCD 15
65.15
64.1
52.53
44.9
52.5
1.96
0.71
49.21
54.73
Fine-tured REHF-Y
[16]
DPO [14
66.61
60.9
53.53
44.8
51.3
1.83
0.73
50.81
54,66
SimPO 75
66.89
61.2
53.07
44,9
50.1
2.07
0.68
49.63
54.30
Ours (Neg. Only)
65.00
62.4
53.53.0.00
44.3-
53.2
1.00
1.98 0.09
0.70
51.94.1.13
55.06.0040
Ours (Pos. Neg.)
66.30
0.30
62.7-1.00
54.80
44.4..0.00 0.50
54.5
3.20
20.36
0.57-u.11
52.37
55.85....0
Fine-tured BPO
18
DPO [4
66.96
63.8
53,13
45.3
52.4
1.84
0.69
52,66
55.71
SimPO 66
66.92
63.2
53.67
45.1
51.5
2.35
0.55
53.68
55.68
Ours (Neg. Only)
67.1
+0.15
60.2
3.60
52.93
0.77
46.6
46.6,1.30
1.00
53.8 1.40
1.97
0.38
0.68
10.13
53.31
0.37
55.66 0.05
Ours (Pos. Neg.)
67.12
o.116
63.2....6
53.07
0.500
|.60
53.2....80 0.80
2.55 40.20
0.52
0.08
53.26
0.42
16.43
Fine-tuned RLAIF-V
DPO [14
66.84
60.8
53.13
44.6
52.9
1.58
0.76
50.16
54.74
SimPO 75
66.32
60.2
52.67
45.1
45.6
2.55
0.59
46.37
52.71
Ours (Neg. Only)
64.70
64.70-2.18
2.14
=0.1
52.87
46.7
1..00
53.0 0.10
1,99
0.41
0.71
0.71-.0.0
52.47
+2:31
45.14.0.440
Ours (Pos. & Neg.)
65.61
1.20
60,9
-.10
50.27
2299
45.8
1..70
53.5
0.60
2.30
+0.72
0.61
0.15
52.68
22.02
54.79 +0.06
Fine-tuned V_Feedback [19]
DPO [4]
66.74
60.5
52.33
44.6
53.1
2.14
0.66
48.27
54,26
SimPO 6S
66.96
62.3
52.87
44.6
51.9
2.33
0.68
48.05
$4.45
Ours (Neg. Only)
65.80 1.
61.2-
,3.13
40.00
53.0
2.21
-0.12
0.66
0.00
$1.79
a..62
55.00
...57
Ours (Pos, & Neg.)
66.25 0.71 61.0-1.00 53.60...78
4..6
54.7
3.15
40.80
0.46
0.30 $1,69......
55.47....0
Table Al: Performance comparison on general and hallucination benchmarks for Qwen 2.5 VL 3B. 'Neg. Only' means only trained negative samples of preference datasets, 'Pos. & Neg.' is trained with both positive and negative samples, 1. indicates lower is better, and denotes that the MMHal values not counted in the Average score. For benchmarks requiring GPT evaluation, follow the same setting as the main table (e.g., GPT-4o 24-05-13).
16

---

Random
Popular
Adversurial
Acc
F1
Acc
F1
Acc
F1
Woodpecker 76
85.51
86.67
83.51
84.33
82.35
83.00
Qwen 2.5 VL. 3B
88.90
87.67
87.87
86.68
86.67
85.55
161
88.93 87.79
86.27 87.33
85.80
Fine-tuned on RLHF-V [16]
DPO 14
88.83 87.58
87.83
86,61
86,60
85,46
SimPO 75
88.07 86.59
87.27
85.82
86.23
84.84
Ours (Neg. Only)
89.57
88.54
88.30
87.32
85.92
Ours (Pos. & Neg.)
89.30
88.18
87.93
86.87
86.87
85.87
Fine tuned on BPO
[8]
DPO 14
89.40
88.29
88.20
87.13
86.87
85.89
88.17
87.23
Ours (Neg. Only)
91.37
90.77
89.07
88.58
86.76
86.51
Ours (Pos. & Neg.)
90.90 90.20
89.33
88.70
86.63
86.23
Fine-tuned RLAIF-V
[17
DPO
89.23 88.08
80
86.99
86.67
85.64
SimPO
89.57 88.46
88.40
87.33
86.90
85.93
Ours (Neg. Only)
91.37
90.80
88.40
88.02
86.07
85.95
Ours (Pos. & Neg.)
90.30 89.56
88.53
87.89
85.80
85.41
Fine-tured VLFeedboock
DPO 14]
87.77 86.21
87.07
85.53
86,00
84.52
SimPO 75
87.03 85.20
86.40
84.59
85,50
83.74
Ours (Neg. Only)
90.03 89.08
88.47
87.58
87.07
86.28
Ours (Pos. & Neg.)
89.27 88.17 88.00
86.96
85,90
85.02
Table A2: Performance comparison on POPE B1
with Qwen 2.5 VL 3B. "Neg. Only" uses only negative samples from preference datasets; "Pos. Neg." uses both positive and negative samples.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction introduce our proposed method and its effective- ness.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in the appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
17

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specing well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tes system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide the implementation details in the experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
18

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
The code will released after the review.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7/
gwwss CCodeSoodieeeeeee ssi mollicyy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
19

---

Answer: [Yes]
Justification:
We provide the experimental setting in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due limited computational resources, we run each experiment once and report the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We run our experiments in NVIDIA A100 80GB, which is included in our implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
20

---

9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://p//aarr....ccco/aaaaaa/aa
Answer: [Yes]
Justification: We reviewed the NeurlPS Code of Ethics and confirm that our research complies with all listed principles.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the social impacts in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
21

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: used open-source datasets and cited them.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pappessii thcode wwwccoo//aaooooo datasat has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
22

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development does not involve LLMs.
Guidelines:
The answer NA means that the method development in this research does involve L.MMs as any important, original, or non-standard components.
Please refer to our LLM policy battps://neurie wtcpppss..../oooooooooe //o//oerrncee//225//111 for what should or should not be described.
23


Reviews:
[{"review_id": "6Ug3n1XhTd", "reviewer": "Reviewer_LwQb", "summary": "This paper presents Decoupling Contrastive Decoding (DCD), a new method for mitigating hallucination in MLLMs with preference optimization. The key idea is to decouple the learning of positive and negative samples from preference datasets by training separate image projection modules for each, thereby directly modeling hallucination patterns and avoiding sacrificing the general reasoning abilities of the model. DCD is evaluated against both training-based and training-free baselines on multiple hallucination and general reasoning benchmarks, showing that it suppresses hallucination comparably to strong baselines like DPO, but better maintains general reasoning ability and overall performance. Extensive ablations and analyses provide further insight into the contributions of positive and negative learning.", "questions": "Please see my comments in the Strengths and Weakness section.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper focuses on a critical problem of reducing the hallucination of MLLMs.\n\n2. The idea of decoupling the learning of positive and negative samples to reduce hallucination while maintaining the general capabilities of MLLMs is interesting and reasonable.\n\n3. Extensive experiments with comparisons among strong baselines are conducted to demonstrate the effectiveness of DCD.\n\n**Weakness**\n\n1. The findings in Section 5.3 that most of the gain could be attributed to negative samples are not very convincing, since the positive samples are seen during supervised fine-tuning, thus the model may already have learned the knowledge of the positives.", "was_revised": true, "final_justification": "My concern has been well-addressed. I will keep my positive score.", "rebuttal": "Thank you for your valuable reviews. We will answer your questions below.\n\n**Model Have learned the knowledge of the positives?** :\n\nIn Section 5.3, we fine‑tune LLaVA‑1.5 (7B) using the BPO preference dataset to evaluate the contributions of positive and negative samples as shown in Table 5. Following your question, we found that approximately 32.4% of BPO’s data examples originate from the LLaVA‑Instruct dataset, i.e., the data overlaps with LLaVA‑1.5’s supervised fine‑tuning (SFT) stage. To address the reviewer’s concern that “the model may already have learned from these positives during SFT”, we performed the following ablation:\n\n- We **removed all BPO samples derived from LLaVA‑Instruct** and replicated the Table 5 experiments using the **de‑overlapped subset**.\n- The results and conclusions remain consistent with the original findings: the **negative-sample variant still outperforms both the base model and positive-only training**, with similar margin improvements.\n\n| **Model** | **MM‑Vet** | **Hallusion** | **POPE Acc** | **POPE F1** |\n| --------- | ---------- | ------------- | ------------ | ----------- |\n| LLaVA‑1.5 | 58.57      | 23.70         | 84.73        | 83.63       |\n| Positive  | **58.73**  | 25.50         | 84.90        | 84.30       |\n| Negative  | 58.68      | **28.90**     | **86.41**    | **85.32**   |\n\nThese results confirm that even after excluding all overlapping data examples, the performance gain remains dominated by negative samples."}, {"review_id": "tIxgCBLsRY", "reviewer": "Reviewer_hgw9", "summary": "The paper proposes a novel approach to reduce hallucinations in multimodal large language models (MLLMs). The author discussed the pros and cons of the training-based and training-free hallucination mitigation methods and lead to their proposed method. They first decouple the modeling of positive and negative training examples to alleviate likelihood displacement problem. To better model the image feature, especially the negative images, the author proposed a simple feature extractor to focus on negative features, instead of adding noise or use image agnostic methods to create negative training examples. The author conduct extensive experimentations and ablations on many benchmark datasets and demonstrate superior and matching performance on most of the metrics.", "questions": "See weakness and please address those.", "limitations": "What is the limitation of the work? How is inference time / run-time compared to training-free method? How well does it work on larger models? The limitations needs to be discussed.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. The paper is well written and well motivated, especially in figure 1.\n2. The author conduct extensive experimentation. The author showed 7 benchmarks with better or matching performance.\n3. The author promise to release the code.\n4. Useful insight and nice case analysis. The performance \n\nWeakness:\n1. Some part of the paper can have better clarity, especially in the caption: for example, figure 1 is confusing with only caption without reading the main text. Figure 2 need more explanation in caption.\n2. Limited architecture choice -- only on LLaVA-1.5-7B.\n3. Did not discuss limitation of the work.\n4. Main contributions section can be improved and highlight the contribution of the paper. line 95 - 101", "was_revised": true, "final_justification": "My comments are properly addressed. The authors show significant efforts on the newly added experimentations. As a results, I increase my rating.", "rebuttal": "Thank you for your valuable reviews. We will answer your questions below.\n\n1. **Better Clarity:**   Thank you for your suggestions. I will update the captions of Figure 1 and Figure 2 in the revision as below.\n\n   - **Figure 1: Comparison between existing hallucination mitigation methods and DCD.** (a) **Training-based method (e.g., DPO [12]):** DPO directly optimizes the likelihood gap between positive (correct) and negative (hallucinatory) responses using preference datasets. However, maximizing this gap (y^+ vs. y^-) can inadvertently lower the probability of both responses, causing likelihood displacement and potential degradation of general reasoning capabilities. Here, v, x, y^+, and y^- denote images, questions, positive responses, and negative responses, respectively; \\theta represents model parameters, and \\alpha is the contrastive decoding coefficient. (b) **Training-free method (e.g., VCD [13]) vs. DCD:** Traditional contrastive decoding (VCD) reduces hallucinations by comparing model outputs from original (v^+) and artificially distorted (v^-, e.g., noise-added) visual inputs at inference time. However, these synthetic perturbations may not align with real hallucination patterns. In contrast, our proposed DCD introduces a trainable negative image projection module, learned explicitly from negative (hallucinatory) samples, to produce meaningful negative visual inputs reflecting authentic hallucination distributions.\n   - **Figure 2: Comparison of DCD method with DPO [12] and VCD [13] in the training and inference stages. Training Stage:** DPO jointly optimizes positive-negative responses, risking likelihood displacement. Our method (DCD) separately learns positive and negative image projections to avoid this issue. **Inference Stage:** VCD uses artificial noise as negative inputs, while our DCD leverages learned negative visual features reflecting authentic hallucination patterns, enhancing effective hallucination suppression.\n\n2. **Results on More Model Architectures**: We further adopt our methods to **Qwen 2.5 VL 3B** and **Qwen 2.5 VL 7B**. Due to the limited time and GPU resources, we only trained with negative samples on **RLHF-V** and **BPO** datasets here and reported on a few benchmarks in the table below. **We will add full experiment results in the revision**. As shown in the table below, our method can mitigate hallucination and maintain general performance in Qwen 2.5 VL 3B & 7B.\n\n   | Model                      | MathVista | MM-Vet   | Hallusion | Random F1 | Random Acc | Popu F1   | Popu Acc  | Adver F1  | Adver Acc |\n   | -------------------------- | --------- | -------- | --------- | --------- | ---------- | --------- | --------- | --------- | --------- |\n   | Qwen 2.5 VL 3B             | 61.00     | 52.5     | 50.68     | 87.67     | 88.90      | 86.68     | 87.87     | 85.55     | 86.67     |\n   | + VCD                      | 64.10     | 52.5     | 49.21     | 87.79     | 88.93      | 86.27     | 87.33     | 84.85     | 85.80     |\n   | + DPO (RLHF-V)             | 60.90     | 51.3     | 50.81     | 87.58     | 88.83      | 86.61     | 87.83     | 85.46     | 86.60     |\n   | + DPO (BPO)                | **63.80** | 52.4     | 52.66     | 88.29     | 89.40      | 87.13     | 88.20     | 85.89     | **86.87** |\n   | **+ Our Negative (RLHFV)** | 62.40     | 53.2     | 51.94     | 88.54     | 89.57      | 87.32     | 88.30     | 85.92     | 86.80     |\n   | **+ Our Negative (BPO)**   | 60.20     | **53.8** | **53.31** | **90.77** | **91.37**  | **88.58** | **89.07** | **86.51** | 86.76     |\n   |                            |           |          |           |           |            |           |           |           |           |\n   | Qwen 2.5 VL 7B             | 68.20     | 60.2     | 55.12     | 86.45     | 87.97      | 85.59     | 87.07     | 84.90     | 86.33     |\n   | + VCD                      | 68.95     | 59.8     | 55.94     | 87.61     | 88.77      | 86.80     | 87.93     | 85.43     | 86.47     |\n   | + DPO (RLHF-V)             | 68.47     | 60.9     | 53.64     | 86.27     | 87.83      | 85.37     | 86.90     | 84.79     | 86.27     |\n   | + DPO (BPO)                | 66.70     | 53.3     | 55.42     | 86.60     | 88.07      | 85.58     | 87.00     | 84.62     | 85.97     |\n   | **+ Our Negative (RLHFV)** | **70.20** | 58.3     | **57.41** | 86.71     | 88.17      | 85.68     | 87.10     | 84.87     | 86.23     |\n   | **+ Our Negative (BPO)**   | 69.80     | **61.4** | 56.21     | **88.90** | **89.87**  | **87.71** | **88.63** | **86.59** | **87.43** |\n\n3. **Limitation**: As we discussed in Appendix Sec. B (the limitation section), the contrastive decoding framework requires twice the computation at inference time because it processes both the original and negative image features simultaneously.\n\n4. **Improve the main contribution section:**\n\n   We will revise our contribution as follows:\n\n   - **Decoupled Learning for Robust Alignment**: We propose *Decoupled Contrastive Decoding (DCD)*, the first framework to separate positive/negative sample optimization from preference datasets in MLLM training. It alleviates the *likelihood displacement* problem in DPO, preserving general capabilities while mitigating hallucinations.\n   - **Vision-Aware Hallucination Suppression:** We introduce a learnable negative image projector trained on *real* hallucinatory samples. Unlike handcrafted perturbations (e.g., VCD), this projector generates distortion grounded in actual MLLM errors, enabling precise suppression of hallucinations.\n   - **Paradigm Shift in Preference Learning:** We reveal that negative samples alone suffice for hallucination mitigation. This challenges the preference learning paradigm — showing that explicit modeling of errors (not just positive alignment) is critical for robustness.\n   - **Comprehensive ablations and results** have demonstrated that our method can achieve competitive performance with training-based methods (e.g., DPO) in hallucination benchmarks while maintaining the general ability."}, {"review_id": "MilVrgzdNz", "reviewer": "Reviewer_SKUk", "summary": "This paper introduces Decoupling Contrastive Decoding (DCD), a novel framework designed to mitigate \"hallucination\" in MLLMs. The core contribution of DCD is its ability to decouple the learning of positive and negative samples within preference datasets, training separate positive and negative image projections in the MLLM. Moreover, this paper provides the negative projection to enable vision-aware negative images during the contrastive decoding inference stage. Consequently, DCD alleviates the likelihood displacement problem in training-based solutions like DPO and robustly generalizes without relying on handcrafted image perturbations in training-free methods like VCD. Extensive experiments demonstrate DCD's effectiveness in suppressing hallucinations while preserving general reasoning capabilities, often outperforming existing methods.", "questions": "### Major question\n1. Clear motivation for the necessity of preference-based fine-tuning and contrastive decoding for MLLM hallucination mitigation (See weakness performance drop)\n2. How well can this method adapt and perform with larger and more recent MLLM backbones (e.g., Qwen-VL, LLaVA-Next), or is its applicability primarily limited to LLaVA 1.5?\n### Minor questions\n3. How well does the proposed method mitigate relational hallucinations (e.g., asserting “a person riding a bicycle” when only a bicycle is present) and attribute hallucinations (e.g., asserting “many red apples” when a green apple under a red apple tree is present)?\n4. in line 189, the $\\theta$ is the parameter of the MLLM while the $\\pi_{\\theta}$ is the policy.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- DCD introduces a decoupling perspective to optimize DPO preference learning for hallucination mitigation, which avoids the likelihood displacement in traditional DPO\n- The method also utilized learnable negative image projections in the contrastive decoding inference stage to further mitigates hallucinations while preserving general reasoning capabilities.\n\n### Weaknesses\n- A central concern of this paper is the lack of a clear motivation for the necessity of preference-based fine-tuning and contrastive decoding for MLLM hallucination mitigation, particularly when Table 1 indicates performance degradation on certain hallucination and general benchmarks after such fine-tuning when compared to the original base models. For completeness, it is necessary to compare with more methods that eliminate hallucinations through data optimization[1] or model optimization[2,3]\n\n    - [1] HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data, CVPR 2024\n    - [2] Woodpecker: Hallucination correction for multimodal large language models\n    - [3] Analyzing and Mitigating Object Hallucination in Large Vision-Language Models, ICLR 2024\n\n- The baseline setting doesn't include any open-source VLLM, such as Qwen-vl, which should be included in the experiment to see the fine-tuning performance for hallucination mitigation. \n\n- The dual-projection training process of DCD may incur higher computational overhead compared to general DPO, due to the need for separate positive and negative projections.\n\n- The main experimental results in Table 1 are reported for only one model size and architecture. To improve completeness, the authors should extend the results to include multiple scales (LLaVA-7b, 13b) and more general baselines (Qwen-2.5-VL)", "was_revised": true, "final_justification": "Although the appendix figure A1 provides the generalization of VCD, I hope to move this part to the main text. Overall, I appreciate the rebuttal for clarification. I will raise the score.", "rebuttal": "Thank you for your valuable reviews. We will answer your questions below.\n\n1. **Clear motivation for preference learning and contrastive decoding.**\n\n   - **Trade-off on Existing Methods**: Existing approaches from both families (preference fine-tuning and training-free contrastive decoding) often exhibit a capability–robustness trade-off when one needs to reduce hallucination rates to meet reliability constraints. Our contribution is precisely to **avoid this trade-off**. The degradations in Table 1 are **motivational baselines**: (i) DPO can cause **likelihood displacement.** (ii) Prior VCD relies on **hand-crafted, vision-only perturbations** misaligned with real vision-text hallucinations.\n   - DCD avoids both by (a) **decoupling** positive/negative learning and training **small image-projection heads while freezing the base**, and (b) using a **vision-aware negative projector learned from real negative samples** instead of synthetic distortions.\n\n2. **Compared with more methods**: Due to benchmark inconsistency ([1],[3]) and time constraints, we're unable to modify their code and run full comparisons, but we will discuss Methods[1,3] in our related works section. We have incorporated Woodpecker [2], which reports results on the POPE benchmark.  We also further adopt our methods to **Qwen 2.5 VL 3B** and **Qwen 2.5 VL 7B**. Due to the limited time and GPU resources, we only trained with negative samples on **RLHF-V** and **BPO** datasets here, and reported on a few benchmarks in the table below. **We will add full experiment results in the revision**. As shown in the table below, our method can mitigate hallucination and maintain general performance in Qwen 2.5 VL 3B & 7B.\n\n   | Model                      | MathVista | MM-Vet   | Hallusion | Random F1 | Random Acc | Popu F1   | Popu Acc  | Adver F1  | Adver Acc |\n   | -------------------------- | --------- | -------- | --------- | --------- | ---------- | --------- | --------- | --------- | --------- |\n   | Woodpecker Otter [2]       | -         | -        | -         | 85.51     | 86.67      | 83.51     | 84.33     | 82.35     | 83.00     |\n   |                            |           |          |           |           |            |           |           |           |           |\n   | Qwen 2.5 VL 3B             | 61.00     | 52.5     | 50.68     | 87.67     | 88.90      | 86.68     | 87.87     | 85.55     | 86.67     |\n   | + VCD                      | 64.10     | 52.5     | 49.21     | 87.79     | 88.93      | 86.27     | 87.33     | 84.85     | 85.80     |\n   | + DPO (RLHF-V)             | 60.90     | 51.3     | 50.81     | 87.58     | 88.83      | 86.61     | 87.83     | 85.46     | 86.60     |\n   | + DPO (BPO)                | **63.80** | 52.4     | 52.66     | 88.29     | 89.40      | 87.13     | 88.20     | 85.89     | **86.87** |\n   | **+ Our Negative (RLHFV)** | 62.40     | 53.2     | 51.94     | 88.54     | 89.57      | 87.32     | 88.30     | 85.92     | 86.80     |\n   | **+ Our Negative (BPO)**   | 60.20     | **53.8** | **53.31** | **90.77** | **91.37**  | **88.58** | **89.07** | **86.51** | 86.76     |\n   |                            |           |          |           |           |            |           |           |           |           |\n   | Qwen 2.5 VL 7B             | 68.20     | 60.2     | 55.12     | 86.45     | 87.97      | 85.59     | 87.07     | 84.90     | 86.33     |\n   | + VCD                      | 68.95     | 59.8     | 55.94     | 87.61     | 88.77      | 86.80     | 87.93     | 85.43     | 86.47     |\n   | + DPO (RLHF-V)             | 68.47     | 60.9     | 53.64     | 86.27     | 87.83      | 85.37     | 86.90     | 84.79     | 86.27     |\n   | + DPO (BPO)                | 66.70     | 53.3     | 55.42     | 86.60     | 88.07      | 85.58     | 87.00     | 84.62     | 85.97     |\n   | **+ Our Negative (RLHFV)** | **70.20** | 58.3     | **57.41** | 86.71     | 88.17      | 85.68     | 87.10     | 84.87     | 86.23     |\n   | **+ Our Negative (BPO)**   | 69.80     | **61.4** | 56.21     | **88.90** | **89.87**  | **87.71** | **88.63** | **86.59** | **87.43** |\n\n3. **High training cost？**\n\n   Our training cost is actually **lower than** DPO’s. Specifically, we only optimize the positive and negative image projections, whereas DPO must train full LoRA parameters. Our training results on BPO datasets show that DPO requires 8 hours 24 minutes on 8 A100 GPU servers, while our method completes each projection training in 2 hours—yielding a total of 4 hours for both projections.\n\n4. **Mitigation of relational and attribution hallucination.**  Our method provides a general approach to mitigating hallucinations by learning from negative samples in preference datasets. By implicitly capturing and suppressing the specific patterns of hallucinations present in the training data, our method can alleviate both relational and attribution hallucinations; in other words,  if such patterns are represented in the negative samples, our method should effectively mitigate both relational and attribution hallucinations.\n\n5. **Typo in line 189:** Thank you for pointing it out. We will correct it in our revision.\n\n[1] HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data, CVPR 2024 \n\n[2] Woodpecker: Hallucination correction for multimodal large language models \n\n[3] Analyzing and Mitigating Object Hallucination in Large Vision-Language Models, ICLR 2024"}, {"review_id": "bzmrPZ1m7o", "reviewer": "Reviewer_YiWX", "summary": "This paper presents Decoupling Contrastive Decoding (DCD), a new framework aimed at mitigating hallucinations in multimodal large language models (MLLMs). The proposed method decouples the learning of positive and negative samples in preference datasets and trains separate positive and negative image projections within the MLLM. The authors argue that this approach alleviates the likelihood displacement issue inherent in training-based methods like Direct Preference Optimization (DPO) and avoids the handcrafted perturbations used in training-free methods such as contrastive decoding. The paper includes extensive experiments across multiple hallucination benchmarks and general reasoning tasks, showing that DCD can suppress hallucinations while preserving general capabilities.", "questions": "1. Could the authors provide a more detailed comparison of DCD with other state-of-the-art models that have been proposed more recently, especially those that claim improved hallucination mitigation?", "limitations": "The paper has several limitations that should be addressed to strengthen its contributions:\n1. As mentioned earlier, the use of LLaVA 1.5-7B limits the impact of this work. To demonstrate the broader applicability and effectiveness of DCD, the authors should validate their method on more up-to-date and advanced MLLMs. This would provide a clearer picture of how DCD performs in comparison to current standards in the field.\n2. While the method shows promise on the evaluated benchmarks, its generalizability to other domains and tasks remains untested. Future work could explore the application of DCD across a wider variety of multimodal scenarios.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper addresses a critical issue in MLLMs—hallucinations—and proposes a method that aims to balance hallucination suppression with the preservation of general reasoning capabilities.\n\n2. The authors provide a thorough experimental evaluation across multiple benchmarks, which helps to demonstrate the effectiveness of their approach compared to existing methods.\n\nWeaknesses: \n\n1. The proposed method appears to be a relatively incremental advancement rather than a novel breakthrough. It builds upon existing contrastive decoding techniques and preference optimization methods without introducing entirely new concepts.\n\n2. The use of LLaVA 1.5-7B as the base model for experiments is a significant limitation. This model is notably outdated compared to more recent and advanced models such as Qwen2.5-VL and InternVL2.5. The lack of validation on these more modern models raises questions about the generalizability and relevance of the proposed method.", "was_revised": true, "final_justification": "I appreciate the rebuttal. Accordingly, I am increasing my score from 2 to 4.", "rebuttal": "Thank you for your valuable reviews. We will answer your questions below.\n\n1. **New concept:** Training-free contrastive decoding and training-based preference learning are two **general paradigms** for aligning MLLMs. Although building on them, we offer **substantive** advances:\n\n   - **Decoupled Contrastive Decoding.** We separate positive/negative learning and decode against a **learned negative distribution.** Notably, a **negative-only** variant (training only the negative projector) already achieves substantial hallucination reduction. To our knowledge, we are the **first** to show that **negative-only preference learning** is sufficient to reduce hallucinations in MLLMs.\n   - **Vision-aware negative projector from real hallucinations.** Unlike VCD’s handcrafted, vision-agnostic perturbations, we learn the projector from **model-generated negative samples**, aligning the contrastive signal with **actual failure modes** and improving robustness.\n\n2. **Advanced MLLM method: Qwen 2.5 VL 3B & 7B:** We further adopt our methods to **Qwen 2.5 VL 3B** and **Qwen 2.5 VL 7B**. Due to the limited time and GPU resources, we only trained with negative samples on **RLHF-V** and **BPO** datasets here, and reported on a few benchmarks in the table below. **We will add full experiment results in the revision**. As shown in the table below, our method can mitigate hallucination and maintain general performance in Qwen 2.5 VL 3B & 7B.\n\n   | Model                      | MathVista | MM-Vet   | Hallusion | Random F1 | Random Acc | Popu F1   | Popu Acc  | Adver F1  | Adver Acc |\n   | -------------------------- | --------- | -------- | --------- | --------- | ---------- | --------- | --------- | --------- | --------- |\n   | Qwen 2.5 VL 3B             | 61.00     | 52.5     | 50.68     | 87.67     | 88.90      | 86.68     | 87.87     | 85.55     | 86.67     |\n   | + VCD                      | **64.10** | 52.5     | 49.21     | 87.79     | 88.93      | 86.27     | 87.33     | 84.85     | 85.80     |\n   | + DPO (RLHF-V)             | 60.90     | 51.3     | 50.81     | 87.58     | 88.83      | 86.61     | 87.83     | 85.46     | 86.60     |\n   | + DPO (BPO)                | 63.80     | 52.4     | 52.66     | 88.29     | 89.40      | 87.13     | 88.20     | 85.89     | **86.87** |\n   | **+ Our Negative (RLHFV)** | 62.40     | 53.2     | 51.94     | 88.54     | 89.57      | 87.32     | 88.30     | 85.92     | 86.80     |\n   | **+ Our Negative (BPO)**   | 60.20     | **53.8** | **53.31** | **90.77** | **91.37**  | **88.58** | **89.07** | **86.51** | 86.76     |\n   |                            |           |          |           |           |            |           |           |           |           |\n   | Qwen 2.5 VL 7B             | 68.20     | 60.2     | 55.12     | 86.45     | 87.97      | 85.59     | 87.07     | 84.90     | 86.33     |\n   | + VCD                      | 68.95     | 59.8     | 55.94     | 87.61     | 88.77      | 86.80     | 87.93     | 85.43     | 86.47     |\n   | + DPO (RLHF-V)             | 68.47     | 60.9     | 53.64     | 86.27     | 87.83      | 85.37     | 86.90     | 84.79     | 86.27     |\n   | + DPO (BPO)                | 66.70     | 53.3     | 55.42     | 86.60     | 88.07      | 85.58     | 87.00     | 84.62     | 85.97     |\n   | **+ Our Negative (RLHFV)** | **70.20** | 58.3     | **57.41** | 86.71     | 88.17      | 85.68     | 87.10     | 84.87     | 86.23     |\n   | **+ Our Negative (BPO)**   | 69.80     | **61.4** | 56.21     | **88.90** | **89.87**  | **87.71** | **88.63** | **86.59** | **87.43** |\n\n3. **Wider Scenarios:** Thank you for your advice. Our DCD explicitly learns generalizable patterns from negative samples, enabling it to broadly extend beyond hallucination mitigation to diverse multimodal scenarios, particularly benefiting tasks where discriminating negative patterns is critical. A potential future extension of our method could be adapted to **chain-of-thought (CoT) correction** by learning negative rationale projections from erroneous intermediate steps (e.g., unsupported leaps or circular reasoning). Applying contrastive decoding specifically to reasoning steps can help reduce flawed logic, while keeping the final answers clear and accurate."}]

Author rebuttal:

Meta-review: This paper introduces Decoupling Contrastive Decoding (DCD), a framework to mitigate hallucinations in MLLMs by separately training positive and negative image projections from preference data. Initial reviews, while appreciating the novel decoupling idea, raised significant concerns regarding experimental validation on outdated models, the clarity of contributions, and the method's novelty. However, the authors provided an exemplary rebuttal, including substantial new experiments on modern architectures (Qwen 2.5 VL), additional baseline comparisons, and a compelling theoretical justification for their approach. This effort successfully addressed all major concerns and led to a unanimous increase in scores from initially skeptical reviewers, forging a strong consensus for acceptance.

# Summary Of Reasons To Publish:
1) The paper's core idea of decoupling positive and negative sample learning is intuitive, well-motivated, and directly tackles the known "likelihood displacement" problem in methods like DPO.
2) Through the rebuttal, the authors have robustly demonstrated the method's effectiveness and generalizability by providing new results on contemporary MLLMs (Qwen 2.5 VL 3B and 7B), directly addressing the primary initial criticism.
3) The authors provided a clear theoretical argument explaining how the decoupled objective avoids likelihood displacement, strengthening the paper's foundations beyond a purely empirical contribution.
4) The work offers a valuable insight by showing that learning exclusively from negative (hallucinatory) samples is sufficient for significant performance gains, a finding that was validated through new ablation studies requested by reviewers.


# Summary Of Suggested Revisions:
1) Integrate the new experimental results on Qwen 2.5 VL models and the Woodpecker baseline from the rebuttal into the main paper to provide a more current and convincing evaluation.
2) Incorporate the theoretical justification for avoiding likelihood displacement and the novelty of the negative-only learning paradigm, as discussed with Reviewer YiWX, to strengthen the paper's contribution section.
3) As agreed with Reviewer SKUk, move key visualizations from the appendix into the main text to better illustrate the method's mechanics and impact on different hallucination types.
4) Update the paper to include the refined figure captions and the clarified discussion on computational efficiency (lower training cost than DPO, higher inference cost) as promised in the rebuttal.
Decision: Accept (poster)