Title: Decoupling Contrastive Decoding: Robust Hallucination Mitigation in Multimodal Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multi-modal Large Language Model

RLHF

Direct Preference Optimization

Contrastive Decoding
Abstract: Although multimodal large language models (MLLMs) exhibit remarkable reasoning capabilities on complex multimodal understanding tasks, they still suffer from the notorious 'hallucination' issue: generating outputs misaligned with obvious visual or factual evidence. Currently, training-based solutions, like direct preference optimization (DPO), leverage paired preference data to suppress hallucinations. However, they risk sacrificing general reasoning capabilities due to the likelihood displacement. Meanwhile, training-free solutions, like contrastive decoding, achieve this goal by subtracting the estimated hallucination pattern from a distorted input. Yet, these handcrafted perturbations (e.g., add noise to images) may poorly capture authentic hallucination patterns. To avoid these weaknesses of existing methods, and realize ``robust'' hallucination mitigation (\ie, maintaining general reasoning performance), we propose a novel framework: Decoupling Contrastive Decoding (DCD). Specifically, DCD decouples the learning of positive and negative samples in preference datasets, and trains separate positive and negative image projections within the MLLM. The negative projection implicitly models real hallucination patterns, which enables vision-aware negative images in the contrastive decoding inference stage. Our DCD alleviates likelihood displacement by avoiding pairwise optimization and generalizes robustly without handcrafted degradation. Extensive ablations across hallucination benchmarks and general reasoning tasks demonstrate the effectiveness of DCD, \ie, it matches DPO’s hallucination suppression while preserving general capabilities and outperforms the handcrafted contrastive decoding methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Decoupling Contrastive Decoding (DCD), a new framework aimed at mitigating hallucinations in multimodal large language models (MLLMs). The proposed method decouples the learning of positive and negative samples in preference datasets and trains separate positive and negative image projections within the MLLM. The authors argue that this approach alleviates the likelihood displacement issue inherent in training-based methods like Direct Preference Optimization (DPO) and avoids the handcrafted perturbations used in training-free methods such as contrastive decoding. The paper includes extensive experiments across multiple hallucination benchmarks and general reasoning tasks, showing that DCD can suppress hallucinations while preserving general capabilities.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper addresses a critical issue in MLLMs—hallucinations—and proposes a method that aims to balance hallucination suppression with the preservation of general reasoning capabilities.\n\n2. The authors provide a thorough experimental evaluation across multiple benchmarks, which helps to demonstrate the effectiveness of their approach compared to existing methods.\n\nWeaknesses: \n\n1. The proposed method appears to be a relatively incremental advancement rather than a novel breakthrough. It builds upon existing contrastive decoding techniques and preference optimization methods without introducing entirely new concepts.\n\n2. The use of LLaVA 1.5-7B as the base model for experiments is a significant limitation. This model is notably outdated compared to more recent and advanced models such as Qwen2.5-VL and InternVL2.5. The lack of validation on these more modern models raises questions about the generalizability and relevance of the proposed method.\n\nQUESTIONS:\n1. Could the authors provide a more detailed comparison of DCD with other state-of-the-art models that have been proposed more recently, especially those that claim improved hallucination mitigation?\n\nLIMITATIONS:\nThe paper has several limitations that should be addressed to strengthen its contributions:\n1. As mentioned earlier, the use of LLaVA 1.5-7B limits the impact of this work. To demonstrate the broader applicability and effectiveness of DCD, the authors should validate their method on more up-to-date and advanced MLLMs. This would provide a clearer picture of how DCD performs in comparison to current standards in the field.\n2. While the method shows promise on the evaluated benchmarks, its generalizability to other domains and tasks remains untested. Future work could explore the application of DCD across a wider variety of multimodal scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Decoupling Contrastive Decoding (DCD), a novel framework designed to mitigate \"hallucination\" in MLLMs. The core contribution of DCD is its ability to decouple the learning of positive and negative samples within preference datasets, training separate positive and negative image projections in the MLLM. Moreover, this paper provides the negative projection to enable vision-aware negative images during the contrastive decoding inference stage. Consequently, DCD alleviates the likelihood displacement problem in training-based solutions like DPO and robustly generalizes without relying on handcrafted image perturbations in training-free methods like VCD. Extensive experiments demonstrate DCD's effectiveness in suppressing hallucinations while preserving general reasoning capabilities, often outperforming existing methods.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- DCD introduces a decoupling perspective to optimize DPO preference learning for hallucination mitigation, which avoids the likelihood displacement in traditional DPO\n- The method also utilized learnable negative image projections in the contrastive decoding inference stage to further mitigates hallucinations while preserving general reasoning capabilities.\n\n### Weaknesses\n- A central concern of this paper is the lack of a clear motivation for the necessity of preference-based fine-tuning and contrastive decoding for MLLM hallucination mitigation, particularly when Table 1 indicates performance degradation on certain hallucination and general benchmarks after such fine-tuning when compared to the original base models. For completeness, it is necessary to compare with more methods that eliminate hallucinations through data optimization[1] or model optimization[2,3]\n\n    - [1] HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data, CVPR 2024\n    - [2] Woodpecker: Hallucination correction for multimodal large language models\n    - [3] Analyzing and Mitigating Object Hallucination in Large Vision-Language Models, ICLR 2024\n\n- The baseline setting doesn't include any open-source VLLM, such as Qwen-vl, which should be included in the experiment to see the fine-tuning performance for hallucination mitigation. \n\n- The dual-projection training process of DCD may incur higher computational overhead compared to general DPO, due to the need for separate positive and negative projections.\n\n- The main experimental results in Table 1 are reported for only one model size and architecture. To improve completeness, the authors should extend the results to include multiple scales (LLaVA-7b, 13b) and more general baselines (Qwen-2.5-VL)\n\nQUESTIONS:\n### Major question\n1. Clear motivation for the necessity of preference-based fine-tuning and contrastive decoding for MLLM hallucination mitigation (See weakness performance drop)\n2. How well can this method adapt and perform with larger and more recent MLLM backbones (e.g., Qwen-VL, LLaVA-Next), or is its applicability primarily limited to LLaVA 1.5?\n### Minor questions\n3. How well does the proposed method mitigate relational hallucinations (e.g., asserting “a person riding a bicycle” when only a bicycle is present) and attribute hallucinations (e.g., asserting “many red apples” when a green apple under a red apple tree is present)?\n4. in line 189, the $\\theta$ is the parameter of the MLLM while the $\\pi_{\\theta}$ is the policy.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-25"}

comments: SUMMARY:
The paper proposes a novel approach to reduce hallucinations in multimodal large language models (MLLMs). The author discussed the pros and cons of the training-based and training-free hallucination mitigation methods and lead to their proposed method. They first decouple the modeling of positive and negative training examples to alleviate likelihood displacement problem. To better model the image feature, especially the negative images, the author proposed a simple feature extractor to focus on negative features, instead of adding noise or use image agnostic methods to create negative training examples. The author conduct extensive experimentations and ablations on many benchmark datasets and demonstrate superior and matching performance on most of the metrics.

STRENGTHS AND WEAKNESSES:
Strength:
1. The paper is well written and well motivated, especially in figure 1.
2. The author conduct extensive experimentation. The author showed 7 benchmarks with better or matching performance.
3. The author promise to release the code.
4. Useful insight and nice case analysis. The performance 

Weakness:
1. Some part of the paper can have better clarity, especially in the caption: for example, figure 1 is confusing with only caption without reading the main text. Figure 2 need more explanation in caption.
2. Limited architecture choice -- only on LLaVA-1.5-7B.
3. Did not discuss limitation of the work.
4. Main contributions section can be improved and highlight the contribution of the paper. line 95 - 101

QUESTIONS:
See weakness and please address those.

LIMITATIONS:
What is the limitation of the work? How is inference time / run-time compared to training-free method? How well does it work on larger models? The limitations needs to be discussed.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-01

comments: SUMMARY:
This paper presents Decoupling Contrastive Decoding (DCD), a new method for mitigating hallucination in MLLMs with preference optimization. The key idea is to decouple the learning of positive and negative samples from preference datasets by training separate image projection modules for each, thereby directly modeling hallucination patterns and avoiding sacrificing the general reasoning abilities of the model. DCD is evaluated against both training-based and training-free baselines on multiple hallucination and general reasoning benchmarks, showing that it suppresses hallucination comparably to strong baselines like DPO, but better maintains general reasoning ability and overall performance. Extensive ablations and analyses provide further insight into the contributions of positive and negative learning.

STRENGTHS AND WEAKNESSES:
**Strengths**

1. The paper focuses on a critical problem of reducing the hallucination of MLLMs.

2. The idea of decoupling the learning of positive and negative samples to reduce hallucination while maintaining the general capabilities of MLLMs is interesting and reasonable.

3. Extensive experiments with comparisons among strong baselines are conducted to demonstrate the effectiveness of DCD.

**Weakness**

1. The findings in Section 5.3 that most of the gain could be attributed to negative samples are not very convincing, since the positive samples are seen during supervised fine-tuning, thus the model may already have learned the knowledge of the positives.

QUESTIONS:
Please see my comments in the Strengths and Weakness section.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03