Title: Alleviating Hallucinations in Large Language Models through Multi-Model Contrastive Decoding and Dynamic Hallucination Detection
Venue: NeurIPS 2025
Accepted: yes
Keywords: Alleviating Hallucinations

Contrastive Decoding

Large Language Model
Abstract: Despite their outstanding performance in numerous applications, large language models (LLMs) remain prone to hallucinations, generating content inconsistent with their pretraining corpora. Currently, almost all contrastive decoding approaches alleviate hallucinations by introducing a model susceptible to hallucinations and appropriately widening the contrastive logits gap between hallucinatory tokens and target tokens. However, although existing contrastive decoding methods mitigate hallucinations, they lack enough confidence in the factual accuracy of the generated content. In this work, we propose Multi-Model Contrastive Decoding (MCD), which integrates a pretrained language model with an evil model and a truthful model for contrastive decoding. Intuitively, a token is assigned a high probability only when deemed potentially hallucinatory by the evil model while being considered factual by the truthful model. This decoding strategy significantly enhances the model’s confidence in its generated responses and reduces potential hallucinations. Furthermore, we introduce a dynamic hallucination detection mechanism that facilitates token-by-token identification of hallucinations during generation and a tree-based revision mechanism to diminish hallucinations further. Extensive experimental evaluations demonstrate that our MCD strategy effectively reduces hallucinations in LLMs and outperforms state-of-the-art methods across various benchmarks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Multi-Model Contrastive Decoding (MCD), a strategy to reduce hallucinations in LLMs. The core of MCD involves integrating a pretrained language model with an \"evil\" model, which is trained to induce hallucinations, and a \"truthful\" model, which is trained to enhance factual consistency. This multi-model approach, combined with a dynamic hallucination detection, aims to strengthen the contrast between hallucinatory and factual tokens during text generation, thereby increasing the model's confidence in producing accurate outputs and further diminishing hallucinations. Experimental results across various benchmarks like TruthfulQA, FActScore, and FACTOR demonstrate that MCD effectively reduces hallucinations and outperforms existing methods.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The paper proposes Multi-Model Contrastive Decoding (MCD) strategy and provides experimental evaluations across multiple benchmarks (TruthfulQA, FActScore, FACTOR) to support its claims of reducing hallucinations in LLMs\n- MCD outperforms current methods in reducing hallucinations, as evidenced by improvements across various benchmarks\n- The paper is easy to follow and organized.\n\n## Weaknesses\n- The fine-tuning process for generating the evil and truthful models introduces additional computational costs, which could be a barrier for practical deployment, especially for very large LLMs\n- The MCD method is very similar to DExperts [1] which contrast outputs from an \"expert\" model and \"anti-expert\" model on top of a base LM. The paper mentions some contrastive decoding methods but omitted this one. This method should be one of the baseline methods for comparison. \n- The performance of MCD is likely sensitive to the choice of hyperparameters α and γ. While the authors mentioned in the appendix that these values are set empirically based on performance on preliminary results, there is a lack of discussion of these optimal values to different tasks. For example, for TruthfulQA multiple choice, $\\alpha$, the hyperparameter for the \"evil\"model,  is set to 0.1, whereas for FACTOR it is set to 1.0 \n\n[1] Liu et al. DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts. ACL 2021\n\nQUESTIONS:\n1. In Table 1, what is True (%) and Info (%), and why do you use True*Info score? I can't find an explanation in the paper for these metrics. \n2. Have you observed any effects (positive or negative) of MCD on other aspects of generation, such as fluency, coherence, style, or diversity? Even a small scale human evaluation would be valuable.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
Authors introduce a multi-model contrastive decoding method that improves upon previous CD methods by adding an additional "truthful" model that is fine-tuned on correct responses. They demonstrate performance improvements in hallucination benchmarks.

STRENGTHS AND WEAKNESSES:
Strengths:
1. Straight-forward to implement; direct extension of prior work
2. Comprehensive evaluations and includes ablations
3. Introduces methods for improving performance like logit extrapolation and a dynamic hallucination detector
4. Includes evaluations on multiple commonly used benchmark datasets like truthfulqa and factscore

Weaknesses:
1. Evaluates only a single model (llama-2-7b-chat)
2. Missing statistical analysis (e.g., error bars) on main experimental results
3. Missing justification and ablation of add-on methods like logit extrapolation

QUESTIONS:
1. Suggest that authors report performance of other models, either in the same 7b class or smaller/larger models to demonstrate generalizability of method performance.
2. Suggest to include more justification and ablation of logit extrapolation method. How much of the model improvement is due to this step alone?
3. Please include error bars with experimental result numbers, i.e., confidence intervals or standard errors.
4. It seems that using a single model for base/truthful/evil introduces high parameter correlation; why not use three different models?
5. Is there a held-out dev set used for selecting the optimal hyperparameters (e.g., Table 9 and 10). If not, then this risks cherry-picking and calls into question how generalizable the choice of hyper-parameters are.
1. Why use gpt-3.5-turbo for the llm-as-a-judge? gpt-4o has demonstrated much more robust performance and less prone to errors.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Multi-Model Contrastive Decoding (MCD), a method to reduce hallucinations in large language models by using three models during inference: a base LLM, an \"evil\" model trained to produce hallucinations, and a \"truthful\" model trained to enhance factuality. The approach modifies the standard contrastive decoding formula by adding a positive term from the truthful model, aiming to increase confidence in factual tokens while suppressing hallucinatory ones. The method also includes dynamic hallucination detection and tree-based revision mechanisms. Experiments on TruthfulQA, FACTOR, and FActScore show improvements over existing contrastive decoding methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Clear Problem Motivation: The paper addresses a well-motivated issue - existing contrastive decoding methods can reduce hallucinations but often lack confidence in factual outputs.\n- Comprehensive Experimental Evaluation: The authors evaluate on multiple benchmarks (TruthfulQA, FACTOR, FActScore) and compare against relevant baselines including both contrastive decoding and representation editing methods.\n- Multiple Technical Components: The approach combines several complementary techniques (multi-model contrastive decoding, logits extrapolation, dynamic detection, tree-based revision) that work together.\n\nWeaknesses:\n\n- Computational and Training Overhead: Training multiple (evil, truthful) models requires extra cost and effort, thus decreasing the applicability to practical usage.\n- Limited Domain Generalization: Experiments are primarily focused on standard knowledge question-answering tasks; the applicability to other domains (math, code) is not discussed.\n\nQUESTIONS:\n1. Computational Efficiency Concerns: I wonder how MCD's computational efficiency compares against simpler baseline approaches. For instance, a straightforward alternative would be to have the LLM generate responses twice and then use either an LLM-as-a-judge framework or directly apply FactScore evaluation to select the superior response. This baseline approach would avoid the overhead of training multiple specialized models (evil and truthful) while potentially achieving similar hallucination reduction benefits. Given that MCD requires 2.30× inference time and the complexity of coordinating three models during generation, it would be valuable to establish whether this sophisticated approach actually outperforms such simpler multi-generation strategies that leverage the base model's inherent variability and post-hoc selection mechanisms.\n\n2. Can MCD generalize to other domains (math, code)? For example, DoLa [1] demonstrates it effectiveness on GSM8k [2].\n\n\n[1] Chuang et al., DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models\n\n[2] Cobbe et al., Training Verifiers to Solve Math Word Problems\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Multi-Model Contrastive Decoding (MCD), a novel decoding strategy for mitigating hallucinations in large language models (LLMs). MCD integrates three models during inference: a base LLM, a hallucination-prone “evil” model, and a factuality-enhanced “truthful” model. A token is assigned high probability only if it is encouraged by the truthful model and discouraged by the evil model, thereby improving contrast between factual and hallucinatory outputs. The authors further propose a dynamic hallucination detection module that identifies problematic tokens during generation and triggers a tree-based revision mechanism to resample and replace low-confidence segments. To train the evil and truthful models, the paper employs Direct Preference Optimization (DPO) and a reverse variant (CDPO) using an augmented hallucination dataset.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well-written and easy to follow, with clear organization and motivation.\n2. The proposed method is conceptually well-grounded, and experimental results on multiple open-ended factuality tasks convincingly demonstrate its effectiveness.\n3. Addressing hallucinations in LLMs is an important and timely problem. The proposed approach provides a practical solution that may benefit the broader community.\n\nWeaknesses:\n1. The novelty of the proposed MCD framework is somewhat limited, as similar ideas have been explored in prior works (e.g., Liu et al., 2021; Liu et al., 2024). That said, to the best of my knowledge, this is the first attempt to apply such a framework specifically to factuality mitigation in LLMs, so I will not weigh this point heavily. However, I strongly encourage the authors to discuss these related works for completeness.\n2. The proposed logits extrapolation component is an interesting idea, but the paper lacks direct empirical evidence isolating its contribution. If this component plays a key role, a dedicated ablation would strengthen the paper.\n3. The use of two additional models and the tree-based revision mechanism introduces considerable computational overhead in terms of both memory and inference latency, which may limit the method’s practical applicability.\n\n[1] Liu et al., 2021 Dexperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts\n\n[2] Liu et al., 2024 Tuning Language Models by Proxy\n\nI also suggest that the authors include the following relevant references on factuality in their manuscript.\n\n[1] Zou et al., 2023 Representation engineering: A top-down approach to AI transparency. \n\n[2] Liu et al., 2024 Enhancing Language Model Factuality via Activation-Based Confidence Calibration and Guided Decoding\n\nQUESTIONS:\nSee the weakness above.\n\nLIMITATIONS:\nIn terms of computational cost, I suggest that the authors mention the inference-time overhead in the limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-07-05"}