Title: Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodel Large Language Model; Direct Preference Optimization; MLLM Alignment
Abstract: Direct Preference Optimization (DPO) has emerged as an effective approach for mitigating hallucination in Multimodal Large Language Models (MLLMs). Although existing methods have achieved significant progress by utilizing vision-oriented contrastive objectives for enhancing MLLMs' attention to visual inputs and hence reducing hallucination, they suffer from non-rigorous optimization objective function and indirect preference supervision. To address these limitations, we propose a Symmetric Multimodal Preference Optimization (SymMPO), which conducts symmetric preference learning with direct preference supervision (i.e., response pairs) for visual understanding enhancement, while maintaining rigorous theoretical alignment with standard DPO. In  addition to conventional ordinal preference learning, SymMPO introduces a preference margin consistency loss to quantitatively regulate the preference gap between symmetric preference pairs. Comprehensive evaluation across five benchmarks demonstrate SymMPO's superior performance, validating its effectiveness in hallucination mitigation of MLLMs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel DPO-based framework, Symmetric Multimodal Preference Optimization (SymMPO), to mitigate hallucinations in VLMs. It is specifically designed to construct preference pairs based on image information. The experiments are thorough and demonstrate the effectiveness of the proposed methods. Overall, I think it is good work that above the acceptance threshold of NeurIPS.\n\nI have some questions and suggestions that I hope the authors will take into consideration. I am willing to raise score if authors answer them appropriately.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The paper is well-organized and easy to follow, with all notations clearly defined. I found the reading experience enjoyable and informative.\n2. The experiments are comprehensive and effectively demonstrate the proposed method's effectiveness.\n3. The methodology is specifically tailored for multimodal models, similar to mDPO, and I believe it provides valuable insights for the community.\n4. The theoretical component, while simple, is logical and adds meaningful support to the overall work.\n\n**Weaknesses:**\n\nI did not notice any significant weaknesses in this paper. Just one point, the coefficient-related hyperparameters seem a bit unusual, such as $\\gamma=1e-4$. I am curious about how these values were selected. Perhaps some ablation studies on this aspect would be helpful. If time is too limited to provide them now, I hope they can be included in the final version of the paper.\n\n**Some minor issues:**\n1. In Section 2.1, while the reward function trained using the Bradley-Terry model was first utilized by PPO-based RLHF algorithms, it was not originally proposed by PPO itself. The authors should cite [1, 2] and other relevant papers. Additionally, this section should be renamed, as \"RLHF with BT reward model\" is more directly related to DPO, compared with PPO.\n2. All practical loss functions in this paper should involve $\\mathbb{E}_{x,m,y \\sim...}$, as the prompt, image, and response are not constants but random variables that should be sampled during training.\n\n[1] Stiennon et al. Learning to summarize from human feedback. \n\n[2] Ouyang et al. Training language models to follow instructions with human feedback.\n\nQUESTIONS:\n1. Could the authors provide more details about the \"contrastive image construction\"? Specifically, what does \"pair each image with its nearest neighbor based on cosine similarity\" mean? Does this imply that $m'$ is sourced from other images within the dataset? If so, what would happen if the dataset is small and no sufficiently similar image can be found?\n2. It appears that the rejected responses are generated by the model being trained, while the preferred responses are generated by DeepSeek-v3 (based on captions generated by Qwen2.5-VL-32B). If this is correct, the performance upper bound of the proposed method should theoretically align with Qwen2.5-VL-32B, right? In that case, it would be helpful if the authors could include the performance of Qwen2.5-VL-32B in Table 1 for comparison.\n3. Building on the previous question, if the preferred responses are generated by DeepSeek-v3, they are effectively off-policy and, according to the analysis in OPA-DPO, may be difficult to learn. What would happen if the authors first conducted SFT on these preferred responses? Would this improve the results? (I fully understand that time during the rebuttal period is limited, and I will not fault the authors if experimental results cannot be provided at this stage.)\n\nLIMITATIONS:\nThe limitations are well-discussed within the paper. I suggest authors to discuss the positive societal impacts of the proposed methods.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the critical issue of hallucinations in Large Vision-Language Models (LVLMs), particularly semantic hallucinations where generated responses include entities unsupported by the visual input. The authors propose a novel framework consisting of two main components. The first is MPD (Masked Prediction Difference), which extracts hallucination-prone components in the decoding process by measuring prediction divergence. The second is Selective Gradient Detachment (SGD), a decoding-phase intervention strategy that suppresses gradients from hallucination components without altering model parameters. The method is model-agnostic, training-free, and designed to maintain overall task performance while reducing hallucinations. Extensive experiments on multiple benchmarks demonstrate the effectiveness of the proposed method, with significant hallucination reduction and minimal performance degradation.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength:**\n\n**1.Motivation:**\nThe paper provides a strong and timely motivation by identifying hallucination as a major limitation in the deployment of LVLMs. The proposed strategy aligns well with practical needs by avoiding additional training or fine-tuning.\n\n**2.Novelty:**\nThe MPD-based extraction of hallucination-inducing components is a fresh perspective. Previous methods often focused on architectural modifications or training-based approaches. The proposed combination of MPD and SGD offers a novel, lightweight inference-time mitigation mechanism.\n\n**3.Writing and Presentation:**\nThe paper is generally well-written, with clear mathematical formulations and well-illustrated figures. The intuition behind the method is conveyed effectively, which helps readers understand the internal mechanics.\n\n**Weakness:**\n\n**1.Lack of Clarity in Problem Formulation:**\nThe authors argue that existing vision-oriented preference optimization methods suffer from two key limitations: (1) a non-rigorous objective function due to the inability to cancel out intractable partition functions when image inputs differ (i.e., $m_w \\ne m_l$)), violating the assumptions of DPO; and (2) indirect preference supervision arising from the use of contrastive images with fixed responses rather than contrastive responses. While these points are theoretically motivated, their practical implications remain obscure due to abstract terminology and limited explanation. Clarifying these limitations with intuitive language and a concrete example would significantly improve the accessibility and persuasiveness of the argument.\n\n**2. Insufficient Evaluation on Standard Benchmarks:**\nThe paper does not report performance on any widely adopted LVLM benchmarks such as MMBench, MME, POPE, SEED, or MM-Vet. This omission significantly limits the credibility of the evaluation, as it is unclear whether the proposed method maintains its advantages across diverse tasks and settings. Without such results, the general effectiveness and real-world utility of the approach remain unconvincing.\n\n**3. Weak Evidence of Generalization Across Models:**\nAll experiments are restricted to the LLaVA model, offering limited insight into how well the method generalizes across architectures. This narrow experimental setup raises concerns about overfitting to a specific model. To establish robustness and broader applicability, it is essential to include results on additional strong baselines such as DeepSeek and MiniGPT-4. The absence of such evidence substantially weakens the empirical contribution of the paper.\n\nQUESTIONS:\n**1.Lack of Clarity in Problem Formulation:**\nThe paper raises two theoretical limitations of existing vision-oriented preference optimization methods, but the explanation is abstract and lacks intuitive grounding. The impact of issues like mismatched partition functions or fixed-response supervision is unclear without concrete examples. Clarifying these points in plain terms with illustrative scenarios would make the problem statement more accessible and compelling.\n\n**2.Missing Evaluation on Standard Benchmarks:**\nThe method is not tested on widely used LVLM benchmarks such as MMBench, MME, POPE, SEED, or MM-Vet. Without results on these datasets, it is difficult to assess the method’s general effectiveness or practical relevance across diverse tasks.\n\n**3.Limited Model Generalization Evidence:**\nAll experiments are conducted on LLaVA, with no evaluation on other strong models like DeepSeek or MiniGPT-4. This narrow scope raises concerns about overfitting and limits claims of generalizability. Broader model-level validation is needed to support the method’s robustness.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SymMPO, a method designed to mitigate hallucination issues in VLMs. SymMPO improves upon existing approaches and constructs hard-negative contrastive image pairs for direct preference supervision, forming a more theoretically-rigorous optimization framework and effectively enhancing visual perception abilities of VLMs. The method incorporates symmetric preference learning and preference margin consistency regularization. Experimental results show that SymMPO outperforms previous methods across several hallucination benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Pros:**\n1. This motivation of this paper is reasonable and straightforward.\n2. The idea of the proposed SymMPO is intuitive and somewhat elegant, especially the design of the symmetric pairwise reward.\n3. The paper writing is good and easy to follow.\n4. Extensive results convincingly demonstrate the effectiveness of SymMPO and its various loss terms in reducing hallucinations.\n\n**Cons:**\n\n1. SymMPO introduces additional computational overhead compared to standard DPO, potentially increasing the computational cost by several times.\n2. The two limitations mentioned in the Introduction essentially refer to the same issue: the inherent conflict between the existing vision-oriented preference learning and the standard DPO framework. Could the authors further explore SymMPO, particularly in terms of whether aspects beyond pairwise rewards might help address existing limitations?\n\nQUESTIONS:\n1. How does the SymMPO approach compare with other methods in terms of training time and scalability, particularly when working with larger datasets or more complex multimodal inputs?\n\n2. I looked at the code provided in the anonymous repository, but it seems that it cannot be executed. Could the authors upload a runnable version of the code for reference?\n\nLIMITATIONS:\nPlease refer to Questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel method, Symmetric Multimodal Preference Optimization (SymMPO), to mitigate hallucinations in Multimodal Large Language Models (MLLMs). It is a symmetric preference learning formulation. This method leverages contrastive image pairs (m, m') along with their respective preferred responses (yw, y'w) to provide direct preference supervision. This symmetric design not only enhances the model's ability to make fine-grained visual discriminations but also ensures the theoretical integrity of the DPO objective by aligning with its standard derivation. Furthermore, the method is augmented with a preference margin consistency regularizer to quantitatively constrain the preference gap between symmetric pairs, enforcing a more precise alignment.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper proposed SymMPO framework is not merely an incremental improvement but an original and elegant solution that restores theoretical consistency with the foundational DPO loss. This demonstrates a high degree of methodological Quality and Originality.\nThe paper introduces a novel symmetric optimization strategy and a preference margin consistency regularizer (LMargin), which together provide a more direct and quantitative supervisory signal for visual understanding. \n\nThis work framework necessitates generating a unique preferred response for every contrastive image, which introduces additional computational complexity and cost compared to methods that reuse the original response. And it relies on CLIP similarity, but this heuristic may not always produce optimally challenging pairs. But it difficult to isolate the true contribution of the SymMPO algorithm itself versus the quality of the input data.\n\nQUESTIONS:\n1. Whether other models can be used to enhance pair screening and acquisition (Besides CLIP).\n2. Which specific stage or component of the \"Caption-Anchored Claim Extraction-and-Rewriting\" pipeline (Section 4.1) is primarily responsible for Fine-Grained task performance deficit? Is the limitation rooted in the initial caption generation, which may lack fine-grained detail, or does it stem from the subsequent claim extraction and rewriting process?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}