Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization
Authors: Wenqi Liu
Xuemeng Song
Jiaxi Li
Yinwei Wei
Na Zheng
Jianhua Yin
Liqiang Nie
Venue: NeurIPS 2025 poster ()
Keywords: Multimodel Large Language Model; Direct Preference Optimization; MLLM Alignment
Primary area: deep_learning

Abstract: Direct Preference Optimization (DPO) has emerged as an effective approach for mitigating hallucination in Multimodal Large Language Models (MLLMs). Although existing methods have achieved significant progress by utilizing vision-oriented contrastive objectives for enhancing MLLMs' attention to visual inputs and hence reducing hallucination, they suffer from non-rigorous optimization objective function and indirect preference supervision. To address these limitations, we propose a Symmetric Multimodal Preference Optimization (SymMPO), which conducts symmetric preference learning with direct preference supervision (i.e., response pairs) for visual understanding enhancement, while maintaining rigorous theoretical alignment with standard DPO. In  addition to conventional ordinal preference learning, SymMPO introduces a preference margin consistency loss to quantitatively regulate the preference gap between symmetric preference pairs. Comprehensive evaluation across five benchmarks demonstrate SymMPO's superior performance, validating its effectiveness in hallucination mitigation of MLLMs.

Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization
Wenqi Liu'
Xuemeng Song2t Jiaxi Li Yinwei Wei
Na Zheng* Jianhua Yin Liqiang Nie
Shandong University Southern University of Science and Technology University of Georgia *National University of Singapore "Harbin Institute of Technology (Shenzhen) bluug_iitooutooo com, sxmustc@gmail com, jhyin@sdu. edw..
Abstract
Direct Preference Optimization (DPO) has emerged effective approach for mit- igating hallucination in Multimodal Large Language Models (MLLMs). Although existing methods have achieved significant progress by utilizing vision-oriented contrastive objectives for enhancing MILMs' attention to visual inputs and hence reducing hallucination, they suffer from non-rigorous optimization objective func- tion and indirect preference supervision. To address these limitations, we propose Symmetric Multimodal Preference Optimization (SymMPO), which conducts symmetric preference learning with direct preference supervision (i.e., response pairs) for visual understanding enhancement, while maintaining rigorous theoretical alignment with standard DPO. In addition to conventional ordinal preference learn- ing, SymMPO introduces a preference margin consistency loss to quantitatively regulate the preference gap between symmetric preference pairs. Comprehensive evaluation across five benchmarks demonstrate SymMPO's superior performance, validating its effectiveness in hallucination mitigation of MLLMs. Our codes are available https://github.comm// wwwpm//wwgit////m/o/ii// bit/SymMPO.
Introduction
Recently, Large Language Models (LLMs) have demonstrated remarkable success in natural language understanding and generation tasks 2 By integrating visual encoders and cross- cross-modal alignment modules to LLMs, Multimodal Large Language Models (MLLMs) 4 have extended capabilities of LLMs to the multimodal field, enabling applications such as visual question answering (VQA), image captioning. and multimodal dialogue systems. Despite their impressive performance, MILMS often suffer from hallucination problems (7) generating outputs inconsistent with the given image input or not relevant the input textual prompt (8).
Due to its impressive performance in improving response quality by aligning LLMs with human preferences, i.e., guiding LLMs to generate outputs humans would judge better (e.g., more safe or contextually appropriate), Direct Preference Optimization (DPO) 9 has been adapted to first construet
address hallucination issues in MLLMs [10] [I 12 [.] Existing methods 10 14
preference pair consisting of hallucination-Fie response you and a hallucinated response y for given multimodal input, including an image m and textual prompt that specifies the generation task (e.g., VQA), and then perform DPO-based response-orienten preference learning (see Figure IT. To strengthen ML..Ms' attention to visual inputs, recent studies further incorporate DPO-based vision- oriented preference learning component [15 [6] 17
as shown in Figure [ This component leverages contrastive triplet pairs (m I. Fw)) and (m, L. Mw) with only images varied, and aims to
18 [9]
*Correpooniing authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Response-Orietted Oriented Preference Leaming (Base Module)
LimitationT: Non- Rigoroo Objective Function
Symmetric Multimodal Proference Optimization (SymMPO)
Intractable partition x(m,,)) be eliminated due to the difference in image Input.
2 ()
Preferred Response
Preferred Response
cannot
r(m,xx,,)) (m,x.,6) (4) r(me (n.))))))) < (mf.n)3))
r( X.)
Viston-Oriented Preference Learning (Vision- Enhancemennnt Module)
Limitation2: Indirect Preference Supervision
x(m,x.
Wig
M(m,
r'me' (mmlaa
4 (nd N. Ma)
ylo
r(m(.xx,y,)
Only utilizee contrastive Images as Indirect preference supervision, fail to explore the preforred responses of these images.
Margin Consistency Regularization
X.3.)
Figure I: Existing methods focus on response- and vision-oriented preference learning (left), where the former serves as the base module, while the latter is optional, specifically for vision understanding enhancement. r(m, y) represents the reward for generating response given the input (m, x). However, current methods face two key limitations: non-rigorous objective function and indirect preference supervision (middle). To address these, we propose SymMPO, which leverages symmetric pairwise preference optimization using contrastive images and their preferred responses, alongside preference margin consistency regularization designed specifically for symmetric optimization (right).
make the likelihood of generating Yie given (Mo,, .) higher than that given (m, x). Here, my is modified version of ma,, obtained by either applying transformations to mh or adding noise to it.
While current methods have demonstrated promising results in MLLM hallucination mitigation by incorporating vision-oriented preference learning, they suffer from two key limitations.
Limitation 1. Non-Rigorous Objective Function. For deriving the objective function, existing works directly replace the shared image input m in the pairwise likelihoods of the response-oriented preference learning objective function with its counterparts (i.e., ma and m²) without rigorous mathematical justification. However, through careful analysis of the standard multimodal DPO loss derivation, we find that the partition functions Z(m..: 2) and (m,, 2mn,wch which involve summing over all possible responses and are computationally expensive to estimate in the implicit reward formulation-canee
be directly canceled out in the loss function when the image inputs differ (i.e., ma * mi). response-oriented preference learning, these partition functions naturally vanish due to their shared input m. However, this cancellation does not hold for vision-orienter preference learning, where the contrastive image inputs my and my are inherently distinct. Nevertheless, cxisting methods directly assume their climinability, which results in misaligned objective function that deviates from the theoretical formulation, ultimately limiting model performance 20 A rigorous mathematical analysis regarding partition functions is provided in Appendix B
Limitation Indirect Preference Supervision. As shown in Figure D existing methods conduct DPO with two image-text-response triplets with fixed prompt and response but contrastive images for vision-oriented preference learning. This design fundamentally relies on contrastive images for preference alignment, which misaligns with DPO's core principle of reward formulation via paired responses (i.c., preferred vs. less preferred answers). This misalignment fails to explicitly model the direct preference relationship between accurate and hallucinated responses conditioned on the given image, leading to suboptimal visual understanding capabilities. fact, we can additionally construct preferred response for the contrastive image my using the same prompt for m... Since m. and my share high similarity, their corresponding preferred responses naturally exhibit strong semantic alignment with subtle variations-mae making them ideal hard negatives for each other. Such contrastive response pairs with minor differences compel the model to perform more thorough multimodal interpretation, thereby effectively reducing hallucinations.
To address these limitations, we propose Symmetrie Multimodal Preference Optimization (SymMPO), to reduce hallucinations in MLLMs. Unlike existing methods that solely rely on contrastive images

---

to conduct the preferred response of the contrastive image given the same prompt of original -mage-prompt- response triplet, enabling vision-oriented preference learning with direct preference supervision (i.e., contrastive response pairs). Specifically, we design symmetric preference optimization formulation, which simultaneously maximizes the likelihood of ranking Ym over V'm given (m, x)) and that of ranking V'n over ymm given (m',x), where m' is an image semantically similar to m (identified via CLIP visual similarity), and Vn, is the preferred response for m' given The symmetric optimization compels the model to thoroughly differentiate between contrastive responses under varying visual contexts. By aligning with standard DPO in terms of using contrastive responses and fixed inputs (m, x),, our method naturally eliminates the two partition functions in the reward formulation, yielding rigorous objective. Further, beyond conventional ordinal preference ranking optimization (i.e. the > V.w). SymMPO introduces preference margin consistency regularization that enforces consistency in magnitude of preference gaps between contrastive responses. Additionally, SymMPO integrates the established response-oriente performance in ensuring response quality, and introduces cost-effective caption-anchored pipeline for constructing response preference pairs, facilitating efficient training and evaluation of our model.
vision-oriented preference alignment, SymMPO additionally introduces
preference learning loss due to its remarkable
Our main contributions can be summarized in three key aspects:
We identify two key limitations in existing DPO-based multimodal preference optimization methods for MLLMs: non-rigorous objective function and indirect preference supervision for vision-oriented preference learning. For the non-rigorous objective function, we also provide detailed mathematical derivation in Appendix B
We propose SymMPO that effectively leverages preferred responses for contrastive images, supporting direct vision-oriented preference learning with a theory--cnsistee symmetric ob- jective function. Additionally, we design preference margin consistency regularization to quantitatively constrain the preference gap between symmetric response pairs, enabling more precise contrastive preference learning.
We empirically validate the superiority of SymMPO over existing approaches through compre- hensive experiments on four well-established MLLM hallucination benchmarks, as well one benchmark designed to evaluate the ability of MLLMs to answer question based on images.
2
Preliminary
To facilitate our method presentation, we first review Proximal Policy Optimization (DPO's founda- tion), standard DPO, and existing multimodal DPO learning paradigm.
2.1 RLHF with Bradley-Terry Reward Model
Reinforcement Learning with Human Feedback (RLHF) 21 22 has become a widely adopted approach for aligning LLMs with human preferences and expectations, enabling the generation of responses that better meet
needs. Among the various RLHF methods, Proximal Policy Optimization (PPO) (23 stands out as one of the most commonly used methods, primarily due to its stability and efficiency in policy optimization. PPO operates in two main stages; first, it trains a reward model that serves as proxy for human feedback; second, it optimizes a policy model based on this reward model guide the LLM toward generating responses that more effectively align with human expectations and preferences.
Specifically, to derive the reward model, PPO employs the widely adopted Bradley- -enly model 24] which maximizes the likelihood of the preferred (winning) response being ranked higher than the less preferred (losing) response y for given input x. This is formalized as follows:
exp(rø(x, Mwr)
(Er > yu (r) = (r, yw) (o v)) exp(ro(x, yw)) - exp(r w))
(1)
where ó represents the parameters of the reward model. a--) denotes the sigmoid function.
Based on the trained reward model ro. PPO optimizes the policy model as follows:
D.oesrD.gooofofeo max E you m (-1x) Tr (r 2) - DD (=)(===))
3

---

where 2 E DD represents the input text sampled from dataset D, and denotes the response generated by the current policy model To parameterized with 0. Wyef fixed reference model used for constraining the policy model through Kullback-Leibler (KI.) divergence D This objective function ensures the policy model mo progressively aligns with human preferences captured by the reward model rg. while maintaining generation stability and preventing mode collapse. The hyper-parameter controls the strength of the KL penalty.
2.2 Standard Direct Preference Optimization
Although the PPO framework demonstrates effective performance in RLHF applications, it involves separate additional reward model training process, limiting the training efficiency [25]. To address this, DPO [9] was introduced to eliminate the need for explicit reward modeling, while preserving remarkable effectiveness aligning LLMss output with human preferences. In particular, DPO introduces an implicit reward formulation that effectively integrates the probabilities yielded by the policy model and reference model, enabling the direct optimization of the policy model through establishing a direct mapping between policy parameters and human preferences via the Bradley-Ters model. particular, the implicit reward formulation integrates the probabilities yielded by the policy model and reference model as follows:
(o(y)z) Krey (vlæ:)
r(x, v) = B Blog
+ B Blog Z(r),
where ((x) = Ey (y)r( exp (37 y)) represents an intractable partition function, and [ is hyper-parameter that controls the deviation from the reference model.
Although the implicit reward involves an intractable partition function, substituting it into the Bradley- Terry model (Equation]) I and applying the negative logarithmic transformation allows us to derive DPO objective function free of any intractable terms, as follows:
CDPO- -.( log or ( (Blog To (T)
B log To /z)) Tref (y1) xr)
Tres (Vw (Vwclz)
where, consistent with Equation I Yo and denote the preferred and less preferred response, respectively, for given input text s.
2.3 Multimodal Direct Preference Optimization
Existing DPO-based multimodal preference optimization methods typically involve two key compo- nents: response-oriented preference learning [14 and vision-oriented preference learning [9] While the former universally adopted for response preference alignment, the latter optional for enhancing visual interpretation.
Response-oriented Preference Learning. The standard DPO is designed for aligning L.MSs to human preferences based pairs of preferred and less preferred responses (i.c., yo and yı). To align MLLMs with human preferences, existing methods typically directly extend the DPO objective by including an image condition as follows,
CDpo.. -D) log a (Blog Te To mm, x))
Tres [m, )
To To (m,, x))
Tref (Yw (m, 3 log
(5)
This objective maximizes the likelihood of the preferred (winning) response you being ranked higher than the less preferred (losing) response for a given multimodal input (m, x). Intuitively, it expects the model capture the preference distinctions based on the overall multimodal input.
Vision-oriented Preference Learning. Despite its effectiveness, the overall response-oriented preference learning cannot guarantee MLLMS accurately interpret the visual content. Therefore, recent studies incorporate vision-oriented contrastive mechanism to enhance the MLLMs' visual understanding and hence generate more accurate responses with less hallucination. Specifically, in addition to LDPO.., they introduce a vision-oriented contrastive objective Lvco defined as:
Trey (Yu (m x) Tref (Vo my, 2²) )1 (6)
Te (Vhrl Mer, x))
3 log Te (Yo mmm x²)
Lvco E E. log log

---

where mwy represents the original image sampled from training dataset, and my denotes a contrastive variant of my generated through certain image manipulation operations (e.g.. cropping and rotation). As can be seen, unlike LDPO.,, (see Equation 5.. this objective takes the image-text pairs with fixed text (i.e., 2) and varied images (i.e., May and my as the input condition, and adopts the same response (i.e., Uw) for both terms. By making the image condition the only variable, this objective encourages the model to learn preference distinctions based solely on visual information, thereby enhancing the MLLM's understanding of visual input.
3 Symmetric Multimodal Preference Optimization
To address the non-rigorous objective function and indirect preference supervision of existing vision-enhanced multimodal preference optimization methods, we propose SymMPO that conducts symmetric pairwise preference learning and preference margin consistency regularization, to enhance visual understanding and reduce MLLM hallucinations. Notably, SymMPO also incorporates the response-oriented preference learning, a well-established universal component for ensuring response quality. Here, omit its details they are covered in Section 2.3
Pair-wise Preference Learning. Unlike existing vision-enhanced multimodal preference alignment methods that use indirect preference supervision (i.e., contrastive images), we seek direct preference supervision by further generating a preferred response y'u for the contrastive image m' given the same prompt æ. m' is designed to differ only subtly from m, thereby leading to similar preferred responses (i.e., Bw and y!..) with minor variations. This intrinsic relationship naturally establishes yo and v' as contrastive pair, for enhancing the MLLMs' visual understanding capabilities. Towards comprehensive preference learning, SymMPO designs the following symmetric pairwise reward modeling function with the Bradley-Terry model:
(y Vm PET(Y's > yw yulm',c mm',))
=r x, x. yw) r(m, x, v'm)) (r(m', xI, 1/11/)) - r(m', F, (..)))).
This function essentially jointly models two pair-wise rewards, each aims to maximize the likelihood that the original preferred response ranks higher than the hard negative (contrastive) response for the given multimodal input. This encourages MILMSs to better interpret the given input image and reduce hallucinations. Then following DPO, we derive the optimization loss by applying the negative logarithmic transformation to the pairwise reward modeling function as follows,
Paair = mm De Nar M
log a (r(m, v, Viw) - (m,,,,,1uv) + log o (r(m', xr, V'w) r(m', x, vur) x,,
(o(y [m, x))
To (2/v |m, x)) Tref (V/n m., x))
=
- Rn DDD
log o log
Tref (Vir (m, x))
log
To(Uull ,
Te T /mm x))
+ loo o (3
Tres (V'v mmrx,))) 3 log
Tres (Vwll /mm, xx)
(8)
Since the above objective function aligns with standard DPO by using response pairs as direct preference supervision, SymMPO naturally cancels out the intractable partition functions that share the same multimodal inputs, leading to a rigorous objective function.
Preference Margin Consistency Regularization. As shown in Equation 7 the Bradley-Terry model only separately constrains r(m, x, fw) > r(m, x, V(w) and r(m', Ir, y'm) > r(m',, ,r, yw.. However, since m' is a perturbed variant of m, the preference margin between r(m, yw) and r(m, x. V(o) should be simílar to that between r(m', xx v'w) and r(m', x, ywo) for near-identical inputs (i.e., (m, and (m', x)). Thus, beyond the traditional ordinal preference learning, we incorporate a preference margin consistency regularization, which quantitatively regulate the response preference margin across contrastive images as follows:
EMargin (s New DDD A(m, I. Vmo V'n)
A (m'', , F. Varr yum)
To x))
To To(y/n m,,x)
A D. Mw V/n) = r(m, x, yw) r(m, x, V(n) log
log
Tre f (Ne (m,x) x)
Tres (B'r [m,
(9)
5

---

Wultinodal Input
Image Caption Generation
The Irage showcases serwne landscape dosinated by Pount Fuji, prosinent volcano located portion of the with [-]-
Sive thorough description of the Image, focusing landrank
Prospt:
Describe the image concisely and comprehensiveey, including all antities, their relationships, quantities, and spatial positions
MIUM
Reference Model
dbi Consistent Claim:
Positive Reaponse: The image features Kourt fujs, prorinent snow-capped volcano [-]. The slopes of the mountain appear rugged and steep, [-1-
The Image festures a large sapa-capped sountain the hackground, [-] sountain sppears to have weite and orange snow on its peak, which [-]-
Japan, snow-capped located
beauty.
M:
dence
green traes the nid-ground.
Integrate and Revrite
The Image features
picturesque landsrape with Multiple peosle can wd throughout the field,
Inconsistent Clain:
The sountain appcars to have white and orange snow on its peak.
Claie 1:
Negative Response:
features
large green
[-], peak adorned with
The image Teatures
picturesque landscape with [-] Hultiple people can be soen scattored throughout the field, [-l.
Clate N:
There with few people scattered
the
orange 4004. [-]
Multiple people can be seen
throughout the
scattered throughout [-].
(2) Response Resampling
Caption-Anchenn Anchored Claim Extraction
(4) flaim-based Response Rewriting
Figure 2: The caption-anchone response preference pair construction pipeline.
where Amm, F. ywu Viee quantifies the preference margin between yhe and yv, given (m, z))
Moreover, to prevent the model from reducing the likelihood of the preferred response for optimizing the likelihood gap between preferred and less preferred responses, which can be harmful to preference alignment, we also adopt the anchored preference regularization [15] 19] as follows:
AnaPO = -m, ww. log a (3 log
Tref (Fw (m, 8) + log o (3 Blog
To (Yw (m, x)
Te To m x))
-
Tref (Vay Imm x.)
(10)
where 5 is the anchor that encourages the model to yield high likelihoods for preferred responses.
Ultimately, the complete optimization objective of SymMPO is formally defined as follows:
Sym = LDro,, + X + Margin + A.A.pPP.
(11)
where X, y, and are weighting hyper-parameters.
4
Experiment
In this section, we present our training data construction process, experimental setup and results.
4.1
Training Data Construction
In this work, we adopt the same of 21.4k image- proget pairs from TPO 226, which aggregates multiple public datasets, including VQA v2 (27]. MSCOCO (28) and TexiVQA 29]. To construct preference data for model optimization, we first generate contrastive responses (ye, yt) for each image-prome ppooppt pair (m. x) to compute Lpro... Subsequently, we create contrastive image m' with its corresponding positive response v'm. forming contrastive triplets (m, x. Vm) and (m', x. v'm) for calculating Pair. LMarginn and Ane.PO. This process essentially involves two key modules: preference response pair construction and contrastive image construction, while y'm, for m' can be easily obtained by the preference response pair construction module.
Preference Response Pair Construction. While previous works have proposed various methods for preference pair construction [13 [T] these approaches typically suffer from either costly API calls commercial MLLMs (e.g., GPT-4V 4 or substantial computational overhead with complex multi-stage processing. Therefore, we design cost-effective Caption-Anchored Anchored Claim Extraction- aed-Reeriting pipeline, as illustrated in Figure comprising four efficient stages: Image Caption Generation, 2) Response Resampling, 3) Caption-Anchore Claim Extraction, and 4) Claim Based Response Rewriting. First, we use an open-source MILMM (e.g., Swven.55VVL3322 30 31) to generate a detailed caption for the image. Next, multiple responses are generated from the reference model. Unlike prior approaches [I] 26), the third stage of our pipeline directly extracts atomic claims by performing fine-grained comparison between each response and the detailed image caption,

---

leveraging a high-performanee LLM (e.g., DeepSeek- V3 [3]). This simplifies the process by remov- ing the need for explicit claim extraction, claim-to- -uquestonn conversion, and per-claim verification as required in previous methods 11 26 Instead, our approach only requires generating the image caption and designing a prompt that instructs the L.M to extract positive claims (consistent with the caption) and negative claims (inconsistent with it). Finally, an LLM (e.g.. DeepSeek-V35 is further used as a rewrite model to generate positive and negative responses based on the extracted consistent and inconsistent claims, respectively.
Contrastive Image Construction. Unlike existing methods that generate the contrastive image m' by applying transformations or noise to the original image m. we adopt alternative construction approach. Specifically, we compute CLIP [33] embeddings for all images in the dataset and pair each image with its nearest neighbor based on cosine similarity. This ensures m' and m share high visual similarity while exhibiting subtle differences, making their positive responses (Jmm Fm more challenging to distinguish, thereby enhancing symmetric pair-wise preference learning.
4.2 Experimental Setup.
Benchmarks. For evaluation, we adopt five established benchmarks: 1) HallusionBench 7
evaluates both language hallucination and visual illusion, featuring 346 images and 1,129 prompts. employs GPT-4 [1 to compare MILM outputs against ground truth, using three evaluation metrics: question-level accuracy (qAce), figure-level accuracy (fAce), and overall accuracy across all prompts (aAcc). For cost efficiency, we substitute the original GPT-4 []] evaluator with DeepSeek-VS V33 [32]. 2) Object-HalBench (34] is benchmark for evaluating common object hallucination in detailed image descriptions. Following [10] and [. we use eight diverse prompts across 300 instances to ensure robust evaluation, reporting both response-level and mention- hallucination rates. 3) MMHal- Bench [14] evaluates response informativeness and hallucination rates using GPT-4 to compare model outputs against human annotations for 96 images. The information score and hallucination rate serve the evaluation metrics. 4) AMBER 35 provides 15k fine-grained annotations with well-designed prompts, enabling comprehensive evaluation across three key aspects: object existence, attribute accuracy, and relational correctness. We report the accuracy and F1 score for this discriminative evaluation. 5) MMStar [36] provides 1.5k image-prompe pairs for evaluating six core capabilities and 18 specific aspects of MLLMs, with results summarized in an overall performance score.
Baselines. For fair evaluation, we first adopt the following PPO/DPO-ber based baselines with publicly available pretrained weights: PPO based method ALaVA-RLHE (14 and several DPO-based methods, including RLHF-V [10] (trained on Muffin 13B (37). ) as well as POVID (38), HALVA (39), HA-DPO [3.. RLAIF-V ITI. TPO 26). OPA-DPO [9) and HSA-DPO 40 (all trained on VL 1.5 B) .. To eliminate confounding factors from differing experimental settings and ensure a rigorous comparison, we also incorporate standard DPO 9 and mDPO [15] in our evaluation. Both methods are trained under the same experimental conditions as SymMPO, including identical training data, parameter configurations, and environment. All models (DPO, mDPO, and SymMPO) for rigorous comparison are trained for 2 epochs with a learning rate of 5e-6 and batch size of 64, using the following hyper-parameters: 3 0.1, 0, À 0.5, 7 le 4, and r 1.0. The training is performed on NVIDIA A100-40GB GPUs.
4.3 Main Results
Table presents the performance comparison across five benchmarks with two model variants of LLL 3 LLaVA-1.5-7B and L.LVVA-L.5-13B. From this table, we have following ob- servations: (1) SymMPO outperforms existing methods across most evaluation metrics for both LLaVA-L.5-7B and LLaVA-L.5-13B. demonstrating its effectiveness in reducing hallucination in MLLMs. (2) The blue-highlighttd rigorous comparisons show SymMPO's consistent superiority over DPO and mDPO across four benchmarks: HallusionBench, MMHal-Benchl AMBER, and MMStar. This advantage stems from our novel dision-oriented contrastive learning framework, which effectively leverages symmetric comparisons through a rigorous objective function derived from standard DPO, thereby enbancing the model's 's visual understanding. (3) Surprisingly. both mDPO and our SymMPO underperform DPO on Object- Bhieet-HaalBenc. This may stem from misalignment between our preference data construction method and Object-HalBennt tion pipeline, we use an open-source MILMM to generate image captions, which serve as references for extracting consistent/inconsitee claims and generating positive/ncgative responses. However,
's evaluation task. In our data construc-
7

---

Table 1: Main experimental results, The best and second results under the same experiment setting are highlighted in boldface and underlined, respectively.
Model
Data Size
Feedback
HallusionBench
Object-HallBeen Hallench
MMH
AMBER
MMStar
qAce1
LAcc
aAcef
Resp...
Ment.)
Score
Hall]
Acc1
F1
Overall!
Muffin-13B
6.15
12.71
+RLHE-V
10
1.4k
Human
9.57
13.87
45.79
8.5
4,9
2.60
56.2
82.0
86.7
31.0
3,95
11.56
41.71
56.5
27.9
2.26
56.2
71.8
74.5
33.3
ALLaVA-RLHF
122K
Self-Reward
5.99
12.13
38.26
55.4
27.3
2,00
66.7
68.7
74.7
+POVID 58
17k
GPT-4V
7.03
9.53
43.31
35.9
17.3
2.28
56.2
78.6
81.9
34.4
*HALVA
GPT-4V
5.49
11.27
42.42
24.6
2.14
60.4
83.5
HHA-PP [3
GPT-4
5.49
11.56
42.
44.9
21.8
1.97
61.5
74.2
78.0
32.6
+RLAIF-V
74.8k
LLaVA-Next
5.93
5.49
36.75
9.9
4,9
3.04
39.6
72.7
84,4
34,6
LLaVA-Nest
+OPA-DPO 19
4.8k
LLaVA-Next
6.37
11.84
42.69
6.1
3.7
2.83
46.9
81.3
85.6
35.1
+DPO []]
21.4k
DeepSeck V3
7.25
7,80
40.21
12.9
8.8
2.44
49.0
71.3
82.6
33.4
DeepSeck-V.
9.53
10.1
2.71
5000
86.3
+SymMPO (Ours)
21.4k
DeepSeek V3
7.25
13.58
44.28
19.5
9.7
2,89
42.7
82.6
87.7
34.8
9.53
H.
14
122k
Self-Reward
8.57
10.11
43.48
45.3
21.5
2.15
66.7
79.7
83.9
33.5
+HALYA 39
21.5k
GPT-4V
8.79
10.11
42.24
47.0
22.9
2.30
57.3
82.9
86.5
33.1
4HSA-DPD
6.15
8,95
2.9
2.55
50.0
79.8
82.8
+OPA-DPO 1b
4.8k
LLaVA-Nest
6.81
12.13
42.60
7.7
4.4
3.05
38.5
84.1
87.5
32.3
DeepScok-V2 V3
10.32
10.69
8.5
2.65
45.8
84.6
+mDPO [S
21.4k
DespSock-V3
9.23
10.69
39.85
20.9
10.8
2.93
43.8
83.8
85.8
35.0
+SymMPO (Ours)
21.4k
DeepSeek-V V3
10.54
10.98
44.55
20.4
10.0
3.01
39.6
84.9
89.1
35.2
Table 2: Ablation studies with LLaVA-1.5-7B.
HallusionBench
Object-HalBenee
MMHal Bench
MMHal-Bench
AMBER
MMStar
Model
qAcct
fAcet
aAcct
Resp-L.
Ment..
Score
Hall
Accf
FIf
Overall
SymMPO
7.25
13.58
44.28
19.5
9.7
2.89
42.7
34.8
wlo-Lpate
6.59
11.84
43.22
18.1
10.6
2.53
50.0
81.7
87.1
33.8
wlo-LMargin
7.03
10.98
44.46
21.1
11.0
2.40
54.2
82.0
87.3
34.5
wlo- AnePP
6.81
11.84
40.83
21.6
11.6
2.39
59.4
79.5
87.4
36.2
these captions tend to focus more on object detection and scene overview rather than fine-grained visual details, thereby introducing noise about subtle visual properties in preference response pairs. While this noise has limited impact on benchmarks that assess hallucination using straightforward discriminative questions (e.g., multiple-chote
yes/no questions). poses a greater challenge for Object-HalBench, which directly evaluates response- and mention-level hallucination in detailed M...M-generated descriptions. Despite this, SymMPO still outperforms mDPO, demonstrating stronger performance in mitigating hallucination in fine-grained descriptions.
4.4
Ablation Study
To evaluate the contribution of each key component in SymMPO, we introduce three variants: w/o- Pair, w/o-LMargins and W/o-CAncPO. where the pair- preference loss, the margin consistency regularization, and the anchored preference regularization are removed, respectively. Table Suble@prreeents the ablation results across five benchmarks using LLaVA-1.5-7B the complete SymMPO model outperforms all three variants on most metrics, demonstrating each component's essential contribution to SymMPO's effectiveness. Specifically, the results confirm: (1) the critical importance of pair-wise preference learning for eliminating MLLM hallucination; (2) the necessity of regulating quantitative preference margins between paired samples beyond qualitative preference alignment; and (3) the positive effect of anchored preference regularization in improving training stability by preventing log probability decline of positive responses during optimization.
the backbone. As
see,
4.5 Impact of Contrastive Image
To investigate the impact of contrastive images on our SymMPO, we also perform experiments on four alternative types of contrastive images using LLaVA-1.5-7B, (1) Black: Completely black images containing no visual information: (2) Cropped: Sub-images randomly cropped from original images, with half the original width and height; (3) Noisy: Original images corrupted with Gaussian noise (a = 0.8); and (4) Synthetic: Inspired by the great success of image captioning and generation

---

(a) Original
(b) Similar
(c) Black
(d) Cropped
(c) Noisy
(f) Synthetic
Figure 3: Samples of the original image and its related contrastive images.
24.
3.23
224
3.00
20.
21.2
61A
192
17.6
2,40
72.2
Simier
Black
Cropped
Malay
Syntheti
2.23
Sinilar
Block
Dropped
Noay
Synthetic
70.0
Similar
Black
Crapped
Nokay
Syndwete
14.8
51.
Warten-eee
103
484
65.6
47
86.0
67.0
25.66 re
85.0
Cropped
Syrdidii
Gropped
Syntiette
Hokyy
Sysdhete
(a) Object-HalBench
(b) MMHHl-Bencch
(c) AMBER
Figure 4: Results of SymMPO and mDPO using different types of contrastive images (t/4: higher/lower better). Orange represents SymMPO, and blue represents mDPO.
models, we synthesize the contrastive images by FLUX.1-dev 41 based on the LLaVA-1.5-7BB generated captions of original images. Figure shows contrastive image examples, where "Similar" refers to our CLIP-based contrastive images used in main experiments.
Figure shows performance comparison between our SymMPO and mDPO with different types of contrastive images on three mainstream benchmarks, including Object Haject-HalBench, HMMHal-Benche and AMBER. From this figure, make the following key observation: (1) SymMPO consistently outperforms mDPO across all types of contrastive images, except for black images. This demonstrates the robustness of our model in handling diverse contrastive inputs. The performance degradation of our SymMPO with black images may stem from their lack of visual information, which results in non-meaningful target responses. With insufficient meaningful signals, SymMPO struggles to perform effective preference optimization. In contrast, mDPO only relies on the varying visual inputs with the same response of the original image for visual understanding enhancement, making it less susceptible to this limitation. (2) Using Similar, Noisy or Synthetic images, demonstrates superior performance over using Black and Cropped images. This possible explanation that the former three types of images preserve semantic similarities with the original images better, enabling more effective visual contrast through symmetric comparison. In contrast, black images suffer from severe information deficiency. while cropped images risk climinating key visual elements relevant to the prompt, both factors weakening their effects visual understanding enhancement.
Limitation
Despite its promising performance in mitigating hallucination of MILMs, SymMPO has two limita- tions. (1) Due constraints our cost-effecttive and efficient preference data construction pipeline, SymMPO's performance on tasks that involve fine-graine visual understanding (e.g., detailed im- age description generation) remains limited (discussed in Section 4.3). (2) SymMPO introduces additional computational overhead as it requires the construction of preferred response for each contrastive image. Although we have designed a cost-effective caption-anchores response preference pair construction pipeline for reducing the computational cost, acknowledge it as limitation.

---

6 Conclusion
In this paper, we propose Symmetric Multimodal Preference Optimization (SymMPO), which effectively addresses the limitations of non-rigorous objective function and indirect preference supervision in existing vision-enhanced multimodal preference optimization methods. The key novelty lies in the symmetric pair-wise preference optimization formulation and preference margin consistency regularization. Extensive experiments conducted across five multimodal benchmarks validate the superiority of our SymMPO framework, while ablation studies confirm the necessity of each key component within the framework. Additional comparisons on contrastive image types highlight the robustness of our model with diverse types.
Acknowledgments and Disclosure of Funding
This research is supported by the National Natural Science Foundation of China (Grants No. 62376137 and 62172261) and the Shandong Provincial Natural Science Foundation (Grant No. ZR2022YQU5 for which we express our heartfelt gratitude. We also deeply appreciate the support and assistance provided by the il eear Laboratory at Shandong University during the course of this research.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:23333087774 2023.
[2] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 2023.
[3] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892- 34916, 2023.
[4] OpenAl. Gpt-4v (ision) system card. 2023.
[5] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint urXiv:2309.16609 2023.
[6] Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1135-18143) 2024.
[7] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-languag models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 14375 14385, 2024.
[8] Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.1 18930, 2024.
[9] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36:53728 0075522--3771,, 2023.
[10] Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han. Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13807-13816, 2024.
[11] Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. Rlaif-v: Aligning milms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405. 2024.
10

---

[12] Mengxi Zhang, Wenhao Wu. Yu Lu, Yuxin Song. Kang Rong, Huanjin Yao, Jianbo Zhao, Fanglong Liu, Haocheng Feng, Jingdong Wang, et al. Automated multi-level preference for milms. In The Thirtt-eighth Annual Conference on Neural Information Processing Systems, 2024.
[13] Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucinations: Enhancing Ivlms through hallucination-awaan direct preference optimization. arXiv preprint arXiv:2311 16839, 2023.
[14] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprini arXiv:2309. /4525, 2023.
[15] Fei Wang, Wenxuan Zhou, James Y Huang. Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. mdpo: Conditional preference optimization for multimodal large language models. arXiv preprint arXiv:2406. 1839, 2024.
[16] Yuxi Xie, Guanzhen Li, Xiao Xu, and Min- Yen Kan. V-dpo: Mitigating hallucination in large vision language models via vision-guided direct preference optimization. arXiv preprint arXiv:2411.02712 2024.
[17] Songtao Jiang, Yan Zhang, Ruizhe Chen, Yeying Jin, and Zuozhu Liu. Modality-fair preference optimization for trustworthy milm alignment. arXiv preprint arXiv:2410./5334, 2024.
[18] Jinlan Fu, Shenzhen Huangfu, Hao Fei, Xiaoyu Shen, Bryan Hooi, Xipeng Qiu, and See-Kiong Ng. Chip: Cross-modal hierarchical direct preference optimization for multimodal Ilms. arXiv preprint arXiv:2501. 16629, 2025.
[19] Zhihe Yang, Xufang Luo, Dongqi Han, Yunjian Xu, and Dongsheng Li. Mitigating halluci- nations in large vision-language models via dpo: On-policy data hold the key. arXiv preprint arXiv:2501.0969 2025.
[20] Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Improved contrastive divergence training of energy based models. arXiv preprint arXiv:2012.01311 2020.
[21] Nisan Stiemnon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning summarize with human feedback. Advances in neural information processing systems, 33:3008-380 -021, 2020.
[22] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27144, 2022.
[23] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1XX.v:/777.064// 2017.
[24] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
[25] Hamish Ivison, Yizhong Wang. Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A Smith, Yejin Choi, and Hanna Hajishirzi. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. Advances neural information processing systems, 37:36602-36633, 2024.
[26] Lehan He, Zeren Chen, Zhelun Shi, Tianyu Yu, Jing Shao, and Lu Sheng. A topic-level self- correctional approach to mitigate hallucinations in milms. arXiv preprint 2..i:2411.112666 2024.
[27] Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEFE conference on computer vision and puttern recognition, pages 5904-6913 2017.
11

---

[28] Tsung- Yi Lin, Michael Maire, Serge Belongie. James Hays, Pietro Perona, Deva Ramanan, Piotr Dollär, and C Lawrence Zitnick. Microsoftt coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 740-755. Springer, 2014.
[29] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317-8326, 2019.
[30] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 23021:250..13923, 2025.
[31] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing rision-languan model's perception of the world at any resolution. arXiv preprint arXiv:2409. 2024.
[32] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 2443:77712.19437, 2024.
[33] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8763. PmLR, 2021.
[34] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. arXiv preprint 1xXivv8809.022 2018.
[35] Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Zhang, and Jitao Sang. An I1m-free multi-dimensional benchmark for milms hallucination evaluation. CoRR, 2023.
[36] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we the right way for evaluating large vision-languag models? arXiv preprint arXiv:2403.203: 20330, 2024.
[37] Tianyu Yu, Jinyi Hu, Yuan Yao. Haoye Zhang, Yue Zhao, Chongyi Wang, Shan Wang, Yinxv Pan, Jiao Xue, Dahai Li, et al. Reformulating vision-language foundation models and datasets towards universal multimodal assistants. arXiv preprint arXiv:23 100.003, 2023.
[38] Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modali- ties in vision large language models via preference fine-tuning. arXiv preprint 241, 2024.
[39] Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan Ô Ankk, and Tomas Pfister. Data-augmented perase-level alignment for mitigating object hallucination. arXiv preprint arXiv.2405.18654, 2024.
[40] Wenyi Xiao, Ziwei Huang, Leilei Gan, Wanggui He, Haoyuan Li, Zhelun Yu, Fangxun Shu, Hao Jiang, and Linchao Zhu. Detecting and mitigating hallucination in large vision language models via fine-graime ai feedback. arXiv preprint arXiv:2404.14233, 2024.
[41] Black Forest Labs. Flux. http:p/p////tttto.oo/oooooooo/eo
howpaaboooobbccccooeeet------- labs/fuux 2024.
[42] Yassine Ouali, Adrian Bulat, Brais Martinez, and Georgios Tzimiropoulos. Clip-dpo: Vision- language models as source of preference for fixing hallucinations in Ivlms. In European Conference on Computer Vision, pages 395-413 Springer, 2024.
[43] Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang, Runtao Liu, Rui Pan, and Tong Zhang. Strengthening multimodal large language model with bootstrapped preference optimization. In European Conference on Computer Vision, pages 382-398. Springer, 2024.
12

---

[44] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li. David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
[45] Weiyun Wang. Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, et al. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:241 1.14422, 2024.
[46] Ziyu Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Haodong Duan, Conghui He, Yuanjun Xiong. Dahua Lin, and Jiaqi Wang. Mia-dpo: Multi-image augmented direct preference optimization for large vision-language models. arXiv preprini arXiv:24 0. 7637,, 2024.
[47] Hongrui Jia, Chaoya Jiang, Haiyang Xu, Wei Ye, Mengfan Dong, Ming Yan, Ji Zhang, Fei Huang, and Shikun Zhang. Symdpo: Boosting in-context learning of large multimodal models with symbol demonstration direct preference optimization. arXiv preprint arXiv:241 2024.
[48] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11461-11471, 2022.
[49] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
13

---

A Related Work
Existing approaches that apply DPO to align MLLMs fundamentally involves two components: (1) preference data construction and (2) optimization strategy. We accordinggy present current multimodal DPO research from these two perspectives.
A.1 Preference Data Construction
Exiting preference data construction approaches primarily fall into two categories: (1) comparative ranking-based methods and (2) hallucination correction- based methods.
The first category constructs preference data by directly evaluating the rankings of candidate responses. For example, CLIP-DPO [2 uses CLIP's s image-text similarity metrics to rank candidate responses and selects pairs with large ranking gaps and similar lengths preference data. Beyond this, AMP [12 leverages MLLMs of varying scales to generate multi-level preference data, assuming that responses from larger MLLMs should rank higher. Furthermore, BPO 43) distorts the input image and pairs the response generated from the distorted image (ranked lower) with the response from the original image (ranked higher) create preference pairs. To eliminate confounding factors such as text style that hinder the model's ability to discern genuine trustworthiness differences within response pairs, RLAIF-V (IT] introduces deconfounded candidate response generation strategy. This strategy generates candidate responses through multiple sampling decoding trials with different random seeds while keeping the input prompt and decoding parameters constant.
The second category constructs preference data by detecting and correcting hallucinated content in MLLM responses. For example, RLHF-V [00] relies human annotators to identify and rectify hallucinatory content, ensuring the creation of higher-quality preference datasets. In contrast, HA-
DPO 13 bypasses human annotators by leveraging GPT-4 [[] to detect and correct hallucination using fine-grained visual annotations from the Visual Genome dataset [44). Similarly, OPA-DPO [19 employs GPT-4V
4 rectify hallucination but identifies them through direct image pair comparisons, eliminating the need for fine-grained annotations and offering greater flexibility. To reduce the high API costs associated with commercial LLMs, HSA-DPO [40) trains a specialized hallucination detection model using sentence-level annotation dataset generated by GPT. Moreover, building on RLAIF-V [T]. TPO 26 introduces topic-level self-correction paradigm. This method works on identifying hallucination at the topic level through sub-sentence clustering, constructing topic-level preference pairs, and generating response preference pairs using a deconfounded ooprc-iver enneuiinn linguistic style consistency.
A.2 Optimization Strategies
Apart from enhancing preference data construction, researchers have also explored various optimiza- tion strategies to improve MLLM preference alignment. For example, MPO [45) introduces a mixed preference optimization model, which effectively combines preference optimization techniques and conventional supervised fine-tuning. To mitigate the hallucination of MLLMs, AMP [12 introduces a multi-level direct preference optimization algorithm, enabling robust multi-level preference learning, while CHiP [18] introduces a cross-modal hierarchical DPO model involving two key optimiza- tions: hierarchical textual preference optimization for capturing fine-graing ane-grained textual preferences and visual preference optimization for cross-modal preference alignment. To address the gradient vanishing problem induced by off-policy data, OPA-DPO 19 proposes an adaptive mechanism that dynamically balances exploration and exploitation during learning. Furthermore, MIA-DPO [46] specifically targets hallucination reduction in multi-image scenarios, where MLLMs process multiple input images simultancously through optimized cross-image attention mechanisms.
One key issue in improving MLLMs' reasoning ability is mitigating their over-reliance on textual prompts while enhancing visual content utilization. To address this issue, SymDPO 47 introduces a symbol demonstration direct preference optimization model for in-context learning. which strengthens MLLMs' visual understanding by replacing textual answers with random symbols, thereby forcing MLLMs to establish mappings between visual information and symbolic responses. For more general multimodal understanding contexts, several studies [15] 16. 17 [8 19 focused on extending DPO with vision-oriented preference learning to improve MILMs* visual signal interpretation. These approaches preserve DPO's structural framework while only introducing visual variation in the
14

---

contrastive image-prompt- prompt-response triplet pairs. For contrastive imgge generation, existing methods employ diverse strategies: mDPO [T5 applies geometric transformations to original images, V-
DPO 16] employs a generative model to replace key visual elements through image inpainting [48]
MFPO 17 OPA-DPO [19] and CHiP [18] perform noise injection original images, while
CHiP
18 utilizes forward diffusion process 49 for generating contrastive images.
Although current vision-enhanced multimodal preference alignment methods have demonstrated great progress in reducing MLLM hallucination, they exhibit two critical limitations: (1) their DPO-based objective function derivations lack mathematical rigor, as they overlook the fact that the intractable partition functions in multimodal scenarios with different vision inputs cannot be directly canceled; and (2) they fail to provide direct preference supervision for DPO-based visual understanding en- hancement. To address these limitations, we propose Symmetric Multimodal Preference Optimization, which effectively utilize the corresponding preferred responses of contrastive images for optimizing the visual understanding capabilities of MLLMs.
B Impact of Partition Function in Multimodal Preference Optimization
Due to space limitations, Section I brieffy argues that existing multimodal DPO methods 15 16
18 [9 non-rigorously ignore two partition functions their vision-oriented contrastive learning mechanisms. Here, we present detailed analysis of these functions' roles in model optimization.
Specifically, according to the standard DPO formulation (Equation can be expressed as follows:
3). the implicit reward of MLLMs
r(m, x. v) 3 :3g To(y)m, 2)
Blog Z(m, x),
Tre/ (y)
(12)
Z(m, ((m,x)=)
Σ
Trey (3)m, x) exp (g pm
where m, x, and y denote the input image, textual prompt, and corresponding response, respectively. Tref. and are the policy model, reference model, and implicit reward function, respectively. (m,,2) 2) the partition function for the multimodal scenario, derived by incorporating an image variable into the single-modal partition function defined by standard DPO [9].
Adopting the mainstream vision-oriented preference learning paradigm that introduces additional contrastive images for preference alignment, we derive the following loss function by substituting the above implicit reward formulation into the Bradley- Terry model (Equation T.
Cyco =
(a (l. log a
(egoor(mm (r(m x, yn:) - r(m,, x. Bw))
To To (mo, - 33 log
(yar (m,, xr) T (Br Mw x)
log a (og
(13)
Tree (Brel Mm
+3 log (m,, x) - B log Z (mm,,
Existing methods directly cancel out (m x)) and Z ((m, 2)) in Equation 6 which is apparently inappropriate based on the above rigorous derivation.
To better understand the role of partition functions in the preference learning process, we calculate the gradient of Lico with respect to f. To facilitate the gradient calculation, for each data instance sampled during training, (z, mw, my, yw) ~ D. we first define:
no (Bur (M x) B log ( (o my, x))
B log
Tres yyw mm x) log (m x) - 3 log Z (m,,
Tre (Nw mi, x)
where u involves the policy model optimization, while e does not. In fact, remains constant across different policy model parameters 0 because [(m, x)) is independent of 0 in its calculation.
15

---

Table 3: Comparison of different learning rates on the performance of SymMPO
HallusionBench
Object-HalBenne
MMHal-Benn
AMBER
MMStar
SymMPO
qAcct
fAcct
aAcc1
Resp-4
Ment..
Scoret
Hall
Acct
Fl1
Overall
1r=5e-5
8.13
10.40
41.09
22.6
13.9
2.29
54.2
80.8
86.1
28.2
1r=Se-6
7.25
13.58
44.28
19.5
9.7
2.89
42.7
82.6
87.7
34,8
Ir=5e-7
8.13
12.42
43.75
20.1
9.8
2.80
49.0
80.8
86.8
33.8
Table Comparison of different on the performance SymMPO
HallusionBench
Object-HalBench
MMflal-Bench
AMBER
MMStar
SymMPO
qAcct
fAcet
aAcct
Resp.p
Ment.
Scoret
Hally.
Acct
FI1
Overall
A=
5.27
10.98
41.27
21.2
12.0
2.61
50.0
80.6
86.4
34.2
0-3.
7.47
10.98
42.60
19.7
10.5
3.07
37.5
81.5
86.7
34.6
A=0.5
7.25
13.58
44.28
19.5
9.7
2.89
42.7
82.6
87.7
34.8
>=0.7
5.93
12.71
41.36
18.1
44.8
82.6
87.7
4=09
7.47
10.98
41.98
21.2
10.7
2.76
46.9
81.8
87.2
34.2
Accordingly, the gradient of Lico with respect 0 is then given by:
aLvco
aLico. du
00
Ou 08
Ou
log (u + c)
Ou
00
(14)
8u
-(1 -oo(u + c))
00
Ou
- (u (u+
00
Similarly, we can derive the gradient of the visual-oriented contrastive objective used in existing
works (i.e., Equation 6 as:
Ou aLvco --(- 00
(15)
00
By comparing the gradients in Equations 14 and 15| we observe that the constant in Equations acts as offset modulating the coefficient of the gradient term 8y Intuitively, since (m, integrates over all possible responses y. it inherently captures the global quality of the response space conditioned xm, Thus, reflects the reference model's global response quality discrepancy between the contexts ax), and (mı, Specifically, larger value of indicates a stronger discrepancy in model behavior between (Mor., .7) and (m, exerting greater inffuence on gradient adjustment for preference alignment optimization. Conversely, a smaller value of implies weaker response quality gap between the multimodal context pair, less impacting the gradient update during training. However, existing methods neglect the two partition functions in visual-oriente optimization. essence, these methods assign the same value (i.e., 0) to all contrastive samples, preventing adaptive weight adjustment for different image contexts. Consequently, the model fails to achieve optimal visual understanding capability, as tends to either wer-at images with limited informative signals or under-reason about images containing complex visual clues.
contrastive
C Additional Ablation Studies
In this section, we detailed the process of selecting hyper- papeeeerrsseters for our experiments.
8, and 8, we adopt the same setup as prior works. Specifically, / is directly set to 0.1, while 7) and are chosen to 1.0 and 0, respectively, in alignment with the configurations used in mDPO and OPA-DPO. For the other hyper-parameters, including the learning rate, À and 7, we determine their values through a grid search. The search is conducted over the following ranges: the learning rate is evaluated [5e-5, 5e-6, 5e-7], À is tested across 10. 0.3,
For the
16

---

Table 5: Comparison of different on the performance of SymMPO
HallusionBench
Object-HalBende
MMHal-Benn
AMBER
MMStar
SymMPO
qAcct
fAcct
aAcc
Resp-4
Ment.
Scoret
Hall
Acct
FI1
Overall
>=1e-2
4.39
10.40
40.83
24.4
15.2
2.15
58.3
78.9
83.5
34.9
>=le-3
6.59
10.40
43.13
21.0
2.70
49.0
83.1
87.6
34,9
1=
7.25
13,58
44.28
19.5
9.7
2.89
42.7
82.6
87.7
34.8
>=1e-5
6.37
10.98
43.75
21.0
10.4
2.70
47.9
82.4
87.8
33.2
0.5, 0.7, 0.9], and ạ is explored within [le-2, le-3, le-4, le-5]. The results from this grid search are presented in Table Table and Table respectively, with all experiments conducted using -LaVA-1.5-7B. Based on these results, the final hyper-paramete values chosen for our experiments are as follows: a learning rate of 5e-6, x=0.5, and y=le-4.
Compared the learning and X, the hyper-parameter v, which governs preference margin consistency regularization, is notably smaller. This may be attributed to the subtle and sensitive nature of relative relationships between preferences across contrastive images. As result, smaller values lead to weaker regularization of preference margin consistency, limiting the effectiveness of SymMPO. Conversely, larger ? values impose excessively strong regularization, which can disrupt preference learning. These observations highlight the critical need to carefully balance the regularization strength achieve optimal performance.
D Case Study
In this part, we present the case study on our constructed preference pair data for model training and generated responses for model evaluation.
On Preference Pair Construction. As illustrated in Figure | each training sample for our symmetric preference optimization comprises two triplets sharing the same prompt and similar but distinct image- response pairs. Empirically, the contrastive images are identified using CLIP similarity scores, while the responses are generated using the aaption-anchored response preference pair construction pipeline described in Section
for corresponding images given the same prompt. Figure illustrates two training examples from our constructed preference dataset, where claims each response that contradict (and are thus potentially hallucinated relative to) the paired response are highlighted in red. These two examples demonstrate that the effectiveness of using CLIP similarity to obtain contrastive image pairs with subtle visual differences, and confirm that the corresponding responses generated by our preference data construction pipeline indeed exhibit strong linguistic alignment with only minor claim variations. Overall, these examples show the effectiveness of our training preference pair construction strategy.
On Hallucination Mitigation. To demonstrate the advantages of SymMPO, we compare the specific responses generated by original LLaVA-1.5-7B and our SymMPO-enhanced version with samples from two widely used benchmarks, AMHallBennh and feature diverse visual questions closely related daily life. Figure and Figure 7 present the corresponding results, with hallucinated content highlighted in red. As can be seen, LLaVA-1.5. 7B+SymMPO consistently generates more accurate responses than the original LLa across both benchmarks. Even for visual questions requiring detailed responses in Object-HalBench, L.La VA- SyMMPPO maintains precise, hallucination--f producing responses with hallucinations. These results confirm the effectiveness of our SymMPO in enhancing Dision-languse understanding and mitigating MLLM hallucinations.
Both benchmarks
responses, while LLL fails by
17

---

Case 1
Prompt: How do these people know each other?
Original Image
Contrastive Image
Response (For Original Image): These people know each other they are family, consisting of two men and woman, who have gathered around a wooden dining table to share meal and spend quality time together. They are sitting three different chairs (green, yellow-green, and red), and the table is set with various items such as plates, bowls, knives, spoons, glasses, and condiments/botttle.. This indicates that they are enjoying dinner together, which is common activity for families to bond and strengthen their relationships.
Response 2 (For Contrastive Image): The group of people in the image, consisting of two men, woman, and child, are likely related or close friends, as they are gathered together for a meal. They are sitting around a dining table with two dark colored bottles. The presence of food and drinks, coupled with the comfortable seating arrangement, indicates that they are part of social gathering or celebration. It is likely that they are enjoying each other's company and sharing meal, which is a common way to bond and strengthen relationships.
Case 2
Prompt: What are the main objects or subjects in the image?
Original Image
Contrastive Image
Response (For Original Image): The main objects or subjects in the image are dense forest with tall, leafy trees featuring vibrant green foliage, suggesting spring or summer. There rocky outcropping composed of layered ledges in light gray to tan hues, with distinct horizontal stratification and partial mass coverage. The ground is covered with grass, moss, and low-lying plants, along with patches of darker soil near the base of the rocks.
Response (For Contrastive Image): The main objects or subjects in the image are trees, fallen leaves, and decomposed log. The scene features forest with mostly slender to medium-sized trees, some with light-colored bark (possibly birch) and others with darker bark. The ground is covered with thick layer of brown and yellow leaves, indicating autumn, while some green foliage remains on the trees. Sparse shrubs and small plants are visible near the base of some trees. A fallen log lies horizontally in the ground, partially decomposed and blending into the natural environment. Soft, diffused sunlight filters through the canopy, creating dappled shadows the ground, enhancing the picturesque autumn setting.
Figure 5: Two training samples from our built preference dataset used optimize SymMPO, with hallucinated elements relative to the other response highlighted in red.
18

---

Case
Case 2
Case
What is the gender of the person riding the motorcycle in the image?
positioned yellow
being washed
yacht image?
LLaVA-1.5-7B
LLaVA-1.5-7B
LLaVA-1.5-78
person riding the motorcycle the image is woman.
toothbrush being washed in the sink.
The yellow boat positioned behind the yacht in the image.
LLVA--15577--SyMMPO
LLaVA-1.5-7B+555 SymMPP
LLaVA-1.5-8--SymMM
There is no person riding the matoreycle in the image. The motorcycle is parked the side the road.
There nothing being washed the sink.
The yellow bost positioned in front of the white yacht the image
Figure Responses generated by LLaVA- L-aVA-1.5-7B 1.5-7B and our SymMPO-enhanced version for data examples from MMHal-Benchh, with hallucinated contents highlighted in red.
Case
LLaVA-1.5-7B
image, He appears working on the possibly preparing use. The quite large and takes up significant portion of the scone. The man positioned on the left side of the boat, while the bost itself spans across the entire width of the image. There are few other people the background, but not work creftsmanship
standing
woaden
construction.
focus the Overall image captures moment wooden boat.
LLaVA-1.5-785 7---111--8--555 SymMPO
image process being standing wooden boat, which appears
wearing with a graphic design on chest and blue jeans, and seems to be actively engaged in the boat-building process.
repaired.
Provide thorough description given picture.
The wooden boat positioned horizontally on the ground, with the man standing be taking place
the left of it. The with vegetation visible background.
semi-rural setting,
Case
LLaVA-1.5-78
The depicts peaceful scene of horse and ponty ponry
in left of the field, while enjoying
green field.
positioned animals appear
right
pasture.
In addition to the horses, there are two people present in scene. One person standing near the left side of the field, while the other person located closer to the right side. They seem to be observing the horses they
LLaVA-1.5 SymMMO
The image depicts beautiful scene of two horses grazing horse is positioned in foreground, while the other slightly behind and the enjoying
field, One
surrounded by
greenery.
What is this photo about?
in the background, there are rolling hills and trees, adding to the picturesque setting. The sey is clear with scattered clouds, creating a sarone and peaceful atmosphere. The image captures moment of tranquiity and harmony between the horses and their natural surroundings
Please answer great detail.
Figure 7: Responses generated by LLaVA- 1.5-7B and SymMPO-enhanced version for data examples from Object- Hallech,, with hallucinated contents highlighted in red.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims presented in the abstract and introduction accurately reflect
the paper's 's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification: We present a detailed analysis of the impact of ignoring partition functions in multimodal preference optimization. For details, please refer Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have fully disclosed the details of our proposed SymDPO, including the objective function for model optimization in SectionßJand the preference data construction process in Section 4. Additionally, we provide all experimental settings in Section 4.2 To ensure reproducibility, we will release the synthetic data, source code, and comprehensive instructions.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
21

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We have provided access the code used for training SymMPO at
while the size (around 10GB) of our constructed preference data is too large be currently released via the above anonymous website. We will publicly release both the training data and code, along with sufficient instructions to ensure accurate reproduction of the experimental results, once the blind review process is completed.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/
mublic/guider httpg/dddde/ooeSummiiaaee
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
7 //nips.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have specified the training and evaluation setup in the experiment section, including hyperparameters, benchmarks used, and compute resources (see Section 4.2)
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: Our work did not conduct statistical significance tests, which is consistent with existing research this domain, as prior studies also do not incorporate such analyses.
Guidelines:
The answer NA means that the paper does not include experiments.
22

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the specific computation resources used for experiments in Sec- tion
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We strictly conform the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiciio))
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There no societal impact of our proposed algorithm as it's about theoretical innovation and contribution.
23

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., denloymemt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: We utilize publicly available datasets to construct our datasets, ensuring that no safety risks or misuse concerns are assiciated with its release.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have strictly adhered to the terms of use and licenses for all assets employed in this paper, including datasets, code, and models. Detailed information regarding asset usage and citations is provided in the Section
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Guidelines: The code will be released alongside this paper, accompanied by prehensive documentation that provides guidance on usage and implementation. De- tailed instructions, including training procedures and licensing, will be made available and in the supplementary materials.
at https://nnonymousssooooo www.::osssopppp.ccccncccecee Agpen.sciencer//SyMMP--B
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not have crowdsourcing experiment.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
25

---

Justification:
We do not have any participants.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality the research, declaration is not required.
Answer: [NA]
Justification:
We do not utilize LLMs in the development of any important or original core methods.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
wtapeiippsccccoofeee / LLMML
26


Reviews:
[{"review_id": "Ra7R8eCjCs", "reviewer": "Reviewer_okF2", "summary": "This paper presents a novel method, Symmetric Multimodal Preference Optimization (SymMPO), to mitigate hallucinations in Multimodal Large Language Models (MLLMs). It is a symmetric preference learning formulation. This method leverages contrastive image pairs (m, m') along with their respective preferred responses (yw, y'w) to provide direct preference supervision. This symmetric design not only enhances the model's ability to make fine-grained visual discriminations but also ensures the theoretical integrity of the DPO objective by aligning with its standard derivation. Furthermore, the method is augmented with a preference margin consistency regularizer to quantitatively constrain the preference gap between symmetric pairs, enforcing a more precise alignment.", "questions": "1. Whether other models can be used to enhance pair screening and acquisition (Besides CLIP).\n2. Which specific stage or component of the \"Caption-Anchored Claim Extraction-and-Rewriting\" pipeline (Section 4.1) is primarily responsible for Fine-Grained task performance deficit? Is the limitation rooted in the initial caption generation, which may lack fine-grained detail, or does it stem from the subsequent claim extraction and rewriting process?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "The paper proposed SymMPO framework is not merely an incremental improvement but an original and elegant solution that restores theoretical consistency with the foundational DPO loss. This demonstrates a high degree of methodological Quality and Originality.\nThe paper introduces a novel symmetric optimization strategy and a preference margin consistency regularizer (LMargin), which together provide a more direct and quantitative supervisory signal for visual understanding. \n\nThis work framework necessitates generating a unique preferred response for every contrastive image, which introduces additional computational complexity and cost compared to methods that reuse the original response. And it relies on CLIP similarity, but this heuristic may not always produce optimally challenging pairs. But it difficult to isolate the true contribution of the SymMPO algorithm itself versus the quality of the input data.", "was_revised": true, "final_justification": "The manuscript and the subsequent rebuttal provided detailed experiments and results, and the authors addressed my concerns. I look forward to further work in this direction.", "rebuttal": "Dear Reviewer okF2,\n\nThank you for recognizing our work. We greatly appreciate your insightful comments and suggestions. Below, we provide detailed answers to your questions:\n\n> **Question 1:** Whether other models can be used to enhance pair screening and acquisition (Besides CLIP).\n\n**Response:** \n\nThank you for this insightful question. Yes, since the primary objective of our contrastive image construction is to pair images that are visually similar but have subtle differences, any model capable of capturing fine-grained visual features and discriminating between nuanced differences is a potential candidate for this task.\n\nFor example, self-supervised vision models such as **DINO** [1] and **DINOv2** [2] could serve as highly effective alternatives to CLIP. While CLIP is designed to align visual and textual representations, models like DINOv2 are optimized particularly for extracting high-quality, fine-grained visual features through self-supervised learning. This specialization makes DINOv2 especially well-suited for identifying images with minor visual differences.\n\nAdditionally, advanced image segmentation models, such as **SAM (Segment Anything Model)** [3] can segment objects or regions within an image, enabling region-level feature extraction. These features could then be used to pair images with specific region-level similarities.\n\nWe acknowledge that while CLIP served as a robust and practical choice for our current implementation, investigating alternative feature extractors is an excellent suggestion for enhancing SymMPO. We will consider this direction in our future work.\n\n \n\n> **Question 2**: Which specific stage or component of the \"Caption-Anchored Claim Extraction-and-Rewriting\" pipeline (Section 4.1) is primarily responsible for Fine-Grained task performance deficit? Is the limitation rooted in the initial caption generation, which may lack fine-grained detail, or does it stem from the subsequent claim extraction and rewriting process?\n\n**Response:** \n\nThank you for your thoughtful question. You are correct that the performance deficit on fine-grained tasks **primarily stems from the initial caption generation stage**, which serves as the foundational step in our \"Caption-Anchored Claim Extraction-and-Rewriting\" pipeline.\n\nBased on our extensive experiments, we found the caption generation process, powered by the Qwen2.5-VL-32B model, exhibits two key limitations that hinder fine-grained task performance:\n\n1. Lack of Fine-Grained Detail: The prompt used in current pipeline is primarily designed to generate global scene descriptions, rather focusing on producing fine-grained details. As a result, the captions often miss critical nuances or specific information. This lack of detailed captions impacts the subsequent claim extraction process. As a result, the generated claims tend to be overly generic, lacking the specificity required for fine-grained tasks [4]. For example, if the model only describes the overall scene without recognizing detailed features such as the shape of a bird’s feet, wings or beak, it would struggle to accurately classify its species.\n2. Propagation of Hallucinations: Hallucination remains a significant challenge in current MLLMs. The Qwen2.5-VL-32B model sometimes generates captions containing **hallucinations**. In these instances, the subsequent claim extraction process may inadvertently incorporate these inaccuracies into claims, leading to distorted or erroneous preference data.\n\nIn summary, these limitations in the initial caption generation stage can trigger a **cascading effect** throughout the preference data construction pipeline, ultimately compromising the quality of the generated preference data, particularly for tasks requiring fine-grained recognition. In the future, we plan to explore solutions to address this challenge. \n\n \n\n**References:**\n\n[1] Caron, Mathilde, et al. \"Emerging properties in self-supervised vision transformers.\" ICCV 2021.\n\n[2] Oquab, Maxime, et al. \"Dinov2: Learning robust visual features without supervision.\" TMLR 2024.\n\n[3] Kirillov, Alexander, et al. \"Segment anything.\" ICCV 2023.\n\n[4] Yucheng Shi, et, al.; “Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Data”. ICLR 2025."}, {"review_id": "jrwACPn8rc", "reviewer": "Reviewer_PFAD", "summary": "This paper proposes SymMPO, a method designed to mitigate hallucination issues in VLMs. SymMPO improves upon existing approaches and constructs hard-negative contrastive image pairs for direct preference supervision, forming a more theoretically-rigorous optimization framework and effectively enhancing visual perception abilities of VLMs. The method incorporates symmetric preference learning and preference margin consistency regularization. Experimental results show that SymMPO outperforms previous methods across several hallucination benchmarks.", "questions": "1. How does the SymMPO approach compare with other methods in terms of training time and scalability, particularly when working with larger datasets or more complex multimodal inputs?\n\n2. I looked at the code provided in the anonymous repository, but it seems that it cannot be executed. Could the authors upload a runnable version of the code for reference?", "limitations": "Please refer to Questions.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Pros:**\n1. This motivation of this paper is reasonable and straightforward.\n2. The idea of the proposed SymMPO is intuitive and somewhat elegant, especially the design of the symmetric pairwise reward.\n3. The paper writing is good and easy to follow.\n4. Extensive results convincingly demonstrate the effectiveness of SymMPO and its various loss terms in reducing hallucinations.\n\n**Cons:**\n\n1. SymMPO introduces additional computational overhead compared to standard DPO, potentially increasing the computational cost by several times.\n2. The two limitations mentioned in the Introduction essentially refer to the same issue: the inherent conflict between the existing vision-oriented preference learning and the standard DPO framework. Could the authors further explore SymMPO, particularly in terms of whether aspects beyond pairwise rewards might help address existing limitations?", "was_revised": true, "final_justification": "The authors have provided further explanations for methodology design and more detailed guidance in code repository, which further enhances the soundness of this work. Considering the current quality of this paper, I have raised my rating to Accept.", "rebuttal": "Dear Reviewer PFAD,\n\nThank you for recognizing our work! We will address your questions as follows:\n\n> **Concern 1:** SymMPO introduces additional computational overhead compared to standard DPO, potentially increasing the computational cost by several times.\n>\n> **Question 1:** How does the SymMPO approach compare with other methods in terms of training time and scalability, particularly when working with larger datasets or more complex multimodal inputs?\n\n**Response:** \n\nThank you for raising this concern regarding the computational overhead of SymMPO. We fully understand the importance of evaluating training time and scalability, particularly when working with larger datasets or more complex multimodal input.\n\nTo address this directly, we conducted a direct comparison of training time of **DPO**, **mDPO**, and **SymMPO** using **DeepSpeed Zero-3** under the same experiment settings. For instance, we trained on a dataset consisting of **21.4k preference pairs** over **2 epochs**, with a uniform batch size of **64**. Due to computational resource limitations, these experiments were conducted on **2 NVIDIA A100-40G GPUs** **with CPU offload**, which is slower than the standard experimental conditions outlined in the paper. The training time comparisons are as follows:\n\n- DPO: 9.6 hours (52 seconds per iteration)\n- mDPO: 10.4 hours (56 seconds per iteration)\n- SymMPO: 13.0 hours (70 seconds per iteration)\n\nThe increased training time of SymMPO primarily stems from the additional computational overhead required for **symmetric sampling and optimization**. These operations were specifically designed to address limitations of existing methods, i.e., (1) Non-rigorous Objective Function and (2) Indirect Preference Supervision. As demonstrated in Table 1 of our paper, experimental results confirm that our approach achieves superior preference alignment. We remain committed to further exploring optimization techniques to enhance SymMPO's scalability while maintaining its theoretical rigor and effectiveness.\n\nThank you for raising this excellent concern. We will address it by including a detailed discussion and analysis of this issue in the revised manuscript.\n\n \n\n> **Concern 2:** The two limitations mentioned in the Introduction essentially refer to the same issue: the inherent conflict between the existing vision-oriented preference learning and the standard DPO framework. Could the authors further explore SymMPO, particularly in terms of whether aspects beyond pairwise rewards might help address existing limitations?\n\n**Response:** \n\nThank you for raising this insightful concern. While the two limitations mentioned in the Introduction both stem from challenges in aligning vision-oriented preference learning with the standard DPO framework, they address distinct aspects of this misalignment:\n\n1. **Non-rigorous Objective Function:** This limitation highlights the theoretical inconsistency due to mismatched partition functions when input pairs differ in visual content. The inability to cancel out the partition function results in optimization that deviates from the theoretical foundation of DPO.\n2. **Indirect Preference Supervision:** This limitation focuses on the supervision signal used in vision-oriented preference learning, where responses are indirectly supervised through visual input comparisons rather than directly optimized via response preferences. This indirect supervision reduces the effectiveness of preference optimization in enhancing multimodal understanding.\n\nSymMPO is specifically designed to address both issues. Through symmetric pairwise preference learning, it cancels out the partition function mismatch by pairing different visual inputs under the same textual context, ensuring theoretical consistency. Simultaneously, it restores direct preference supervision by leveraging response pairs as rejected responses, aligning with the original design intent of DPO.\n\nRegarding exploring aspects beyond pairwise rewards, SymMPO introduces the preference margin consistency regularization to quantitatively regulate the preference gap between symmetric pairs. This extends optimization beyond simple pairwise comparisons by imposing constraints on the magnitude of preference differences, encouraging more stable and nuanced learning.\n\n \n\n> **Question 2:** I looked at the code provided in the anonymous repository, but it seems that it cannot be executed. Could the authors upload a runnable version of the code for reference?\n\n**Response:** \n\nThank you for bringing this to our attention. We sincerely apologize for the inconvenience you experienced while attempting to execute our code.\n\nTo address this, we have thoroughly reviewed and updated the anonymous repository to ensure its usability. Specifically:\n\n1. The README.md file has been revised to include clear, step-by-step instructions for setting up the environment, installing dependencies, and running the code.\n2. We have added a **small demo dataset** alongside a corresponding script, allowing users to quickly validate the core functionality of our pipeline.\n\nUnfortunately, due to storage limitations of Github, we were unable to include the full training dataset. However, we are committed to ensuring full reproducibility. Upon acceptance of the paper, we will publicly release the complete dataset, all model weights, and any remaining resources needed to replicate our results.\n\nWe hope this update resolves the issue, and we appreciate your understanding and feedback on this matter. Please do not hesitate to reach out if further issues arise."}, {"review_id": "wun33sbwT8", "reviewer": "Reviewer_8fTk", "summary": "This paper tackles the critical issue of hallucinations in Large Vision-Language Models (LVLMs), particularly semantic hallucinations where generated responses include entities unsupported by the visual input. The authors propose a novel framework consisting of two main components. The first is MPD (Masked Prediction Difference), which extracts hallucination-prone components in the decoding process by measuring prediction divergence. The second is Selective Gradient Detachment (SGD), a decoding-phase intervention strategy that suppresses gradients from hallucination components without altering model parameters. The method is model-agnostic, training-free, and designed to maintain overall task performance while reducing hallucinations. Extensive experiments on multiple benchmarks demonstrate the effectiveness of the proposed method, with significant hallucination reduction and minimal performance degradation.", "questions": "**1.Lack of Clarity in Problem Formulation:**\nThe paper raises two theoretical limitations of existing vision-oriented preference optimization methods, but the explanation is abstract and lacks intuitive grounding. The impact of issues like mismatched partition functions or fixed-response supervision is unclear without concrete examples. Clarifying these points in plain terms with illustrative scenarios would make the problem statement more accessible and compelling.\n\n**2.Missing Evaluation on Standard Benchmarks:**\nThe method is not tested on widely used LVLM benchmarks such as MMBench, MME, POPE, SEED, or MM-Vet. Without results on these datasets, it is difficult to assess the method’s general effectiveness or practical relevance across diverse tasks.\n\n**3.Limited Model Generalization Evidence:**\nAll experiments are conducted on LLaVA, with no evaluation on other strong models like DeepSeek or MiniGPT-4. This narrow scope raises concerns about overfitting and limits claims of generalizability. Broader model-level validation is needed to support the method’s robustness.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength:**\n\n**1.Motivation:**\nThe paper provides a strong and timely motivation by identifying hallucination as a major limitation in the deployment of LVLMs. The proposed strategy aligns well with practical needs by avoiding additional training or fine-tuning.\n\n**2.Novelty:**\nThe MPD-based extraction of hallucination-inducing components is a fresh perspective. Previous methods often focused on architectural modifications or training-based approaches. The proposed combination of MPD and SGD offers a novel, lightweight inference-time mitigation mechanism.\n\n**3.Writing and Presentation:**\nThe paper is generally well-written, with clear mathematical formulations and well-illustrated figures. The intuition behind the method is conveyed effectively, which helps readers understand the internal mechanics.\n\n**Weakness:**\n\n**1.Lack of Clarity in Problem Formulation:**\nThe authors argue that existing vision-oriented preference optimization methods suffer from two key limitations: (1) a non-rigorous objective function due to the inability to cancel out intractable partition functions when image inputs differ (i.e., $m_w \\ne m_l$)), violating the assumptions of DPO; and (2) indirect preference supervision arising from the use of contrastive images with fixed responses rather than contrastive responses. While these points are theoretically motivated, their practical implications remain obscure due to abstract terminology and limited explanation. Clarifying these limitations with intuitive language and a concrete example would significantly improve the accessibility and persuasiveness of the argument.\n\n**2. Insufficient Evaluation on Standard Benchmarks:**\nThe paper does not report performance on any widely adopted LVLM benchmarks such as MMBench, MME, POPE, SEED, or MM-Vet. This omission significantly limits the credibility of the evaluation, as it is unclear whether the proposed method maintains its advantages across diverse tasks and settings. Without such results, the general effectiveness and real-world utility of the approach remain unconvincing.\n\n**3. Weak Evidence of Generalization Across Models:**\nAll experiments are restricted to the LLaVA model, offering limited insight into how well the method generalizes across architectures. This narrow experimental setup raises concerns about overfitting to a specific model. To establish robustness and broader applicability, it is essential to include results on additional strong baselines such as DeepSeek and MiniGPT-4. The absence of such evidence substantially weakens the empirical contribution of the paper.", "was_revised": true, "final_justification": "This rebuttal has successfully addressed my concerns. The authors have provided clear and detailed explanations for the theoretical issues, showing how their proposed method, SymMPO, overcomes them. The explanations for both the mismatched partition function and indirect preference supervision are now much more intuitive and grounded.\n\nFurthermore, the authors have effectively justified their evaluation strategy. By focusing on a comprehensive set of hallucination-centric benchmarks and comparing their approach to prior work in this specific domain, they have demonstrated that their evaluation is both thorough and aligned with the paper's primary objective. The commitment to including other standard benchmarks in future work is also reassuring.\n\nFinally, the authors' reasoning for their choice of the LLaVA model for experiments is logical. They have correctly pointed out that using a widely adopted and representative architecture ensures direct comparability with other established studies. Their plan to extend these experiments to other models in the future further strengthens their argument.\n\nTherefore, I am maintaining my original rating of borderline accept. The authors' rebuttal has satisfactorily resolved the weaknesses I identified in my initial review.", "rebuttal": "Dear Reviewer 8fTK,\n\nThank you for your feedback. We provide our responses to your questions as following.\n\n> **Question 1:** **Lack of Clarity in Problem Formulation:** The paper raises two theoretical limitations of existing vision-oriented preference optimization methods, but the explanation is abstract and lacks intuitive grounding. The impact of issues like mismatched partition functions or fixed-response supervision is unclear without concrete examples. Clarifying these points in plain terms with illustrative scenarios would make the problem statement more accessible and compelling.\n\n**Response:**\n\nWe sincerely thank you for your critical feedback. We acknowledge the importance of providing an intuitive explanation of these theoretical issues discussed. While these theoretical aspects are challenging to quantify, we will strive to elucidate these limitations with clearer and more comprehensible language.\n\n1. **Mismatched Partition Function**\n\nIn the context of multimodal scenarios, the partition function $Z(m,x)$ in DPO is defined as $\\sum_y\\pi_{ref}(y\\vert m,x)\\exp(\\frac{1}{\\beta}r(m,x,y))$. This term integrates over all possible responses y given the input image $m$ and text $x$, and captures the overall quality of the response space conditioned on $(m,x)$. However, since $Z(m,x)$ requires computation over all possible responses, it is intractable. DPO resolves this by constructing preference data pairs, e.g., $(m,x,y_w)$ and $(m,x,y_l)$, and subtracting the implicit rewards of these pairs. This subtraction between the implicit reward of the chosen response $y_w$ and the implicit reward of the rejected response $y_l$ cancels out the shared partition function $Z(m,x)$.\n\nIn vision-oriented preference optimization approaches, the preference data pairs are constructed with different visual inputs $m_w$ and $m_l$, while keeping the response $y_w$ identical. The goal is to encourage the model to better understand and prioritize visual content. However, the partition functions $Z(m_w,x)$ and $Z(m_l,x)$ differ due to the distinct visual inputs, i.e., $m_w$ and $m_l$. **Consequently, these terms cannot be directly canceled out as in standard DPO.** Prior studies have overlooked this issue, failing to account for the influence of the partition function and leading to inconsistencies between the theoretical formulation and practical implementation. This mismatch results in suboptimal optimization outcomes.\n\nIn contrast, our proposed Symmetric Multimodal Preference Optimization (SymMPO) addresses this issue by pairing responses corresponding to different images under the same textual input as each other’s rejected responses. By leveraging identical multimodal inputs across paired data, SymMPO naturally cancels out the partition function terms, ensuring consistency between theory and implementation.\n\n2. **Indirect Preference Supervision**\n\nStandard multimodal DPO optimizes preference by leveraging triplets $(m,x,y_w,y_l)$, where the model is guided to generate $y_w$ over $y_l$ given the image $m$ and text $x$. In vision-oriented preference optimization methods, preference pairs are constructed with different visual inputs $m_w$ and $m_l$, but identical responses $y_w$. The intention is to improve the model’s focus on visual content through comparative learning. **However, this approach deviates from the original design of DPO, which relies on response-based supervision signals to directly enhance model performance.** As a result, these methods struggle to effectively improve the model’s visual understanding capabilities.\n\nIn comparison, SymMPO adopts the same direct supervision strategy as original DPO. By treating responses associated with different images under the same textual input as symmetric preference pairs, SymMPO achieves direct preference supervision for vision-oriented optimization. This ensures alignment with the theoretical foundation of DPO while effectively enhancing the model’s preference optimization across varying visual inputs.\n\n \n\n> **Question 2:** **Missing Evaluation on Standard Benchmarks:** The method is not tested on widely used LVLM benchmarks such as MMBench, MME, POPE, SEED, or MM-Vet. Without results on these datasets, it is difficult to assess the method’s general effectiveness or practical relevance across diverse tasks.\n\n**Response:**\n\nThank you for this suggestion regarding evaluation on standard benchmarks.\n\nOur primary research goal with SymMPO is to specifically address and mitigate multimodal hallucination. Consequently, we focused on leading hallucination-centric benchmarks，including **HallusionBench**, **Object-HalBench**, **MMHal-Bench** and **AMBER**, which have been widely adopted in prior work [1-6] aimed at **mitigating MLLM hallucination through DPO**. Additionally, we evaluated our method on **MMStar**, which assesses the model’s ability to combine visual information with textual reasoning to answer questions correctly. This aligns closely with our goal of improving joint vision-language understanding while reducing hallucination.\n\nThe benchmarks you suggested can be categorized as follows:\n\n**MMBench, SEED, and MM-Vet** serve as excellent and widely recognized benchmarks for assessing **general multimodal understanding** in MLLMs rather than hallucination issues. To the best of our knowledge, these benchmarks have not been utilized in prior studies related to hallucination mitigation.\n\n**MME and POPE**, in contrast, are highly relevant benchmarks for evaluating **object-level hallucination**. However, since these benchmarks are less frequently adopted by prior studies in this domain, we prioritized using five widely adopted benchmarks in our current experiments. Below, we provide a comparative table summarizing benchmark usage in existing work versus our work.\n\n| Model       | HallusionBench | ObjHal       | MMHal        | AMBER        | MME  | POPE         |\n| ----------- | -------------- | ------------ | ------------ | ------------ | ---- | ------------ |\n| mDPO [1]    |                | $\\checkmark$ | $\\checkmark$ | $\\checkmark$ |      |              |\n| TPO [2]     |                | $\\checkmark$ | $\\checkmark$ | $\\checkmark$ |      |              |\n| RLAIF-V [3] |                | $\\checkmark$ | $\\checkmark$ | $\\checkmark$ |      |              |\n| OPA-DPO [4] |                | $\\checkmark$ | $\\checkmark$ | $\\checkmark$ |      | $\\checkmark$ |\n| CHiP [5]    | $\\checkmark$   | $\\checkmark$ | $\\checkmark$ | $\\checkmark$ |      |              |\n| HALVA [6]   | $\\checkmark$   |              | $\\checkmark$ | $\\checkmark$ |      |              |\n| SymMPO      | $\\checkmark$   | $\\checkmark$ | $\\checkmark$ | $\\checkmark$ |      |              |\n\n \n\nAs shown, most of existing work only employed three hallucination-centric benchmarks, whereas ours incorporates four. We believe this more comprehensive evaluation setup better validates our model's generalization capability. We fully recognize the relevance of these benchmarks, especially MME and POPE, and plan to include them in future evaluations to further demonstrate SymMPO's generalizability and effectiveness across a broader range of tasks and datasets.\n\n \n\n> **Question 3:** **Limited Model Generalization Evidence:** All experiments are conducted on LLaVA, with no evaluation on other strong models like DeepSeek or MiniGPT-4. This narrow scope raises concerns about overfitting and limits claims of generalizability. Broader model-level validation is needed to support the method’s robustness.\n\n**Response:**\n\nThanks for your question. We selected the LLaVA family for evaluation based on the following considerations:\n\nFirstly, many related studies on multimodal DPO [1-6] have consistently utilized the LLaVA family as their primary evaluation baseline. By centering our experiments on LLaVA, we ensure direct comparability with these established body of works. Notably, two recent CVPR 2025 and ICLR 2025 publications [4, 6] employed the same evaluation approach, focusing on the 7B and 13B LLaVA variants to assess model generalization capabilities.\n\nSecondly, the LLaVA family represents a foundational and widely-used class of MLLMs, characterized by a classic MLLM architecture (a vision encoder connected to a language model via a projection layer). Experimental results obtained from LLaVA are therefore more likely to generalize to other MLLM models. \n\nWe fully acknowledge the importance of demonstrating SymMPO’s effectiveness on structurally diverse MLLM families to strengthen claims of generalizability, and plan to explore them in future work.\n\n \n\n**References**\n\n[1] Fei Wang, et, al.; “mDPO: Conditional Preference Optimization for Multimodal Large Language Models”. EMNLP 2024.\n\n[2] He, Lehan, et al. \"A topic-level self-correctional approach to mitigate hallucinations in mllms.\" arXiv preprint arXiv:2411.17265 (2024).\n\n[3] Yu, Tianyu, et al. \"Rlaif-v: Aligning mllms through open-source ai feedback for super gpt-4v trustworthiness.\" CVPR 2025.\n\n[4] Yang, Zhihe, et al. \"Mitigating hallucinations in large vision-language models via dpo: On-policy data hold the key.\" CVPR 2025.\n\n[5] Fu, Jinlan, et al. \"Chip: Cross-modal hierarchical direct preference optimization for multimodal llms.\" ICLR 2025.\n\n[6] Sarkar, Pritam, et al. \"Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignment.\" ICLR 2025."}, {"review_id": "eiR7jHvmgG", "reviewer": "Reviewer_rbzU", "summary": "This paper introduces a novel DPO-based framework, Symmetric Multimodal Preference Optimization (SymMPO), to mitigate hallucinations in VLMs. It is specifically designed to construct preference pairs based on image information. The experiments are thorough and demonstrate the effectiveness of the proposed methods. Overall, I think it is good work that above the acceptance threshold of NeurIPS.\n\nI have some questions and suggestions that I hope the authors will take into consideration. I am willing to raise score if authors answer them appropriately.", "questions": "1. Could the authors provide more details about the \"contrastive image construction\"? Specifically, what does \"pair each image with its nearest neighbor based on cosine similarity\" mean? Does this imply that $m'$ is sourced from other images within the dataset? If so, what would happen if the dataset is small and no sufficiently similar image can be found?\n2. It appears that the rejected responses are generated by the model being trained, while the preferred responses are generated by DeepSeek-v3 (based on captions generated by Qwen2.5-VL-32B). If this is correct, the performance upper bound of the proposed method should theoretically align with Qwen2.5-VL-32B, right? In that case, it would be helpful if the authors could include the performance of Qwen2.5-VL-32B in Table 1 for comparison.\n3. Building on the previous question, if the preferred responses are generated by DeepSeek-v3, they are effectively off-policy and, according to the analysis in OPA-DPO, may be difficult to learn. What would happen if the authors first conducted SFT on these preferred responses? Would this improve the results? (I fully understand that time during the rebuttal period is limited, and I will not fault the authors if experimental results cannot be provided at this stage.)", "limitations": "The limitations are well-discussed within the paper. I suggest authors to discuss the positive societal impacts of the proposed methods.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. The paper is well-organized and easy to follow, with all notations clearly defined. I found the reading experience enjoyable and informative.\n2. The experiments are comprehensive and effectively demonstrate the proposed method's effectiveness.\n3. The methodology is specifically tailored for multimodal models, similar to mDPO, and I believe it provides valuable insights for the community.\n4. The theoretical component, while simple, is logical and adds meaningful support to the overall work.\n\n**Weaknesses:**\n\nI did not notice any significant weaknesses in this paper. Just one point, the coefficient-related hyperparameters seem a bit unusual, such as $\\gamma=1e-4$. I am curious about how these values were selected. Perhaps some ablation studies on this aspect would be helpful. If time is too limited to provide them now, I hope they can be included in the final version of the paper.\n\n**Some minor issues:**\n1. In Section 2.1, while the reward function trained using the Bradley-Terry model was first utilized by PPO-based RLHF algorithms, it was not originally proposed by PPO itself. The authors should cite [1, 2] and other relevant papers. Additionally, this section should be renamed, as \"RLHF with BT reward model\" is more directly related to DPO, compared with PPO.\n2. All practical loss functions in this paper should involve $\\mathbb{E}_{x,m,y \\sim...}$, as the prompt, image, and response are not constants but random variables that should be sampled during training.\n\n[1] Stiennon et al. Learning to summarize from human feedback. \n\n[2] Ouyang et al. Training language models to follow instructions with human feedback.", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer rbzU,\n\nThank you for recognizing the value of our paper, which is really encouraging to us. We also greatly appreciate your thoughtful comments on the minor issues, as they provide valuable suggestions for improvement. We will carefully revise the manuscript in accordance with your recommendations.\n\nWe address the specific questions raised in your review below.\n\n> **Weakness:** the coefficient-related hyperparameters seem a bit unusual, such as $\\gamma$=1e-4. I am curious about how these values were selected. Perhaps some ablation studies on this aspect would be helpful. If time is too limited to provide them now, I hope they can be included in the final version of the paper.\n\n**Response:**\n\nThank you for your question.\n\nFor hyper-parameters $\\beta$, $\\eta$ and $\\delta$, we use the same setup following prior works:\n\n- We set $\\beta$=0.1 directly.\n- Consistent with mDPO and OPA-DPO, we use $\\eta$=1.0 and $\\delta$=0.\n\nFor the hyper-parameters, including learning rate, $\\lambda$ and $\\gamma$, we select their values **through grid search**. Specifically, we searched within the following ranges: \n\n- Learning rate: [5e-5, 5e-6, 5e-7],\n- $\\lambda$: [0.1, 0.3, 0.5, 0.7, 0.9],\n- $\\gamma$: [1e-2, 1e-3, 1e-4, 1e-5].\n\nAfter performing the grid search, the best results were obtained with learning rate=5e-6, $\\lambda$=0.5, and $\\gamma$=1e-4.\n\nFrom our experiments, we observe that **too small $\\gamma$ values** lead to weak preference margin consistency regularization and limit the effectiveness of SymMPO, and **too large $\\gamma$ values** impose excessively strong regularization which interferes with preference learning.\n\nUnfortunately, due to character limitations, we are unable to provide detailed comparative results of our grid search. However, we fully acknowledge the value of sensitivity analysis on these hyper-parameters and promise to include them in future updates to the paper.\n\n \n\n> **Question 1:** Could the authors provide more details about the \"contrastive image construction\"? Specifically, what does \"pair each image with its nearest neighbor based on cosine similarity\" mean? Does this imply that is sourced from other images within the dataset? If so, what would happen if the dataset is small and no sufficiently similar image can be found?\n\n**Response:**\n\nThanks for your question. As you consider, the paired image is indeed sourced from the same dataset. The \"contrastive image construction\" operates as follows:\n\n1. Feature Extraction: For every image in our source dataset, we first extract its visual feature vector using the CLIP model.\n2. Similarity Search: We then compute the cosine similarity between this feature vector of the target image and the feature vectors of all other images in the dataset.\n3. Pairing: The image with the highest cosine similarity score is selected as its \"nearest neighbor\" of the target image. This pair of images is then used for the contrastive training process of SymMPO.\n\nWe fully acknowledge the point you raised: if the dataset is small or lacks sufficient visual diversity, it may be challenging to find a meaningful \"nearest neighbor\" that satisfies the requirements of our contrastive strategy. \n\nIn fact, to justify the robustness of our framework, we have evaluated our model with **multiple contrastive image construction methods: (1) Black; (2) Cropped; (3) Noisy; and (4) Synthetic** (see Section 4.5). The first three represent naive approaches, while the Synthetic method is our contribution (detailed in Section 4.5). \n\nFigure 4 reveals that despite its sensitivity to the data distribution, our \"Similar\" method (nearest neighbor pairing) outperformed all alternatives approaches, none of which depend on data distribution. This success likely stems from two factors:\n\n1. The paired images are derived from real data and share semantically meaningful features with the target image. \n2. Even when the paired image is suboptimal due to the absence of sufficiently similar images, it can still function as a random contrastive sample, providing more or less contrastive signals for vision enhancement.\n\nIn the future, we plan to refine this **contrastive image construction method.** For instance, we can apply a similarity threshold for the \"Similar\" method and fall back to the \"Synthetic\" method (the second-best performer) in cases where no adequately similar pair exists. This hybrid approach offers a robust solution for datasets of varying sizes and diversity levels.\n\n \n\n> **Question 2:** It appears that the rejected responses are generated by the model being trained, while the preferred responses are generated by DeepSeek-v3 (based on captions generated by Qwen2.5-VL-32B). If this is correct, the performance upper bound of the proposed method should theoretically align with Qwen2.5-VL-32B, right? In that case, it would be helpful if the authors could include the performance of Qwen2.5-VL-32B in Table 1 for comparison.\n\n**Response:**\n\nThank you for this insightful question regarding our experimental design and analysis. You are correct in observing that the preferred responses in our preference data are generated through a pipeline where captions are sourced from **Qwen2.5-VL-32B** and rewritten by **DeepSeek-V3**. \n\nRegarding the upper bound, the assertion that the performance of Qwen2.5-VL-32B represents an upper bound of our trained model may not be entirely accurate, as the rewriting process is performed by DeepSeek-V3 rather than Qwen2.5-VL-32B. As discussed in Section 4.3, the image captions generated by Qwen2.5-VL-32B may lack fine-grained detail or contain hallucination, which could lead to incorrect consistent/inconsistency claim judgments by DeepSeek-V3 and consequently affect the final response quality. For instance, when the base model’s initial responses include erroneous claims not explicitly mentioned in the captions, DeepSeek-V3 may fail to correct them and propagate these mistakes during rewriting due to its lack of direct image context access. In contrast, Qwen2.5-VL-32B, with its powerful vision understanding capability, could potentially recognize and correct such errors if used for rewriting. Consequently, the synthetic preferred responses do not directly reflect the end-to-end capabilities of Qwen2.5-VL-32B. These additional processing steps and adjustments mean that the performance upper bound of our proposed method with SymMPO is likely **lower** than the raw outputs of Qwen2.5-VL-32B. Here, we clarify that we chose DeepSeek-V3 over Qwen2.5-VL-32B for response rewriting because initial experiments showed that Qwen2.5-VL-32B's rewritten responses exhibited weaker alignment with the base model's outputs in both format and style. In contrast, DeepSeek-V3 better maintains the required stylistic consistency, which led to its selection as our rewriting model.\n\nRegarding comparisons, directly comparing Qwen2.5-VL-32B with our method would be less meaningful, as Qwen2.5-VL-32B would undoubtedly outperform other baselines by a significant margin. This is due to its more advanced architecture, larger parameter scale, and training on more extensive, diverse, and proprietary datasets. Thus, our work focuses on improving multimodal preference optimization within the LLaVA-1.5 family, consistent with existing studies.\n\nWe understand the potential value of including Qwen2.5-VL-32B's performance as a reference point. While such a comparison may fall outside the immediate scope of our evaluation goals, we recognize the value it could add for broader contextual understanding. We will consider exploring and including such results in future iterations of this work.\n\n \n\n> **Question 3:** Building on the previous question, if the preferred responses are generated by DeepSeek-v3, they are effectively off-policy and, according to the analysis in OPA-DPO, may be difficult to learn. What would happen if the authors first conducted SFT on these preferred responses? Would this improve the results? (I fully understand that time during the rebuttal period is limited, and I will not fault the authors if experimental results cannot be provided at this stage.)\n\n**Response:** \n\nThis is an exceptionally insightful question, and we sincerely thank you for raising this critical point. We also deeply appreciate your understanding of the time constraints during the rebuttal phase.\n\nYou are absolutely correct that the analysis in **OPA-DPO** highlights a key challenge for DPO when working with off-policy data, particularly when there is a significant stylistic or distributional gap between the model's own outputs and the preferred responses. This gap can result in a large reverse KL-divergence penalty, which may hinder the effectiveness of preference learning.\n\nTo address this issue, we designed our rewriting process (outlined in **Section 4.1**) with specific practice to mitigate this challenge. We explicitly instructed LLM rewriter to **preserve the original format and style** of the base model's outputs as much as possible. This design aims to minimize the distributional gap and ensure that the preference data remains as \"close-to-on-policy\" as feasible.\n\nWhile this approach reduces the off-policy challenge to some extent, we acknowledge that it is an **indirect solution** and may not completely eliminate the issue. We strongly agree with your suggestion that **first conducting supervised fine-tuning (SFT) on the preferred responses before applying DPO** represents a more direct and principled solution. Fine-tuning the policy model on these preferred responses would effectively shift its output distribution closer to the target distribution defined by the preference data.\n\nUnfortunately, due to the time constraints of the rebuttal period, we were unable to conduct experiments to validate this approach. Nonetheless, we fully recognize the importance of your suggestion and are excited to explore this in future research."}]

Author rebuttal:

Meta-review: This paper presents Symmetric Multimodal Preference Optimization (SymMPO), a new method to mitigate hallucinations in Multimodal Large Language Models (MLLMs). After rebuttal, it received scores of 4445. Generally, all the reviewers are positive about the paper, commenting that (1) the proposed SymMPO framework is novel and somewhat elegant, especially the design of the symmetric pairwise reward; (2) the motivation is reasonable and straightforward, the paper writing is good and easy to follow; and (3) experiments are comprehensive. Therefore, the AC would like to recommend acceptance.
Decision: Accept (poster)