DA-DPO: Cost-efficient Difficulty-aware Preference Optimization for Reducing MLLM Hallucinations
Authors: Longtian Qiu
Shan Ning
Chuyu Zhang
Jiaxuan Sun
Xuming He
Venue: Accepted by TMLR ()

Abstract: Direct Preference Optimization (DPO) has shown strong potential for mitigating hallucinations in Multimodal Large Language Models (MLLMs). However, existing multimodal DPO approaches often suffer from overfitting due to the difficulty imbalance in preference data. Our analysis shows that MLLMs tend to overemphasize easily distinguishable preference pairs, which hinders fine-grained hallucination suppression and degrades overall performance.
To address this issue, we propose Difficulty-Aware Direct Preference Optimization (DA-DPO), a cost-effective framework designed to balance the learning process. DA-DPO consists of two main components: (1) Difficulty Estimation leverages pre-trained vision–language models with complementary generative and contrastive objectives, whose outputs are integrated via a distribution-aware voting strategy to produce robust difficulty scores without additional training; and (2) Difficulty-Aware Training reweights preference pairs based on their estimated difficulty, down-weighting easy samples while emphasizing harder ones to alleviate overfitting.
This framework enables more effective preference optimization by prioritizing challenging examples, without requiring new data or extra fine-tuning stages. Extensive experiments demonstrate that DA-DPO consistently improves multimodal preference optimization, yielding stronger robustness to hallucinations and better generalization across standard benchmarks, while remaining computationally efficient. The project page is available at https://artanic30.github.io/project_pages/DA-DPO/.

Published in Transactions on Machine Learning Research (12/2025)
DA-DPO: Cost-efficient Difficulty-aware Preference Opti- mization for Reducing MLLM Hallucinations
Longtian Qiu
gintt@shhnnghiteehh.....
ShanghaiTech University, Shanghai, China
Shan Ning
singhhan22022@aaannhhhhhhchhhhh
Shanghai University, Shanghai, China Längang Laboratory, Shanghai, China
Chuyu Zhang
shangghhy&ssannghhcitee........
ShanghaiTech University, Shonghai, China
Jiaxuan Sun
Shanghai Tech University, Shanghai, China
Xuming He
herm hermm@shaongaatteehhet..c
ShanghwiTech University, Shanghai, China
Shanghoi Engineering Research Center of Intelligent Vision and Imaging
Reviewed on OpenReview: https: // openreview. net/ forun? id= M52CgPPog
Abstract
Direct Preference Optimization (DPO) has shown strong potential for mitigating halluci- nations in Multimodal Large Language Models (MLLMs). However, existing multimodal DPO approaches often suffer from overfitting due to the difficulty imbalance in prefer- ence data. Our analysis shows that MT.LMs tend to overemphasize easily distinguishable preference pairs, which hinders fine-grained hallucination suppression and degrades overall performance. To address this issue, we propose Difficulty-Aware Direct Preference Opti- mization (DA-DPO), a cost-effective framework designed to balance the learning process. DA-DPO consists of two main components: Di)fffcculyy Estimation leverages pre-trained vision language models with complementary generative and contrastive objectives, whose outputs are integrated via distribution-aware voting strategy to produce robust difficulty scores without additional training: and (2) Difficulty-Aware Training reweights preference pairs based on their estimated difficulty, down-weighting casy samples while emphasizing harder ones lo alleviate overfitting. This framework enables more effective preference op- timization by prioritizing challenging examples, without requiring new data or extra fine- tuning stages. Extensive experiments demonstrate that DA-DPO consistently improves multimodal preference optimization, yielding stronger robustness to hallucinations and bet- ter generalization across standard benchmarks, while remaining computationally efficient. The project page is available at Ittpp::/aartanic3..giiitt wwt.ggtthhu.... httthub.iooprroeet_ppeeees///
1 Introduction
Recent advancements in Multimodal Large Language Models (MLLMs) (Liu et al. 2023a Open 2023 et al. 2024) have significantly improved vision-languagg tasks, such as image captioning Lin et al. 2014).
visual qucstion answering (Agrawal al. 2015 Mathew et al.] 2021 Marino et al.] 2019). By combining powerful large language models with state-of the-art vision models, MLLMs have enhanced multimodal understanding and reasoning. However, persistent challenge for ML.LMs is their tendency to produce

---

Published in Transactions on Machine Learning Research (12/2025)
responses that are not reliably grounded in visual inputs, often resulting in 'hallucinations' where descriptions include non-existent or inaccurate visual details. This limitation affects the reliability of ML.MMs, posing a significant barrier for applications that require high factual accuracy.
(Rafailov Recent efforts have turned to Direct Preference Optimization ( et al., 2024 as a promising approach to mitigate hallucinations in MLLMs. DPO encourages models to align their outputs with prefer- ence data that favor faithful responses and reduce hallucinations. Crucially, the effectiveness of DPQ hinges the quality of pairwise preference data. To address this, early approaches (Sun et al. 2023b Yu et al.,
2024a) rely on manual annotation, but such data collection is both labor-intensive and difficult to scale, More recently, several works (Pi et al. 2024 Li et al., 2023d Zhou et al.) 2024c Yang ct al.) 2025 have proposed automated strategies for constructing multimodal preference data. These methods exploit trained models to produce pairwise preference data at scale, significantly increasing data coverage across diverse scenarios and thereby improving the model's ability to reduce hallucinations.
Despite their elfectiveness in reducing hallucinations, vanilla DPO methods trained on existing pairwise preference data often lead to noticeable degradation in general multimodal capabilities, as shown in Figure Ia We attribute this limitation to an imbalance between easy and hard samples in the training data, illustrated in Figure 1b Easy samples typically involve clearly distinguishable faithful and hallucinated responses, whereas hard samples require more nuanced reasoning to differentiate. This imbalance leads to a training bias where models overfit to easy cases while failing to learn from more challenging examples. We provide a detailed empirical analysis of this phenomenon in Section 3 showing that while models quickly adapt to easy samples, they struggle to generalize to hard ones, ultimately limiting the effectiveness of preference-based alignment.
To address this limitation, we propose difficulty-aware training framework that dynamically balances the contribution of easy and hard samples during preference optimization. A key challenge in implementing this strategy is the lack of explicit supervision for estimating sample difficulty. We tackle this by introducing a lightweight, training-free strategy: by aggregating signals from multiple pre-trained vision language models (VT.Ms) trained under diverse paradigms, we obtain robust difficulty scores that estimate the difficulty of pairwise preference data without explicit training a specific model. These difficulty scores are then used to reweight preference data, enabling effective difficuly-awwrre training that emphasizes harder samples while preventing overfitting to easier ones.
Specifically, we propose Difficulty Aware Direct Preference Optimization (DA-DPO), a framework that consists of two steps: difficulty estimation and difficulty-aware training. The first step assesses the difficulty of each pairwise preference sample using multiple VLMs. In particular, we leverage both contrastive VLMs (e.g., CLIP (Radford et al., 2021)) and generative VLMs (e-g., LLaVA (Liu et al. 2023b)) to estimate difficulty from complementary perspectives. Their outputs are aggregated through a distribution-aware voting strategy, in which the weight of each VLM is adaptively derived from its observed classification reliability over the training data. Building on these scores, the second step performs difficulty-aware training by dynamically adjusting the optimization strength of each sample in DPO. Specifically, the difficulty scores adjust the degree of divergence permitled between the learned policy and the initial policy. This mechanism strengthens learning from challenging samples while limiting unnecessary drift on trivial ones.
We conduct experiments on three popular MLLMs with different scales and abilitics. To provide a com- prehensive comparison, we report the performance comparison and analysis on two sets of benchmarks,
hallucination benchmarks (Wang et al.) 2023a Rohrbach et al. 2018 Sun et al. 2023b Li et al.
2023e and
general MLLM benchmarks (Hudson & Manning 2019 Liu et al. 2023b Fu ct al. 2023 Li ct al. 2023a
which demonstrate the effectiveness of our approach.
Our main contributions are summarized as follows:
We conduct analysis on the multimodal preference optimization training and empirically demonstrate the existence of an overfitting issue, which can lead to suboptimal performance.
We propose a cost-cffective framework that leverages vision-language models (VLMs) to estimate the sample difficulty without additional training and utilize the estimation to improve preference modeling via diffieulty-wware training.
2

---

Published in Transactions on Machine Learning Research (12/2025)
Iwhesraeton Berchmon's
Compotersiee Benchmar
diared onliityy in iniiiing Wh witer This bekawar commin for buth bidd specics they knows naterfow.
14:300
enpoging in the shamed acthvity of flying the dry, This o not common
toey Samale
aponis as ovena are waturtowl duste deciling mis...
X
What shared inforest of both swans and dicks in this inage?
Reject crever En nag rmoge meer and dadaa one eejiping
Hand Gerel'
Thes behavior, typical for dacke leee commaniy a3-orwad ewans, ane pr monrrry fanege wetto..
a
(b)
Figure (1a) Performance Comparison of DPO and DA-DPO. We provide the performance im- provements of DPO and DA-DPO compared to the LLaVA v1.5 7B without preference optimization. The Hallucination indicates the performance on hallucination benchmarks, and Comprehensive indicates the performance of 4 comprehensive MLLM benchmarks. The details described in the experiments section. (1b) Easy and hard pairwise samples: "Easy Samples' have large score gap due to clear differences between preferred and dispreferred responses, while 'Hard Samples' show minor differences, making them more valuable for learning.
We evaluate method on hallucination and comprehensive benchmarks, and experimental results show that it significantly enhances the performance of varions MLLMs in cost-efficient manner.
2 Preliminaries
In this scction, we provide brief overview of the Reinforcement Learning from Human Feedback (RLHF) to Direct Preference Optimization (DPO) pipelines.
RLHF Reinforcement Learning from Human Feedback (RLHF) is widely used framework for aligning LLMs with human values and intentions. The standard approach (Bai et al. 2022a] Ouyang et al.) 2022) first trains a reward model and then optimizes a KT-regularized reward objective to balance preference alignment with output diversity. The optimization can be written as:
max D. (000--KKiA [K[[T(y) [ror((|=)]] 8g
(1)
where Tyo is reference policy (typically the SFT model) and 3 controls the trade-off between reward maximization and staying close to Tref. The objective is usually optimized with PPO (Ouyang et al. 2022
Pair-wisc Preference Optimization Despite the success of the above RLHF, PPO is challenging to optimize. To enhance the efficiency of PPO, DPO [Rafailov el al. 2024 reparameterizes the reward function with the optimal policy:
Toly xx) Tref(y [z) + 3 log Z(x),
r(z v) = 3 log
where (z) denotes the partition function ensuring proper normalization. The hyperparameter 3, analogous to the KL weight in Eq. 1 controls the trade-off: larger value encourages me remain closer to the reference policy, preserving generalization and robustness, while smaller value places greater emphasis preference alignment but risks overfitting
Building on this reward formulation, we can directly integrate it into the Bradley-Terry model, which treats pairwise preferences probabilistically: By doing so, we can optimize the preference objective without learning a separate reward model. The optimization objective is described as:
min EE [log (r(x, y.) - r(æ, yr))),
(3)
where r(z, y) can be any reward function parametrized by ne, such as those defined in Eq. ( (2) and Ve and yr denote the chosen and rejected responses in pairwise preference data, respectively.
3

---

Published in Transactions on Machine Learning Research (12/2025)
3 Multimodal Preference Optimization Analysis
In this section, we present a systematic investigation of the prevalent overfitting challenge in multimodal preference optimization. Through empirical analysis, demonstrate that models exhibit a tendency to overfit to simpler training samples, while progressively reducing their effective learning from harder instances. This phenomenon is particularly pronounced in pairwise training paradigms like DPO (Rafailov et al. 2024). This overfitting behavior ultimately compromises model performance when applied to diverse real-world scenarios. We substantiate these findings with quantitative evidence drawn from training dynamics and reward trend analyses.
Reward Dynamics with CLIP Estimation
Reward Dynamics with MLLM Estimation
Reward Dynamics with Fused Estimation
160
DPO
23.06% DPO Top 15%
DnD 75-32%%
DPG Tep 50 73%
Te 50-
250
DAO
250
290
6-39%
BA-RG-Pa Ra 75-100%
DN-PPG Rig 75-2286
26-00- Top 15----0
Reward
300
010 23-2%%
Raward
BYC Tp
75250-999
1.00
06.090
330
500
-50
-90
500
750
1860
1250
1500
3750
2000
2250
2500
500
750
1000
1350 1500
1750
2000
2259
2550
500
750
1000
1250
1500
1750
2000
2250
2008
Training Iteration
Training Iteration
Training Iteration
AUG Reward Gap with CLIP Estimation
AUG of Reward Gap with MLLM Estimation
AUG Reward Gap with Fused Estimation
4.33
DPO
DPO
DA-DPO
4.30
DA-DPO
DA-DPO
Reward Gap
4.25
Reward Gap
4.0
3.5
Reward
6.20
1.5
3.0
4.95
45
300
750
1000
1250
1500
3150
2002 1250
2502
530
150
1000
2250
1500
1150
2000
2250
2550
1000 3250 1300
1750
2000 2250 2500
Training iteration
Training Iteration
Training Iteration
Figure 2: Reward Dynamics and Grea-Under-Gap (AUG) Between the Easiest and Hardest Samples. We present the reward trajectories held-out validation set of LLaVA-v1.5-BB for both DPO and DA-DPO trained on the BPO dataset. The first row depicts how the rewards of data buckets with different estimated difficulty levels evolve over training iterations. The difficulty is estimated using three distinct proxies, as detailed in Section|.] Here, the Easiest samples (Top 75-100% in the legend) correspond to samples with the largest gap between chosen and rejected responses. The second row reports the Area Under Gap (AUG), quantifying the cumulative reward gap between the easiest and hardest samples across training, serving as a compact indicator of reward disparity dynamics. Shaded regions in the plots indicate the standard deviation across three independent sceds, though the effect is subtle due to the low variance induced by training randomness.
Analysis Setting To systematically analyze overfitting in preference optimization, we construct a con- trolled evaluation sctup. We split the dataset into a training set and a held-out validation set with a ratio of 90% to 10%. Both DPO and DA-DPO models are trained on the 90% training portion, while their reward performance is periodically evaluated the validation set across training iterations. Since there is no oracle annotation for sample difficulty, we estimate the difficulty of each validation sample using three different proxy metrics, as introduced in Section 4.1 Based on the difficulty ranking derived from each proxy, we partition the validation samples into four equally sized buckets, ranging from the easiest to the hardest. This setup allows us to examine how the model's reward evolves for samples of varying difficulty, thereby providing insights into overfitting behavior during preference optinization. To assess the statistical reliability of the results, we repeat the experiments with three different random seeds and report the corresponding standard deviations.
Reward Dynamics Analysis We analyze the reward dynamics from two complementary perspectives. The first perspective examines how the rewards of samples with different difficulty evolve throughout the training process. Based three different proxy metrics (as detailed in Section 4.1p, the validation samples
4

---

Published in Transactions on Machine Learning Research (12/2025)
CLIP
Cheees crower demonatrates emplayes efforts persoralized
Chosen onswer: Interacting with giraf fes zebros, and other anincl is exhibit several benefits children viaiters.
office cubicle that reflects their
preferences.
0.12
0.43
CL Score Gup
dep Question Retevance
Image Relevance
MILM Score Gap
Reject answer There is computer monetor the desk dispaaynng scenic landscope. Survounding the moniter, postcorde planed
Queattion: What feotures of this office cubicle indioate the emplayed's personal touch?
0.58
0.08
Question: What the bonefits inferacting with these animals?
Reject domesticated
store adoptien center can also provide aeveral benefits for individuas...
walla cubicle..
MLLM
Figure 3: Illustration of the two contrastive and generative VLMs. As shown on the right side, CLIP captures the misalignment between the image and the answer such that the CLIP score gap is large when chosen and rejected answers differ in image relevance. However, as shown on the left side, the MLLM is better at capturing the logical connection between the question and answers, such that the MLLM score gap is large when chosen and rejected answers differ in question relevance.
are ranked by difficulty and partitioned into multiple buckets, from the easiest to the hardest. As shown the first row of Figure 2 we observe that for both DPO and our proposed DA-DPO, the rewards of harder buckets consistently remain lower than those of casier buckets. This trend indicates limited capacity to fit hard samples, which typically require fine-grained understanding capabilities. Moreover, observe that in DA-DPO, the reward of the easier buckets increases more slowly compared to that in naive DPO. This slower growth is a result of DA-DPO's adaptive weighting mechanism, which adjusts the importance of each training instance based on its estimated difficulty. By doing so, DA-DPO effectively reduces over vver-optimization on casier samples, thereby alleviating the overfitting tendency frequently observed standard DPO training.
To further quantify this phenomenon, we introduce second perspective: the Area Under Gap (AUG) between the easiest and hardest samples. The AUG is computed by first taking the difference between the reward of the easiest bucket and that of the hardest bucket at each evaluation point across training iterations, and then integrating this gap over the entire training trajectory. This provides cumulative measure of the reward disparity between easy and hard samples throughout training. A larger AUG indicates a stronger optimization bias toward easier samples. As shown in the second row Figure 2 the AUG in DA-DPO remains consistently smaller than that in naive DPO, demonstrating that DA-DPO achieves a more balanced optimization across samples of varying difficulty. These analyses collectively highlight the presence of overfitting in multimodal preference optimization and the effacacy of DA-DPO in mitigating it.
4 Methods
In this section, we introduce the DA-DPO framework, which addresses the overfitting issue in standard multimodal DPO training in a cost-efficient manner. In Section [.1 we first discuss the estimation of preference data, where CLIP and MLLMs to evaluate the difficulty of preference data. In Section 4.2 explain how the data difficulty estimation from pretrained VLMs informs and guides the difficulty-aware DPO training.
4.1 Data Difficulty Estimation
The key challenge in evaluating the difficulty of preference data is the lack of explicit supervision. To address this, we propose a lightweight, training-frce strategy that leverages pre-trained contrastive and generative VLMs to estimate sample difficully from complementary perspectives, as illustrated in Figure To combine the estimates from multiple models, we adopt a distribution-aware voting strategy, where each model's contribution proportional to preference classification accuracy on the training set. This results in a robust difficulty score for each sample without requiring additional model training. The details are described in the following sections.
5

---

Published in Transactions on Machine Learning Research (12/2025)
Contrastive Estimation CLIP is trained on web-scale image captions via contrastive training objectives and is proven to contain generalized knowledge regarding image and text relevance. We utilize CLIP to evaluate the difficulty of pairwise DPO data. Specifically, we first compute the CLIP text embeddings for the chosen response Ve and rejected response yr (denoted as fe and fr, respectively), and the CLIP image embedding for the image DPO data m (denoted as vm). The CLIP scores ce and e, represent the imago-text relevance of both responses computed as follows:
= Cos Um), = -osSSim(fr: Um).
(4)
We then introduce CLIP score gap cy. which reflects the difficulty of DPO sample; larger gap indicates that the chosen and rejected responses are easily distinguishable in terms of image-text relevance. Formally, the CLIP score gap by is defined as follows:
Cg = Co Cy.
(5)
To integrate this metric with other VLM-based estimates, we normalize cg to a common scale via dataset-level Gaussian projection. Given a dataset of N samples, the normalized score is computed as follows:
0g = $ Cg Mey
(6)
Ggg
where Me, and oe, are the mean and variance of all CLIP score gaps Cg in the DPO dataset. The normalized CLIP score gap 0g E [0, 1], and (() denotes the cumulative distribution function of the standard Gaussian distribution.
Generative Estimation Recent Multimodal large language models (MLLMs) are built on large language models (LLMs) and learn to interact with the visual world by connecting visual encoder to an LLM with language modeling objectives. For simplification, we denote this as a generative VLM. Although such modols are prone to hallucination, they provide an informative evaluation of DPO data difficulty from another perspective complementary to CLIP. We extract the MLLM score for the chosen responses Ye and rejected responses yr, denoted as m. E and m, E TR which defined as follows:
T
mc Σ
log (P(vllm.vl ve...., vē- ;ret)))
(7)
my Σ log (Ply (ty//mm./..... .i..., ;
=ret))
(8)
where vi and oy are the 8hh tokens in the chosen and rejected responses, and T is the sequence length. To evaluate the difficulty of pairwise DPO data, the MLLM score gap my is computed as follows:
mg
m. my.
For a similar reason as the normalization of CLIP score, we utilize a Gaussian normalization to acquire a normalized MLLM score gap my E [0, 1]. The process is defined as follows:
mg = + mg Mmng
(10)
Omg
where Pmy: On is the mcan and variance of all the MLLM score gaps my in the DPO datasct.
Distribution--ware Voting Fusion To this end, we evaluate the pairwise DPO data from two perspec- tives, resulting in two difficulty scores. As shown in Figure the two VLMs perform differently preference classification: CLIP excels caption-related preference data, while MLLM performs better VQA-related

---

Published in Transactions on Machine Learning Research (12/2025)
VOA
Caption
Text Caption
Overall
Figure 4: Preference classification comparison. Preference classification evaluates whether the pre- trained VLMs output higher reward score for the chosen answer compared with the rejected answer. We report the classification accuracy on three subcategories and the overall performance on the BPO training dataset.
data. We propose a data-driven voting strategy to adaptively combine the difficulty scores based on the pref- erence classification results. Specifically, we use the classification accuracies of CLIP and MLLM, denoted clsc and clsm; to determine the weight of bela for each DPO sample, described below:
clsc
cls
3 = clsc + clsm
bg clsc + clsm
myg) +1,
(11)
where we add constant term of 1 improve the stability of the training process. Without this term, when both by and my close to zero, the resulting 3 would also approach zero, potentially leading to training collapse.
4.2 Difficulty-wware Training
After estimating the difficulty of the preference pairs, obtain robust score that reflects the difficulty of the pairwise DPO data. Building previous work (Wu et al. 2024), we perform difficult-aware DPO training by adaptively calibrating the 3 in Eq. I This approach allows us to adjust the weight of each training sample, reducing overfitting on easy samples compared to standard DPO training. Formally, the proposed difficult--waare DPO objective is defined as:
LPA-DPO - -B [logo (Br(x, 3 Br(x, -
(12)
where 3 is the difficulty-waare scaling factor, r(z, y) denotes the reward function defined in Eq. (2) which itself incorporates the temperature coefficient, B, Ue and are the chosen and rejected responses, respectively, and (-) is the sigmoid function.
5
Experiments
5.1
Implementation Details
Training Sctup To validate the effectiveness of the proposed methods, we use the pair preference datascts from BPO (Pi el al. 2024). This dataset contains 180k pairwise preference data, where the negative responses are generated by the Weakee prompting and LLM Error Injection. For training parameters, we train the model for 1 epoch and set the to 0.2. For LLaVA V1.5, we follow previous work (Pi et all, 2024) to adopt the LoRA (Hu et al. 2021) training with rank 32 and LoRA alpha 256. The learning rate is set to 2e-6. For VA-Oneeiisio we use the recommended official training script to perform full fine- tuning where the learning rate is 5e-7. The training takes about hours for LLaVA V1.5 7B models and 22 hours for LLa Va-AneVisiio 7B. For the analysis section, we preserve the lop and bollom 20% scores as high-confidennce CLIP-based predictions.
Choice of VLMs For the choice of VLMs, we employ the CLIP (Radford et al. 2021 VITL/140336. For the preferred and negative text. responses in each data sample, we encode the longest possible text from

---

Published in Transactions on Machine Learning Research (12/2025)
Table 1: Results on hallucination and comprehensive benchmarks. For each column, the t symbol indicates that higher values are better, while the symbol indicates that lower values are better. The AMB, Gen. and AMB. Dis. refer to the generative and discriminative components of the AMBER benchmark. The symbol denotes the evaluation results of the official model weights from BPO (Pi et al. 2024 We clarify that focuses on utilizing CLIP knowledge to construct preference data, whereas we leverage CLIP to address the overfitting issue in standard DPO training. The mDPOf provides an alternative approach to improving multimodal preference optimization by addressing the over-prioritization of language preference, which is orthogonal to the DA-DPO approach.
AMB. G
AMB. P ObjHal
MMHal
GQASeed POPE GOAASeL MMEE" MME
C. 1Cov. Hal (Cog. + F1 C. UCs JScore HallRate F1 t Score t Acct Acct Scoret Scoref
Reference Only (Not directly comparable)
CogVLM
5.6
57.2
23.6
1.3
mPLUG-Owl2
10.6
52.0
39.9
4.5
56.1
InSIructBLIP
8.8
52.2
38.2
4.4
58.2
58.8
1212.8
Qwen-VL.
5.5
49.4
23.6
1.9
36.0
2.89
0.43
59.3
1487.6
360.7
GPT-4V
4.6
67.1
30.7
2.6
13.6
7.3
3.49
0.28
VLaVA-v1.5-7B
7.8
51.0
36.4
4.2
74.7
54.7
15.9
2.19
0.57
85.8
63.8
62.0
66.1
1510.7
307.5
+ HA-DPO
7.2
33.6
19.7
2.6
39.9
19.9
1.97
0.60
+ CLIP-DPO
3.7
47.8
16.6
1.3
85.8
1468.7
+ mDPO
4.4
52.4
24.5
2.4
35.7 9.8 2.39
0.54
Main
Resulls
LLaVA-v1.5-7B
7.8
51.0
36.4
4.2
74.7
54.7
15.9
2.19
0.57
85.8
63.8
62.0
66.1
1510.7
307.5
+ DPO
5.5
58.4
35.7
2.0
83.9
43.3
10.0
2.24
0.53
84.3
70.5
45.3
57.8
1409.4 315.0
+ DA-DPO
4.3
57.4
28.0
2.1
85.6
39.7
9.9
2.22
0.50
85.9
75.4
59.8
64.8
1406.6
323.2
LL
7.1
52.4
33.9
3.8
88.6
56.3
15.8
2.22
0.57
86.9
71.2
63.0
67.5 1496.7
308.9
+ DPO
6.6
61.9
45.7
2.5
85.5
39.0
8.8
2.05
0.55
74.1
60.9
53.5
1450.6 276.4
+ DA-DPO
5.1
59.0
32.2
1.9
87.3
37.0
8.3
2.39
0.48
85.6
74.5
61.5
61.4 1474.2
276.4
LLVVA-OV-7B
8.4
74.8
70.4
9.8
90.6
41.3
8.1
2.76
0.37
87.2
80.0
59.6
75.8 1580.4
406.4
+ DPO
7.7
66.5
59.0
4.3
91.9
34.3
8.6
2.61
0.38
86.1
72.9
55.0
73.7 1545.4 364.6
+ DA-DPO
7.0
64.1
51.8
3.7
91.7
28.0 8.0
2.78
0.30
86.0
73.7
59.2
75.6 1551.4
381.8
the start and truncate the rest the text. For MILM, we use the LLaVA v1.5 7B (Liu et al. 2023a to compute the probability of responses given the image and question. We also provide an ablation study on the choice of these models in Section 5.5
5.2 Evaluation Benchmarks
To comprehensively evaluate the impact of preference optimization on MLLMs, we select two types of benchmarks. Hallucination benchmarks measure the model's ability to reduce factual errors, which is the primary goal of multimodal preference alignment. Comprehensive benchmarks assess general multimodal capabilities, ensuring that improvements in hallucination do not come at the cost of overall performance.
Hallucination Evaluation. Following previous works (Pi et al. 2024 Wang al. 2024a Ouali et al.
2024), we comprchensively evaluate the DA-DPO on various hallucination benchmarks such as AM- BER Wang el al., 2023a), MMHalBench (Sun el al.) 2023b),, Object HalBench (Rohrbach el al. 2018)) and POPE (Li et al. 2023e 1) AMBER provides a multiddmmeniional framework suitable for assessing both gencrative and discriminative tasks. 2) MMHalBench is a question-answering benchmark with eight question
types and object topics. We follow the official evaluation scripts with GPT-4. 3) Object HalBench is a
standard benchmark for assessing object hallucination, and follow the settings in (Wnng et al. 2024a

---

Published in Transactions on Machine Learning Research (12/2025)
Table 2: The impact of estimation from different VLMs. We report the performance of the model trained with different strategics on hallucination benchmarks such as the AMBER benchmark and the comprehensive benchmark SeedBench. The row with the blue color indicates the adopted strategy.
VLMs
AMB.
AMB. D
Seed
CLIP
MLLM
C2 +
Cov. f
Hal. 4
Cog.
F1 f
Score
DPO
5.5
58.4
35.7
2.0
83.9
57.8
4.7
58.0
31.9
2.0
85.2
63.8
Ours
/
4.6
57.6
29.9
2.3
85.3
64.3
4.3
57.4
28.0
2.1
85.6
64.8
Table 3: Results of LLaVA v1.5 7B trained with VLFeedback and LLaVA-RLHF dataset. The VLFeedback is an automatically constructed dataset by collecting responses from multiple VLMs and filtering the results with GPT4-V. The LLa LLaVA-RLHE is a human-annotated dataset.
AMB.
AMB. D ObjHal
MMHal
POPE GQA Seed' MME" MME
C. J Cov, 1 Hal. Cog. 4
F1 1
C. C. J. Score 1 HalRate 1
F1 f
Score 1
Acct
Acct
Scoret
Scoret
LLaVA-v1.5-7B
7.8
51.0
36.4
4.2
74.7
54.7 15.9
2.19
0.57
85.8
63.8
62.0
66.1
1510.7
307.5
Vi.eestback
DPO
6.5
55.1
34.5
2.3
84.6
49.0
13.0
2.19
0.65
84.6
72.1
59.8
63.5
1368.5
294.2
DA-DPO
5.6
53.0
29.7
2.7
85.8
48.3
12.8
2.23
0.53
84.2
73.0
60.3
65.0
1422.7
297.1
L.La VA RL.P
DPO
7.3
50.8
33.7
3.9
86.6
58.3
16.9
2.10
0.57
84.4
68.7
60.7
64.5
1415.9
343.9
DA-DPO
5.8
50.7
27.6
3.0
86.6
48.0
13.8
2.02
0.59
84.5
71.3
60.9
64.8
1464.1
301.4
4) POPE utilizes polling- based query to evaluate the model's hallucination. score of three kinds of questions in POPE.
We report the average F1
Comprehensive Evaluation. For evaluating MLLM helpfulness, we use: 1) LLaVA-Bench [Liu et al. 2023b), a real-world benchmark with 60 tasks assessing LLaVA's 's visual instruction -orrruttioo-oolooiinng and question- answering abilitics. We use official scripts to compute scores with GPT-4. 2) Seedbench (Li ct al.) 2023a which consists of 14k multiple-choice VQA samples to evaluate the comprehensive ability of MLLMs. 3) MME (Fu et al., 2023) which measures both perception and cognition abilities using yes/no questions. 4) GQA Hudson & Manning (2019) evaluates real-world visual reasoning and compositional question-answering abilities in open-ended answer generation format.
5.3 Baselines
The proposed DA-DPO framework designed to improve pairwisc preference optimization by introducing
the pretrained VLMs cost-efficient manner. We mainly compare DA-DPO with standard DPO (Rafailov
et al.
2024) under three MLLMs, LLaVA V1.5 7B/13B (Liu al. 2023a) and LLaVA-OneVisooe 7B (Li
et al.
2024h. Additionally, we provide results from other multimodal LLMs, such GPT4V (OpenAI
2023
CogVLM (Wang et al.] 2023b), mPLUG-Ow12 (Ye et al.
2023 InstructBLIP (Dai et al. 2023
VT. Bai et al. 2023)) HA-DPO (Zhao et al. 2023). CLIP-DPO Ouulii et al. 2024), mDPO (Wang et al.
Owen-
2024a) for reference.
5.4
Results Analysis
Hallucination Benchmarks To demonstrate the effectiveness of the proposed methods in reducing hallu- cinations, we present evaluation results on various hallucination benchmarks in Table We observe that the DA-DPO improves model performance on most benchmarks compared to DPO, such as the HolRate in AM- BER generative decrease from 35.7 to 28.0 when training with LLaVA v1.5 7B. Moreover, the performance
9

---

Published in Transactions on Machine Learning Research (12/2025)
Table 4: Choices of VLMs. The VITL indicate the CLIP VITL/140336 and EVA 8B is the EVA-CLIP 8B. The LLaVA 7B is the LLaVA v1.5 7B and the OV 7B is the LLa LLaVA-Onevision 7B.
Model Choice
AMB, G
AMB.
Seed'
CLIP
MLLM
C, 4
Cov.
Hal. 4
Cog. +
F1 f
Score f
VITL
LLaVA 7B
4.3
57.4
28.0
2.1
85.6
64.8
EVA 8B
LLaVA 7B
4.9
57.1
31.8
2.6
83.4
56.2
VITL
OV 7B
1.6
57.7
31.4
2.0
85.7
65.2
EVA 8B
OV 7B
4.3
55.6
25.5
2.0
86.0
65.5
Table 5: Comparison of DA-DPO with direct filtering basclines (10%, 25%, 50% easy samples removed). Metries with f indicate higher is better; with & indicate lower better.
AMBER
AMBER P
POPE
MMHal Scoret HalRate]
ObjHall
LLaVA Scoref
GQA
Seed
MME
Model
C.
Cov.
Hal. Cog.
Fif
Fif
C. J C2
Acuf
Accf
Pt
Ct
DPO
5.5
58.4
35.7
2.0
83.9
84.3
2.23
0.53
43.3 10.0
70.48
45.3
57.7
1409.4
315.0
Filter 10%
5.6
63.2
39.6
2.6
85.9
84.9
2.22
0.51
39.0
9.3
72.49
53.4
50.3
1365.1
300.0
Filter 25%
5.3
61.3
37.8
2.3
85.8
84.3
2.29
0.50
38.3
9.0
67.04
53.3
51.1
1318.6
330.3
Filter 50%
5.3
62.2
38.3
2.0
86.0
85.5
2.03
0.56
42.7 10.2
71.21
56.3
52.2
1359.7
306.7
DA-DPO
4.3
57.4
28.0
2.1
85.6
85.9
2.22
0.50
37.7 9.8
75.38
59.8
64.8
1406.5
323.2
of three MLLMs is improved significantly on the Object Hallucination benchmark, which demonstrates that the DA-DPO greatly reduces the hallucination in the caption.
Comprehensive Benchmarks We evaluate the proposed methods on five comprehensive benchmarks. The results, shown in Table 11 indicate that while preference optimization reduces hallucination, perfor- mance general abilities suffers. However, DA-DPO mitigates this degradation most benchmarks when compared to standard DPO. Furthermore, DA-DPO significantly enhances conversational ability, with per- formance on the LLa VA-Bench reaching 75.4, compared to 70.5 for DPO using LLaVA v1.5 7B.
5.5 Ablation Study
Impact of the Diffculttyaware Training To better understand the proposed methods, we conduct an ablation study to assess the effectiveness of each design. As shown in Table 2 we begin with the standard Direct Preference Optimization (DPO) (Rafailov et al. 2024). We then add the reward score from the CLIP to control sample weight during training, followed by results for DPO with only the MLLM's reward score. Finally, we combine both reward scores using the adaptive fusion strategy. The results show that using either CLIP or MLLM's reward score improves both Hallate and CHAIR,, with the combination of both achieving the best performance, highlighting the necessity of cach framework design.
Influence of Preference Datasets To demonstrate that the proposed methods mitigate data bias different types of multimodal pairwise preference data, we use VLFoedBack (Li et al. 2023d), which consists of 80k responses generated by MLLMs with varying levels of ability, and rated by GPT4-V (OpenAT, 2023). This dataset is automatically generated through different approach compared to BPO (Pi et al. 2024), and it covers the mainstream data gencration pipcline for multimodal preference data. We follow the settings of mDPO (Wang et al. 2024a) to select 10k preference data samples. Moreover, we trained on LLaVA- RLHF (Sun et al., (2023b), the most widely used human-annotated multimodal preference dataset with 10k samples. As shown in Table 3 DA-DPO outperforms DPO on most benchmarks, demonstrating that overfitting is a common issue in multimodal preference data, and DA-DPO effectively alleviates this problem in cost-efficient manner.
Choices of the VLMs We present an ablation study on the selection of VLMs. These models estimate the difficulty of pairwise preference data and guide difficulty-aware preference optimization during training
10

---

Published in Transactions on Machine Learning Research (12/2025)
Table 6: Performance across difficulty buckets estimated using CLIP-based and MLLM-based proxies. We fine-tune LLaVA-1.5-7B using vanilla DPO, where the training data in each bucket sampled from the BPO dataset based on the estimated reward gap. The Top 0 0-25% bucket corresponds samples with the smallest reward gaps.
AMB. G
AMB. D
ObjHal
POPE
Bucket
C,1
Cov.t
Hal.
Cog.l
F11
C.4
C,I
F11
CLIP Estimation
Top 0 25%
5.3
62.0
37.6
2.2
86.8
41.3
10.8
85.6
Top 25 50%
5.1
62.7
37.5
2.0
86.2
43.0
9.7
85.8
Top 50-75%
5.2
61.1
37.9
2.4
85.7
51.0
10.8
85.8
Top 75-100%
5.4
55.0
38.9
2.5
85.6
52.7
12.9
85.7
MLLM Estimation
Top 0-25%
4.9
59.1
35.4
2.5
85.6
46.7
11.8
85.7
Top 25 50%
4.6
53.9
32.3
1.9
86.6
42.3
9.0
85.8
Top -5%
5.4
55.5
30.5
2.0
84.6
48.3
11.4
86.5
Top 75 100%
5.5
54.3
32.4
2.2
84.8
46.3
10.8
86.3
in the proposed framework. We experiment with two CLIP models of varying parameter scales: CLIP ViT- L/14@336 (Radford et al.) 2021) and EVA-CIIP 8B (Sun et al. 2023a). For MLLMss, we use LLaVA v1.5 7B (Liu al., (223aa) and VLLVA-OneVision 7B (Li al. 2024). As shown in Table { we observe that performance remains similar across VLMs with different capabilities, which is attributed to the Gaussian normalization in Eq. (10) and Eq. ( This normalization ensures that the VLMss only provide a ranking of preference data, making our proposed framework robust to variations in the choice of VLMs.
Comparison with Direct Filtering Baseline To further validate the effectiveness of our sample re- weighting strategy, we compare DA-DPO against a baseline that directly filters out easy samples from the training data. Specifically, we remove 10%, 25%, and 50% of the easy samples from the BPO dataset based our difficulty metric and train a model using the DPO algorithm. Table summarizes the performance of DA-DPO and the direct filtering baselines across wide range of benchmarks. We observe that DA-DPO consistently out perperfrrms the filtering-based approaches on most metrics, especially on hallucination--elatee benchmarks such as POPE, MMHal, and Object Hallucination. Notably, the performance of the direct filtering bascline fluctuates across different filtering ratios, with no consistent improvement as more casy samples are removed.
We attribute the limited effectiveness of direct filtering to its inherent trade-off: while it removes truly uninformative samples, it also discards portion of valuable training data, thereby reducing the diversily and coverage of preference information. In contrast, DA-DPO addresses this issue by softly dowg-weighting casy samples instcad of hard filtering, preserving data diversity while emphasizing informative examples. This allows DA-DPO to maintain robustness across benchmarks with varying difficulty and annotation styles.
Estimated Reward Gap and Model Hallucination To examine the relationship between the reward gap and hallucination behavior, we design a controlled validation experiment. Specifically, we utilize CLIP- based and MLLM-based proxies to estimate the reward gaps of training samples and evenly divide the data into four buckets according to the estimated gap range. We then train vanilla DPO model on each bucket independently to analyze how sample difficulty affects preference optimization. As shown in Table G we observe Ihat the largest reduction in hallucination occurs for samples in the 25 50% bucket, while the exlent of hallucination reduction gradually decreases as the reward gap increases. Interestingly, the bucket with the smallest reward gap (0-25%) does not yield the most significant hallucination improvement. We attributc this to noisy or confusing samples within this subset during data construction such cases where the "rejected" answer is actually semantically correct-wie which may interfere with effective model optimization.
11

---

Published in Transactions on Machine Learning Research (12/2025)
6 Related Works
6.1 Vision-Language Models
Vision-Language Models (VLMs) (Radford et al. 2021 Jia et al. 2021 Li et al. 2022 have substan-
tially advanced multimodal understanding, achieving strong performance across a wide range of downstream
tasks (Guo et al., 2022 Ning et al. 2023 Contrastive VLMs, such as CLIP (Radford et al.)
2021 align
images and text via largo-scale contrastive objectives and demonstrate impressive zero-shot transfer on recog-
nition lasks NNukrai el al. 2022 Qiu et al. 2024 Liao el al. 2022 Ning el. al. 2023
Yao el al.
2022 Peebles
& Xie
2022 With the rise of large language models (LLMs), subsequent works integrate LLMs with visual
encoders to enhance image-conditioned text generation LLi et al.
2023b
Dai et al.
2023
Liu et al.
2023b;
Zhu et al. 2023 Chen et al. 2023 Lin et al.
2023), enabling instruction following and open-ended reasoning.
More recent efforts (Liu et al. 2023a Li et al. 2024 Gao et al. 2024 Bai et al. 2025) further improve visual
capabilities by leveraging higher-resolutee
inputs, multi-image contexts, and even video sequences. Despite
these advances, VLMs remain prone to hallucinations when aligning visual evidence with textual responses, motivating preference-based optimization approaches to improve faithfulness.
6.2
Preference Alignment in Large Language Models
The alignment problem
Leike et al.
2018 aims to ensure that agent bohaviors are consistent with human
intentions. Early approaches leveraged Reinforcement Learning from Human Feedback (RLHF) (Bai et al.
2022 Nakano et al. 2021 Ouyang et al.) 2022 Scheurer et al.) 2023 Stiennon et al. Ziegler ct al. 2019 where policy optimization methods such as PPO (chhulann to maximize human-labeles rewards. More recently, Direct Preference Optimization (DPO) [Rafailov et al. 2024 reformulates aligmment as a direct optimization problem over offline preference data, avoiding the need for reinforcement learning. Extensions such as Gibbs-DPO (Xiong et al. 2023) enable online preference optimization, while 3-DPO (Wu et al. 2024) addresses sensitivity to the temperature parameter 3 by introducing dynamic calibration with a reward model. However, such approaches are less effective multimodal domains, where reward models are vulnerable reward hacking (Sun et al., 2023b). Other works explore alternative strategies, such as self-rewarding mechanisms or data reweighting (Wang et al. 2024b Zhou et al. 2024a), to mitigate issues like overconfident labeling and distributional bias in preference data.
2022a
b Glaese et al.
2020
Wu et al.
2021
ct al.
2017) were used
6.3 Preference Alignment in Multimodal Models
Recent efforts have extended preference aligument from language-only models to the multimodal domain. A major line of work focuses on constructing multimodal preference datasets. (Sun al., 2023b; Yu eet al. 2024a) collect human annotations, while others rely on powerful multimodal models such as GPT-
4V (Li ct al. 2023c Yu et al. 2024b Zhou et al. 2024d
Yang al. 2025) to generate preference signals.
However, both human annotation and large model inference incur prohibilive costs, limiting scalability. To
2024 explore automatic or self-training methods for preference data generation. These works senthesize dis-preferred responses from corrupted images or misleading prompts, enabling the model to learn preferences without external supervision. Another direction (Ouali et al. 2024)) leverages CL.IP to score diverse candidate responses, ranking preferred versus dis- preferred outputs using image text similarity. From the perspective of training objectives, most multimodal alignment methods adopt the standard DPO objective
address this, alternative approaches (Deng et al.
2024 Pi et al.
(Li et al. 2023c Zhao et al.) 2023 Zhou et al., 2024b) to optimize preferences paired data. Other approaches employ reinforcement learning (Sun et al, 2023b) or contrastive learning (Sarkar et al.) 2024 Jiang et al. 2024) to improve alignment. To further reduce overfitting to language-only signals, Ouuali et al.) 2024) exlends DPO by jointly optimizing both textual and visual preferences. Despite these advances, multimodal DPO remains vulnerable to overfitting, espocially when trained on imbalanced preference data. In this work, we introduce a general difficulty- aware training framework that explicitly accounts for sample difficulty, thereby mitigating overfitting and improving robustness in multimodal preference optimization.
12

---

Published in Transactions on Machine Learning Research (12/2025)
7 Conclusion
In this work, present an empirical analysis of the overfitting issue in multimodal preference optimization, which often stems from imbalanced data distributions. To address this, we introduce DA-DPO, a cost- efficient. framework consisting of difficulty estimation and difficulty-waree training. Our method leverages pretrained contrastive and generative VLMs to estimate sample difficulty a training manner, and uses these estimates to adaptively reweight emphasizzn harder samples while preventing overfitting easier ones. Experiments across hallucination and general-purposs benchmarks demonstrate that this paradigm effectively improves multimodal preference optimization.
Limitations Despite these promising results, framework relies the assumption that pretrained VLMs provide reliable evaluations of preference data. Although the adaptive voting strategy shows ro- bustness on existing datasets, its generalizability to domains that differ substantially from the pretraining objectives of these VLMs remains uncertain. Future work may explore integrating domain-adaptive or self- improving mechanisms to further enhance robustness.
8 Acknowledgements
This work was supported by NSFC 62350610269, Shanghai Frontiers Science Center of Human-centered Ar- tificial Intelligence, and MoE Key Lab of Intelligent Perception and Human-Machine Collaboration (Shang- haiTech University). This work was also supported by HPC platform of ShanghaiTech University.
References
Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Devi Parikh, and Dhruv Batra. Vqa: Visual question answering. International Journal of Computer Vision, 123:4 31, 2015.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large rision-languse model with versatile abilities. ArXiv, fbs//238.11290 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint Xrx::22502.113 13923, 2025.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2fx::2224405880 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional Harmlessness from ai foodback. arXiv preprint ar orXi:2221.080733 2022b.
Jun Chen, Deyao Zhul Xiaoqian Shen1 Xiang Li, Zechun Lin2 Pengchuan Zhang, Raghuraman Krishnamoor- thi2 Vikas Chandra2 Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: Large language model as a unified interface for languael multi-task learning. arXiv preprint arXiv:2310.09) 1011000.047.. 78, 2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Hual Tiong, Junqi Zhao, Weisheng Wang, Boyang Al- bert Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. ArXiu, abs/2205.06600, 2023.
Yihe Deng, Pan Lu, Fan Yin, Ziniu Hu, Sheng Shen, James Zou, Kai-Wei Chang, and Wei Wang. Enhancing large vision language models with self training on image comprehension. arXiv preprint orXiv:2405.1977 19716, 2024.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint XXx::2366.1339, 2023.
Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, Kaipeng Zhang, Wenqi Shao, Chao Xu, Conghui Hc, Junjun He, Hao Shao, Pan Lu, Hongsheng Li, and Yu Qiao. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models. ArXiv, abs/ abs/2402.05933, 2024.
Amelia Glaese Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiu:2200.14977 2022.
Ziyu Guo, Renrui Zhang, Longtian Qiu, Xianzheng Ma, Xupeng Miao, Xuming He, and Bin Cui. Calip: Zero-shot enhancement of clip with parameter-free attention. arXiv preprint arXim:2209. 14169, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 09685, 2021.
Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and com- positional question answering. 2019 IEEE/CVF Conference Compuler Vision and Pollern Recognation (CVPR), pp. 6693 6702, 2019.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. ArXiv, abs/2102.05911 2021.
Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition, pp. 27036-27046, 2024.
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.0787/ 2018.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. ArXiv, abs//240800336,, 2024.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal Iims with generative comprehension. ArXiv, abs/2307.16125, 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint rrXi::2311.15017 2023b.
Lei Li, Zhihui Xie, Mukai Lú, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. arXio preprint arXiv:2312 10665, 2023c.
Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. ArXiv, hbs/2312.10665, 2023d. URL https://api. semanticscholar Prrg/CorpusI:::000
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training, In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 10965 10975, 2022.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXio preprint arXiv:2305. 10355, 2023e.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang, Xiaobo Li, and Si Liu. Gen-vlkt: Simplity association and enhance interaction understanding for hoi detection. 2022 IERE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20091 20100, 2022.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Ziknick. Microsoft coco: Common objects in context. In Compuler Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 62, 2014, Proceedings, Part V 13, pp. Springer, 2014.
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, Jiaming Han, Siyuan Huang, Yichi Zhang, Xuming He, Hongsheng Li, and Yu Jiao Qiao, Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. ArXee, abs/2311.07577 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. ArXie, abs/ abs/2310..3374, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. orXiv preprint arXiv:2304.088. 08485, 2023b.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3190-3199, 2019.
Minesh Mathew, Dimosthenis Karalzas, and CV Jawahar. Docvqa: A dalaset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2021.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted guestion-answerii with human feedback. arXie preprint arxiv:21 093393 2021.
Sha Ning, Longtian Qiu, Yongfei Liu, and Xuming He. Hoielip: Efficient knowledge transfer for hoi detection with vision-language models. 2023 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 13507-23511 2023.
David Nukrai, Ron Mokady, and Amir Globerson. Text-only training for image captioning using noise- injected clip. ArXiv, abs 0bs/2211.00075 2022.
OpenAL. Vision openai api. https://plattooms http:://paattoompoppaalcoooaaoop httpaa/cc/mcccco//ss////////////
2023.
Yassine Ouali, Adrian Bulat, Brais Martinez, and Georgios Tkimiropoulos. Clip-dpo: Vision-language models as a source of preference for fixing hallucinations in Ivims. arXie preprint Xrxix:2408. 104333 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almcida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730 277444, 2022.
William S. Pecbles and Saining Xie. Scalable diffusion models with transformers. 2023 IEEE/CVF Inter- national Conference on Computer Vision (ICCV) pp. 4172 4182, 2022.
Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang, Runtao Liu, Rui Pan, and Tong Zhang. Strengthening multimodal large language model with bootstrapped preference optimization. ArXio, abs/2403.08730, 2024. URL https://aappp semanticscholar wwt.r///rr...ooooooooooo/////110
Longtian Qiu, Shan Ning, and Xuming He. Mining fine- aieeggraindd image-text alignment for zero-shot captioning via text-only training. ArXiv, 00//22001023/4 2024.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Rafacl Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea. Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning, In Conference on Empirical Methods in Natural Longuage Processing, 2018. URL http::::/app.semanticsccse http://t.c.cctooooooooorrorroopa :tt
Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan Ô Arı,, and Tomas Pfister. Mitigating object hallucination via data augmented contrastive tuning. arXiv preprint arXiv:2405.1865, 18654, 2024.
Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with language feedback at scale. arXiv preprint arXiv:2303.16755 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimiza- tion algorithms. arXiv preprint arv: 7707003347 00344, 2017.
Nisan Stiemnon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-300 2020.
Quan Sun, Yuxin Fang, Ledell Yu Wo, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. ArXiu,
2023a.
Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309. 14525, 2023b.
Fei Wang. Wenxuan Zhou, James Y. Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. mdpo: Conditional preference optimization for multimodal large language models. Arxiv, Abs/2406.11833 2024a.
URL https://api. http:/pppp..mmmnicccccccccchooaa URR.Attp:://ppp...mmnttcccccoooo
org/ IDtg/orppssI:::000 270560448
Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. An l1m-free multi-dinensional benchmark for milmss hallucination evaluation. arxiv preprint arXiv:2311.07397, 2023a.
Weihan Wang, Qingsong Lv, Wemmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang, Cogylm: Visual expert for pretrained language models. ArXiv, 0///2211103007 2023b. URL https: //api. semmmeiccnniccchllrrr
ID : 265034288
Zhaoyang Wang, Weilei He, Zhiyuan Liang, Xuchao Zhang, Chetan Bansal, Ying Wei, Weitong Zhang, and Huaxiu Yao. Cream: Consistency regularized self-rewarding language models. ArXiu, abs/ Abs/2410.118 12735,, 2024b. URL https://api. semanticscholar. wttp://orpus...:2237755555
Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human foodback. arXiv preprint arXiv:2109. 10862, 2021.
Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiang- nan He. -dpo: Direct preference optimization with dynamic. arXiv preprint arXiv:2407.08660 08639, 2024.
Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang. Gibbs sampling from human feedback: A provable kl-constrained framework for rlhf. arxie preprint arXi::2322.11450 2023.
16

---

Published in Transactions on Machine Learning Research (12/2025)
Zhihe Yang, Xufang Luo, Dongqi Han, Yunjian Xu, and Dongsheng Li. Mitigating hallucinations in large vision-language models via dpo: On-policy data hold the key. 2025 CREE/CVF Conference Compuler Vision and Pattern Recognition (CVPR), pp. 10610 10620, 2025.
Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, W. Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: Dictionary-cnriched visual-concept paralleled pre-training for open-world detection. ArXie, hos/2209.094000 2022.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Mingshi Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owi2: Revolutionizing multi-modal large language model with modality collaboration. 2024 IEEE/CVF Conference Computer Vision and Pottern Recognation (CVPR), pp. 13040-13051, 2023.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen Hc, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Ta Zheng, Maosong Sun, al. Rlhf-v: Towards trustworthy milmss via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13807 13816, 2024a.
Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. Rlaif-v: Aligning milms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint ar2::2405.11220, 2024b.
Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond halluci- nations: Enhancing Ivims through hallucination-awarre direct preference optimization. ar preprint arXiv:2311.16833 2023.
Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, and Chenguang Zhu. Wpo: Enhancing rlhf with weighted preference optimization. In Conference on Empirical Methods in Natural Language Processing, 2024a. URL https://api. http:tttt////pp.eemmatiti http:/ppp..emmnticcccclloollaa http://ccccoaaaaorrorrrrrppspppp org/CorpusID:
270559089
Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, and Chenguang Zhu. Wpo: Enhancing rlhf with weighted preference optimization. arXiv preprint arXiv:2406.1188 2024b.
Yiyang Zhou, Chenhang Cui, Rafacl Rafailov, Chelsca Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference fine-tuning. ArXiv, abs/2402.11411, 2024c.
Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint arXic:2402.11411, 2024d.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint ar arxiv:2904. 10592, 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.0553 08593, 2019.
17

---

Published in Transactions on Machine Learning Research (12/2025)
Table 7: Fusion Ratio Ablation. The Fusion Ratio indicate the weight for the Eg and my Equation The row with blue color is the weight of adaptive score fusion weight.
Fusion Ratio
AMB.
AMB. P Seed'
CLIP MLLM C, & Cov.
Hal. Cog.
4
F1 f Score
20%
80%
4.9
57.7
30.9
2.0
85.4
63.2
40%
60%
4.3
57.5
28.3
2.0
85.6
64.0
53%
17%
4.3
57.4
28.0
2.1
85.6
64.8
60%
40%
4.5
57.6
28.4
2.1
85.3
63.8
80%
20%
4.6
58.3
28.6
2.0
85.5
62.8
Table 8: Performance with different 3 for DA-DPO and DPO. We provide the results of DPO and our diffcuult-awaree DPO trained in the BPO dataset. We mark the reported hyperparameter with blue. For DA-DPO, we chose the best performance 8, which is 0.2. For DPO, we follow previous work reports that B equals to 0.1.
AMB.
AMB. D
ObjHal
POPE
GQA
Seed' MME MMEC
C, Cov.
Hal. & Cog. +
FIf
C.
C.
FI
Score
Acc t SSoore t
Score 1
DA-DPO
8 = 0.1
4.6
59.0
32.5
2.0
85.4
40.4
10.2
85.6
58.6
64.2
1398.2
312.8
8 0.2
4.3
57.4
28.0
2.1
85.6
39.7
9.9
85.9
59.8
64.8
1406.6
323.2
3 0.3
4.7
53.6
25.8
2.4
86.3
38.6
9.5
84.8
56.4
58.4
1414.2
315.6
3 0.4
4.9
54.2
28.4
2.6
86.0
41.6
10.4
85.2
58.4
62.2
1378.2
306.9
DPO
8 = 0.1
5.5
58.4
35.7
2.0
83.9
43.3
10.0
84.3
45.3
57.7
1409.4
315.0
8 0.2
4.8
58.5
30.0
2.0
83.4
44.2
10.2
83.8
58.5
47.9
1369.1
299.6
8 = 0.3
4.9
56.6
29.4
2.0
84.4
45.3
10.8
84.3
56.8
54.7
1387.1
310.7
0.4
4.9
56.5
30.8
2.0
84.5
45.3
12.1
85.2
58.9
55.4
1402.4
312.3
A Influence of Adaptive Score Fusion
The proposed framework integrates two types of pretrained VLMs to assess the difficulty of pairwise pref- erence data. We design an adaptive score fusion mechanism to determine the weight of each reward model without requiring hyperparameter selection from the data perspective. As shown in Table 7 we present the results for multiple fixed fusion scores. We observe that the adaptive score fusion achieves the best performance.
B Sensitivity of B
We propose difficulty-awan preference optimization strategy that aims to alleviate the weefitting-to-eass sample issue during alignment training. To achieve this, we dynamically calibrate the 8, which is described in the main paper. However, our method affects the scale of the 3 in the DPO objective. As shown in Table S we provide the ablation of the 3 between the DPO and DA-DPO, From the results, we observe that varying I leads to mixed performance across different benchmarks for both DPO and DA-DPO. Nevertheless, the proposed DA-DPO consistently achieves better overall performance compared vanilla DPO, indicating that our difficulty-aware calibration effectively enhances robustness while mitigating hallucinations.

---

Published in Transactions on Machine Learning Research (12/2025)
EPO Oetaas
(a) BPO
(b) VLFeedBack
Figure 5: The visualization correlation between pretrained VLMs. We present the normalized
MLLM and CLIP scores for the same preference data sample three datasets, BPO and VLFeedBack. The x-axis the CLIP score and the y-axis the ML.LM score. The red line indicates the MLLM and CLIP's predictions are the same.
Table 9: Classification accuracy of fusson scores on the BPO and VLFeedback datasets.
Dataset
RM Type
VQA
Caption
Text VQA
Overall
BPO
CLIP
0.5877 0.8478
0.8947
0.9827 0.6282
0.7732
BPO
MLLM
0.5053
0.6415
VLFeedback
CLIP
0.5827 0.8104
0.6352
0.5897
VLFeedbacc
MLLM
0.5193
0.7715
C Pretrained VLMs Correlation
In our proposed DA-DPO framework, we utilize the CLIP and the MLLM to evaluate the difficulty of pref- erence data from different perspectives. To validate this claim, we provide the score of difficulty correlation between the two VLMs. As shown in Figure the scores from the two models exhibit no significant positive correlation, as evidenced by their weak correlation coefficient. This indicates that the two models capture different aspects of the response evaluation, and their scoring patterns do not align consistently across the datasets.
D Fusion Score Classification Accuracy
To provide comprehensive understanding of the score fusion process, supplement the main text with detailed numerical results and evaluation metrics. Score fusion is conducted at the dataset level, where distinct fusion scores are employed for the BPO and VLFeedback datasets.
Evaluation Metric. We compute classification accuracy on pairwise preference data using the following criterion: for each sample, the VLM assigns a score to both the chosen and rejected answers. sample is considered correctly classified the chosen answer receives higher score than the rejected one.
Results. Table reports classification accuracy for each category (VQA, Caption, and Text VQA, where applicable) and the overall accuracy both datasets. We evaluate two types of VLMs: vision encoder (CLIP) and multimodal large language model (MLLM). Specifically, we use EVA-CLIP 8B as the CLIP model and LLaVA-1.5 7B as the MLLM.
Based on the overall accuracy, we designate the CLIP-based classification score (cls_c) and MLLM-based classification score (cls_m) m) follows: For the BPO datasct, cls_c 0.7732 and cls 0.6415; for the VLFeedback dataset, cls_c _ 0.5897 and cls m 0.7715. These results demonstrate the complementary strengths of CLIP and M-.L-- reward estimations across different categories.
2

---

Published in Transactions on Machine Learning Research (12/2025)
Table 10: Compute and Time Estimation for Reward Model and Proxies. FLOPs are shown in scientific notation, and Wall-clock hours are estimated on an NVIDIA A100 GPU. The data for the reward model training set to 180k.
Task
FLOPs
Wall-clock hours (A100)
Traditional Approach
Training Reward Model
2.6 x 10
241.9
Reward Model Estimation
5.1 10
40
DA-DPO
CLIP Estimation
1046
0.7
MLLM Estimation
5.1 10
40
Table 11: Ablation of data difficulty normalization. We report the performance of different normal- ization strategies trained with LLaVA-v1.5 7B on the BPO datasel.
AMB. G
AMB. p ObjHal
POPE
GQA
Seed!
MME
MME
C, Cov.
Hal. 4. Cog. +
F1 f
C.
C,
F1 Score f Acc f
Score
Score
DPO
5.5
58.4
35.7
2.0
83.9
43.3
10.0
85.6
45.3
57.7
1409.4
315.0
Ranked-based
4.8
57.2
31.3
1.9
85.7
42.7
10.7
86.1
59.5
62.1
1404.8
302.5
Tength-controlled
4.7
57.0
29.3
1.9
85.9
37.7
9.6
85.9
60.0
64.1
1402.6
316.0
Gaussian Normalization
4.3
57.4
28.0
2.1
85.6
39.7
9.9
85.9
59.8
64.8
1406.6
323.2
E
Efficiency Comparison
To validate the efficiency advantage of our proposed framework, we compare the computational cost in terms of FLOPs and wall-clock time between traditional reward model training and evaluation, and our estimation- based approach.
summarized in Table [0] training a full reward model requires over 6 times the wall- clock time of MLLM-basee estimation and approximately 345 times that of CLiP-based estimation. These results clearly demonstrate that method achieves significantly higher efficiency compared to conventional approaches.
F Difficulty Estimation Normalization Ablation
To evaluate the impact different normalization strategies on data difficulty estimation, we conducted experiments with two alternative approaches. The first strategy, Ranked- based, normalizes scores by ranking the data and projecting the values lincarly into the J0, 1] range, instead of using Gaussian normalization. The second strategy, Lengll-controlled, is applied during the MLLM generative estimation: the sum of log probabilities in Equations 7 and 8 is divided by the response token length to mitigate length bias. As shown in Table II all three normalization strategies achieve performance improvements over vanilla DPO. We also observe that the Ranked- approach slightly underperforms the other two strategies, suggesting that preserving the original distribution of estimated scores is beneficial for effective difficulty estimation.
G Experiments with more Model Variants
To validate the effectiveness of our proposed method beyond the LLaVA scries, conduct a small-scale experiment on Qwen2.5-VL 3B. We use a subset. of 50k samples from the BPO dataset to train both the baseline DPO and our proposed DA-DPO. The results are presented in Table [2] We observe that DA- DPO achieves better reductions in hallucination metrics compared DPO, while retaining performance on comprehensive benchmarks. This demonstrates that DA-DPO is a general framework, not limited LLa models.
3

---

Published in Transactions on Machine Learning Research (12/2025)
Table 12: Comparison of hallucination and general benchmarks across methods. Results are re- portod on Qwen2.5-VI VL 3B and models trained with DPO and DA-DPO. The proposed DA-DPO consistently reduces hallucination metrics while maintaining improving general ability.
AMB.
AMB.
ObjHal
POPE
GQA
Seed
MME
MME
C. + Cov.
Hal. + Cog- 4
f
C2 4 C,
f
Score 1 Acc
Score
Score f
Qwen2.5-Ve 5L 3B
7.5
68.0
48.8
5,0
89.6
30.7
7.6
88.6
62.5
75.9
1607.9
625.7
DPO
6.6
62.5
47.7
3.4
87.3
30.0
8.4
78.5
55.5
75.9
1558.5
616.4
DA-DPO
6.5
57.6
42.3
2.6
88.3
25.0
7.3
80.6
56.4
76.0
1561.5
623.9
The image features yourg girl riding yellow bicycle pink sidewal.. The sidewelk estends from the left edge the image to the right edge, providing clear path for the ride her bike. scene captures moment of outdoor activityy and lehure the girl.
DA-DPO
features
riding yellow bloycle sddwalk... The bicycle positioned niiigg towards the left.. The atmosphere the activity.
DPO
towards
adding
features ridlin bicycle sidewalk, focus the There other people background, but they
background...
LLaVA v1.5 7B
Question: Describe the image.
liickngg athletidism
10
DA-DPO
Anather
7
captures lively with the foreground... There other background, closer edges possibly obsenvng
field. towards him,
LLaVA 7B
players
ticking
image.
likeyy
Figure 6: Visualization of Predictions. We present the outputs from the proposed method alongside baseline models to highlight the characteristics of our approach. The results are derived from the generative component of AMBER, which is designed to detect hallucinations in image captions.
Visualization To provide a better understanding of our method's performance, we present visualizations of the model's outputs, comparing them with the results obtained from both the DPO and reference models in Figure These visualizations highlight the ability of our approach more effectively reduce hallucina- tions. By examining the outputs, is evident that our method aligns better with the expected responses, demonstrating superior accuracy in scene understanding and coherence in generated content.


Reviews:
[{"review_id": "PJtpWNPKUw", "reviewer": "Reviewer_fFru", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents DA-DPO, which leverages external difficulty proxies to dynamically steer the divergence penalty strength in the DPO algorithm. With CLIP and multi-modal large language models (MLLMs) as the two difficulty proxies, the authors propose the score margin for the chosen and rejected responses given the image as a heuristic to determine the appropriate divergence penalty strength. The experimental results are shown across different sizes and models on multiple benchmarks, both for hallucinations and comprehensive benchmarks. While the paper provides comprehensive experiments, the motivation and method remain unclear. Overall, the paper is promising in direction and provides valuable empirical results, but would benefit from clearer motivation and a more rigorous treatment of the difficulty concept.", "claims_and_evidence_yn": "No", "claims_explanation": "1. **Unclear connection between \"*difficulty score*\" and preference difficulty**: As the title suggests, DA-DPO builds on an assessment of sample “difficulty” in multimodal preference learning. The paper introduces two proxies: (1) the CLIP-score margin (cosine similarity margin between chosen and rejected responses given the image), and (2) the MLLM log-likelihood margin. However, their interpretation remains unclear.\n\n   - Why should a larger CLIP-score margin imply easier distinguishability, and in turn, why does this make the sample an easier instance for preference optimization?\n   - In what sense does the MLLM log-likelihood margin reflect difficulty?\n\n   While the high-level intuition is understandable, the connection to preference learning is neither theoretically grounded nor empirically substantiated. For example, the CLIP margin reflects image–text relevance, which does not necessarily capture the implicit reward separability discussed in Section 3. Similarly, the log-likelihood margin $m_c - m_r$ may serve as a proxy under reference-free algorithms such as SimPO [1] or ORPO [2], where it aligns with the logistic loss term. But in DPO, the implicit reward $\\log \\frac{\\pi\\_\\theta(y|x)}{\\pi\\_\\mathrm{ref}(y|x)}$ always starts with zero, making the connection less direct. A stronger theoretical or empirical justification is needed, and the manuscript currently does not cite related work that could support this interpretation.\n\n2. **Missing analysis on the reward dynamics and over-optimization**: Section 3 motivates DA-DPO by suggesting that standard DPO can suffer from over-optimization (or overfitting) when difficulty variance is high, and that difficulty-aware weighting mitigates this. However, the analysis presented is based on training-set reward trajectories. To substantiate the claim, evaluation on held-out or validation sets would be essential [3-5]. Without such evidence, it remains unclear whether DA-DPO actually alleviates over-optimization in the sense of improved generalization, rather than simply altering training dynamics.\n\n3. **Additional questions**\n   - In Table 1, results across the “Main Results” section appear mixed, especially on comprehensive benchmarks. Could the authors provide more insight into why these tendencies arise?\n   - In Appendix C, the authors report DA-DPO performances with varying $\\beta$. However, as in Equation (11), $\\hat{\\beta}$ is a dynamic function of difficulty scores and the static $\\beta$ is not directly present in $\\mathcal{L}_\\mathrm{DA-DPO}$. How should this ablation be interpreted?\n   - Why is the method specifically targeted at hallucination mitigation? The method itself is rather designed for addressing the difficulty-awareness in DPO, not hallucination-specific, especially in Sections 3 to 4. Is it because the gains are more notable in the hallucination benchmarks?\n\n&nbsp;\n\n**References**\n\n[1] Meng et al., 2024, \"SimPO: Simple Preference Optimization with a Reference-Free Reward.\"\n\n[2] Hong et al., 2024, \"ORPO: Monolithic Preference Optimization without Reference Model.\"\n\n[3] Gao et al., 2023, \"Scaling Laws for Reward Model Overoptimization.\"\n\n[4] Rafailov et al., 2024, \"Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms.\"\n\n[5] Shi et al., 2024, \"Understanding Likelihood Over-optimisation in Direct Alignment Algorithms.\"", "audience_yn": "Yes", "audience_explanation": "The paper's approach clearly has empirical value, especially regarding its wide scope of evaluation from hallucination-specific benchmarks to comprehensive ones. Also, the direct alignment algorithms that are specifically targeted to MLLMs are not yet widely studied compared to the plain language models. Thus, the empirical results can draw the interest of the audience.", "requested_changes": "1. **Clarify the definition of difficulty**: Provide theoretical or empirical justification for why CLIP-score and MLLM log-likelihood margins correspond to preference difficulty in DPO.\n2. **Over-optimization analysis**: Add validation or held-out results to demonstrate that DA-DPO mitigates over-optimization in generalization, not just in training dynamics.\n3. **Interpretation of $\\beta$ ablation**: Clearly explain how varying $\\beta$ in Appendix C interacts with Equation (11), and what the reported ablation actually measures.\n4. **Empirical support for difficulty proxies**: Include correlation or binning studies showing how the proposed difficulty scores relate to optimization success or generalization error.\n5. **Mixed benchmark results**: Offer analysis or discussion of why DA-DPO improves some benchmarks but not others.\n6. **Clearer connection between the method and hallucination**: While the title emphasizes hallucination mitigation, it is hard to find the connection between hallucination and the core motivation/method of the paper; it can only be found in the experiments.", "broader_impact_concerns": "No specific concerns regarding the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hHWxXm4mSL", "reviewer": "Reviewer_T6ss", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes DA-DPO, a framework for reducing hallucinations in multimodal large language models (MLLMs) by addressing overfitting in Direct Preference Optimization (DPO). The key idea is that MLLMs tend to overfit on “easy” preference pairs, while failing to learn from “harder” and more nuanced examples. To mitigate this, DA-DPO introduces:\n* Difficulty Estimation: a training-free method that leverages both contrastive (e.g., CLIP) and generative (e.g., LLaVA) vision-language models to assign difficulty scores to preference pairs, fused via distribution-aware voting.\n* Difficulty-Aware Training: reweighting preference data based on difficulty, down-weighting easy samples and emphasizing harder ones, thereby improving robustness without requiring new data or extra fine-tuning stages.\n\nKey strengths:\n* Clear identification of a real weakness in current multimodal preference optimization (bias toward easy samples).\n* Cost-efficient framework that does not require additional human annotations or training of new models.\n* Extensive experiments on multiple hallucination and general benchmarks showing improved robustness and reduced hallucinations.\n* Ablation studies confirm the contribution of each component (CLIP vs. MLLM estimators, fusion strategy, filtering baselines).\n\nKey weaknesses:\n* Some figures are confusing or hard to read (e.g., Figure 1 and Figure 3).\n* Lack of clarity about the proportion of easy/hard/ambiguous samples in the datasets.\n* Limited discussion on why DA-DPO’s results are not directly comparable to prior approaches like HA-DPO, CLIP-DPO, or mDPO, although these often show stronger raw results in Table 1.\n* Hardware details are incomplete, making reproducibility harder.\n* Presentation issues (minor formatting problems, e.g., “LoRA” written with a lowercase o).", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper provides empirical evidence of overfitting on easy samples through reward dynamics analyses. Benchmarks across hallucination (AMBER, MMHal, ObjectHal, POPE) and general MLLM evaluations (SeedBench, GQA, LLaVA-Bench, MME) consistently show that DA-DPO outperforms standard DPO, particularly on hallucination metrics. Ablations (e.g., CLIP-only, MLLM-only, fusion strategies, filtering baselines) convincingly demonstrate that both difficulty estimation and difficulty-aware weighting are necessary. However, the comparability of results with alternative optimization methods (HA-DPO, mDPO) is insufficiently addressed.", "audience_yn": "Yes", "audience_explanation": "The problem of hallucinations in MLLMs is central to ongoing research, with both theoretical and practical implications. Preference optimization is a rapidly developing area, and a cost-efficient approach that improves robustness without additional supervision is of clear interest to researchers working on alignment, multimodal reasoning, and scalable training methods.", "requested_changes": "Figures:\n* Improve Figure 1 readability (larger font, avoid overlaps).\n* Revise Figure 3 to more clearly separate CLIP-based vs. MLLM-based scoring examples.\n\nClarity:\n* Explicitly report how many samples are classified as easy, hard, and ambiguous.\n\nComparisons and baselines:\n* Clarify why HA-DPO, CLIP-DPO, and mDPO results are “not directly comparable,” especially since they achieve higher scores in some benchmarks.\n\nReproducibility:\n* Provide details on hardware (GPUs, memory, batch size) for reported training times.\n* Discuss training cost more explicitly to strengthen the claim of “cost-efficient.”\n\nMinor corrections:\n* Fix capitalization of “LoRA.”\n* Improve caption descriptions in figures to guide interpretation.", "broader_impact_concerns": "The method primarily addresses hallucinations in multimodal models, which is a positive contribution toward more reliable AI. Broader impacts are relatively limited compared to other alignment methods (e.g., RLHF with human data), since the approach relies on existing pretrained models. However, reliance on pretrained VLMs (CLIP, LLaVA) for difficulty estimation may inherit their biases, which could indirectly affect model alignment quality. This limitation should be more explicitly acknowledged.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IufcvIJfFH", "reviewer": "Reviewer_u6Ks", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces DA-DPO, a modification to Direct Preference Optimization (DPO) for multimodal large language models (MLLMs). The central idea is to compute a per-pair difficulty score without training any auxiliary models, using two complementary signals: (1) a contrastive image–text gap from a pretrained CLIP model and (2) a generative likelihood gap from a reference MLLM. These normalized signals are fused into a single scalar that scales the DPO logit per example, thereby down-weighting “easy” preference pairs and emphasizing “hard” pairs. The paper argues that this directly addresses an observed tendency of standard DPO to overfit easy pairs, which contributes to visual hallucinations. Experiments across several LLaVA-family models report reduced hallucination rates and smaller capability regressions relative to vanilla DPO. Additional ablations indicate benefits over simple data filtering and show that the two difficulty signals are complementary.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Overall, the paper’s claims are substantiated for the most part, with several focused gaps that could be addressed.\n\n**(C1) Empirically demonstrate the existence of an overfitting issue**\n\n* **What is substantiated:** The paper’s Section 3 presents a training-dynamics analysis showing that models learn more from “easy” pairs than from “hard” ones, using reward trends and relative reward-gap plots as seen in Figure 2 and Figure 6.\n\n* **Where it falls short:**\n    * The analysis defines difficulty using only the tails of the CLIP score distribution and discards mid-range cases, which could exaggerate the observed gap-shrinking effect.\n    * The paper does not present a classical train-vs-held-out divergence consistent with standard definitions of overfitting.\n    * The reward curves appear to be from a single seed without uncertainty bands, so their robustness is unclear.\n    * The paper asserts, but does not demonstrate, a within-run link between smaller reward gaps and fewer hallucinations.\n    * The evidence for the initial analysis relies on a single proxy (CLIP), which may reflect a bias of that particular scorer.\n* **What is needed:** This claim would be better substantiated by analyzing the full difficulty spectrum (e.g., ranked buckets) and varying the cutoff thresholds. The analysis could be strengthened by computing buckets on a held-out split to show easy-vs-hard performance there. For robustness, the paper should report results from $\\geq 3$ seeds with mean $\\pm$ standard deviation and perhaps a concise area-under-gap summary. To establish a clearer mechanism, the analysis could correlate checkpoint-wise gap reductions to changes in hallucination metrics. Finally, the dynamics analysis should be replicated with an MLLM-based and a fused proxy (ideally with length-controlled likelihoods) to ensure the finding is not proxy-dependent.\n\n**(C2) Propose a cost-effective framework that uses pretrained VLMs to estimate difficulty and improves preference modeling via difficulty-aware training**\n\n* **What is substantiated:** The paper introduces a drop-in DPO modification (Eq. 12) that scales the logit per example using normalized CLIP and MLLM gap scores, fused by a distribution-aware rule. No auxiliary models are trained, and ablations show CLIP-only, MLLM-only, and fused variants, with fusion generally being the strongest approach.\n\n* **Where it falls short:**\n    * The “cost-effective” claim is qualitative; the paper provides training times but lacks a full FLOPs or wall-clock accounting for the data scoring and training stages.\n    * Fusion weights are learned on the training distribution and may simply mirror its task mix (e.g., caption vs. VQA), rather than capturing true, generalizable reliability. If the weights encode dataset proportions, the “difficulty-aware” gains could fade or flip under a different data mix, weakening the claim that the framework is broadly effective.\n    * The effects of normalization choices and text length/verbosity on the difficulty scores are not stress-tested. If the difficulty signals are sensitive to scaling choices or reward longer answers, the method may be optimizing for artifacts of the scoring procedure rather than genuine hardness.\n* **What is needed:** This claim needs stronger support. The paper should add a compute table (FLOPs and wall-clock time) for CLIP scoring, MLLM likelihood scoring, and fine-tuning, with a comparison to a baseline like training a small reward model. The fusion mechanism could be validated by re-estimating weights on a held-out split or per-task bucket, alongside ablations with fixed (50:50) weights. The paper should also include ablations on the normalization scheme (e.g., rank/percentile-based) and use length-controlled (per-token) likelihoods for the MLLM score to ensure robustness.\n\n**(C3) Significantly enhance hallucination metrics while preserving (or improving) general ability across multiple MLLMs**\n\n* **What is substantiated:** The paper’s main tables show consistent reductions on hallucination metrics alongside smaller drops (and occasional gains) on comprehensive benchmarks across LLaVA-v1.5-7B/13B and OneVision-7B. Comparisons to filtering baselines favor the proposed reweighting approach over data deletion.\n\n* **Where it falls short:**\n    * Statistical and tuning robustness are unclear. The paper appears to report single-seed results. While an appendix table shows a sweep for the DPO temperature ($\\beta$), the main comparisons may not be against the optimal baseline setting. Without uncertainty estimates and fairer baseline comparisons, the reported improvements could be due to noise or suboptimal baselines.\n    * Generalization and evaluator robustness are not demonstrated. The evidence is confined to the LLaVA model family and a single evaluation setup; alternative model architectures and automatic graders are not examined. The claimed gains may not transfer beyond the tested models or could be dependent on a particular evaluator's artifacts.\n* **What is needed:** The claim would be better supported by reporting multi-seed results with uncertainty and paired statistical tests. The paper should include $\\beta$ sweeps for vanilla DPO for each model/dataset and ensure comparisons are made against each baseline’s best performance, ideally visualized with metric-versus-$\\beta$ plots. To show broader applicability, a small-scale run on a non-LLaVA model (e.g., Qwen-VL/InternVL) would be valuable. Finally, rescoring a subset of outputs with a different automatic evaluator (or a small human audit) would confirm that the improvements are robust to the choice of evaluator.", "audience_yn": "Yes", "audience_explanation": "This work addresses a practical, high-impact pain point, multimodal hallucination, via a minimal change to a widely adopted alignment method (DPO). Its training-free difficulty estimation and per-example scaling are straightforward to integrate into existing pipelines, offering a scalable alternative to data filtering that preserves data diversity. The analysis of easy-pair overfitting and the curriculum-like remedy provide conceptual clarity that can inform future work on per-example temperatures, multi-signal difficulty estimation, and robust preference optimization in multimodal systems.", "requested_changes": "1. **[Definite] Statistical robustness:** The paper would be strengthened by providing results over at least three random seeds with uncertainty (mean $\\pm$ s.t.d. or confidence intervals) and using paired tests for key metrics.\n2.  **[Definite] Baseline tuning fairness:** For transparency and fairness, the paper should include $\\beta$ sweeps for vanilla DPO per model and dataset, and compare DA-DPO against each baseline at its best setting. Adding metric-versus-$\\beta$ plots would be beneficial.\n3.  **[Definite] Cost and efficiency accounting:** To better substantiate the “cost-effective” claim, the paper should include a compute table with FLOPs and wall-clock times for the data scoring and fine-tuning stages, ideally with a comparison to training a small reward model.\n4.  **[Definite] Difficulty definition and robustness:** The analysis of difficulty would be more robust if it examined the full difficulty spectrum (e.g., ranked buckets), varied tail cutoffs, constructed buckets on a held-out split, and replicated the dynamics analysis with MLLM-based and fused proxies (using length-controlled per-token likelihoods).\n5.  **[Definite] Portability beyond LLaVA:** To demonstrate the method's generality, adding a small-scale experiment on a non-LLaVA model (e.g., Qwen-VL or InternVL) would be a valuable addition.\n6.  **[Optional] Mechanistic validation:** To provide deeper insight into the method's mechanics, the authors could log per-example gradient norms to correlate (a) difficulty with gradient magnitude and (b) checkpoint-wise easy–hard gap reductions with changes in hallucination metrics.\n7.  **[Optional] Evaluator robustness:** To confirm that the improvements are not an artifact of the evaluator, the paper could include a re-scoring of a subset of outputs with an alternative automatic evaluator or a small-scale human audit.\n8.  **[Minor] Presentation and clarity:** For readability, Figure 1a should be enlarged (including axes/legend fonts). The paper should also ensure consistent notation for key terms like $\\beta$ and $\\hat{\\beta}$, define fusion weights on first use, standardize acronyms, and label all axes and units clearly in plots.", "broader_impact_concerns": "N/A", "additional_comments": "I think the paper is valuable and with the addition of a few clarifications, would be a good addition to the community. Here are a few additional minor comments -\n\n* **[Minor] Figure 1a readability:** The paper should enlarge Figure 1a and increase the font sizes for its axes and legend. The current graph is difficult to read.\n* **[Minor] Define fusion terms on first use:** The weights used in the fusion rule (e.g., `cls_c`, `cls_m`) should be clearly defined upon first use, and the notation should remain consistent throughout the paper.\n* **[Minor] Acronyms and capitalization:** All acronyms (e.g., AMBER, POPE, MME) should be defined on first use. Capitalization should be consistent for model names and terms (CLIP, LLaVA, OneVision, VQA, OCR).\n* **[Minor] Hyphenation/style pass:** Hyphenation should be standardized (e.g., “cost-efficient,” “per-example,” “state-of-the-art”). The paper would also benefit from a light copy-editing pass for consistency.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)