DAMA: Data- and Model-aware Alignment of Multi-modal LLMs
Authors: Jinda Lu
Junkang Wu
Jinghan Li
Xiaojun Jia
Shuo Wang
YiFan Zhang
Junfeng Fang
Xiang Wang
Xiangnan He
Venue: ICML 2025 poster (2025)
Keywords: Multi-modal Large Language Models
Preference Optimization
Primary area: applications->computer_vision

Abstract: Direct Preference Optimization (DPO) has shown effectiveness in aligning multi-modal large language models (MLLM) with human preferences. 
However, existing methods exhibit an imbalanced responsiveness to the data of varying hardness, tending to overfit on the easy-to-distinguish data while underfitting on the hard-to-distinguish data. 
In this paper, we propose Data- and Model-aware DPO (DAMA) to dynamically adjust the optimization process from two key aspects: (1) a data-aware strategy that incorporates data hardness, and (2) a model-aware strategy that integrates real-time model responses. By combining the two strategies, DAMA enables the model to effectively adapt to data with varying levels of hardness.
Extensive experiments on five benchmarks demonstrate that DAMA not only significantly enhances the trustworthiness, but also improves the effectiveness over general tasks. For instance, on the Object HalBench, our DAMA-7B reduces response-level and mentioned-level hallucination by 90.0% and 95.3%, respectively, surpassing the performance of GPT-4V.

DAMA: Data- and Model-aware Alignment of Multi-modal LLMs
Jinda Lu
Junkang Wu
Jinghan Li'
Xiaojun Jia? Shuo Wang'
Yifan Zhang? Junfeng Fang
Xiang Wang" Xiangnan He
Abstract
Direct Preference Optimization (DPO) has shown effectiveness in aligning multi-modal large lan- guage models (MLLM) with human preferences. However, existing methods exhibit an imbalanced responsiveness to the data of varying hardness, tending overfit the easy-to-distinguis/ data while underfitting the hard-to-disiin hard-to-distinguiish data. In this paper. we propose Data- and Model-aware aware DPO (DAMA) to dynamically adjust the optimiza- tion process from two key aspects: (1) a data- aware strategy that incorporates data hardness, and (2) model-aware strategy that integrates real-time model responses. By combining the two strategies, DAMA enables the model to effec- tively adapt to data with varying levels of hard- ness. Extensive experiments on five benchmarks demonstrate that DAMA not only significantly enhances the trustworthiness, but also improves the effectiveness over general tasks. For instance, the Object HalBench, our DAMA-7B reduces response-level and mentioned-leva hallucination by 90.0% and 95.3%, respectively, surpassing the performance of GPT-4V. Code is available at: https: /ttp//gtttti git.tub......g.gg..g com/ in jadlu/ u DAMA.
Prompt What time of day is it the image?
Vw It is not possible to determine the exact time two boats clear blue sky suggest that likely during daytime.
Preferred
yr boats cityscape
leisure activities
Rejected
perhaps aftermoon
yr boats few birds
Hard-to-distinguish
groups of people presumed daytime
disy-to-distinuiss
Rejected
Early Stage
Later Stage
Image Text Sim Score
Reward
Reward
E
Figure 1. (1) Preference data (Prompt, Image, Preferred response ye. Rejected response 3) with different hardness: "easy-to- distinguish" data denotes large image-Text sim score gap be- tween y and yo: "hand-to-distingustt data indicates low gap between and you (2) Implicit reward across the optimiza- tion stage: the reward gap for "casy-to-distinguish" data enhances significantly during optimization, while for "hard-to- haaddto-distinguish" data, the gap remains low.
1. Introduction
broader applicability (Liu et al., 2023a; Yu et al., 2024b). To address this, direct preference optimization (DPO) (Rafailov et al., 2024; Wu et al., 2024b:a) has been adapted into MILM alignment (Yu et al., 2024c; Wang al., 2024; Lu et al., 2025), achieving encouraging performance with moderate computational costs.
Recent advances in Multimodal Large Language Models (MLLMs) have demonstrated remarkable visual understand- ing capabilities on the basis large language models (Liu et al., 2023b; 2024a). However, despite their effectiveness, the hallucination issue generating outputs inconsistent with the image content and human preference limits their
DPO methods (Lu et al., 2025; Yu et al., 2024c) collect pref- erence data consisting of an image, a prompt, and two re- sponses (yw, 31). The preferred response yo better aligned with the visual content (large Image-Tes Text similarity score), while the rejected response yı contains more hallucinated content (small Image-Texx Text similarity score). They prioritize preferred responses yo over the rejected ones yr using DPO with a hyperparameter 3, which balances retaining the refer- ence model काe and incorporating new preferences into the
*Corresponding Authors. 'MoB Key Lab of BIPC, University of Science and Technology of China "Nanyang Technological Uni- versity "Institute of Automation, University of Chinese Academy of Sciences National University of Singapore. Correspondence to: Xiang Wang <xiangwang1223 @e233.mmailcomm gom,, Xiangnan He <hexn @ pepuustc..ee www.:@ussttucc........ edu.en>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

DAMA: Data- and Model-aware Alignment of Multi-modal LLMs
updated model me (Gu al., 2024; Xie et al., 2024).
gling to capture the "hard-to-distinguish"" data.
However, our analysis reveals that current methods exhibit imbalanced responsiveness in handling the data with varying hardness during the optimization process, resulting in sub- optimal performance. As illustrated in Figure 1, for "easy- to-distinguish" data (large Image-Text similarity score gap between preferred Yu and rejected yi), the reward gap am- plifies during training, indicating stronger alignment. Con- versely, for haard-to-distinguish data (small Image-Text similarity score gap), the reward gap stagnates, suggesting limited capability to distinguish Yu from y/- This implies that current methods, which employ optimization strategies using a static across data with varying hardness (Sun et al., 2024; Yu et al., 2024c), could fail to capture the learning dynamics inherent in multimodal preference data.
We introduce DAMA, a data- and model-aware strategy that dynamically adapts the optimization process based on both data hardness and model responsiveness, facilitating more nuanced and adaptive optimization.
We demonstrate DAMA's effectiveness through compre- hensive evaluations across various benchmarks and model sizes. For instance, on the VLa7B model, it achieves more than 16% performance gains in Non-Response lev- els compared to vanilla DPO.
2. Preliminary
In this section, we briefly review the MLLM preference learning procedure, which starts by sampling pairwise pref- erence data with supervised fine-tuned (SFT) model and then optimizing on such data. Specifically, we categorize this process into the following aspects:
To address this imbalanced responsiveness issue, we pro- pose Data- and Model-aware direct preference optimization (DAMA), which dynamically adapts 3 to both data hard- ness and model's responsiveness, enabling adaptively adjust model's s learning behavior based on the inherent preference dynamics and model's 's real-time responses. Specifically, we propose two novel mechanisms:
Preference construction. Preference learning of an MLLM æ begins by sampling pair-wise responses (yer: 31) from an SFT model WSFT, given an image I and a prompt question ær This can be formalized as (ywr ut) [I, x). These pairs are then labeled, either by hu- mans or other models, to indicate preference, such that Vie is preferred over y1 given (I, denoted as (Nw > xI)
Data-aware Preference Optimization (Section 3.1): We quantify data hardness via CLIP-based image-text similar- ity scores (Radford et al., 2021), decomposing responses into sub-sentences for granularity aware estimation. Then we normalize and transform the scores into probabilities to enable effective hardness estimation. By dynamically scaling 3 inversely with hardness, we enforce stronger regu- larization on "easy-oo-distinguish" samples (large B) while relaxing constraints for haadd-to-iistingushh" ones (small 3). preventing overfitting and underfitting, respectively.
TTF
Preference optimization. Given pair-wise preference data (ywr 9t) ~ TSFT (M/T(y/ z), the preference optimization process traditionally involves explicit reward modeling fol- lowed by policy optimization.
In this work, we employ Direct Preference Optimization (DPO) (Rafailov et al., 2024), which simplifies this process by directly optimizing a policy against the preference data without an explicit reward model. Instead, DPO defines an implicit reward function, formalized as:
Model-aware Preference Optimization (Section 3.2): We estimate the model's s responsiveness through reward gaps between preferred (yw) and rejected responses yt, incor- porating this into the optimization process by dynamically adjusting B. Larger /3 values are assigned to larger reward gaps between yo and yı, indicating the model's proficiency in those cases, while smaller 3 values correspond to smaller reward gaps, suggesting that the model needs to improve its responsiveness on those cases. This adaptive strategy al- lows the model to focus more on cases with less confidence, while maintaining its performance on well-learned cases.
To (y/ I,
r ZTv = 3 log
(I)
Tiet ( x)
Therefore, the DPO loss can be defined as:
To (New ( II, xx)
Lapo - -E ()) (log (B log
Toof (Ur [I,))
(2)
To (yrl II, x)
-B
]].
Teel (y1 [I, x))
By combining these strategies via element-wise multipli- cation, DAMA enables real-time adaptation to both data hardness and model responsiveness, demonstrating strong alignment performance across various evaluation bench- marks. Our contributions are summarized as follows:
3. Approach
In this section, we describe our DAMA in detail. Specifi- cally, we first illustrate our data-aware preference optimiza- tion, then we describe our model-aware preference optimiza- tion, and finally, we show our combination strategies for robust preference optimization. Our approach algorithm is listed in Algorithm.
We pioneer the exploration of imbalanced responsiveness in Multi-modal LLM (MLLM) alignment, where MLLMs tend to overfit on "easy-to-distinguiis data while strug-
2

---

DAMA: Data- and Model-aware Alignment of Multi-modal LLMs
(1) Response Split
(2) Hardness Calculation
(3) Optimization
sentences
Probability
Preference Instonce
(S,
Data hordness
Modifying P
chosen y
reject
LLM to Split
chosen set Sy
reject set S₁
Visual Encoder MLLM (Faw)
Sub-sentence
MLLM W
FCLIp
Figure Overview of our data-aware preference optimization. For each preference instance: (1) We first break the preferred and rejected response into sub-sentences by prompting a large language model (LLM); (2) Next, we estimate the similarity scores between each sub-sentence and given image using the CLIP classifier, and then calculate the differences between the preferred and rejected response as the hardness of the data; (3) Finally, we incorporate the estimated hardness into the preference optimization process by modifying /3 in Equ (2), allowing the model to adjust based on the data hardness.
3.1. Data-aware Preference Optimization
corresponding score by the softmax probabilities as
An overview of our data-aware preference optimization is shown in Figure 2. Given preference instance from the dataset D as {II, 2. yee } ~ D, where I, x, yw, Y1 de- notes the image, question, preferred response, and rejected response, respectively, firstly splits the responses into sim- ple and self-contained sub-sentences. Next, it calculates the image-text similarity scores between the sub-sentences and the image by the CLIP classifier. Then, it combines the scores of each response and compares the difference between the preferred and rejected responses as the data hardness. This hardness is embedded into the preference optimization process by modifying the in Equ (2). The following are detailed descriptions.
P P
([ Cw C, l)
Softmax
(4)
P. E R2 and P1 E R" represents the probabilites. The dif- ference between the preferred and rejected probabilities demonstrates the data hardness. A large difference im- plies that the preference data is "rasy-to-distinguiss", where the rejected response includes more elements that are not present in the image, conversely, small difference suggests that the preference data is thard-to-distinuuit rejected response exhibits minimal hallucination. Then we define the hardness based on the probabilities difference as:
and the
P
q
We employ the CLIP classifier I'LIP (Radford et al., 2021), to calculate similarity scores. For each preference instance, we aim to effectively capture the similarity between the re- sponses (ym, 31) with the given image I, while alleviating the 77 token length constraints in CLIP. To achieve this, we decompose the complex responses, which contain various objects and relations, into simple and self-contained sub- sentences. Concretely, prompt the open-source large lan- guage model, such as LLaMA-3 (Dubey et al., 2024), to split (Vw, 31) into sub-sentences S = [Sw.yli = 1,2,... p} and S; = [S = 1,2,.. ....... 4). where p and q denotes the number sub-sentences for S., and S,.
8
P
PL
aD (0)0//(()))
(6)
where an denotes the data hardness, 5 measures the differ- ence between Par and Pt, and 8 denotes the mean difference across the dataset. The Sigmoid function o-)) is employed to transform the response divergence P and the mean 8 into the range (0, 1) for convenient comparison.
Finally, we adapt 3 of Equ (2) to incorporate the hardness into the optimization procedure, and each preference in- stance corresponding to a specific B as:
Subsequently, we employ the CLIP classifier I'LIP cal- culate the similarity score between the given image I and the sub-sentences Sw, R S as:
8D = 3 ap-
This adjustment allows the model to optimize based on the data hardness, further enhancing its adaptability to the data.
C.. T TCLP (I,Sw )
C Tuur (I,
3.2. Model-aware Preference Optimization
An overview of our model-aware preference optimization is shown in Figure 3. Given a batch of preference in- stances from dataset D, as B = {[I, F; Uw. Vt,)) 1, 1,2,...NN ~ D, it firstly calculates the reward gaps be- tween the preferred yus and rejected yt, and then filters
where C E R and C E R represents the corresponding similarity scores of S. and S,, respectively. To effectively quantify the difference between preferred response C,, and rejected response C for each instance, we normalize the
3

---

DAMA: Data- and Model-aware Alignment of Multi-modal LLMs
where (R;- R)2² implies the squared distances from the
(1) Reward Modeling
mean, and r represents the sorted K-th distance. With the filtering, current responsiveness of no can be formalized as:
T
ref
chosen Yw1
reject Ye1
Implicit Reward Modeling by MLLM my and Teet
Rg N- K Σ M, Rs,
1
(11)
N Preference instances
aM = (RB))O(())
(12)
(3) Optimization
(2) Outlier Filtering
a(-) is the Sigmoid function, which transforms both the filtered gaps Rg and the estimated mean R into the range (0, 1) for convenient comparison, and am refers to the esti-
Model Responsiveness
reward gap
filtered
Modifying B
K selected
mated model responsiveness.
MLLM Te.
We then integrate aM into the optimization procedure by modifying 3 of Equ (2) as:
instances
Figure 3. Overview of our model-aware preference optimization. Given N preference instances: (1) we first calculate the reward gap of each instance using the implicit reward model; (2) To ensure stable modeling, we filter out the outliers (i.e. the instance with excessively high or low gaps) and then estimate the average gap: (3) To enable the model to be aware of its current responsiveness, we integrate such estimation into the preference optimization process by modifying / in Equ (2).
BM = 3 ay-
(13)
By utilizing 3a to optimize To with the filtered batch M,, the model can effectively adapt to its current responsiveness to the preference data.
Finally, we update the estimated mean R using a moving average after optimization over batch B as follows:
R y- R (1 -7) Rs.
(14)
out the outliers (i.e. the instance with excessively high or low gaps) for stable estimation. Such estimations are embed- ded into the preference optimization process by integrating into 3 in Equ (2), enabling the model to be aware of its current responsiveness. Details of model-aware preference optimization follows.
the momentum 7 is set to 0.9, and R is initialized to 0.
3.3. Combining for Preference Optimization
In this section, we present the combination of both data- and model-aware strategies for robust preference optimiza- tion. Specifically, given batch of preference instances B = ((I, Fi, Vw Ji. = 1.2, N} ~ 'D, where I, Hi, Uu..i Ut, denotes the image, question, preferred response, and rejected response, respectively, we first compute data hardness offline based on our data-aware preference opti- mization, and then obtain the corresponding instance wise hardness by Equ (6) as aß the reward gaps using Equ 12 ax- To facilitate the op- timization process to be both data- and model- we propose an element-wise combination strategy. Specifically, we combine of and (M as:
We employ current implicit reward gaps between the pre- ferred and rejected responses of the given B instances to measure the current model's responsiveness. Specifically, the reward gap R for the i-th instance in B formalized as:
R₂- log
ne (Mo, (Z.2.) (Me Z..m..) B log
(8)
RRV. Next, we calculate
where Te and Tref represent the optimizing model and ref- erence model, respectively. We then normalize the reward gaps using the estimated mean as follows:
R, = RR/R,
(9)
where R represents the estimated average reward gap, and R2 is the normalized one for the i-th instance.
a == OM-
(15)
where a E RV represents the combined factor. Subse- quently, we adjust 3 of Equ (2) to incorporate both compo- nents into optimization procedure as:
However, the estimation remains sensitive to outliers despite normalization, especially in the full fine-tuning scenario, where the batch size is relatively small. To mitigate this issue, we filter out instances with exceptionally high or low gaps using a mask vector M E R.V. defined as:
Bc = B 3.00 a.
(16)
where 3c E R and each preference instance in B corre- sponds to a specific 3. Finally, our combination strategy can be achieved by employing the 3c to optimize no with
1, (R, R)² ST,
M,
(10)
0, (Ra - R)² >T,
4

---

DAMA: Data- and Model-aware Alignment of Multi-modal LLMs
the filtered batch B M. where M is obtained by Equ (10). Thus, the optimization process can become more adaptive, allowing the model to refine its preferences based on both pre-computed data hardness and time model respon- siveness, further enhancing the robustness.
(4) LLaVA Bench consists of 24 images and 60 questions including conversation, detailed description, and complex reasoning ability (Liu et al., 2023b).
(5) MM- Vet is designed to evaluate six integrated compe- tencies, including OCR, recognition, knowledge, language generation, spatial awareness, and math (Yu et al., 2024d).
4. Experiment
Implementation Details. For both LLaVA-1.5 7B and 13B models, we employ full parameter-tuning over the prefer- ence dataset with four epochs. Specifically, for reproducibil- ity, we adopt the same hyperparameters as provided in the official VLaVA GitHub repository ' The batch size N is set to 16, the selected size K is set to 12, and the penalty hyperparameter 3 is set to 0.1 by following (Rafailov et al., 2024; Yu et al., 2024c). All experiments are conducted with four A100 80GB GPUs, and four epochs of fine-tuning cost seven hours for both backbones.
In this section, we elaborate the effectiveness of our Data- and Model-aware Direct Preference Optimization (DAMA). Specifically, we first introduce the details of our experi- mental settings. Next, we illustrate the ablation studies, and finally, we compare the results with the state-of-the-ee methods over various benchmarks.
4.1. Experimental Settings
In this section, we describe the experimental settings. Backbone: We employ the LL. 7B and 13B for per- formance comparison (Liu et al., 2023b).
4.2. A Ablation Studies
In this section, we evaluate the effects of different compo- nents of our DAMA. To this end, we utilize the VLa 7B model the backbone. For clear illustration, we report both the response and the mentioned -evell non-hallucination rate on the Object Hallucination benchmark, where the non- hallucination rate is defined as 100% hallucination rate. The following are detailed illustrations.
Dataset: Our focus is not on the preference data construc- tion, thus we directly utilize the released dataset by (Yu et al., 2024c), which contains 22k preference data totally.
Baselines. In this work, we compare against state-of-the-arr baselines across various categories:
(1) Hallucination-speecfie baselines. In this category, we mainly compare with VCD (Leng et al., 2024), Less-is-more (Yue al., 2024), OPERA (Huang al., 2024), and CCA- LLaVA (Xing et al., 2024).
Influences of different components. The performance of various components of DAMA is reported in Table 1, where "DPO" refers to Direct Preference Optimization (Rafailov et al., 2024). "MDPO" represents our Model-aware Pref- erence Optimization, "D"PO" denotes Data-aware Prefer- ence Optimization, and "DAMA" corresponds to our com- bined strategy. The experimental results demonstrate: (1) All strategies significantly outperform the baseline method (DPO), especially our final "DAMA", achieving more than 10% response level performance gains, highlighting the ef- fectiveness of our method: (2) Compared with "MDPO", the performance gain of "D"PO" is relatively modest, suggest- ing that the quality of the preference data is already high, which further validates the efficacy of the preference data construction strategy (Yu et al., 2024c).
(2) Preference Optimization- based baselines. In this cate- gory, we mainly compare with HA-DPO (Zhao el al., 2023), POVID (Zhou et al., 2024a), LLaVA-RLHF (Sun et al., 2024), RLHF-V (Yu et al., 2024b). RLAIF-V (Yu ct al., 2024c), AMP-MEG (Zhang et al., 2024), CSR (Zhou et al., 2024b), V-DPO (Xie et al., 2024), and TPO (Gu et al., 2024). (3) Commercial bascline. We include GPT-4V as strong reference to evaluate the performance gap between the open- source and commercial models.
Benchmarks. We conduct experiments on five henchmarks, including three hallucination benchmarks reflecting trust- worthiness, and two general benchmarks:
(1) Object HalBench is employed to evaluate object hal- lucination by detailed descriptions of the image content. We report the response-level and mentioned-level non- hallucination rates to evaluate its capability to reduce hallu- cination (Rohrbach et al., 2018).
Influences of probability transformation in data-aware preference optimization. Table 2 summarizes the perfor- mance of different response inconsistency estimation strate- gies, where "CLIP Scores" denotes that we directly estimate based on C. and C, with ô = Ei= Cw.s/ / CL and "CLIP Probs" represents our strategy, which transforms C and C₁ into probabilities. The results indicate that: firstly, both strategies improve the performance, underscor- ing the effectiveness of integrating data inconsistency into the optimization process, which allows the model to better
(2) AMBER is a multi-diimensionl hallucination bench- mark which contains more than 15k samples. We report the Accuracy and Fl metric by its discriminative component (Wang et al., 2023).
(3) MMHal- assesses response-level hallucination rate and informativeness by GPT-4 compare model outputs with human responses and object labels (Sun et al., 2024).
"htt:://gthabb.commaataariiiiiti LivLLLA
5

---

DAMA: Data- and Model-aware Alignment of Multi-modal LLMs
Table Experimental results of different components of DAMA.
Table 3. Experimental results of different filtering strategics.
OВJECT HALBENCH
OВJECT HALBENCH
METHOD
METHOD
RESPONSE (f)
MENTION (1)
RESPONSE (1)
MENTION (1)
LLAVA-1.5-7B
47.75
73.08
No FILTER
86.66
92.62
+DPO
78.29
89.48
BОTTOM
76.36-10.30
87.20-5.42
+D*PO
82.54+4.25
90.64+1.16
TOP
86.34-0.32
92.48-0.14
+MDPO
88.00+9.71
93.74+4.28
BOTTOM TOP
88.00+1.34
93.74+1.12
+DAMA
90.87+12.58
95.33+5.85
Response
9250
Table 2. Experimental results of different preference inconsistency
construction strategies.
07.50
METHOD
OBJECT HALBENCH
8500
RESPONSE (f)
MENTION (1)
Multiplication
DPO
78.29
89.48
(2.5
Addition
Addition Ratio
+CLIP SCORES
80.65+2.36
89.62+0.14
80.00
+CLIP PROBS
82.54+4.25
90.64+1.16
00
0.1
02
04
0.5
066
0.7
0.8
0.9
1.0
Figure 4. Experimental results of the combination strategies with the response-level non halluccinationn rates.
handle varying levels of data hardness, thereby improving overall robustness; furthermore, transforming Cw and C₁ into probabilities yields a larger performance gain, as prob- abilities smooth the estimation 5, mitigating noise from large gaps between Cw and C₁ and preventing the influ- ences by the outliers.
and = 1.0 refers to the "D"PO". We can observe that multiplication outperforms the weighted sum by a signif- icant margin. This because the weighted sum balances the two components without effectively merging their influ- ences, whereas multiplication amplifies their contributions, allowing for more robust and superior performance gains.
Effects of the data filtering in model-aware preference optimization. Table 3 presents the performance of differ- ent data filtering strategies, where "No Filter" denotes that we directly utilize the mean gaps to estimate the model state without filtering, "Bottom" shows that we remove the N K samples with the largest distances in the batch, "Top" is filtering the N - K samples with the smallest distances, "Bottom & Top" refers to our filtering strategy, which filters both extremes based on the squared distances. Specifically, we can observe that: firstly, filtering solely from the bottom or top leads to performance degradation, indicating that such data introduces bias in estimating the model state. More- over, filtering only the bottom samples results in significant performance drops due overfitting the top-ranked data, which misguides the estimation of model responsiveness to focus excessively on potentially less representative in- stances. Furthermore, filtering both bottom and top samples yields performance improvements, demonstrating the effec- tiveness our proposed strategy, as it balances the influence of extreme data points.
Sensitivity to hyper- parameter. Table 5 shows the ef- fects of varying K, where we select K = 8, 10, 12, and 14 responses from batch of 16, where no filter denotes that K is set to 16. We observe that the performance improves as K increases from 8 to 12, peaking K = 12, and then slightly declines as K approaches 16. Notably, K performs worse than the baseline (K 16), which can be attributed to the introduced bias by too few samples (50% samples). small These suggest that moderate samples-can effectively number of low-quality or outlier improve performance, Based on this observation, we adopt 12 our experiments.
4.3. Comparison with state-of-the-art methods
In this subsection, we compare our method with state-of-the- art methods under three trustworthy benchmarks: Object HalBench, AMBER, and MMHal-Bente and two general benchmarks: LLa VA-Behhe and MM-Vet. We compare our method against baselines from various aspects, including hallucination-specife baselines, preference optimization- based baselines, and the commercial baseline GPT-4V. Our DAMA is adapted to both the LLaVA- 1.5 7B and 13B mod- els. The experimental results, along with results vanilla DPO and the improvements achieved by DAMA over DPO,
Effects of the combination strategy. Figure 4 and Fig- ure 5 illustrate the performance of different combination strategies from the Response and Mention, respectively. "Multiplication" refers to combining both effects using a weighted sum strategy, where a - -) aM + p. aa with ranging from 0.0 1.0, where p = 0.0 is the MDPO,
6

---

DAMA: Data- and Model-aware Alignment of Multi-modal LLMs
Table Performance comparisons with state-of-the-ar methods different benchmarks. We report non-hallucination rates in different levels including response level (Non-Rsp.) and mentioned- (Non-Man Men.) for Object HalBench (Rohrbach et al., 2018). Hall. refers to the Hallucination Rate MMHal Bench (Sun al., 2024). best results all methods are indicated bold, and second best results underlined. The compared results are sourced from (Yu et al., 2024c; Gu et al., 2024), and the reported results of LLaVA-1.5 DPO, and DAMA are evaluated using GPT-4-turbx-2224--000009
ОВJECT
MMHAL- BENCH
LLAVA BENCH(1)
AMBER
METHOD
SIZE
HENCH
MM-VET(1)
(ON-RSP(") (ON-MEN.
Acc(f)
FI(
SCORES(T)
HALL.())
METHOD (HALLUCIAATIONNSPPEIII
VCD (CVPR)34)
7B
$1.2
75.7
71.8
74.9
2.12
54.2
61.6
LESS-IS-MORE (AC1/24)
7B
59.7
82.2
72.4
75.8
2.33
50.0
OPERA
7B
54.9
77.7
75.2
78.3
2.15
54.2
61.3
CCA-LLAAVA (NIPS124)
7B
53.3
76.2
77.7
81.9
1.92
61.5
64.3
METHOD (PREFERENCE OPTIMIZATION)
HA-DPO (AXXiv/23)
60.1
75.2
79.9
1.98
60.4
POVID
7B
51.9
75.6
82.9
87.4
2.08
56.2
68.2
31.7
RLHE-V (CVPR(24)
7B
74.8
78.5
2.02
60.4
68.0
32.3
RLAIF-V (ARXIV'24)
7B
89.5
94.8
76.8
84.5
2.95
32.3
(NIPS'24)
7B
73.2
76.
2.05
60.4
68.9
31.0
V-DPO (EMNLP'22)
7B
81.6
2.16
56.0
TPO (A4221124)
7B
79.3
85.0
2.47
51.0
70.2
33.0
TPO (A4XIV'24)
13B
83.9
88.0
2.72
45.8
72.8
36.2
LLAVA- (ACI)24)
13B
61.9
81.1
79.7
83.9
2.02
62.5
95.6
RLHF-V (CVPRR134)
138
92.5
76.7
38.5
AMP-MEG (NIP5124)
13B
68.3
79.4
79.5
84,6
3.08
36.5
LLAVA-1.5
7B
47.8
71.2
73.9
77.7
1.95
63.5
62.3
31.6
DPO
7B
78.3
89.5
75.5
79.2
2.15
$7.0
64.6
30.4
+ DAMA
7B
90.9
95.3
83.3
87.0
2.76
41.0
68.0
32.8
IMPROVEMERR EMENTS (%)
16.11
%
+10.3%
+9.8%
+2.44%
+28.1
+5.3%
+7.9%
LLAVA-1.5
13B
50.0
76.4
71.2
73.0
2.36
56.0
66.1
36,
+ DPO
13B
78.5
2.59
48.0
74.0
35.4
DAMA
13B
89.1
94.4
84.3
88.1
2.89
43.0
75.1
36.4
IMPROVEMENTS (%)
+5.1 %
+2.1
5.8%
4.6 %
+15.4%
+10.4%
+1.4 %
+2.8 %
GPT-4V
86.4
92.7
83.4
87.4
3.49
28.1
98.0
67.7
Mantion (f)
96.00
Table 5. Experimental results of diffcrent selected number(K)) with the batch size N=16.
94.00
OBJECT HALBENCH RESPONSE (t)
METHOD
98:00
MENTION (t)
Multiplication
No FILTER
86.66 85,34-1.32
92.62
91.00
K = 8
92.25-0.37
Additiin
Adkittion
Ratio
K 10
87.50+0.84
93.19+0.57
90:00
00
01
0.2
03
041
0.5
0.6
Q.7
0.8
0.9
10
K = 14
87.96+1.30
93.45+0.83
K = 12
88.00+1.34
93.74+1.12
Figure 5, Experimental results of the combination strategies with the mentioned-level non-halluce con-hallucination rates.
hallucination on the generative Object HalBench by 90.9%
and 95.3%. Moreover, DAMA-13B achieves 84.3% Ac-
are listed in Table 4.
curacy and 88.1 Fl score on the discriminative AMBER benchmark. These attained results surpass those of GPT-4V, fully demonstrating the effectiveness of DAMA.
From the experimental results, we can observe that: (1) Compared DPO, our DAMA achieves substantial per- formance improvements over all compared benchmarks. It's noteworthy that on MMHal-Benchh it achieves more than 10% improvements for the 13B model, and more than 28% gain for 7B models, which is significant; (2) DAMA achieves new state-of- -f-te-of-the-art the-art over various benchmarks. Our DAMA-7B reduces the response-level and mentioned-level
5. Related Work
In this section, we first briefly summarize the hallucination mitigation methods for multi-modal large language models, and then, we illustrate related methods for aligning with

---

DAMA: Data- and Model-aware Alignment of Multi-modal LLMs
human preferences. Finally, we enumerate the differences between ours and related methods.
LLAV DPO DAMA
Log Probability Gaps
17.14
1537
12.42
5.1. Hallucination mitigation in MILMS
1.99
Hallucination, as a key indicator of trustworthinnss, refers to that the MLLM outputs are not aligned with the image content (Liu et al., 2024b). Current mitigation strategies generally fall into the following four key areas:
3148
-12
hand-toodisttnnuussh
Easy-to
Data cleaning. Works such as LRV-Instruction (Liu et al., 2023a) and Hallucidoctor (Yu et al., 2024a) identify noise within instruction tuning data as a cause of hallucination and propose fine-grained data cleaning and curation strategies. Visual representation enhancement. Methods like (Jain et al., 2024; Tong et al., 2024) suggest that insufficient visual cues as a cause of hallucination, and they incorporate more intricate visual features to enrich the visual representations. Inference-time enhancement. Works like VCD (Leng et al., 2024) and MARINE (Zhao et al., 2024) employ visual contrastive decoding mechanisms to enhance the model's focus visual details, often by contrastively sampling from the original and visually distorted distribution.
Categories
Figure 6. Log probability gaps of different models (LLaVA-1155 7B, DPO, and DAMA) between "easy-to-distinuuis"" (top 25% image-text similarity), dinisttiiiss (bottom 25% image- text similarity), and "normal" data (middle 50%).
Zhang al., 2025) employ closed-source models such as GPT-4V (Achiam et al., 2023) for annotation.
Emphasizing visual detail. Recent approaches that con- struct vision-based rejected samples via image manipulation include (Xie et al., 2024; Wang et al., 2024; Lu et al., 2025) (c.g.. object removal or replacement (Lu et al., 2025)). Ad- ditionally, works like (Gu et al., 2024; Cui al., 2025) identify and emphasize key language tokens from prefer- ence responses to better attend visual details.
Preference Optimization. Inspired by the great success of preference optimization in language models (Rafailov et al., 2024; Shao et al., 2024), recent works utilizing pref- erence optimization have demonstrated significant perfor- mance in hallucination mitigation (Iuu al., 2025; Yu et al., 2024b;c). They achieve this by designing strategies to pri- oritize the preferred responses (better aligned with visual content) over the rejected responses (containing more hallu- cinated content).
Differences. Compared with existing MLLM preference optimization strategies, which focus on data curation and introduce fine-grained regularizations, our work aims to de- sign an improved optimization strategy that can effectively leverage model responses and preference data, enabling the model adaptively respond to the data, and thereby enhancing the alignment performance.
5.2. Preference Optimization
To align model outputs with human preferences, preference optimization methods have garnered significant attention (Ouyang et al., 2022; Rafailov et al., 2024; Wu et al., 2024b). As a pioneering work, RLHF (Ouyang et al., 2022) firstly trains a reward model on preference data and then optimizes the policy with PPO (Schulman et al., 2017). However, obtaining effective reward model remains challenging. To address this, DPO (Rafailov et al., 2024) simplifies the process with an implicit reward function, allowing for direct optimization on preference data. 3-DPO (Wu et al., 2024b) further enhances DPO by introducing dynamic/ B to more effectively leverage preference data of varying quality.
6. Discussion and Future Work
6.1. Discussion
Although DPO has demonstrated its effectiveness in miti- gating hallucination issues in multi-modal LLMs (Yu et al., 2024b;c), inherently treats all training samples equally. As illustrated in Figure 6, this tends bias the model toward "easy-to-distingussh" data-thhss, with larger log probabil- ity gaps-while neglecting the more challenging "hard-to- distinguish" data, which often exhibit negative log probabil- ity gaps. This imbalance constrains the model's expressive capacity and generalization ability.
In MLLMs, current efforts typically focus on two aspects:
To address this limitation, we introduce DAMA, data- and model-aware strategy that enhances the model's sensitivity to data hardness. As shown in Figure 6, by explicitly ac- counting for data hardness and model responses, DAMA achieves consistently higher log probability gaps, with no- table gains on "hard-to- -istiiiggiiiiihh"" This leads to a more balanced optimization process and substantially enhances the model's 's overall reliability.
Collecting high-quality preference data. Existing works employ diverse strategies for preference construction. These include human-based methods (Yu et al., 2024b; Sun et al., 2024), and large language model-based strategies (Yu et al., 2024c; Li et al., 2024). Among the latter, RLAIF-V (Yu et al., 2024c) utilizes open-source models (e.g., LLaVA- NeXT (Liu al., 2024a)), while works like (Ii et al., 2024;
8

---

DAMA: Data- and Model-aware Alignment of Multi-modal LLMs
6.2. Future Work
References
While DAMA marks significant advancement towards adaptive and robust MLLM alignment, several promising avenues remain for future exploration:
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L.,, Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
(1) Adaptive strategies in online scenarios. DAMA cur- rently operates on offline data; future work could investigate its application in online settings, leveraging approaches like GRPO (Huang et al., 2025) to enable continuous model refinement with evolving preference data.
Cui, C., Zhang, A.. Zhou, Y., Chen, Z., Deng, G., Yao, H., and Chua, T.-S. Fine-grained verifiers: Preference modeling as next-token prediction in vision-language alignment. In The Thirleenth International Conference on Learning Representations, 2025. URL https:// openreview net
(2) Advanced hardness and responsiveness estimations. Our current estimation of data hardness (CLIP-based simi- larity (Radford et al., 2021)) and model responses (reward gaps) is simple. Future research could explore more sophis- ticated metrics, e.g. semantic entropy (Farquhar et al., 2024), fine-grained visual representations (Tong et al., 2024). (3) Scalability to broader architectures and modalities. crucial next step involves assessing DAMA's efficacy and scalability commercial MILMs (Google, 2025; OpenAL, 2025), diverse multi-modal architectures (e.g., Omni model (Xu et al., 2025), Unified model (Deng et al., 2025)), and reasoning models (Kimi et al., 2025), which promises to broaden its practical impact significantly.
2i==cJ01k2f]jj
Deng, C., Zhu, Li, K., Gou, C., Li, F., Wang, Z., Zhong, S., Yu, W., Nie, X., Song, Z., Shi, G., and Fan, H. Emerg- ing properties in unified multimodal pretraining, 2025. URL https: / arxiv. wewxiv.rrg/reeeee org/ //s///////// 2505. 14683.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407. 200.0.77777, 2024.
Farquhar, S., Kossen, J., Kuhn, I.., and Gal, Y. Detecting hallucinations in large language models using semantic entropy. Nature, (30(8017))625-63)) 2024.
7. Conclusion
This paper introduced Data- and Model-aware Direct Prefer- ence Optimization (DAMA), simple yet effective strategy to address the imbalanced responsiveness to varying data hardness in Multimodal Large Language Models (MLLMs). DAMA adaptively adjusts the model's learning behavior through two key strategies: (1) data-aware strategy that dy- namically incorporates data hardness, and (2) model-aware strategy that integrates the model's 's real-time responses. Ex- periments across five benchmarks and various model sizes consistently demonstrated DAMA's effectiveness.
Google. Gemini 2.5: Our most intelligent ai model. Large language model, 2025. URL https: //eepmind/ google/models/g//e//e//o
Gu, J., Wang, Y., Cao, M., Bu, P., Song, J., He, Y., Li, S., and Zheng, B. Token preference optimization with self-calibrated hinhol-anchored rewards for hallucination mitigation. arXiv preprint Mxi::2412.114488 2024.
Huang, Q. Dong, X., Zhang, P., Wang, B., He, C., Wang, J., Lin, D., Zhang, W., and Yu, N. Opera: Alleviat- ing ballucination in multi-modal large language models via over-trust penalty and retrospection vetrospection-allocation allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13418-13423 2024.
Impact Statement
This paper aims to enhance the robustness of multi-modal large language model alignment, thereby advancing the field of visual understanding. By tackling challenges like hallucination, we strive to improve the reliability of AI systems in alignment with human preferences. We hope that our research will contribute to facilitating the development of Al systems that are both effective and reliable, ultimately delivering meaningful societal benefits.
Huang, W., Jia, B., Zhai, Z., Cao, S., Ye, Z., Zhao, F,, Xu, Z., Hu, Y., and Lin, S. Vision-rl: Incentivizing reasoning capability in multimodal large language models, 2025. URL https:/ arxiv. org/ hte//ba//55000 06749.
Jain, J., Yang, J., and Shi, H. Vooder: Versatile vision en- coders for multimodal large language models. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27992-28002, 2024.
Acknowledgment
This paper is supported by the National Science and Technol- ogy Major Project (20232D0121 and the National Nat- ural Science Foundation of China (U24B20180, 62121002).
Kimi et al. Kimi kl.5: Scaling reinforcement learning with 1mms, 2025. URL http:ttss://aaxxi 1501.1259.
9

---

DAMA: Data- and Model-aware Alignment of Multi-modal LLMs
Leng, S., Zhang, H., Chen, G., Li, X., Lu, S., Miao, C., and Bing, I.. Mitigating object hallucinations in large vision- language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13872-13882, 2024.
Rohrbach, A., Hendricks, L. A., Burns, K., Darrell, T., and Saenko, K. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pp. 2018.
Li, L., Xie, Z.,, Li, M., Chen, S., Wang, P., Chen, L.,, Yang, Y., Wang, B., Kong, L., and Liu, Q. Vlfeedback: A large- scale ai feedback dataset for large vision- vision-language models alignment. In Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pp- 6227-6246, 2024.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Shao, Z., Wang, P., Zhu, Q. Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., and Guo, D. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models, 2024. URL https: http:///xxvvvvvo org/ /tw./////400 03300.
Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., and Wang. L. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference Learning Representations, 2023a.
Sun, Z., Shen, S., Cao, S., Liu, H., Li, C., Shen, Y., Gan, C., Gui, L.-Y., Wang, Y.-.., Yang. Y., al. Aligning large multimodal models with factually augmented rlhf. In Findings of the Association for Computational Linguis- tics, 2024.
Liu, H., Li, C., Wu, Q., and Lee, J. Visual instruction tun- ing. Advances in neural information processing systems, 2023b. URL https: http:::///ppeeeeiiiiee wwwp://w.eeee/rreoo id=w0H2xGH1kw.
Tong, S., Liu, Z., Zhai Y., Ma, Y., LeCun, Y., and Xie, S. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
Liu, H., Li, C.,, Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reasoning, ocr, and world knowledge, 2024a.
2024.
Liu, H., Xue, W., Chen, Y., Chen, D., Zhao, X., Wang, K., Hou, L.. Li, R., and Peng. W. A survey on halluci- nation in large vision- laiounlangguage models. arXiv preprint arXiv:2402.00255 2024b.
Wang. F., Zhou, W., Huang, Y., Xu, N., Zhang, S., Poon, H., and Chen, M. mdpo: Conditional preference opti- mization for multimodal large language models. In Pro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 8078-8088, 2024.
Lu, J., Li, J., Gao, Y., Wu, J.. Wu, J., Wang, X., and He, X. Adavip: Aligning modti Iims via adaptive vision-enhanced preference optimization. arXiv preprint arXiv:2504. 15619, 2025.
Wang, J., Wang, Y., Xu, G., Zhang, J., Gu, Y., Jia, H., Yan, M., Zhang, J., and Sang. J. An lim-free multi-dimensional benchmark for milms hallucination evaluation. arXiv preprint arXiv:2311.073997 2023.
OpenAl. Introducing openai 03 and 04-mini. Large lan- guage model, 2025. URL https: https:tt//ppeeeteteeetttt /wwp/appnaaaa/////eee/.ae/.a.... com/ index/introduiiin -3-and---a-ao----- and 04 -imini/
Wu, J., Wang, X., Yang, Z., Wu, J., Gao, J., Ding, B., Wang, X., and He, X. a-dpo: Adaptive reward margin is what direct preference optimization needs. arXiv preprint arXiv:2410. 10148, 2024a.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
Wu, J., Xie, Y., Yang, Z., Wu, J., Gao, J., Ding, B., Wang, X., and He, X. 3-dpo: Direct preference opti- mization with dynamic B. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b. URL https: /pwpreveeeeeeeeeeeeeeeeeeeeeeeee net/ forum? id=2
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763, PMLR, 2021.
Xie, Y., Li, G., Xu, X., and Kan, M.-Y. V-dpo: Mitigating hallucination in large vision language models via vision- guided direct preference optimization. In Findings of the Association for Computational Linguistics: EMNLP, pp. 13258-13273, 2024.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
10

---

DAMA: Data- and Model-aware Alignment of Multi-modal LLMs
Xing, Y., Li, Y., Laptev, I., and Lu, S. Mitigating object hallucination via concentric causal attention. Advances in neural information processing systems, 2024.
In Proceedings of the 62nd Annual Meeting of the Associ- ation for Computational Linguisties, pp- 11766-11781, 2024.
Xu, J., Guo, Z., He, J., Hu, H., He, T., Bai, S., Chen, K., Wang, J., Fan, Y., Dang, K., Zhang, B., Wang, X., Chu, Y., and Lin, J. Qwen2.5-omni technical report, 2025, URL ht https: /ttp//aaxvvv...///////////...av. wwwooogg////... 2503. 20215.
Zhang, M., Wu, W., Lu, Y., Song, Y., Rong, K., Yao, H., Zhao, J., Liu, F,, Sun, Y., Feng, H., al. Automated multi-level preference for milms. Advances in Neural Information Processing Systems, 2024.
Yu, Q., Li, J., Wei, I.., Pang. I.., Ye, w., Qin, B., Tang. S., Tian, Q., and Zhuang. Hallucidoctor: Mitigating hallu- cinatory toxicity in visual instruction data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 100944-1993,, 2024a.
Zhang, R., Wei, X., Jiang, D., Guo, 7., Li, S., Zhang, Y., Tong, C., Liu, J., Zhou, A., Wei, B., al. Mavis: Mathematical visual instruction tuning with automatic data engine. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL https: / //openreview. net
Yu, T., Yao, Y., Zhang, H., He, T. Han, Y., Cui, G., Hu, J. Liu, Z., Zheng, H.-T., Sun, M., et al. Rlhf-v: To- wards trustworthy milms via behavior alignment from fine- tne-ggained correctional human feedback. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13807-13816, 2024b.
Zhao, L., Deng, Y., Zhang, W., and Gu, Q. Mit- igating object hallucination in large vision-language models via classifier-free guidance. arXiv preprint arXiv:2402.0880 2024.
Zhao, Z., Wang. B.. Ouyang, L., Dong, X., Wang, J.. and He, C. Beyond hallucinations: Enhancing Ivims through hallucination direct preference optimization. arXiv preprint arXiv:2311.16833 2023.
Yu, T., Zhang, H., Yao, Y., Dang, Y., Chen, D., Lu, X., Cui, G., He, T., Liu, Z., Chua, T.-S., et al. Rlaif-v: Aligning milms through open open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint 242v 2024c.
Zhou, Y., Cui, C., Rafailov, R., Finn, C., and Yao, H. Align- ing modalities in vision large language models via prefer- ence fine-tuning, In ICLR 2024 Workshop on Reliable and Responsible Foundation Models, 2024a. URL https: / //operreiiee ew. net/ / forum? d=GRGvCO
Yu, W., Yang, Z., Li, L.,, Wang, J., Lin, K., Liu, Z.,, Wang. X., and Wang. L. Mm-vet: Evaluating large multimodal models for integrated capabilities. In International Con- ference on Machine Learning, pp. 57730-57754. PMLR, 2024d.
Zhou, Y., Fan, Z., Cheng, D., Yang, S., Chen, Z., Cui, C., Wang, X., Li, Y., Zhang, L., and Yao, H. Calibrated self- rewarding vision language models. In Advances in Neural Information Processing Systems, 2024b. URL https: / //openrvvi ew net/ forum? d=nXYedTTFIT.
Yue, Z., Zhang, L., and Jin, Q Less is more: Mitigating mul- timodal hallucination from an eos decision perspective.

---

DAMA: Data- and Modcl-aware Alignment of Multi-modal LLMs
A. Appendix
Distribution of over the data
Min: 0,5555
Mean: 0.0999
Max: 1426
1750
Median: 0.0992
Std: 0.0288
1250
Frequency
1000
500
250
Bo = Bxdx
Figure Dynamic data-aware fo over each data point, we also the mean, std, min, max value.
training
Naa:: 358
0.14
0.12
0.10
0.08
2000
2000
3000
4000
3000
Figure 8. Dynamic model-aware SM over each training step, we also show the mean, std, min, and max value.
A.1. Prompt templates split the sentences
In this section, we introduce the prompt template split the sentences into sub-sentences with LLaMA3 (Dubey al., 2024). The prompt is: You task is to: Analyze the provided question-unswer pair based on the image, extract all factual statements from the answer, and rewrite them into self-contained sentences. Requirements for each sentence are: 1. complete, each sentence must be self- coffconnta;e:; 2. factual (omit opinions, subjective statements); 3. concise (no more than 77 tokens). Format your result strictly a5:### Facts:- Fact I (e.g.. "A red shoe sits on a wooden floor.") Fact 2 (e.g.. "The shoe has laces and a white sole." # Duestion-ansser pair: Question: "I "(Question) Answer: " "(Answer)"
expert in extracting facts from the given question-answar pair about image. Your
12

---

DAMA: Data- and Modcl-aware Alignment of Multi-modal LLMs
Dynamic over training steps
0.20
STDL: 0303
6.15
0.30
6.03
Dynam over mach Anorago of oves lhc duta oach
1092
2000
4020
Triniin Stops
Figure! 9. Dynamic & by combining the data- and model-aware strategies, also show the mean, std, min, and value.
A.2. Dynamic 3 with different strategies.
this section, we analyze the dynamic 3 with our data- and model-aware strategies.
Data-aware strategy (Figure 7.): The range of data-aware ÅD falls within (0.0524, 14248) with the original / initialized as 0.1. Moreover, the mean value is 0.0999 with standard deviation of 0.0288. These observations demonstrate that maintains proximity to the original, while adaptively adjusting based on the data characteristics, enabling more effective capture of the data.
Model-aware strategy (Figure 8): as the training progresses, the model-aware ÂMM gradually converges to the original B. Meanwhile, we find that the fM fluctuates within a moderate range of (0.0530, 0. 0.1580), demonstrating controlled adaptivity. These observations suggest that as the model training stabilizes, its responsiveness becomes more consistent and eventually approaches steady state.
Combined strategy (Figure 9): Combining both the data- and model-aware strategies yields a more dynamic range of combined Ac, spanning from 0.0177 0.2261, which is wider than either. Moreover, while the value eventually stabilizes around 0.1, notice that the mean value during the training stage is a bit lower than BM. This suggests that it relaxes the constraints during training based on the data hardness, enabling the model to better capture fine- ene-ggained data patterns and thereby adaptively enhancing its responsiveness to data characteristics.
A.3. Qualitative analysis
In this section, we provide qualitative analysis between our Data- and Model-aware preference optimization (DAMA) and the DPO method (Rafailov et al., 2024). The case studies are shown in Figure 10 and Figure 11, and we also include the evaluations of GPT-4. From the case studies, we can observe two key points: (1) while both methods generate correct outputs, DPO tends to be more general, whereas DAMA offers richer, more detailed descriptions. This can be attributed to the integration of data hardness, quantified by the CLIP score between the Image and responses, into the optimization process, enabling DAMA capture and describe more nuanced visual details; and (2) evidenced by the GPT-4 evaluation, DAMA generates more vivid and immersive visual descriptions, this can be attributed to the enhancement of the responsiveness to the visual content, resulting in a more expressive and contextually aligned representation of the scene.
13

---

DAMA: Data- and Modcl-aware Alignment of Multi-modal LLMs
Algorithm Algorithm DAMA.
Input: Preference dataset D, hyper-parameter 3, SFT model TSFT, CLIP classifier TLLP-
Output: The optimized model me.
Initialize model me and reference model Tie as TSFT.
for {IZ, x; Mws }1) in D do
S (3 }. S/ < (ur); obtains 5 with (I,Sw) S.). {I, S
> Equ (3) - (5); Equ (6);
aD - (0)/o(()
end for
repeat
for B = {[I, Fe, Ve Ut,s) ( do
obtain R, with yo and yr.
Equ (8);
obtain Rg with R4;
Equ (9) - (11);
0M < (RBB)/ (R):
> Equ (12);
- an aM, where a = foD 3c + 3
>> Equ (15);
>> Equ (16);
Compute loss w.r.t 8c, ne; Compute the gradient and update the model no.
b Equ (2);
R 7 R (1 7) Rgi
Equ (14);
end for
until The optimization is converged.
Q: Provide thorough description of the given image.
DPO: The image features a bathroom with a unique and eye- catching element a large green plant climbing up the wall. The plant adds a touch of nature to the space, making it feel more alive and vibrant.
In the bathroom, there is toilet located towards the right side of the image. A shelf can also be seen in the room, possibly for storing bathroom essentials. The presence of the plant and the shelf contribute to the overall cozy and inviting atmosphere of the bathroom.
The image features a bathroom with a unique and eye- catching element a large green plant climbing the wall. The plant covers significant portion of the wall, adding a touch of nature to the space. In the bathroom, there is a toilet located towards the right side of the image. A potted plant can also be seen the room, further enhancing the greenery in the space. The bathroom appears have a modern design, giving it contemporary feel.
DAMA:
Prompt: Given two different descriptions of a given image, with the question "Provide a thorough description of the given image", tell me which one you like, and show me reasons.
GPT-4: DAMA is more effective due to the richness in its description. It not only describes the plant's effect but gives clearer picture of its scale and placement, creating a more vivid visual. The added mention of another potted plant and the modern design gives a stronger sense of the room's style and enhances the overall atmosphere. These details would likely help someone visualize the space better.
Figure 10. Qualitative Comparisons between DAMA and vanilla DPO. Data source: Object HalBench.
14

---

DAMA: Data- and Modcl-aware Alignment of Multi-modal LLMs
What is this photo about? Please answer in great detail.
DPO: In the image, a person is performing a trick on a snowboard in the snow. They are in the middle of the action, showcasing skills, The snowboarder is the main focus of the scene, and their snowboard is prominently visible beneath them.
The background features building, adding to the wintery atmosphere of the scene. The snowboarder's impressive trick and the presence of the building create dynamic and exciting image.
The image features snow snow boarder performing a trick in the snow, with the snowboard in mid-air near a ramp. The snowboarder is the main focus of the scene, showcasing skills.
DAMA:
There are several other people and a building present in the background, possibly watching the snowboarder's performance or engaging in their own snowboarding activities. The overall atmosphere of the image conveys an exciting and dynamic winter sports scene.
Prompt: Given two different descriptions of an given image, with the question "What is this photo about? Please answer in great detail", tell me which one you like, and show me reasons.
DAMA provides a more comprehensive, detailed, and immersive portrayal of the image. It not only highlights the snowboarder's skill and trick but also fleshes out the background, giving a sense of the environment and the people around. The extra information about the ramp, the other people, and the overall excitement makes the scene feel more dynamic and real, which is why prefer it over DPO.
GPT-4:
Figure 11. Qualitative Comparisons between DAMA and vanilla DPO. Data source: Object HalBench.
15


Reviews:
[{"review_id": "WvJZCBXdLL", "reviewer": "Reviewer_ERND", "summary": "Authors propose a variant of DPO where the Beta hyperparameter is adapter dynamically depending on model and data- awareness. Author postulate the existence of easy and hard to distinguish example in alignment training settings, and therefore propose dynamic strategy to adjust those. Evaluation is reported on 5 benchmarks, including Object HalBench.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response to Reviewer $\\color{blue}\\text{ERND}$:\n\nWe highly appreciate your insightful comments, which help us a lot to better scrutiny and polish our work! The following are point-to-point responses.\n\n> **Q1. Implementation with more advanced models (e.g., LLaVA 1.6 and LLaVA-OneVision) makes DAMO more compelling.**\n\nThank you for your kind suggestion. We extend DAMO to more advanced models like LLaVA 1.6 and LLaVA-one-vision with a more comprehensive alignment dataset[1]. Due to the GPU constraint, **only the MLP layer is finetuned for LLaVA-one-vision**, and all models are finetuned with one epoch. As shown in Table 1, DAMO still demonstrates consistent performance gains across these advanced models.\n\nTable 1: Performance on the Object-hal bench.\n\nModel | Response | Sentence | Average Length\n--|--|--|--\nLLaVA-1.6 | 84.36 | 91.10 | 194.75\nLLaVA-1.6 + DAMO | **85.61** | **92.39** | 182.06\nLLaVA-OneVision | 81.48 | 90.29 | 223.08\nLLaVA-OneVision + DAMO | **85.82** | **91.22** | 243.86\n\n\n> **Q2. Measurements over more helpfulness benchmarks further support our effectiveness.**\n\nThank you for your kind suggestion regarding model helpfulness. Our evaluation on **LLaVA-Bench and MM-Vet** (Table 4 of the paper) shows that both our 7B and 13B models achieve competitive performance, with notable improvements of 6% and 9% over their baseline counterparts, respectively.\n\nFurthermore, we extend the helpfulness evaluation to the MME Bench with advanced models. As shown in Table 2, DAMO consistently improves both perception and cognition capabilities across different model architectures, demonstrating its effectiveness in enhancing both helpfulness and hallucination resistance.\n\nTable 2: Performance on the MME bench.\n\nModel | Perception | Cognition \n--|--|--\nLLaVA-1.6 | 1498 | 286 \nLLaVA-1.6 + DAMO | **1503** | **291** \nLLaVA-OneVision | 1565 | 335\nLLaVA-OneVision + DAMO | **1571** | **339**\n\n**Notes:** We sincerely appreciate your valuable feedback regarding the helpfulness and advanced models. Our extensive experiments demonstrate that DAMO achieves consistent and significant improvements across:\n\n1. Multiple model architectures (LLaVA-1.5 / 1.6, LLaVA-one-vision)\n2. Extensive benchmark suites, including:\n\t- Helpfulness (MME, MMVet, LLaVA-Bench)\n\t- Hallucination (Object-Hal, AMBER, MM-Hal)\n\n**Most importantly**, DAMO serves as a plug-and-play mechanism that can be seamlessly integrated into various architectures with different alignment data, while maintaining minimal computational overhead. This versatility and efficiency, combined with consistent performance gains across different settings, support the effectiveness and practical value of DAMO. \n\n> **Q3. Adding essential related work.** Understanding Alignment in Multimodal LLMs: A Comprehensive Study, Amirloo et al., 2024.\n\nThank you for providing this related work. We have added this method to our revised paper, `As a representative method, BDHS significantly advances the MLLM community by pioneering the alignment techniques into the advanced LLaVA 1.6 architecture, effectively bridging the gap between theoretical MLLM alignment research and practical applications.`\n\n> **Q4. Clarification about the model-aware strategy.**\n\nSorry about the confusion. Let us clarify our model-aware strategy that adjusts $\\beta$ according to the implicit reward gap between the preferred $y_w$ and rejected $y_l$. Specifically:\n\t1. A larger $\\beta$ is assigned to a larger reward gap between $y_w$ and $y_l$, which indicates that the model has already grasped this type of response well.\n\t2. A smaller $\\beta$ is assigned to a smaller reward gap between $y_w$ and $y_l$, which suggests that the model needs to improve its responsiveness on such cases.\nThis adaptive scaling mechanism helps the model focus more on cases with less confidence, while maintaining its performance on well-learned cases.\n\n> **Q5. Presentation refinement.**\n\n* Please do not mix serif (body-text, Figure 4) and non-serif font (Figure 1, 2, 3).\n\nThank you for pointing this out. We have unified the font in our revised paper.\n\n* L081: \"Similarly to Section 3.1\". It is odd to see this statement in Section 1.\n\nThank you for your kind suggestions. We have removed such descriptions and polished this as discussed in **Q4**.\n\n* All Section 2 could be removed ...\n\nWhile this section provides essential background for researchers new to preference alignment in MLLMs, we have significantly condensed it to improve paper conciseness.\n\n* Equation 8, is there an extra dot after D?\n\nSorry about the confusion. The first dot is the dot product, and the second dot is the period at the sentence end. We have modified as $\\beta_{D} = \\beta \\times \\alpha_{D}$.\n\n* one could appreciate to have a panel of responses in the appendix ...\n\nThank you for constructive suggestions, we have added these analyses to the revised paper.\n\n[1] Yi-Fan Zhang etal MM-RLHF: The Next Step Forward in Multimodal LLM Alignment, 2025.", "claims_and_evidence": "- Authors claim that introducing more regularization through Beta for easy to distinguish examples, and more regularization at the batch level help the model to learn from preferences, leading to improved results on Hallucinations benchmarks. However, this claim might be problematic to verify because all the experiments have been conducted on LLaVA 1.5, which is model lagging fairly behind on current VLM benchmarks compared to more modern alternatives. One could have probably found this work more compelling if those experiments would have been conducted on a more modern LLaVA (e.g LLaVA 1.6) or different models (e.g. InternVL2.5 or QwenVL2). \n- Besides, this work only focuses on hallucinations, while there is clearly a link with the helpfulness of the responses given by aligned models. A non-helpful model with short answers will always lead to less hallucinations, and hence improved scores on the reported Benchmarks. The claims made in this work would be better supported with an extended view of the problem, ie. not just the hallucinations. \n- Finally, while the formalization proposed in this work is extensive, and at time even a bit scholar like in Section 2, one could have preferred having a more extended experimental setup with more than just LLaVA 1.5 and more than one training dataset.", "methods_and_evaluation_criteria": "- Authors focus on hallucinations, which is a common pattern in recent Multimodal Alignment papers, but seem to ignore Helpfulness altogether, where the 'preferred response' is more helpful than the 'rejected response'. However, one could argue there is a clear relationship between helpfulness and hallucinations: it's easy to trick the hallucinations benchmarks reported in this work with shorter answers where the likelihood to produce hallucinations is inherently smaller but so the helpfulness. One could have appreciated benchmarks beyond hallucinations, along report on the responses length. \n- In that regard, it is not surprising to see no improvement on the LLaVA-Bench benchmark, which arguably might be the only benchmark that might give a measure of the helpfulness of a model.", "theoretical_claims": "- See 'Claims And Evidence'.", "experimental_designs_or_analyses": "- The LLaVA 1.5 model has been used in numerous multimodal alignment papers in 2024. This is surprising to see it explored in this work yet another time, with a training dataset that was already introduced in the RLAIF-v paper. There is a possible experimental flaw in constraining this work in that small experimental setting.", "relation_to_broader_scientific_literature": "Numerous papers have been published in 2023 and 2024 about using LLaVA 1.5 for alignment. Today, in 2025, one hopes to see the community explore further experimental settings beyond LLaVA 1.5. This work here explores the pair LLaVA 1.5 + RLAIF-V preferences that have been already explored in a previous work. The novelty lies only in the defining of the regularization parameter B of the DPO loss.", "essential_references_not_discussed": "- Understanding Alignment in Multimodal LLMs: A Comprehensive Study, Amirloo et al., 2024, performed alignment evaluation on LLaVA 1.6. This is unclear why the work evaluated here (a) report experiment in LLaVA 1.5 instead of LLaVA 1.6, and (b) does not compare results with Amirloo et al. 2024 on that very similar topic.", "other_strengths_and_weaknesses": "- Two important weaknesses of this work are (a) the lack of experimental novelty, - this work uses LLaVA 1.5 like numerous other papers before, and train it on the RLAIF-V preferences, exactly like Yu et al, 2024, (b) the single dimension view of the problem, where the focus is on hallucinations, - leaving alone the helpfulness of the model. It is likely that the aligned model might be less helpful and provide vague answers, explaining the higher scores on hallucinations benchmark, but the lack of improvement on LlaVA-Bench.", "other_comments_or_suggestions": "- Please do not mix serif (body-text, Figure 4) and non-serif font (Figure 1, 2, 3).\n- L081: \"Similarly to Section 3.1\". It is odd to see this statement in Section 1. Is that intended? \n- Generally the Paragraph L079-L086 does not explain what is Model-aware Preference Optimization. We learn it is similar to another section, but it's not stated how exactly the gap between chosen vs. rejected is used to scale Beta. Can you try to (a) make the overall process explicit, and (b) avoid referring future sections when the structure of the paper is not yet introduced?\n- All the Section 2 could be removed from the paper has that topic has been already presented in many other papers? For instance (Tang et al., 2024 - Generalized Preference Optimization: A Unified Approach to Offline Alignment) or (Tang et al., 2024 - Understanding the performance gap between online and offline alignment algorithms) are clear reference in that matter. In particular, it is a bit scholar to introduce PPO when this work is about offline direct alignment.\n- Equation 8 is there an extra dot after D?", "questions_for_authors": "Dear authors, \n- Have you considered evaluating the helpfulness of your model along the hallucinations? Optimizing only for hallucinations might introduce an important bias in the helpfulness of the responses of the aligned model. One can easily trick Hallucinations benchmark with overly short or vague answers. The lack of improvement on Llava-Bench, arguably the only benchmark evaluating the helpfulness, tend to show that direction. \n- Have you considered using another model than LLaVA 1.5, which by today standard is fairly old and has been already explored by 10+ other multimodal alignment papers since 2023?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xRrMtVCnfK", "reviewer": "Reviewer_TUJo", "summary": "The paper examines the inherent property of DPO regarding its imbalanced responsiveness to data with varying difficulty levels and proposes Data and Model-aware DPO (DAMO) to address this issue. Experiments across various benchmarks demonstrate that DAMO enhances both trustworthiness and general task performance.\n\n**update after rebuttal**\n\nThe authors have effectively addressed my principal concerns pertaining to the implementation details and have supplemented their work with additional experiments that underscore the generalizability of DAMO. Therefore, I have opted to retain my initial rating.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response to Reviewer $\\color{red}\\text{TUJo}$:\n\nWe highly appreciate your insightful comments and acknowledgment of our contributions! Your constructive criticism is invaluable in refining our work! We organize your concerns into the following 3 aspects:\n\n> **Q1. Clarification about sub-sentence construction.**\n\nThank you for pointing this out! We provide our detailed prompt template as: \n```\nYou are an expert in extracting facts from the given question-answer pair about an image. Your task is to: Analyze the provided question-answer pair based on the image, extract all factual statements from the answer, and rewrite them into self-contained sentences.\n\\n\\n Requirements for each sentence are: \\n1. complete, each sentence must be self-contained; \\n2. factual (omit opinions, subjective statements); \\n3. concise (no more than 77 tokens).\n\\n\\n Format your result strictly as:\\n### Facts:\\n- {Fact 1 (e.g., \"A red shoe sits on a wooden floor.\")}\\n- {Fact 2 (e.g., \"The shoe has laces and a white sole.\")}\\n- ...\\n\\n### Question-answer pair: Question: \"{Question}\" Answer: \"{Answer}\"\n```\nFor the very limited number of sub-sentences with more than 77 tokens (2/113156 for preferred, 1/114268 for rejected), we apply truncation over them.\nHere is a representative example demonstrating our sub-sentence construction:\n```\n\"question\": \"Is this book related to Literature & Fiction?\",\n\"answer\": \"No, this book is not related to Literature & Fiction. It is a religious or theological book, as evident from the title \\\"What Love Is This? Calvinism's Misrepresentation of God\\\" by Dave Hunt.\"\n\"facts\": [\n\"The book is not related to Literature & Fiction.\",\n\"The book is a religious or theological book.\",\n\"The title of the book is \\\"What Love Is This? Calvinism's Misrepresentation of God\\\" by Dave Hunt.\"]\n```\n> **Q2. Evaluation of different sub-sentence granularities.**\n\nThank you for pointing this out! To validate this, we modify the prompt template by replacing the `77` in `no more than 77 tokens` with `60` and `50`.\n\nTable 1: Token Length of the segmented sub-sentences from the preferred responses (22626 responses in total) \n\n| Tokens | > 77 | 60-77 | 50-60 | 40-50 | 30-40 | 20-30 | < 20 | Total sub-sentences|\n|-|-|-|-|-|-|-|-|-|\n| `less than 77 tokens` | 0.002% | 0.005% | 0.047% | 0.540% | 5.161% | 30.659% | 63.586% | 113,156 |\n| `less than 60 tokens` | 0.007% | 0.004% | 0.048% | 0.536% | 5.176% | 30.778% | 63.451% | 113,040 |\n| `less than 50 tokens` | 0.006% | 0.004% | 0.040% | 0.532% | 5.194% | 30.734% | 63.490% | 113,085 |\n\nTable 2: Token Length of the segmented  sub-sentences from the rejected responses (22626 responses in total) \n| Tokens | > 77 | 60-77 | 50-60 | 40-50 | 30-40 | 20-30 | < 20 | Total sub-sentences|\n|-|-|-|-|-|-|-|-|-|\n| `less than 77 tokens` | 0.001% | 0.014% | 0.066% | 0.554% | 5.228% | 30.630% | 63.507% | 114,268 |\n| `less than 60 tokens` | 0.001% | 0.010% | 0.067% | 0.562% | 5.264% | 30.552% | 63.544% | 114,247 |\n| `less than 50 tokens` | 0.006% | 0.004% | 0.048% | 0.524% | 5.194% | 30.734% | 63.490% | 114,218 |\n\nTables 1 and 2 show the statistics of the segmented sub-sentences. The statistics strongly support the robustness of our approach, with over 99% of sub-sentences containing fewer than 40 tokens, and more than 63% having fewer than 20 tokens.\n\nDue to the inherent inability of the LLM to precisely handle the length constraints [1], we manually truncate the sub-sentences exceeding the length constraints at different scales. The distributions of $\\delta$ are in Figures 1,2,3 of **[the Anonymous link](https://anonymous.4open.science/r/Rebuttal_DAMO-05C2/README.md)**, and subtle differences in $\\delta$ can be observed over different segmentation strategies. \n\nTable 3: Performance over object-hall bench.\n\n method | response | mention \n-|-|-\n `less than 77 tokens` | 82.54 | 90.64\n `less than 60 tokens` | 82.25 | 90.08\n `less than 50 tokens` | 81.78 | 90.20\n\nMoreover, we also evaluate the model trained with the given different $\\delta$ in Table 3, and observe subtle differences. These comprehensive statistics and experimental results demonstrate DAMO's robustness across different segmentation granularities.\n\n\n> **Q3. Typo Correction.**  Line 208 (right): \"the momentum $\\gamma$ is set to 0.9, and $\\bar{H}$ is initialized to 0.\" $\\bar{H}$ should be $\\bar{R}$.\n\nWe acknowledge this typographical error and have corrected this in the revised paper.\n\nThank you again for your valuable and insightful suggestions. We welcome any additional questions and would be happy to discuss them further.\n\n[1] Yuan et al. \"Following length constraints in instructions.\"", "claims_and_evidence": "All pivotal claims in the paper are supported by empirical interpretation or systematic experimental validation.", "methods_and_evaluation_criteria": "The proposed method DAMO is rational in tackling the imbalanced responsiveness issue, and the benchmark selection also makes sense.", "theoretical_claims": "All theoretical claims (the function and design of data-aware/model-aware preference optimization), including the proofs of key formulas, are accurate and validated.", "experimental_designs_or_analyses": "The ​performance validation in the article (e.g., the selection and analysis of ​hallucination benchmarks reflecting trustworthiness and ​general benchmarks) and the ​construction of ablation studies are ​methodologically sound and comprehensive.", "relation_to_broader_scientific_literature": "The paper's key contributions are primely related to Direct Preference Optimization (Rafailovetal., 2024).", "essential_references_not_discussed": "Works directly relevant to contextualizing the paper’s contributions are appropriately cited and discussed.", "other_strengths_and_weaknesses": "**Strengths:** The article demonstrates ​high writing quality with ​a well-organized content structure, ensuring ​ease of comprehension throughout.\n\n**Weaknesses:** The paper ​lacks implementation details for the ​response split procedure in data-aware preference optimization. Specifically, the ​prompt templates used in LLaMA3 and ​examples of partitioned sub-sentences should be reported.", "other_comments_or_suggestions": "Line 208 (right): \"the momentum γ is set to 0.9, and **$\\bar{H}$** is initialized to 0.\" **$\\bar{H}$** should be **$\\bar{R}$**.", "questions_for_authors": "1. Is the ​probability difference in ​Equation (6) sensitive to the ​LLM’s sub-sentence segmentation strategy, particularly regarding the ​number of sub-sentences generated?\n\n2. If such sensitivity exists, how does the ​segmentation granularity empirically impact the ​performance of DAMO?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hp01xA06hQ", "reviewer": "Reviewer_gFQs", "summary": "In this paper, the authors propose DAMO, an innovative data- and model-aware alignment strategy for Multi-modal Large Language Models (MLLMs), Specifically, a data-aware strategy is introduced to enhance the model's adaptability with data hardness, a model-aware is proposes to facilitate a more effective optimization with model's current responses. The authors conduct extensive experiments for evaluation, and the promising results demonstrate their effectiveness.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response to Reviewer $\\color{green}\\text{gFQs}$: \n\nWe sincerely thank you for your invaluable and constructive feedback. We particularly appreciate your positive acknowledgement of our novelty, clear organization, and extensive experimental validations. Below we provide the point-to-point responses to address your concerns about our approach. \n\n> **In-depth Analysis of $\\beta$.** To deepen the understanding of the proposed approach, the authors could provide more detailed analysis to demonstrate how the data- and model-aware strategy influences performance...\n\nThank you for pointing this out. Your suggestion is invaluable for this paper. To address this, we show the dynamic $\\beta$ concerning the data hardness, model responses, and the combination in the Figures 4,5,6 of **[the Anonymous Link](https://anonymous.4open.science/r/Rebuttal_DAMO-05C2/README.md)**.\n\n- **1. $\\beta_{D}$ for the data hardness (Figure 4 in [Anonymous Link](https://anonymous.4open.science/r/Rebuttal_DAMO-05C2/README.md)).**\nFrom Figure 4, we can observe that the range of $\\beta_{D}$ falls within (0.0524, 0.1428) with the original $\\beta$ initialized as 0.1. Moreover, we observe that the mean value is 0.0999 with a standard deviation of 0.0288. These observations demonstrate that $\\beta_{D}$ maintains proximity to the original $\\beta$, while adaptively adjusting based on the data characteristics, enabling a more effective capture of the data.\n\n- **2. $\\beta_{M}$ for the model responses (Figure 5 in [Anonymous Link](https://anonymous.4open.science/r/Rebuttal_DAMO-05C2/README.md)).**\nFrom Figure 5, we can find that as the training progresses, the $\\beta_{M}$ gradually converges to the original $\\beta$. Meanwhile, we find that the $\\beta_{M}$fluctuates within a moderate range of (0.0530, 0.1580), demonstrating controlled adaptivity. These observations suggest that as the model training stabilizes, its responsiveness becomes more consistent and eventually approaches a steady state.\n\n- **3. $\\beta_{C}$ by combining both (Figure 6 in [Anonymous Link](https://anonymous.4open.science/r/Rebuttal_DAMO-05C2/README.md)).**\nFrom Figure 6, we can observe that combining both the data- and model-aware strategies yields a more dynamic range of beta values, spanning from 0.0177 to 0.2261, which is wider than either. Moreover, while the value eventually stabilizes around 0.1, we notice that the mean value during the training stage is a bit lower than $\\beta_{M}$. This suggests that it relaxes the constraints during training based on the data hardness, enabling the model to better capture fine-grained data patterns and thereby adaptively enhancing its responsiveness to data characteristics.\n\nThank you again for your kind suggestions and supportive feedback. If you have any additional questions, we would be pleased to discuss them with you.", "claims_and_evidence": "The authors' claims regarding improving Multi-modal Large Language Model (MLLM) alignment through data- and model-aware strategy are well-supported. Extensive experiments across multiple datasets with multiple metrics demonstrate consistent improvements compared to baseline methods. The results effectively demonstrate that enhancing MLLM alignment via the data- and model-aware strategy improves the alignment performances.", "methods_and_evaluation_criteria": "The proposed method logically integrates data hardness and model responses by modulating the $\\beta$, strengthens effectiveness withour introducing additional computations. The evaluation criteria for different benchmarks are standard and appropriate for MLLM tasks. The use of multiple benchmarks, along with comparisons to existing methods, validates the evaluation framework's robustness.", "theoretical_claims": "The theoretical foundation of the method is sound. The approach is based on the DPO method, which is well-supported by experimental results.", "experimental_designs_or_analyses": "The experimental designs are comprehensive and valid. The authors evaluate their method on diverse benchmarks, demonstrating consistent performance improvements across different scenarios.", "relation_to_broader_scientific_literature": "This work makes significant contributions to the field of multi-modal large language models and preference alignment. The proposed method advances the state-of-the-art by demonstrating how data hardness and model responses can be effectively and efficiently integrated into MLLM alignment.", "essential_references_not_discussed": "The paper adequately cites relevant works in multi-modal large language models and preference alignment.", "other_strengths_and_weaknesses": "Strengths:\n1. Integrating data- and model-knowledge to MLLM alignment is an innovative idea, bringing new perspectives to the MLLM community. \n2. The paper is well-organized, the figures to illustrate the problems and approaches are clearly.\n3. Extensive evaluations, encompassing both quantitative metrics and qualitative assessments, demonstrate the effectiveness of the paper.\n\nWeaknesses: \nTo deepen the understanding of the proposed approach, the authors could provide more detailed analysis to demonstrate how the data- and model-aware strategy influences performance, for instance, visualize and analyze the dynamics of $\\beta$ during the training procedure for further illustration.", "other_comments_or_suggestions": "See weaknesses.", "questions_for_authors": "See weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces DAMO, a data- and model-aware enhancement to align multi-modal LLMs. The method adaptively tunes alignment strength based on data difficulty and model confidence, addressing a key gap in prior work. While initial concerns of reviewers focused on limited model and benchmark diversity, the authors provided substantial new experiments on LLaVA 1.6 and OneVision, alongside helpfulness metrics and fine-grained beta analysis, convincingly addressing all major criticisms. I recommend acceptance.
Decision: Accept (poster)