What Matters in Data for DPO?
Authors: Yu Pan
Zhongze Cai
Huaiyang Zhong
Guanting Chen
Chonghuan Wang
Venue: NeurIPS 2025 poster ()
Keywords: DPO
Fine-tuning
Alignment
Primary area: deep_learning
TL;DR: In this paper,  we study how preference data distribution influences DPO, from both theoretical and empirical perspectives.

Abstract: Direct Preference Optimization (DPO) has emerged as a simple and effective approach for aligning large language models (LLMs) with human preferences, bypassing the need for a learned reward model. Despite its growing adoption, a fundamental question remains open: what characteristics of preference data are most critical for DPO performance? In this work, we provide a systematic study of how preference data distribution influences DPO, from both theoretical and empirical perspectives. We show that the quality of chosen responses plays a dominant role in optimizing the DPO objective, while the quality of rejected responses may have relatively limited impact. Our theoretical analysis characterizes the optimal response distribution under DPO and reveals how contrastiveness between responses helps primarily by improving the chosen samples. We further study an online DPO setting and show it effectively reduces to supervised fine-tuning on the chosen responses. Extensive experiments across diverse tasks confirm our findings: improving the quality of chosen responses consistently boosts performance regardless of the quality of the rejected responses. We also investigate the benefit of mixing the on-policy data. Our results interpret the mechanism behind some widely adopted strategies and offer practical insights for constructing high-impact preference datasets for LLM alignment.

What Matters in Data for DPO?
Yu Pan
Zhongze Cai?
Guanting Chen
Huaiyang Zhong*
Chonghuan Wang
University of Sydney Imperial College London
UUniversity of North Carolina at Chapel Hill University of Texas at Dallas
Virginia Tech
Abstract
Direct Preference Optimization (DPO) has emerged as a simple and effective ap- proach for aligning large language models (LLMs) with human preferences, by- passing the need for a learned reward model. Despite its growing adoption, a fundamental question remains open: what characteristics of preference data are most critical for DPO performance? In this work, we provide a systematic study of how preference data distribution influences DPO, from both theoretical and em- pirical perspectives. We show that the quality of chosen responses plays domi- nant role in optimizing the DPO objective, while the quality of rejected responses may have relatively limited impact. Our theoretical analysis characterizes the op- timal response distribution under DPO and reveals how contrastiveness between responses helps primarily by improving the chosen samples. We further study an online DPO setting and show it effectively reduces supervised fine-tuning on the chosen responses. Extensive experiments across diverse tasks confirm our findings: improving the quality of chosen responses consistently boosts perfor- mance regardless of the quality of the rejected responses. We also investigate the benefit of mixing the on-policy data. Our results interpret the mechanism behind some widely adopted strategies and offer practical insights for constructing high- impact preference datasets for LLM alignment.
Introduction
The importance of aligning large language models (LLMs) with human preferences cannot be over- stated. Two leading paradigms for achieving this alignment are Reinforcement Learning from Hu- man Feedback (RLHF) (Bai et al., 2022: Ouyang et al., 2022) and Direct Preference Optimization (DPO) (Rafailov et al., 2023). The key difference lies in whether an explicit reward model is trained (as in RLHF) whether the model itself is optimized directly using preference data (as in DPO). Significant effort has been devoted to improving the performance of these methods by constructing more effective preference datasets. Common techniques include rejection sampling (i.e., generat- ing multiple responses and selecting the best or worst, see Khaki et al. 2024), annotator rewrit- ing/editing, and iterative use of on-policy data (Tajwar al., 2024).
However, despite the empirical progress, fundamental questions about what properties of preference datu actually malter for alignment remain underexplored. For example: Do chosen and rejected responses contribute symmetrically during optimization? How does the contrastiveness between response pairs affect learning? Under what conditions does incorporating on-policy data lead to gains?
In this paper, we provide systematic study of the role of preference data in DPO, combining the- oretical analysis with empirical validation. We begin by analyzing how the preference dataset's coverage of the high quality responses influences the gradient of the DPO loss. We begin by exam- ining how the coverage of high-quality responses in the preference dataset influences the gradient
"Corresponding to: yu.pan @udddddydyyedu.ar
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

of the DPO loss. Insufficient coverage of such responses can hinder optimization, as the DPO ob- jective lacks an explicit gradient signal to promote high-reward outputs when they are absent from the comparisons in DDPO Then, we analyze the optimal distribution that minimizes the DPO loss and how it is shaped by the distribution of the preference dataset. Our theoretical results show that the quality of the chosen responses plays dominant role in DPO performance, while the quality of rejected responses has a more limited impact. We further demonstrate that the widely adopted strategy of increasing contrastiveness between responses is effective primarily because it tends to elevate the quality of the chosen responses. Moreover, we examine a simplified online DPO setting in which high-quality chosen responses remain fixed, and rejected responses are generated in an online fashion. We show that this setting essentially reduces to supervised iin-tuning on the chosen responses, further highlighting the central role of the quality of the chosen responses.
We empirically validate our theoretical insights across multiple tasks and datasets. When fixing the chosen responses and varying the quality of the rejected ones, we observe little change in DPO performance. In contrast, when fixing the rejected responses and increasing the quality of the chosen ones, DPO performance consistently improves. Additionally, when holding the quality gap between chosen and rejected responses constant, improving the absolute quality of the chosen responses leads to better outcomes. Finally, we investigate how mixing on-policy and offline data affects performance under varying levels of offline data quality.
Preliminaries
Supervised Fine-tuning (SFT). SFT is is typically the first stage in adapting pre-trained LLM to downstream tasks. Given dataset DSFT consisting of high-quality cnaructiionrespoone pairs (x, y) (Ouyang et al. 2022), the objective is to maximize the log-likelihood of the the demonstration data. Specifically, SFT is minimizing the loss function:
LSFT (0; DSFT) = - -Exxy)>>ar [log To To [)]]
Reinforcement Learning from Human Feedback (RLHF). After SFT, fine-tuning with human preference data is widely used further align the model. RLHF begins by training reward model r(x, y) to reflect human preferences based on a preference dataset ( Ni,..i,,, 11,) where yi and yı, denote the preferred and rejected response, respectively. The policy my is then optimized, typically using reinforcement learning algorithms such as PPO (Schulman et al. 2017,0uyang et al. 2022), by minimizing the following objective:
Yi
CRLHF ( - -Ex (1t (x),) + 5 (T (x)l))))
Directed Preference Optimization (DPO). To simplify the process of RLHF, particularly to get rid of the training of reward model, Rafailov et al. (2023) realize that the reward function can be represented by the learning policy:
re y) = B[log r log [x)( + log Zx(x),
(3)
where 7(x) = Ey ( xx) exp y), is the partition function. Based on the Bradley-
(BT) preference assumption (Bradley and Terry 1952), together with the pairs of chosen and rejected responses, DPO fine-tunes the language model by optimizing the following loss function:
To TV xx.
LDPO (0: DDPO) = Ex vx) Preo log a
log
33 log
(4)
T x)
Teer (yr((
where yw and yt are the chosen response and the rejected response respectively, and (xr) I+e
The global minimizer of Eqs. (2) and (4) under BT assumption has been well understood to be
Ty (yl x) X (y)(() exp
( (x..)))
(5)
where r* is the true reward model. Rafailov et al. (2023) also derive the derivative of the DPO loss (4) with respect to the parameters 0:
VeLoro Doro)
(T D1)>
-x((x) Vo log (x(x) - Volog Moggoo(ye (y:
(6)

---

where y) 3 log the reference model Tel-
xoly|x] denotes the implicit reward induced by the language model To and [x]()))
In this work, we aim to better understand the role of the data distribution DDro, and identify the key factors that contribute to successful DPO training.
3 Related Works
RL-based LLM Alignment and DPO. Following SFT, RL policy gradient methods are then em- ployed align model outputs with human preferences encoded through reward modeling. While early RL approachs like TRPO (Schulman et al., 2015) and PPO (Schulman et al., 2017) established foundational frameworks, their computational intensity motivated the development of resource- efficient alternatives such as RAFT (Dong et al., 2023), RRHF (Yuan et al., 2023), SLiC (Zhao et al., 2023), and ORPO (Hong et al., 2024). DPO (Rafailov et al., 2023) cleverly reinterprets the RL objective through contrastive loss by eliminating explicit reward value while maintaining stable training dynamics with reduced computational demands. The DPO framework has subsequently in- spired multiple variants including KTO (Ethayarajh et al., 2024), IPO (Azar et al., 2024), and CPO (Xu et al., 2024a). Recent work by Shao et al. (2024) attempts to unify alignment-ster training paradigms through generalized perspective.
Data Quality in LLM Alignment. The critical role of data quality in L.LM alignment has been rigorously established across training paradigms. Early studies (Zhou et al., 2023a) demonstrated its decisive impact in fine-tuning contexts, while the contemporary focus on reasoning (Muennighoff et al., 2025) further underscores the performance gains attainable through carefully curated align- ment data. Empirical evidence specifically in DPO training (Morimura et al., 2024; Wu et al., 2024; Ivison et al., 2024) reveals two critical insights: (1) DPO exhibits stronger sensitivity to data quality compared to traditional RL methods like PPO; (2) strategic selection of high-quality samples im- prove DPO training performance. While Khaki al. (2024) and Gou and Nguyen (2024) suggest larger preference gaps improve DPO, Patmaik et al. (2024) and Xiao et al. (2025) find moderate gaps beneficial. However, systematic understanding of the role of data quality remains lacking in the literature.
On-policy DPO. Another fruitful stream of literature that is related to the proper usage of data is on-policy DPO implementations (Yuan et al., 2024; Chen et al., 2024; Guo et al., 2024; Rosset et al. 2024; Tajwar et al., 2024; Pang et al., 2024). Empirical analyses by Xu et al. (2024b) reveal that distributional mismatch between training data and the base model's original domain dispropor- tionately impacts DPO compared to PPO. On-policy DPO actively samples the intermediate model generations, which serves as an adaptive distributional bridge to mitigate out-of-domain degrada- tion. Despite the potential benefit of on-policy DPO, excessive reliance on the on-policy data is very likely to induce training instability that can lead to a significant drop in model performance (Lambert al., 2024; Deng et al., 2025). Feng et al. (2025) propose PILAF, theoretically -rrounddd sampling strategy for online and iterative DPO, which shares our work's 's conceptual focus on attributing opti- mization signals to the data distribution. Their method uses policy interpolation to explicitly align the training gradient with the true oracle objective and help stablize the training process. While trials have also been made to balance the on-policy and off-policy data integration (Wang et al., 2025), understanding when and how on-policy data can be helpful also remains to be further explored.
4 DPO Interpretation
In this section, we provide theoretical insights into what characteristics of a dataset matter most for DPO performance, and explain why some widely used data generation strategies are effective.
4.1 The Role of Distributions of Chosen and Rejected Samples
We begin by analyzing the standard DPO setup, where Doro is generated in two steps. First, triplet (x, YI, Y2) is sampled from the distribution D,, = x Y x V2. Then, a preference label assigned by the BT model, which identifies the more preferred response as y. and the less preferred one as y:- One important perspective we want to highlight for understanding the role of DDPo is coverage. Specifically, let us consider the classical solution Eq. (5) which is heavily decided by the optimal
3

---

reward model r". When our DDPO fail to include examples representative of responses with high true rewards in terms of r", then this optimal reward r* may not be identifiable, especially in the regime of "high reward", from the data alone. This lack of coverage can complicate the optimization process, as there no explicit gradient signal within the DPO objective to increase the likelihood of high-reward responses if they are absent from the preference comparisons in DDP-
To formalize this, let us fix prompt x and focus on a high-reward response yι, i.e., r' yn) is high. For a generating policy (-(xx), a higher value of ha (yn (x) corresponds to greater likelihood of generating high -ugh-uulity responses. Intuitively, yı is not covered by the dataset, DPO has no mechanism to increase its likelihood. The following theorem
Theorem 4.1. Let ne, be the policy trained with gradient descent on the DPO loss (4) step t under the preference data DDPo. Then for a given high-reward response (x, Yn), the likelihood To (yn) (x) will not change if yn is not in the support of Du.. That
yf $ supp(Du)) To (ynlx) TT,(yxlx))
Theorem 4.1 reveals that if yn is not in the support of D... Te, will not get the signal to converge towards the high quality response. Therefore, without sufficient coverage, DPO cannot promote desirable behaviors, regardless of how well the loss is minimized. In practice, this suggests that data selection and filtering strategies should not only focus on clear preferences but also ensure that high-reward responses are adequately represented to enable generalization. Following Theorem 4.1, we have a closer investigation on when the Te, (y|x) changes, the following proposition provides formal characterization of how the DPO training process updates the likelihood of responses, depending on how the current models preference ranking aligns with the true preferences, which may be of independent interest.
Proposition 4.2. Following the notation Theorem 4.1, y E supp (D)), the change in likelihood Tec+1 (yn (x) satisfies:
(i) _e.-, (yn(x) > (vnlx) if PYE-D.IN.Y (yn > Y2|x) >
pATTO.) YgnDD x.y, (yn > Y2|x);
(ii) Ne (ynlx) (vnlx) if PYSND..x. (yn Y2|x) ..TTO))
b.y (yn > Y2|x));
(ii To (ynlx) To, (ynlx) if PYIND.NNA (yn Y2|x) < YONDA x.y (yn > Y2(x),
where P FYsNDalxNNn (yn > Y2(x) denotes D.s. [Plyn > X)), Y2-Dalx.y. (yh > [ (yn > Y2 Plyn, > Y2/x) is the true probability the yı is more preferable, PBT stands for the BT model parametrized by the current model O and Du IX Y1 denotes the conditional distribution Y2 given X and Y1 under Du.
Y21x) represents EyInDa
In Proposition 4.2, PYa-DeIx.ys (yn > Y2(x) measures the average preference probability for yn observed in the dataset under the true preference model, and (BT(())) "x.-D_lx.,, (yn > Y2|x) stands for the preference probability predicted by the current model. Proposition 4.2 reveals when y is present in the dataset and the current model underestimates how good yı is relative to alternatives, DPO will increase its likelihood. Conversely, if the model overestimates y 's quality, it will receive a negative update, decreasing its likelihood.
Up till now, the above analysis, as well as many classical results in the literature, relies on the as- sumption that the dataset DDPO satisfies the BT model. However, in practice, we are often provided with a preference dataset without the ability to verify whether this assumption holds. In the fol- lowing, provide another perspective DPO loss, regardless of whether the BT assumption is valid. We denote the marginal distribution of the chosen response yv. and the rejected response yı of DDPO (-1x) and ( (x) respectively. For simplicity, we assume that yw and y: are independently drawn from T Th (- and m-(-xx). The
directly analyzing the distribution that minimizes the
following theorem characterizes the optimal policy that minimizes the DPO loss in Eq. (4).
Theorem 4.3. Denote the policy induced by the O minimizing the DPO loss function in Eq. (4) as Ty x). We can have
Tw (x(x)
TDpO(y|x xx) x
-Ter(y(x)) (x). m (y|x)
(7)

---

The proof of Theorem 4.3 is based on taking the functional derivative of the DPO loss and the detailed proof is delayed to Appendix. Theorem 4.3 reveals that DPO modifies the reference policy Tee based on the ratio between the chosen and rejected distributions, which is generated from DDPO- DPO places more density mass where Th exceeds m. The hold of Eq. (7) does not rely on the hold of BT assumption. If the BT assumption truly holds, Eq. (7) will align with the well understood minimizer of the DPO loss (5) in certain cases.
Moreover, note that (y(x) is also the optimal solution to the following optimization problems.
Proposition 4.4. The distribution tppo x) coincides with the solution minimizing the following loss function:
Tw (y(x)
(o Dnpo) Px ((1x)
log mi (y(x))
+ (Dxr Tnd)
(8)
Dxz TT (w)) (To (m)) + 3 (I [wr)
Together with Theorem 4.3 and Proposition 4.4, there are several insights that we want to highlight.
DPO may deviate from RLHE Rafailov et al. (2023) derive the DPO objective from RLHF by the observation the language model is secretly a reward model under the BT assumption. However, when the BT assumption does not necessarily hold, Eq. (8) implies that, in a more general sense, DPO implicitly performs reward learning with a specific reward function defined by nw and Ip:
Tw (ylx) m (y(x)
F(y|x) log
As a result, the distribution DDPO plays an essential role and DPO and RLHF may converge to very different policies.
The quality of the chosen responses matters. Eq. (7) suggests that DPO performance is fundamentally limited by the quality of the chosen responses. More straightforwardly, when 3 and the rejected samples are generated from the reference model, DPo x) is just T (y |x). It is intuitively unreasonable to expect the language model generating the responses whose quality is much better than the chosen samples after DPO. Such an intuition has also been reflected in many DPO practices. For example, Dong et al. (2024b) showcase the effectiveness of using a response improver to polish what the current model the chosen samples.
generates and use
The quality of the rejected responses may not always be critical. When T. (y xx) and T. (y((x) are very similar or even the same, DPO lacks a learning signal. However, the re- jected distribution may not always take the fundamental role. As shown in Figure 1, imag- ine now we have two different distributions on the rejected samples 7f(y|x) x) and nr (y x) that are only different from each other on the area where Twe (y(x) is small. Although xf(y) xx) and nr (y|x) are very different, the ratios x. (x((y)) (ylx))//eeex) (r/x)///r ///((//xx ()) and nw (y(x)/ (y x) can still be similar. Thus, (x(x)x and n/ (x(x) may still lead to similar performances of DPO. In the literature, there have been some numerical results that implicitly implying such an idea. For example, Khaki et al. (2024) compare different preference data gener- ation policies, including two based on & generated answers. One is called Best-vs- worrst where the chosen response is the best among the l generated and the rejected response is the worst. The other one is Best-vs Best-vs-randee vs-random where the chosen response is again the best and the rejected response is randomly chosen from the rest k - responses. Interestingly, Khaki et al. (2024) report similar performances of Best-vs-worst and Best-vs-random across sev- eral different tasks. More numerical evidence is presented in our Section 5.
The role of contrastiveness between chosen and rejected samples in DPO. Conventional wisdom in the field suggests that larger preference gap between chosen and rejected re- sponses enhances DPO training performance (Khaki et al., 2024; Gou and Nguyen, 2024). From Eq. (7), when nw (y(x) and (y/xx) are nearly identical, DPO receives little use- ful signal to learn from. In this sense, greater contrastiveness helps avoid such degen- erate cases. However, following what discussed above, once sufficient contrastiveness is achieved, further degrading the quality of the rejected responses may yield limited returns. A more fundamental benefit of contrastiveness appears to be that encourages higher- quality chosen responses. This also interprets why increasing the number of candidates
5

---

x(()))
x(()
(y(x)
Figure 1: An illustration example on when the quality of rejected samples are not essential.
in rejection sampling tends to improve the performance: increases the likelihood of se- lecting better chosen samples, while the specific choice of rejected responses is often less importantconsistent with the findings in Khaki et al. (2024).
4.2 Online DPO with Fixed Chosen Samples
In this subsection, consider the setting where the chosen samples yo generated before DPO training and keep fixed. For example, yu could be high-quality outputs written by human annotators or produced by stronger, well-aligned language model. In contrast, the rejected responses yr are generated from the current policy Ty and are updated as training progresses. Formally, the distribution of the dataset can be written DDPO Dx x л (y [x) ( where x' denotes the fixed distribution over chosen responses. Under the distribution D DrFo, we can show that such simplified online DPO is explicitly (almost) conducting SFT on the chosen samples. This is formalized in the following theorem.
Theorem 4.5. Under the data distribution Drro. the derivative of the DPO loss function satisfies
B²
VeLopo Dpro) = +
3 ((xx))))xx Ve m x)] +
(TolI(TTOllTT + e3.
3 Dxl (To(
Vo -F(x.)) (log (y())
SFT
where represent a quantity in the order of B, Ex is a three-order error, and the approximation holds when B chosen be small.
In current practice, the parameter B is chosen between 0.03 and 0.1 under many circumstances, as seen in prior work such as Rafailov et al. (2023), as well as implementations from Cerebras.Ar AI (Vishnevskiy, 2023) and Anyscale (Wang et al., 2024). In fact, through our dataset construction, it is also likely for eØ to be positive. Therefore, even when é is not negligibly small, the insights below can still be valid under many cases. The proof of Theorem 4.5 is based on the Taylor expansion of (z), along with the fact that yw and y: are independently generated in Dpro-
Theorem 4.5 establishes that the gradient of LDPO DDP) closely approximates the gradient of the SFT loss, with an additional regularization term penalizing the divergence between wo and wee-- Intuitively, when the chosen samples are of high quality and the rejected samples are generated from the current model, DPO effectively reduces SFT on the chosen examples. This result further reinforces our earlier observations:
The quality of chosen samples critical for DPO. Since DPO in this setting behaves like SFT on the chosen responses, the performance ceiling is determined by their quality. If the chosen samples are not of sufficiently high quality, collecting DPO data online may offer limited benefit.
Contrastiveness may not always be essential. In our setup. training progresses and the model improves, the quality of rejected responses increases, naturally reducing the gap between chosen and rejected responses. Theorem 4.5 suggests that this reduction in contrastiveness does not significantly affect training, since the model primarily learns from the chosen samples.

---

5
Numerical Evidence
In this section, through controlled experiments and quantitative analysis, we demonstrate a strong alignment between the derived theoretical insights and the empirical findings.
5.1 Experiment Settings
Base Model. In the following numerical experiments, we are utilizing Allen-AI's open-sourced Llama-3. 1..3..1.0 1---uul--------P--- checkpoint (Lambert et al., 2024) as the base model for DPO training (Sun et al., 2024). This model is exclusively supervised senerissedfinttueedd (SFT) on a mix of publicly avail- able and transparent SFT data from Meta's official pre-trained model (Dubey et al., 2024), making it possible to guarantee no data overlap during the SFT and DPO stages.
Datasets. According to the data recipe of our base model, we select two public datasets, LAION- Al's Open Assistant 2 (Köpf et al., 2023) and OpenBMB's s UltraFeedback (Cui et al., 2023), as the prompt datasets for our DPO training. We carefully curate the datasets to make sure the prompts are unique and not seen during the SFT stage.
Data Processing. For our experiments, we include multiple responses per prompt. For Open Assistant retain only first-turn dialogues and filter out prompts with fewer than responses. For UltraFeedback, we consider two variants: the original version cetrafeeddbck-oriignne al.. 2023), which provides 4 responses per prompt, and the tulu3 version (ultrafeedback-tul) (Lambert et al., 2024), which provides 2 responses per prompt. We focus on prompts that appear in both variants. After filtering, the Open Assistant and UltraFeedback datasets contain 4,603 and 41,633 prompts, respectively. Besides the mentioned sources of responses, to ensure the abun- dance of the dataset for comparison, we also leverage the responses generated by the Mistral series model (Meng et al., 2024; Jiang et al., 2023). For each completion pair, i.e., prompt and one of its responses, we use the Skywork-Reward-Gemmt -erraard-Gemaa--22----- 22-r---v 2 (Liu et al., 2024) model as an oracle to assign quality scores. These scores serve as proxy for data quality, enabling us to rank or categorize the samples and construct DPO datasets with different controlled qualities. We paired each prompt with five responses of varying quality, labeled as best, high, medium, low, and worst. We also use the filtered prompts of U1 illtraFeedbba dataset for on on-policy response generation. For a detailed explanation, please refer to Appendix B.2.
(Cui
Evaluation. To comprehensively assess the capabilities of our models, we employ a suite of stan- dard evaluation benchmarks that measure diverse aspects of model performance. Based on estab- lished practices in the field, we include AlpacaEval-2 (Dubois et al., 2024), MMLU (Hendrycks et al., 2020), IFEval (Zhou et al., 2023b), TruthfulQA (Lin et al., 2021) and GSM8K (Cobbe et al., 2021) for estimating models' abilities of general conversation, multitask understanding, instruction following, being truthful and informative, and mathematical reasoning, respectively.
For more details about the data, training and evaluation, please refer to Appendix B. Unless oth- erwise specified, all benchmark results reported in this work are calculated as the average of three independent runs with different random seeds, ensuring the reliability.
5.2 Chosen response quality dominates DPO training performance
In this part, we investigate the relative impact of chosen and rejected response quality on DPO training performance. According to the above analysis, we hypothesize that the chosen response plays a more critical role in determining the effectiveness of DPO training. To validate, we construct several DPO datasets with different qualities of the chosen and the rejected responses based filtered Open Assistant 2 and UltraFeedback datasets. Recall that for each query, we have five responses of different qualities. Concretely, each DPO pair is synthesized under two guiding principles:
Fixed Chosen, Varied Rejected: Among the multiple responses under each prompt, we lock in response of the highest quality as the chosen response, then pair with rejected responses whose quality is systematically degraded from the relatively high to low quality.
Fixed Rejected, Varied Chosen: We hold the rejected response to be at the lowest quality tier, while the chosen response is systematically varied from moderate to high quality.
7

---

As mentioned before, the quality of responses is revealed by the reward model scores. The detailed statistics can be found in Appendix B. To evaluate the effectiveness of DPO training, we compare the DPO-trained models with the SFT checkpoint (the untrained base model). The results are shown Table I.
Dataset
Configuration
GSM8K
LC-AE2
MMLU
IFEval
TruthfulQA
N/A
SFT Baseline
76.8
12.7
62.1
74.3
46.8
Best/Worst
78.4
20.9
62.8
72.3
48.4
Open Assistant 2
Best/Low
78.6
19.2
62.6
72.7
47.4
(Fixed Best)
Best/Medium
79.3
19.3
62.8
71.4
49.1
Best/High
78.4
19.6
62.7
72.1
47.5
Low/Worst
77.5
15.2
61.2
66.5
47.1
Open Assistant 2 (Fixed Worst)
Medium/Worst
78.2
17.0
61.3
70.4
48.0
High/Worst
78.2
17.4
61.2
70.2
47.6
Best/Worst
78.4
20.9
62.8
72.3
48.4
Best/Worst
80.4
36.5
64.8
77.4
62.2
UltraFeedback (Fixed Best)
Best/Low
80.8
34.5
63.4
76.9
58.6
Best/Medium
80.2
34.2
63.3
76.7
59.4
Best/High
79.0
33.6
62.5
76.0
58.7
Low/Worst
79.3
25.8
61.4
76.5
56.1
UltraFeedback (Fixed Worst) High/Worst
Medium/Woon Worst
78.6
26.7
62.1
75.8
58.0
79.5
30.9
63.7
77.0
61.3
Best/Won Worst
80.4
36.5
64.8
77.4
62.2
Table 1: Results across different datasets and DPO data mixtures. Data mixture types in the "Con- figuration" column are formatted as Chosen/Rejected. Quality tiers are ranked from highest to lowest Best, High, Medium, Low, Worst. "L.C-AE2" is the abbreviation for Length-Controlled AlpacaEval-2 benchmark.
Dataset
Configuration
LC-AE2
MMLU
IFEval
TruthfulQA
GSM8K
N/A
SFT Bascline
12.7
62.1
74.3
46.8
76.8
Continual SFT
18.7
60.4
71.5
46.9
78.7
Open Assistant
Online-DPPO
19.0
60.6
71.8
47.5
78.6
Continual SFT
35.8
61.6
74.1
57.1
79.5
UltraFeedback
Online-DPO 37.6
62.0
74.5
58.0
79.7
Table 2: Results of the chosen-fixee fixed online DPO and continual SFT training.
Our experimental results reveal clear asymmetric impact of chosen and rejected response quality. We find that the quality of the chosen response is the primary determinant of the model's final perfor- mance, effectively setting knowledge ceiling. This is demonstrated by the strong, monotonic im- provement in the fixed-worst setting on both datasets. the chosen response quality increases from Low to Best, performance shows a universally climbing pattern for each of the benchmarks. This confirms that high-quality positive examples are essential for reaching high performance. Mean- while, the role of the rejected response is more nuanced. When the chosen response is fixed to the best quality, the performance does not exhibit a monotonic trend as the quality of the rejected re- sponse increases or decreases, which indicates the quality of the rejected sample alone may not be reliable indicator DPO performance.
To empirically validate Theorem 4.5, we also test its central prediction: that DPO with fixed cho- sen responses and on-policy rejected responses approximates Supervised Fine-Tuning (SFT) on the chosen data alone. We compare two setups: (1) Online-DPO as described in Section 4.2, and (2) Continual SFT, where we perform SFT exclusively on the high-quality chosen responses from the preference set. We use the best response group mentioned in Table as the training dataset. The

---

results are presented in Table 2. Across both datasets, the performance profiles of Online-DPO and Continual SFT are nearly identical. This striking similarity provides strong empirical support for our theory, confirming that in this setting, the DPO learning signal is overwhelmingly derived from the chosen responses, effectively reducing the process to SFT.
5.3 Preference gap and exposure bias might not always be essential
Building on our understanding of the impact of response quality, we now turn investigate two additional factors frequently discussed in the context of DPO training: preference gap and exposure bias. Conventional wisdom suggests that a larger preference gap between chosen and rejected re- sponses enhances DPO training performance (Khaki et al., 2024; Gou and Nguyen, 2024) and that exposure bias arising from on- on-policy data also helps improve the model's s ability learn preferences (Guo et al., 2024; Dong et al., 2024a). However, our findings respectively challenge these hypothe- ses, demonstrating that neither the preference gap nor exposure bias might not be as critical as previously believed. Instead, the quality of the chosen response emerges as the primary determinant of model performance, overshadowing the influence of these factors.
To investigate the relative importance of preference gap versus chosen response quality in DPO training. we create six specialized datasets derived from the ultrafeodback-origine original dataset through controlled modifications. The core experimental design comprises two phases:
1. We first construct four baseline datasets with two orthogonal dimensions, i.c., the prefer- ence gap size (large and small) and the chosen response quality (high and low). This yields four combinations: large sap/high-quality (LG-HQ), large gap/low-qualit, (LG-LQ), small gap/high-quality (SG-HQ), and small gap/low-quality (SG-LQ). Rejected responses systematically adjusted in each pair to maintain precise gap sizes while preserving the original quality hierarchy.
2. To isolate the effect of chosen quality from gap magnitude, introduce two additional counterfactual datasets: the first one, LG-HH-inverse, maintains LG-HQ's high chosen quality (identical absolute scores) but reduces its gap, and the second one. SG-HQ-inverse, preserves SG-HQ's ' high chosen quality while expanding its gap.
With these strategically mismatched conditions, we enable direct attribution of performance varia- tions to either gap magnitude or chosen quality dominance. We then conduct DPO training these datasets and compare the outcome models' performance.
Our controlled experiments and their results depicted in Table 3 reveal one of our main observations: the quality of the chosen response the dominant factor driving DPO performance, significantly outweighing the influence of the preference gap. This conclusion is supported by three key obser- vations from our experiments First, when controlling for the preference gap (Part 1), elevating the quality of the chosen response yields the most substantial performance improvements, delivering a +7.1 to +8.7 point gain in LC-AE2. Second, while widening the preference gap does provide a benefit, its impact is comparatively modest, contributing smaller +3.0 to +4.6 point increase (Part 2). Finally, our counterfactual analysis (Part 3) provides the clearest evidence: when the chosen responses identical, isolating the effect of the gap by swapping the rejected response yields only a minimal gain of +0.8 to +1.4 points. Collectively, these findings strongly suggest that DPO's s ef- fectiveness is primarily rooted in quality anchoringlearning the characteristics of the high-quality chosen responserather than in margin maximization.
To explore the impact of exposure bias, we further conduct on/off-polict data mixture experiments on the UltraFeedback dataset. We utilize the prompt datasets and rejection sampling technique to generate on- on-policy rejected responses and mix these on-policy responses of different quality with existing off-policy data at different ratios. We then evaluate the performance of models trained on these mixed datasets.
Our investigation of exposure bias reveals a nuanced interaction between the inclusion of policy data and the quality of the chosen response. As shown in Table 4, introducing on-policy responses yields substantial gains in LC-AE2 and GSM8K metries. However, such benefit strictly depends on the base data quality: low-quality configurations show minimal improvement despite equivalent on-policy proportions. This confirms that exposure bias mitigation only amplifies existing quality foundations rather than compensating for low-quality chosen responses. Notably, our implemen-

---

Configuration
Chs
Avg.Diff
LC-AE2
Score
MMLU
IFEval
GSM8K
Part 1: Effeet of Chosen Quality (Gap size is held constant)
LG-HQ (High Quality)
-2.98
3.54
33.0
64.9
76.9
81.2
+8.7
LG-LQ (Low Quality)
-5.15
3.45
24.3
61.9
73.8
80.5
SG-HQ (High Quality)
-3.56
1.34
28.4
64.0
74.2
80.8
+7.1
SG-LQ (Low Quality)
-6.59
1.49
21.3
62.6
72.3
78.0
Part 2: Effect of Preference Gap (Chosen quality held constant)
LG-HQ (Large Gap)
-2.98
3.54
33.0
64.9
76.9
81.2
+4.6
SG-HQ (Small Gap)
-3.56
1.34
28.4
64.0
74.2
80.8
LG-LQ (Large Gap)
-5.15
3,45
24.3
61.9
73.8
80.5
+3.0
SG-LQ (Small Gap)
-6.59
1.49
21.3
62.6
72.3
78.0
Part 3: Effect of Gap via Counterfactuals (Chosen quality is identical)
LG-HQ (Large Gap)
-2.98
3.54
33.0
64.9
76,9
81.2
+1.4
LG-HQ-inv (Small Gap)
-2.98
1.92
31.6
64.7
76.5
81.3
SG-HQ-inv (Large Gap)
-3.56
4.51
29.2
64.2
75.1
80.5
+0.8
SG-HQ (Small Gap)
-3.56
1.34
28.4
64.0
74.2
80.8
Disentangling the effects of chosen quality and preference gap. This analysis compares the performance delta (by LC-AE2) from improving quality versus widening the gap. The gain from higher chosen quality (Part 1: +7.1 to +8.7 points) is consistently and significantly larger than the gain from wider preference gap (Part +3.0 to +4.6 points). The counterfactuals in Part further confirm this, showing that altering the gap while keeping quality constant has only a minor effect (+0.8 +1.4 points).
Table 3:
Avg. Chs
On-Pol. %
LC-AE2
MMLU
1FEval TruthfulQA
GSM8K
0.93
0
34.5
63.4
76.9
58.6
80.8
4.18
0
25.8
61.4
76.5
56.1
79.3
-0.33
10%
39.4
63.2
76.7
56.7
82.2
0.93
20%
39.2
63.4
76.2
56.4
81.9
4.18
10%
27.7
61.3
76.3
56.2
80.0
4.18
20%
27.4
61.5
76.5
56.2
79.6
Table Results across data mixtures of different on-policy data ratios. The "On-Pol. stands for on-policy data ratio in percentage.
tation adopts the commonly used on-policy data ratios in the literature, as excessive reliance on such data is very likely to induce training instability that can lead to significant drop in model performance (Lambert et al., 2024; Deng et al., 2025).
6 Conclusion
This work provides theoretical and empirical analysis of the role preference data in DPO. We demonstrate that the quality of chosen responses is the primary driver of DPO performance, whereas the quality of rejected responses plays comparatively less important role. Our results also clarify the mechanism behind commonly used practices such increasing contrastiveness, showing that their effectiveness stems largely from improving the quality of chosen responses. Our empirical studies across multiple tasks confirm these insights, highlighting that improving the absolute quality of chosen responses consistently yields better outcomes. These findings provide practical guidance for building preference datasets and raise important considerations for future alignment strategies, including better data selection, more targeted annotation protocols, and extensions to more complex preference structures.
10

---

References
Azar, Mohammad Gheshlaghi, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, Daniele Calandriello. 2024. A general theoretical paradigm understand learn- ing from human preferences. International Conference on Artificial Intelligence and Statistics. PMLR, 4457-4455.
Bai, Yuntao, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint axXiv:220.05880
Bradley, Ralph Allan, Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method paired comparisons. Biometrika (9/4)) 324-345.
Chen, Zixiang, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, Quanquan Gu. 2024. Self-play fine-tuning converts weak language models strong language models. arXiv preprint 240.00330..1335
Cobbe, Karl, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2
Cui, Ganqu, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback. CoRR.
Deng, Xun, Han Zhong, Rui Ai, Fuli Feng, Zheng Wang, Xiangnan He. 2025. Less is more: Im- proving IIm alignment via preference data selection. arXiv preprint arXiv:2502. 14560..
Dong, Hanze, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, Tong Zhang. 2023. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint crXiv:230..06767
Dong, Hanze, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, Tong Zhang. 2024a. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863
Dong, Qingxiu, Li Dong, Xingxing Zhang, Zhifang Sui, Furu Wei. 2024b. Self-boosting large language models with synthetic preference data. arXiv preprint arXiv:2410.06611.
Dubey, Abhimanyu, Abhinav Jauhri. Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The Ilama herd of models. arXiv preprint arXiv:2407.2178:
Dubois, Yann, Balázs Galambosi, Percy Liang, Tatsunori B Hashimoto. 2024. Length-contronn alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475.
Ethayarajh, Kawin, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, Douwe Kiela. 2024. Kto: Model
alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.013366
Feng, Yunzhen, Ariel Kwiatkowski, Kunhao Zheng, Julia Kempe, Yaqi Duan. 2025. Pilaf: Optimal
human preference sampling for reward modeling. arXiv preprint arXiv:2502.042270
Gou, Qi, Cam-Tu Tu Nguyen. 2024. Mixed preference optimization: Reinforcement learning with data selection and better reference model. arXiv preprint arXiv:2403. 19443
Guo, Shangmin, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. 2024. Direct language model alignment from online ai feedback. arXiv preprint 24X1:22402.0499.22
Hendrycks, Dan, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Ja- cob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300
Hong, Jiwoo, Noah Lee, James Thome. 2024. Orpo: Monolithie preference optimization without reference model. arXiv preprint arXiv:2403.07691

---

Ivison. Hamish, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A Smith, Yejin Choi, Hanna Hajishirzi. 2024. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. Advances in neural information processing systems 37 36602-36633.
Jiang, Albert Q.. Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, William El Sayed. 2023. Mistral 7b. arXiv preprint 1rX.0:2310.06825
Khaki, Saeed, JinJin Li, Lan Ma, Liu Yang, Prathap Ramachandra. 2024. Rs-dpo: A hybrid rejection sampling and direct preference optimization method for alignment of large language models. arXiv preprint arXiv:2402. 10038.
Köpf, Andreas, Yannic Kilcher, Dimitri Von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen, Oliver Stanley, Richárd Nagyfi, et al. 2023. Ope- nassistant ponversations-demoeraiiiinn large language model alignment. Advances in Neural In- formation Processing Systems 36 47669-47681
Lambert, Nathan, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, al. 2024. T\" ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15/24
Li, Dawei, Renliang Sun, Yue Huang, Ming Zhong, Bohan Jiang, Jiawei Han, Xiangliang Zhang, Wei Wang, Huan Liu. 2025. Preference leakage: A contamination problem in llm-as-a-judge. arXiv preprint ar arXiv:2502.015344
Lin, Stephanie, Jacob Hilton, Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958.
Liu, Chris Yuhao, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, Yahui Zhou. 2024. Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410. 18451.
Meng, Yu, Mengzhou Xia, Danqi Chen. 2024. SimPO: Simple preference optimization with reference-free reward. arXiv preprint arXiv:2405. 14734
Morimura, Tetsuro, Mitsuki Sakamoto, Yuu Jimnai, Kenshi Abe, Kaito Ariu. 2024. Filtered direct preference optimization. arXiv preprint arXiv:2404. 13846
Muennighoff, Niklas, Zitong Yang, Weijia Shi, Xiang Lisa Li, Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, Tatsunori Hashimoto. 2025. sl: Simple test-time scaling. arXiv preprint arXiv:2501.19393
Ouyang, Long, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35 17730-27744
Pang, Richard Yuanzhe, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, Jason We- ston. 2024. Iterative reasoning preference optimization. Advances in Neural Information Pro- cessing Systems 37 116617-116637.
Pattnaik, Pulkit, Rishabh Maheshwary, Kelechi Ogueji. Vikas Yadav, Sathwik Tejaswi Madhusud- han. 2024. Enhancing alignment using curriculum learning & ranked preferences. Findings of the Association for Computational Linguistics: EMNLP 2024. 12891-12907.
Rafailov, Rafael, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems 36 53728-53744
12

---

Rosset, Corby, Ching-An An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, Tengyang Xie. 2024. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprini arXiv:2404.03715.
Schulman, John, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz. 2015. Trust region policy optimization. International conference on machine leurning. PMLR, 1889-1897.
Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint urXiv:177.00344.
Shao, Zhihong, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
Sun, East, Yan Wang, Lan Tian. 2024. arXiv:2409.15355.
Block-attention for efficient rag. arXiv preprint
Tajwar, Fahim, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Ste- fano Ermon, Chelsea Finn, Aviral Kumar. 2024. Preference fine-tuning of llms should leverage suboptimal, on-policy data, arXiv preprint arXiv:2404.14367
Vishnevskiy, Alexander. 2023. rect preference optimization. haw.uuuinnng-laggu----odddededdd hatpaggdderccc-peeeeeeeeeeepeaaa 2025-03-23.
Fine-tuning language
models
using di-
URL https://www. cerebras. ai/blog//
Accessed:
Wang, Franklin, Sumanth Hegde, Kourosh Hakhamaneshi. 2024. Direct preference op- timization with synthetic data on anyscale. URL https://www. wwwscale com/blog/ wwwpereeneeoppiieee...re
www.ooattttttttttttttttttaaeee Accessed: 2025-03-23.
Wang, Yunan, Jijie Li, Bo-Wen Zhang, Liangdong Wang, Guang Liu. 2025. ing distribution shift and data quality for enhanced preference optimization. arXiv preprint arXiv:2503.15880
Inco-dpo: Balanc-
Wu, Junkang, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, Xiangnan He. 2024. 8-dpo: Direct preference optimization with dynamic Advances Neural Information Processing Systems 37 129944-129966.
Xiao, Yao, Hai Ye, Linyao Chen, Hwee Tou Ng, Lidong Bing. Xiaoli Li, Roy Ka-wei Lee. 2025. Finding the sweet spot: Preference data construction for scaling preference optimization. arXiv preprint arXiv:2502.1682.
Xu, Haoran, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, Young Jin Kim. 2024a. Contrastive preference optimization: Pushing the boundaries of IIm performance in machine translation. arXiv preprint 24Xiv:2401.08411
Xu, Shusheng, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang. Chao Yu, Yi Wu. 2024b. Is dpo superior to ppo for IIm alignment? a comprehensive study. arXiv preprint 1047:9240..0079.
Yuan, Weizhe, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, Jason Weston. 2024. Self-rewarding language models. Proceedings of the 41st International Conference on Machine Learning. ICML' 24, JMLR.org. 19.
Yuan, Zheng, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, Fei Huang. 2023. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint 0533:22
Zhao, Yao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, Peter J Liu. 2023. Slic- hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10420
Zhou, Chunting, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023a. Lima: Less is more for alignment. Advances in Neural Information Processing Systems 36 55006-55021.
13

---

Zhou, Jeffrey, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, Le Hou. 2023b. Instruction-followinng evaluation for large language models. arXiv preprint arXiv:211..07911
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract, we claim that this work, we provide a systematic study of how preference data distribution influences DPO, from both theoretical (discussed in Section 4) and empirical (discussed in Section 5) perspectives. In the introduction, we also mainly talk about the theoretical and the empirical contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It is fine to include aspirational goals motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Theorem 4.1 is based on the very commonly used BT assumption of DPO. We also discuss the case in Section when the BT assumption violated (Theorem 4.2). We also clearly pointed out that the setting considered in Section 4.2 is a simplified setting. We do not propose any algorithms. Hence, the computational efficiency, and privacy and fairness concerns of the algorithms are not applicable.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
15

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
All the formal proofs provided in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Ye]]
Justification: We carefully present the models, data, hyper-parameters used in Section 5 and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, deseribing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way accomplish this, but reproducibility detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), relcasing of model checkpoint, or other means that are appropriate to the research performed.
also be provided via
While NeurlPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky some cases, in which case au- thors welcome to describe the particular way they provide for reproducibility.
16

---

reproducible, they In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our code is publicly available on GitHub (Nhattattter:DDODatta) and the data is hosted on HuggingFace.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release code and data, understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips../ //nips.cc/ppli haapp.ccc/ubbll www.cc/cpppcc..cc.cc.coccco.oltt httpppplbllppdeeeeeeppppppppappe http://///gdiee//eueuubbbbb Cooe//ubeeieeeeeeeeeee ississsonn ssionPolicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All the details have been presented in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We have reported in Appendix and Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative еπτог rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Appendix B.5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: https://meett httpe://neuui.peeeee wwwpehwwwww....pppeeeeeeeeeeeeee wwtpccccccccctteeeeeeecec hwwbblii//chiccccieeiiieee
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics. Our research does not cause potential harms or harmful consequences.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper tries to better understand the role of data for the very standard DPO, which is not tied particular applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: is not relevant.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere usage guidelines restrictions access the model or implementing safery filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the data and models them.
open sourced. We have properly credited
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswithcode cww/ aatasatss has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification:
We will include an anonymized zip file.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and sereenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: is not relevant.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https fttps://neuripe: httt:::://orrpps...co/ffeeeeeeee cc///onferrenccc//////11111 for what should or should not be described.
21

---

A Technical Details
A.1 Proof to Theorem 4.1
For brevity, let us denote the relative logit of To and Fref by So(2, y) = log ma(ulz) Then
[(,, D) - E. loool/-ogo(((((/00 Mw) - f0 m))l)]-
With the gradient update
Bet1 02 VoL(B,,
(10)
the policies no.n (gla) and no, (z)(r)) have the following relationship
fe (Ir, v) (I, agge, (z, y) + O(a²)
via second-order approximation where go y) denotes the functional derivative of L with respect to fe, that is, (0 y) In order to compute go. we explicitly express the dependency of to fo as [[ Since
5C (z, y). Note that is usually taken around 10-5, so O(a²) 9) is negligible.
ofe
[f + ff] = (( MND [- log a (S(f(z, yw) + ef(z, Bwo f(x, v1) ef ((t.:)))]
for test function f. the functional differential is
0 de
8clf. fl Lif + cf
c=0
BED [1-o (3((z,)w)) f(x,, (f(2, vw) f(r, ui)]
(11)
Suppose ED is taken over population. Let define (111, 82(x) = (ww.)ixww..llll)--. aai (x).w)l)))))) Then (11) extends to
SC = -3 |||
1 ( ( ((x [(x U2) 32)))x(x)) (Py. 821x) + XXX (V2, v1l)))
P(y1 > y2 d2)
= -Iʃ -3
e --
+ Pla Px (21) > y2(x)dyzf yn unyd
+ *ʃ 1te (1
e -110(2.111
(z) (2 PP) > y2(x))d (ar ya dyyde.
As the functional derivative of is defined by equation 8L fl = If 5 (x. f(x y) dydr, it fol- lows that
8C
(z. v) = -3 /
+ PX > ddy
o.fo
+3/ /
E ((oo))))))))
+ PX)0 ~ )
-3 /
(3) 1+-- Px
> valx)dd2
e (lool..))))-a))))e))
P Px
Vn PP 1x)
3 /
3Uo fo
33 px (=
1t - px (x (z)) (2) v2(x))y2
+33 / e --llolea)))-((a px (x) 1
22 12 9222))dd
(12)
= --3 / px px(x)22(1, v2 wolp)> + 3
(otee Px (xx (4))2(y 22 v2l dyy
(13)
22

---

In (12), dummy variable y1 in the last integral is substituted by y2- (13) uses P(y > ()) + P(y x 82|z) 1.
sc be further reduced by realizing that (13) is expectation with respect y2 conditional distribution as follows. (13) is integrated by y2 with the term px(x)q(y v2(z). (X, Y1, Y2) Du. then this the joint density. However, x and y are arguments of 8fs only y2 is being integrated. In order to simplify the integrals, we define Dul (x, Yi) = (a. y) the conditional distribution of Y2 given X and Y1 under D... Equivalently, the density of 22 Y)) (z.y) (x)) where Px,yi (I, y) J px(x)q(y) y/2 V2(rrdy2 is the PX.y, (2,8)
and
marginal distribution of (X, 1)) on D.. Intuitively, it amounts to first sampling (X, Y1, Y2) D. and then considering only the case where (X, V1) = (x. y). With this definition, the first integral in (13) is
f (x)())2((.va)))ee)) > yalx)dy2 (x.-D..KXM)) Y2/=))]
(14)
The second integral in (13) reduced using the same distribution. Recall that the BT model with reward r is defined as
exp(r(x, (1))
pBT (y1 V2(x) = exp(r(x, y1)) + exp(r(x, v2))
((((2,0)) r(z, 82)).
Let us denote P5T the BT model with reward (S0. that is,
P + val=) ( vi) - So(2,,22)).
Then
1 (loo fo(m.))) Px (x) (v)2 v2lx)dy2 = / o [(B(Fo(x.)) - fo(a (2)), (x)2 82lz)dy2
f (y 12 (z) 2g) 22 r)y2
f
EY [P 37 (y Y2 vlx (2,
(15)
Plugging (14) and (15) in (13) leads to
(o (r, y) = ---ys=D..i.. [P(y > Y2/z) - PBT > Y2/x)|2, v) 2px.yi (z,
(16)
A.2 Proof to Theorem 4.3
For simplicity, let us consider a fixed context x, and we take the functional derivative of the DPO loss in Eq. (4) with respect x))),
@L (0; DDPO)
Olog
Bogg 5oly (x) Farly xx) Tarld?
B log so x))
= E Eyy Nt)-Doo
(17)
imo (y(x)
Note that the derivates are only non-zero either when = Ym or = yı- Therefore, when y = Y (x()) ( log mo(y (x) Blog )n)(Sr)
alog
ax)ly
8y=y=
(x)(
= -(1-0 log (
To(y)(>)))
8y=y..
$ log
(18)
Thet (v/1x)
To (v(vix) x))
where 8 the Kronecker delta, and the equality holds since Olog o (=))/Oz = 1 - (2)). Similarly, we can have that when y
yı.
i log a ( log xoly B Blog =o(v marly
str(y( (x)
8y y
Oxo(
=- -1110
BBog To (v(x)) (rer(y
T To(y|x))
x))
dy=y1
B log Teet (y|x)
(19)
To Ty(y)x)
23

---

By plugging Eqs. (18) and (19) into Eq. (17), we can get
@LDpO ( DDPO)
Two (y x)
3 log =o(vi(x))
To x
mrrr(//x) ]
Ey (-
(ye((y) B log
Tol (y xx)
Ti (y ix)
Blog xo(y|x))
2o(vwwl (x)
By (ym=m(((
Blogg
( (x)
(r(x(y/
TTeel (Yw x)
Tue (y x)
To (x(() Blog To(y x)
18 By ( (-1b))
3 log
Tol (y ix)
Trer (y:
Trer(y|x)
Tw (y(x)) (x)
(y(xx) (x(lxx
(8l0g log Ty (x))
3 log [y(()
Tror (xx) )
Ex, B
Te(y" (y(c)
(y(x) (x(((xx)
(y(x)
Tue (y x)
Blgg To (yilx))) B log
To x
EE Ey (-1x))
To (y xx)
(yr
T
Tw (y|x))
(y(xx) The (y/lxx)
c log calorbe))
log (r(
To(y)|x) (y:
To (y(x)))
E Bx)mmx((11x)
(glog
33 log T x
(y(c)
Tw (y|x) (x/(xx)
Two (y x)
Blog (yilx)) Tiat (x)
8 Blog To(y|x)
E B
Te(y (
xx)
Tn ( x
(-)xx((-1x)) (
E
log mylsin)
mo (xl)) +log (ycle) Sull)
+.8 log To (x))
log (x(3)) (erly)))) (at(1/1 a (Blog
To(vi|x) Tee (y/(x)
B log
To(y|x)) [x))
Terly (x) )
where the second equality is by changing the probability measure, and the third equality holds due the fact that (--x) = (x)) Therefore, by setting ( for every y as,
Tww (ylx) $
To ( x (yr(y(x)
T x)
the functional derivate 8Coro (0: Deeo) is always O, since - log [](s[x) (n [o(yl[x) is going to be equal to O. We finish the proof.
T- (y|x) +log (x(x)))
(xilx) +B +Blog (y())x))
Tar(y)
3 log [x]
A.3 Proof to Theorem 4.5
First, apply Taylor Expansion to (2)) = 1/(1 + e =)
1
a(2)
4 + o(23),
since the second derivate of o()) at z = 0 is equal to O. Another useful fact that we will heavily rely on is that
E Drx.yo. Vo log [y()(())] = E (x Vo log (y]]x]]
-EXND
To To
TO To
ExxDx
ExxDa
Vo
To(yilx)dy)
0,
(20)
where the first equality holds due how we construct Doro and the last inequality because of J The (yi xx)y; = 1.
24

---

From Eq. (6), we can have
Vo VoLoro DoFO)
-3 (x.y v.) Doeo
(o(Fo(x, -o( loo log Vo log ()]||(]]]]
(2 2 (fo(x, -1xxxx))))xxxxxx)))) (o (Vologo(yyll)) Vo Vologmo(yixx)]
1
BE v.) Doen
B y.. Deo
Vo lo (ym
4 1 (x.. v y1 Dam [(fo(x, vi) - fo (x, (x)) [o log mo(yu>))) - Vo log (x(((())
3 E Di l ((f((x,))) fx( mo
B x
Vo oo
4
B
[x.y Vi (x.y
Foo log To
4
3
+
E
[ (xy Vo log [y|]) + V (Tell
2
4
where the second and the third equations are based on the repeated use of Eq. (20), and the last equality holds because
log Te(
VolDKI (Fo (res) V (x
Teer Tye (x)
(yi x)
BxNDx
Tx(x) log
Trer (y xx) dy:
To x Tree (y:
ExxDx
Vo TT(yi(x) log
dyi
/ (oloyix)Vologgooooe
Ex
log
V Vo=o(y)(x)dy))
+ ExNDa
Teer (yer(y x)
1 3 E (. (e y:) Ve log To
where the last quality is using Eq. (20) again. We finish the proof. Note that the term (fo(x, - fo ym)) is likely to be positive since yı is generated from (-(x).
B
Implementation Details
B.1 Dataset Details
As explained in the section 5.1, we use a reward model to annotate all the completion samples. The quality score distributions of the used datasets in the paper are given in Table
Some may doubt the reliance on reward model (RM) scores as quality criteria, given known cali- bration limitations that prevent these scores from being perfect human preference estimators. We justify this design through two principled arguments. First, our experiments require aggregating re- sponses from heterogeneous sources, where unified quantitative metric becomes indispensable for quality-aware sample reorganization. Second, empirical evidence from our LLM-as-a-judde com- parison validates RMS practical superiority: the Tulu3 UltraFeedback dataset, RM and L.M evaluators disagreed on response quality rankings for 10,000 prompts. Crucially. when training DPO models on datasets filtered by each method, RM-based selection achieved better performance, demonstrating its operational robustness. Recent work has also revealed the possible limitation of -a-uue- methods (Li al., 2025). Although we do not claim RM scores are intrinsically perfect, their empirical stability and cross-domain applicability make them functionally optimal choice for our multi-soun souti-sooure quality stratification objectives.
B.2 On-policy Data Generation
In the context of Direct Preference Optimization (DPO), on-policy data refers to preference pairs generated using the policy model that is currently being trained recent checkpoint thereof.
25

---

Statistical Measures
Dataset
Type
Mean
Std
Min
25%
Med
75%
Max
C
-3.00
3.02
-20.9
-4.00
-3.30
-1.66
19.4
OA2 Best+
R
-8.03
3.84
-24.4
-10.1
-7.44
-5.40
7.31
C
-3.00
3.02
-20.9
4.80
-3.30
-1.66
19.4
OA2 Best+Low
R
-7.01
2.91
-21.6
-8.94
-6.97
-4.91
6.19
C
-3.00
3.02
-20.9
-4.80
-3.30
-1.66
19.4
OA2 Best+Medium
R
-5.32
3.06
-22.0
-6.11
-5.12
-3.48
13.9
C -3.00
3.02
-20.9
4.80
-3.30
-1.66
19.4
OA2 Hest+High
R
4.80
2.77
-22.0
-636
-472
-3.14
13.9
C -7.01
2.91
-2166
-8,94
-6.97
-4.91
6.19
0A2 Low+Worst
R
-8.03
3.84
-2.44
-0.11
-7.44
-5.40
7.31
C -5.32
3.06
-2.00
-691
-5.12
-3.48
13.9
OA2 Medium+Worss
R
-8.03
3.84
-24.4
-10.1
-7.44
-5.40
7.31
C 4.80
2.77
22.00
6636
4.72
-3.14
13.9
OA2 High+Wors
R
-8.03
3.84
24.44
11011
-7.44
-5.40
7.31
C -0.33
4.91
19.5
4.19
1191
1.34
21.0
UF Best+Wors:
R
-8.54
3.36
24.5
10.8
-.56
-6.19
6.28
C -..33
4.91
19.5
4.19
1.91
1.34
21.0
UF Best+Low
R
4.90
3.07
21.5
6,94
4.90
33.00
16.5
C -0.93
4.91
19.5
4.19
1191
1.34
21.0
UF Best+Medium
R
4.18
3.77
23.6
6.53
4.34
-2.33
18.2
C -0.93
4.91
19.55
4.19
-1.11
1.34
21.0
UF High
R
-3,22
4.18
21.5
6.06
3.67
-.14
18.75
C 4.90
3.07
21.5
6694
4.90
-3.00
16.5
UF Low+Worst
R
-8.54
3.36
-2.55
-00.8
-856
-6.19
6.28
C
4.18
3.77
23.6
6.53
4.34
-2.33
18.2
UF Medium+
R
-8.54
3.36
22.55
-0088
-8.56
-6.19
6.28
C
3322
4.18
-2.55
6.06
-3677
-1.14
18.75
UF High+Wort Worst
R
-8.54
3.36
-24.5
10.8
-8566
-6.19
6.28
C 2.98
3.70
21.5
5.28
-3.34
-1.32
18.2
UF LG-HQ
R
6653
3.37
-24.1
8..99
-6.47
-4.25
11.0
C
-.16
3.67
-23.6
-7.0
-5.44
-3.22
18.2
UF LG-LQ
R
8.61
3.35
-24.2
-1088
-8.62
-6.28
6.28
C
-3566
3.76
-23.6
-.84
-3.80
-.78
18.2
UF SG-HQ
R
4.90
3.07
-21.5
-694
-4.91
-3.00
16.5
C
-6.59
3.32
-23.6
-8.81
-6.56
-4.34
14.9
UF SG-LQ
R
-8.08
3.41
-24.2
-10.3
-8.12
-5.69
11.3
C
-0.93
4.91
-19.5
4.19
-191
1.34
21.0
UF HQ-On-Pol.10%
R
4.90
3.07
-21.5
-6.44
4.91
-3.00
16.5
C
-4.18
3.77
-23.6
-6.53
-4.34
-2.33
18.2
UF L--OOnPol.200%
R
-8.54
3.36
-24.2
-10.8
-8.56
-6.19
6.28
Table The overall dataset quality score distribution (chosen (C) rejected (R)). "OA2" and "UF" stand for Open Assistant 2 and UltraFeedback, respectively. The LG-HQ-inv. dataset utilizes the LG-HQ's chosen part and SG-HQ's rejected part, while the SG-HQ-inv. uses the SG-HQ's chosen part and SG-LQ's rejected part. The datasets in the exposure bias experiments share the same chosen responses within the same chosen quality level, respectively.
26

---

Training with such data allows the model to learn from its own evolving capabilities, Several meth- ods leverage on-policy data, including fully online approaches like online DPO (Guo et al., 2024) and online iterative DPO (Dong et al., 2024a). While potentially effective, these online methods of- ten require frequent interaction with preference judge (e-g., a human annotator or a reward model) during the training loop, which can significantly increase computational and annotation costs.
An alternative strategy, which balances the benefits of on-policy data with practical constraints, involves generating a batch of on-policy data offline before commencing or resuming DPO training. This generated data can then be mixed with existing off off-policy datasets (Lambert al., 2024; Deng et al., 2025). In this paper. we adopt a similar offline generation approach, closely following the methodology described by Lambert al. (2024). The specific process for generating our on on-policy preference data is detailed in Algorithm 1.
Algorithm On-Policy Data Generation Process
Input: SFT model checkpoint TSF,, Reward model Tú, Offline preference dataset Dofftine
((pi, Mon,s vr.)) { On-policy data ratio p. Generations per prompt k.
2: Output: Mixed preference dataset Dmixed-
3: Initialize Doo-policy O.
4: Sample a subset of prompts Pon S {p: such that Pn = Lo x J.
5: Let Lon = i p; E } be the indices of the selected prompts.
6: Let Dremaining
{(pts Mu (1,s)) ) i LonJ.
7: for each index i E Ion do
8: Let = Ps- yu Vu 31 = yl
9: Generate & candidate responses {ıバ} using pi 3// ~ TSFT(-Ip)
Using specified
sampling parameters
Score each generated response: s/ = re (p.pv') for == 1....,k. Identify the best on-policy response: Bbest arg maxy; {8/}} J. Let sbest r (p. Hbea). Retrieve the score of the original preferred response: 8w T (p, yw). if hhes > Sw. then
12:
14:
Add new preference pair (p, Wheat Mwe to Don-policy. > Replace original chosen response
else
16:
Add new preference pair (p, yuu Vhea) to D Don-policy. Replace original rejected response
end if
18:
end for
Combine the datasets: Dmixed
Dremaining J Dan policy-
20:
return Dixxed-
The generation process utilized multinomial sampling with parameters specified in Table 6. The reward model Ts used for scoring the generated responses in step 9 of Algorithm is the same reward model used to create the initial offline preference dataset D. Domline
Parameter
Value
Sampling Method
Multinomial Sampling
Sampling Temperature
0.6
Max Generation Length
1024 tokens
Responses per Prompt (k)
8
Table 6: Parameters for On-Policy Response Generation.
B.3 Training Hyperparameters
For all DPO experiments, we adopt the standard DPO training pipeline using the Huggingface frame- work with the following hyperparameters:
Optimizer: AdamW (31 = 0.9, 32 0.99) with no weight decay
Learning Rate: Linear warmup with ratio to peak of 5 10-7, followed by cosine
decay
27

---

Batch Size: A global size of 32 via gradient accumulation over 4 steps
Duration: 2 epochs
DPO Beta: 0.1
Sequence Length: 2048
Precision: bfloat16
For the continual SFT training mentioned in Table 4, we adjust its peak learning rate to x 10-6 and AdamW $2 0.95, and keep other optimizer and batch size parameters the same the DPO setting.
B.4 Evaluation
We leverage the Tulu3 evaluation pipeline except for AlpacaEval-2 2 to exclude the possible evalua- tion data leakage. For the AlpacaEval-2 assessment, we adopt a generation config of beam-search multinomial sampling with num beams=3 and temperature=1.0 O. For the other benchmarks, use the default configuration as described in Lambert et al. (2024). The score metrics used in our experiments are presented in Table 7.
Benchmark
Core Metric
Setting Details
LC-AE2
Length-Controlled Win-Rate
Alpaca-Eval 2.0 version. O-shot.
MMLU
Accuracy
5-shot setting.
TruthfulQA
MC2
6-shot setting.
IFEval
Instruction-Follote Accuracy
O-shot setting.
GSM8K
Exact-Match Accuracy
8-shot setting.
Table 7: Overview of Evaluation Benchmarks and Metrics. All the evaluations run with only attempt, i.e., under the pass@1 setting.
B.5 Compute Resources
All experiments were conducted on a server with 128 CPU cores, 1024 GB memory, 96 TB SSD storage and 8 NVIDIA H20 GPUs. Under these conditions, each training step in the experiments takes approximately 10 seconds.
Running the full set of evaluation benchmarks (excluding Alpaca-Eval) on single GPU requires ap- proximately hours, and Alpaca-Ever evaluation times vary between 10 and 30 minutes per model, due to network fluctuations and API request limits.
28


Reviews:
[{"review_id": "FSNKArkyIf", "reviewer": "Reviewer_aWhg", "summary": "This paper provides the first systematic study on which properties of preference data matter most for DPO in llm alignment. Through theoretical analysis and extensive experiments, the authors show that the absolute quality of chosen responses in the preference pairs plays a dominant role in DPO training, while the quality of rejected responses has only a limited impact once basic contrastiveness is ensured. The paper further reveals that increasing contrastiveness or mixing on-policy data is effective primarily because it raises the quality of chosen responses. These findings offer practical guidance for constructing high-impact preference datasets for DPO, highlighting the importance of curating high-quality positive responses to maximize alignment performance across diverse benchmarks.", "questions": "My questions are stated in the weakness section.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The paper provides the first systematic theoretical and empirical analysis of how preference data properties affect DPO, offering clear guidance on what truly matters for DPO-based LLM alignment.\n- The findings deliver practical advice for data curation—demonstrating that improving the quality of chosen responses is the primary driver of performance, while the quality of rejected responses is far less important once minimal contrastiveness is achieved.\n- The work demystifies common practices such as increasing the preference gap or on-policy mixing, showing that their main benefit comes from raising positive response quality.\n\n**Weaknesses**\n- The paper does not propose any new algorithm or model, focusing entirely on analysis rather than methodological innovation.\n- The results are specific to DPO, and may not directly generalize to other alignment frameworks such as IPO, ORPO, SimPO, etcs.\n- The paper highlights the importance of curating high-quality chosen responses but does not address the practical challenges of reliably collecting or evaluating such data in real-world settings.", "was_revised": true, "final_justification": "This paper provides a solid theoretical analysis and experiments. However, the main finding, while rigorously demonstrated, aligns with established intuition and could be perceived as lacking in significant novelty. Therefore, I leave my score as borderline accept.", "rebuttal": "We sincerely appreciate your effort and your time in reviewing our paper. Please see the below for our responses.\n\n### No new model/algorithm/methodological innovation\n\nWe sincerely appreciate your encouragement to translate our observations into new methods or algorithms. However, we respectfully disagree with the premise that contributions must involve algorithmic novelty to be considered innovative. A core contribution of our paper lies in theoretical clarity and novel insights into the role of data in the behavior of DPO, a widely adopted yet not well understood method. Our work characterizes the impact of various preference data structures on DPO’s performance—results that were previously unknown and that address pressing open questions in the community. Foundational analysis has a long-standing tradition of driving progress by revealing when, why, and how existing methods work. We believe our contribution fits firmly within this tradition. \n\n### Specific to DPO\n\nWe fully acknowledge that our analysis focuses specifically on DPO. However, we believe this focus is well-justified for several reasons. First, DPO is one of the most foundational and widely adopted paradigms in preference-based fine-tuning. Many recent alignment methods are either inspired by or closely related to DPO. While our results are centered on DPO, they have broader implications for understanding and improving these related methods. Second, our work provides a clear and extensible framework—particularly in the numerical analysis—that can serve as a starting point for investigating the role of data in other alignment approaches. To further support this, we will release our codebase, which can be easily adapted to analyze or benchmark other alignment methods beyond DPO.\n\n### Practical challenges in collecting high-quality data\n\nThank you for raising this important and practically relevant point. Our paper is primarily analytical in nature, and its goal is to explain the impact of data quality on the performance of DPO. While implementation-level data curation strategies are important, they might be outside the scope of this work. However, we do identify concrete properties—such as the informativeness of chosen–rejected margins and the effect of suboptimal preference scores—that guide future data collection. Rather than offering ad-hoc heuristics, our framework provides principled criteria for evaluating whether a dataset is suitable for effective DPO training."}, {"review_id": "DttQum4e8d", "reviewer": "Reviewer_jmf2", "summary": "The authors talk about what matters in data for DPO with some theories and experimentations. From the theory side, the authors show that where the  high-quality chosen responses dominate the gradient, and when the samples are generated on policy, DPO becomes SFT. The experimentations try to support the claims such as the chosen sample quality is the only thing that matters in DPO, and preference gap and exposure bias, which are commonly believed to be important given past literature, also have limited impact.", "questions": "see weakness", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength: \n1. This is an interesting and important topic to have both theoretical insights and experiments to support it. The paper offers both and provides good insights for practitioners to construct DPO data when training their models.\n\nWeakness:\n1. The biggest concern and weakness is that, although the axis of experiments design was promising, specifically “Fixed Chosen, Varied Rejected” and “Fixed Rejected, Varied Chosen.”, the experiment results are not comprehensive enough to support the theories. I would like to see a more informative analysis that includes additional quality levels, such as at least 3 different levels of responses for chosen and rejected (9 experiments in total). Currently table 1 only has 3 rows, which is insufficient to reach the conclusions the author claimed. \n2. Regarding theorem 4.5, I would like to see experimentation results to support it, which shows that the performance of the SFT results vs the DPO with on-policy samples. \n3. Regarding Table 2, it does seem that there is some difference between LG-HQ-inv & LG-HQ, SG-HQ & SG-HQ-inv. I think the main problem is still the lack of comprehensive experiments to draw strong conclusions about the role of gap size versus chosen quality, though I think this table is already better than Table 1.\n4. The experiments' terminologies are confusing and introduced unclearly. For example, for Table 1, the caption is telling me “The “Type” column abbreviates combinations of chosen and rejected sample quality, where “Best” and “Worst” denote the quality of chosen and rejected samples, respectively, and “High” or “Low” indicates the quality of the paired response.” which is unclear what those levels refer to. It would be great if the terminologies are specified in a clearer way.", "was_revised": true, "final_justification": "The new experiments, analysis, and practical guide addressed most of my concerns. I am willing to give an accept for this paper.", "rebuttal": "### **1. More comprehensive numerical experiments**\nThank you very much for your constructive suggestion to conduct more comprehensive numerical experiments. Following your recommendation, we paired each prompt in both the OpenAssistant 2 and UltraFeedback datasets with five responses of varying quality, labeled as *best, high, medium, low*, and *worst*. For each dataset, we conducted two sets of experiments: (1) fixing the best response as the chosen one and varying the rejected response among the remaining four, and (2) fixing the worst response as the rejected one and varying the chosen response.\n\n#### **Open Assistant 2: fix the best as the chosen and vary the quality of the rejected.**\n\n| **Dataset** | **Configuration** | **GSM8K** | **LC-AE2** | **MMLU** | **IFEval** | **TruthfulQA** |\n|:------------|:------------------|:----------|:----------|:---------|:---------|:---------------|\n| Open Assistant 2 (Fixed Best) | Best/Worst | 78.4      | 20.9      | 62.8     | 72.3     | 48.4           |\n| Open Assistant 2 (Fixed Best) | Best/Low  | 78.6      | 19.2      | 62.6     | 72.7     | 47.4           |\n| Open Assistant 2 (Fixed Best) | Best/Medium | 79.3      | 19.3      | 62.8     | 71.4     | 49.1           |\n| Open Assistant 2 (Fixed Best) | Best/High | 78.4      | 19.6      | 62.7     | 72.1     | 47.5           | \n\nThe performance is actually quite similar across all five tasks, suggesting that the quality of the rejected response may not be a critical factor. This also implies that the chosen–rejected quality gap may not be fundamentally important. For instance, on GSM8K and TruthfulQA, using the medium-quality response as the rejected one yields the best performance. Similarly, on IFEval, using low-quality responses as the rejected ones leads to slightly better results.\n\n#### **Open Assistant 2: fix the worst as the rejected and vary the quality of the chosen.**\n\n| **Dataset** | **Configuration** | **GSM8K** | **LC-AE2** | **MMLU** | **IFEval** | **TruthfulQA** |\n|:------------|:------------------|:----------|:----------|:---------|:---------|:---------------|\n| Open Assistant 2 (Fixed Worst) | Low/Worst | 77.5      | 15.2      | 61.2     | 66.5     | 47.1           |\n| Open Assistant 2 (Fixed Worst) | Medium/Worst | 78.2      | 17.0      | 61.3     | 70.4     | 48.0           |\n| Open Assistant 2 (Fixed Worst) | High/Worst | 78.2      | 17.4      | 61.2     | 70.2     | 47.6           |\n| Open Assistant 2 (Fixed Worst) | Best/Worst | 78.4      | 20.9      | 62.8     | 72.3     | 48.4           |\n\nWhen the chosen response is fixed to be of the best quality, the model consistently achieves the highest performance across all tasks. This highlights the critical role that the quality of the chosen response plays in driving learning.\n\n#### **UltraFeedback: fix the best as the chosen and vary the quality of the rejected.**\n\n| **Dataset** | **Configuration** | **GSM8K** | **LC-AE2** | **MMLU** | **IFEval** | **TruthfulQA** |\n|:------------|:------------------|:----------|:----------|:---------|:---------|:---------------|\n| UltraFeedback (Fixed Best) | Best/Worst | 80.4      | 36.5      | 64.8     | 77.4     | 62.2           |\n| UltraFeedback (Fixed Best) | Best/Low  | 80.8      | 34.5      | 63.4     | 76.9     | 58.6           |\n| UltraFeedback (Fixed Best) | Best/Medium | 80.2      | 34.2      | 63.3     | 76.7     | 59.4           |\n| UltraFeedback (Fixed Best) | Best/High | 79.0      | 33.6      | 62.5     | 76.0     | 58.7          |\n\nSimilarly, the performance does not exhibit a monotonic trend as the quality of the rejected response increases or decreases. This confirms that the quality of the rejected sample alone may not be a reliable indicator of DPO performance.\n\n#### **UltraFeedback: fix the worst as the rejected response and vary the quality of the chosen response.**\n| **Dataset** | **Configuration** | **GSM8K** | **LC-AE2** | **MMLU** | **IFEval** | **TruthfulQA** |\n|:------------|:------------------|:----------|:----------|:---------|:---------|:---------------|\n| UltraFeedback (Fixed Worst) | Low/Worst | 79.3      | 25.8      | 61.4     | 76.5     | 56.1           |\n| UltraFeedback (Fixed Worst) | Medium/Worst | 78.6      | 26.7      | 62.1     | 75.8     | 58.0           |\n| UltraFeedback (Fixed Worst) | High/Worst | 79.5      | 30.9      | 63.7     | 77.0     | 61.3           |\n| UltraFeedback (Fixed Worst) | Best/Worst | 80.4      | 36.5      | 64.8     | 77.4     | 62.2           |\n\nThe set of experiments show that increasing the quality of the chosen response leads to a consistent, monotonic improvement in DPO training performance.\n\nHowever, when we fix the worst-quality response as the rejected one, increasing the quality of the chosen response not only improves the quality of the chosen sample itself but also enlarges the gap between the chosen and rejected responses. To further disentangle the effect of this quality gap from the absolute quality of the responses, we design an additional set of experiments. Specifically, we construct four new datasets—**LG-HQ, LG-LQ, SG-HQ, and SG-LQ**. The LG-HQ and LG-LQ datasets have similarly large chosen–rejected gaps but differ in the absolute quality of their responses. In contrast, SG-HQ and SG-LQ have similarly small gaps but also differ in response quality.\n\n| **Configuration** | **Avg.Chs** | **Avg.Diff** | **LC-AE2** | **Δ Score** | **MMLU** | **IFEval** | **GSM8K** |\n|:------------------|:------------|:-------------|:-----------|:------------|:---------|:---------|:----------|\n| LG-HQ (High Quality) | -2.98 | 3.54 | 33.0 | **+8.7** | 64.9 | 76.9 | 81.2 |\n| LG-LQ (Low Quality)  | -5.15 | 3.45 | 24.3 | | 61.9 | 73.8 | 80.5 |\n| SG-HQ (High Quality) | -3.56 | 1.34 | 28.4 | **+7.1** | 64.0 | 74.2 | 80.8 |\n| SG-LQ (Low Quality)  | -6.59 | 1.49 | 21.3 | | 62.6 | 72.3 | 78.0 |\n\n“Avg.Chs” and “Avg.Diff” refer to the average quality score of the chosen responses and the average quality gap between the chosen and rejected responses, respectively. The Δ Score is the performance difference between the HQ and the LQ on LC-AE2. Across all four tasks, we observe consistently better performance when the chosen responses are of higher quality. Moreover, the magnitude of improvement is comparable across both large-gap and small-gap settings (8.7 vs. 7.1, 3.0 vs. 1.4, 3.1 vs. 1.9, 0.7 vs. 1.2), demonstrating the generalizability of this observation regardless of the gap size.\n\nTo double check our findings on the non-essential effect of the quality gap, we construct two additional datasets. The first, **LG-HQ-inverse**, retains the high chosen response quality of LG-HQ but reduces the chosen–rejected gap. The second, **SG-HQ-inverse**, preserves the high chosen response quality of SG-HQ while increasing the gap.\n\n| **Configuration** | **Avg.Chs** | **Avg.Diff** | **LC-AE2** | **Δ Score** | **MMLU** | **IFEval** | **GSM8K** |\n|:------------------|:------------|:-------------|:-----------|:------------|:---------|:---------|:----------|\n| LG-HQ (Large Gap)       | -2.98 | 3.54 | 33.0 | **+1.4** | 64.9 | 76.9 | 81.2 |\n| LG-HQ-inv (Small Gap) | -2.98 | 1.92 | 31.6 | | 64.7 | 76.5 | 81.3 |\n| SG-HQ-inv (Large Gap) | -3.56 | 4.51 | 29.2 | **+0.8** | 64.2 | 75.1 | 80.5 |\n| SG-HQ (Small Gap)       | -3.56 | 1.34 | 28.4 | | 64.0 | 74.2 | 80.8 |\n\nExpanding the gap does lead to some performance improvement, but the gains are notably limited compared to those achieved by improving the quality of the responses. In fact, on GSM8K, we even observe a slight decrease in performance when the gap is increased.\n\nWe hope these additional results address your concerns.\n\n### **2. The numerical performance of SFT and online DPO**\n\nHere, we present the numerical results supporting Theorem 4.5, including evaluations on two datasets. We find that continual SFT and online DPO (with a fixed chosen response) exhibit similar performance across all five tasks. We sincerely thank you for this insightful comment, which has helped us improve the comprehensiveness of our experimental evaluation.\n\n| **Dataset** | **Configuration** | **LC-AE2** | **MMLU** | **IFEval** | **TruthfulQA** | **GSM8K** |\n|:------------|:------------------|:-----------|:---------|:---------|:---------------|:----------|\n| N/A | SFT Baseline | 12.7 | 62.1 | 74.3 | 46.8 | 76.8 |\n| Open Assistant 2 | Continual SFT | 18.7 | 60.4 | 71.5 | 46.9 | 78.7 |\n| Open Assistant 2 | Online-DPO | 19.0 | 60.6 | 71.8 | 47.5 | 78.6 |\n| UltraFeedback | Continual SFT | 35.8 | 61.6 | 74.1 | 57.1 | 79.5 |\n| UltraFeedback | Online-DPO | 37.6 | 62.0 | 74.5 | 58.0 | 79.7 |\n\n### **3. Terminologies**\nThank you for your great catches. We will carefully update them in our next version."}, {"review_id": "HfbzTONQXL", "reviewer": "Reviewer_fHiL", "summary": "This paper provides a theoretical analysis of how the distribution of preference data influences DPO training. The key insight is that the quality of the chosen response plays a central role, whereas the quality of the rejected response and the degree of contrastiveness between the two responses appear to be less critical. The empirical results presented in the paper support these theoretical findings.", "questions": "See Strengths and Weaknesses.", "limitations": "See Strengths and Weaknesses.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Major comments:**\n\nThe paper is theoretically solid and well-supported by empirical results. It offers a novel and insightful perspective on how preference data influences DPO training.\n\nThe theoretical part is well-written and well-structured. One minor suggestion is that it would be helpful to include a concrete example of of $\\pi_w$ and $\\pi_l$, say $\\pi_w(y\\mid x) = \\pi_{\\text{ref}}(y\\mid x) \\exp\\left(r(x, y)/\\beta\\right)$ and $\\pi_l = \\pi_{\\text{ref}} \\exp\\left(-r(x, y)/\\beta\\right)$, to help readers better understand how these distributions reflect response quality and help avoid potential confusion between $\\pi_w(y), \\pi_l(y)$ and $\\pi_{\\text{ref}}(y_w), \\pi_{\\text{ref}}(y_l)$. Besides, I’m not fully convinced by the example mentioned in line 198-204, which is used to support the claim that \"the quality of rejected responses may not always be critical.\" Since the likelihood ratio $\\pi_w(y)/\\pi_l(y)$ between best vs worst and best vs random is different, especially when the reward is extremely high.\n\nRegarding the empirical section, additional clarity is needed. It is not explicitly stated what the numbers in the tables represent. I assume they reflect expected rewards, but this should be made explicit. For GSM8K, metrics such as pass\\@1, pass\\@k, or major\\@k are more standard.\n\n**Minor comments:**\n\n* Line 208: One of the occurrences of $\\pi_w$ should likely be $\\pi_l$.\n* Line 216: The word \"importantconsistent\" should be split into two words.", "was_revised": true, "final_justification": "I had two major concerns about the paper. First, the empirical results lacked clarity, say, the metrics being reported were not clearly specified. Second, the example used to support the claim that \"the quality of the rejected responses may not always be critical\" was not appropriate. Both concerns have been addressed by the authors.\n\nI believe the theoretical contributions of the paper offer novel insights into DPO which could provide practical guidance on how to design effective training data for it. Since my main concerns have been resolved, I decide to increase my score.", "rebuttal": "### **1. More details on the empirical section**\n\nWe sincerely thank the reviewer for the great comments and suggestions. \nWe will make sure that in the revised version, we will include the exact metric of each benchmark used in our experiments. And the following is for your reference.\n\n| Benchmark | Core Metric | Setting / Details |\n| :--- | :--- | :--- |\n| LC-AE2 | Length-Controlled Win-Rate | Alpaca-Eval 2.0 version. 0-shot. |\n| MMLU | Accuracy | 5-shot setting. |\n| TruthfulQA | MC2 | 6-shot setting. |\n| IFEval | Instruction-Following Accuracy | 0-shot setting. |\n| GSM8K | Exact-Match Accuracy | 8-shot setting. |\n\n*Caption: Overview of Evaluation Benchmarks and Metrics. All the evaluations are run with only 1 attempt, i.e., under the pass@1 setting.*\n\n### **2. Concrete examples of $\\pi_w$ and $\\pi_l$**\nWe greatly appreciate your valuable comments and will follow your excellent example in the revised version. We also recently found similar ideas of your example in prior work [1], which we will cite and discuss in the updated manuscript.\n\n### **3. Example in line 198-204**\nWe sincerely apologize for any confusion our presentation may have caused. Our intention was to use the example in lines 198–204 as **numerical** support, rather than very rigor theoretical justification, for the claim that \"the quality of rejected responses may not always be critical.\" We fully agree that the likelihood ratios $\\pi_w(y)/\\pi_l(y)$ between best vs. worst and best vs. random can indeed differ. However, despite our efforts, deriving analytical results to characterize these differences seems to be challenging. We will highlight this as an important direction for future work.\n\n\n**Reference:**\n\n[1] Feng, Y., Kwiatkowski, A., Zheng, K., Kempe, J., & Duan, Y. (2025). Pilaf: Optimal human preference sampling for reward modeling. arXiv preprint arXiv:2502.04270."}, {"review_id": "fT3AOTX4UV", "reviewer": "Reviewer_RvM1", "summary": "This paper analyzes how the distribution of preference data influences DPO, from both theoretical and empirical perspectives. The authors find that the quality of the chosen responses plays a dominant role in optimizing the DPO objective, and that online DPO effectively reduces to SFT on the chosen responses.", "questions": "1. Lines 35–38 are duplicated.\n\n2. Line 208 should use $\\pi_l$ instead of $\\pi_w$.", "limitations": "See weakness", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\nThis paper investigates how the selection of *chosen* and *rejected* data in DPO affects performance, which is a very important question.\n\n**Weaknesses:**\n1. The paper points out that *“The quality of the rejected responses may not always be critical when $\\pi(y_w|x)$ is small.”* Could the authors analyze the actual magnitude of $\\pi(y_w|x)$ observed in their experiments? Is it always very small? After one epoch of training, could $\\pi(y_w|x)$ increase? If so, shouldn’t DPO be trained for more epochs, yet in practice people usually train for only one epoch.\n\n2. How should we interpret the fact that the chosen–rejected margin in DPO increases as training progresses? If it is equivalent to SFT, then the model should converge to the SFT solution.\n\n3. Under Theorem 4.5, DPO is essentially doing SFT, so training the model for a long time should not cause collapse, but in contrast, online DPO can easily overfit and produce long repetitive outputs.\n\n4. Table 1 should include SFT results on the *chosen* responses to support the paper’s core claim: *“the quality of chosen responses plays a dominant role in optimizing the DPO objective and online DPO effectively reduces to SFT on the chosen responses.”*\nIn Table 2, the LG-HQ and SG-HQ results show that larger gap sizes are better; similarly, LG-LQ and SG-LQ also show that larger gaps are better. This contradicts the authors’ claim that *“DPO optimization primarily operates through quality anchoring rather than margin maximization.”*", "was_revised": true, "final_justification": "Issues Resolved:\n1. Whether pi is small\n2. The increase of the chosen-rejected margin\n3. The collapse of online DPO\n4. Numerical results\n\nIssues Not Resolved:\n\n* On the Usefulness and Implications of the Specific Online DPO Variant. The authors argue that fixing the chosen response while resampling only the rejected response provides a clean setting to disentangle effects. However, my concern is that this setup is too narrow: it only shows a special case rather than giving insights into general online DPO. Thus, the claimed “non-trivial insight” mostly reiterates that chosen responses matter, without extending understanding to broader DPO mechanisms.\n\n* Online DPO vs. Offline DPO vs. SFT. The authors try to reconcile the seeming contradiction by carefully distinguishing datasets and training stages. But this sidesteps my original concern: if offline DPO consistently outperforms SFT, and their variant of online DPO reduces to SFT, then offline DPO would likely outperform their online DPO variant. Their clarification doesn’t resolve the tension, it just reframes the comparison.\n\n* On the Intuitiveness of the Conclusions. The authors defend their conclusion as surprising, but from my perspective, the claim that “chosen response quality matters more than rejected response quality or contrastiveness” feels intuitive. What’s missing is a quantitative measure of how much more. Without that, the contribution seems qualitative and unsurprising.\n\nIn summary, I am inclined to give a negative score, but since two other reviewers expressed positive opinions and I have not fully reviewed the proof process, I will lower my confidence.", "rebuttal": "### **1. Whether $\\pi(y_w|x)$ is small**\n\nWe apologize if we misunderstood your original comment and have done our best to respond based on our current understanding. We had difficulty locating the exact sentence “The quality of the rejected responses may not always be critical when $\\pi(y_w|x)$ is small.” The closest related statement we could identify appears around line 190, where we discuss the roles of the distributions $\\pi_w(y|x)$ and $\\pi_l(y|x)$, rather than the specific probability $\\pi(y_w|x)$ of generating a particular response $y_w$.\nOur intended point was the following: if two rejected response distributions, $\\pi'_l(y|x)$ and $\\pi''_l(y|x)$, differ only in regions where $\\pi_w(y|x)$ is small, then the corresponding likelihood ratios $\\pi_w(y|x)/\\pi'_l(y|x)$ and $\\pi_w(y|x)/\\pi''_l(y|x)$ may still be similar. As a result, the model may exhibit similar behavior or performance under both distributions, despite the differences in rejected samples.\n\nThat said, following your suggestion, we carefully reviewed all the experiments we conducted. In most of our DPO runs, the probability assigned to the chosen response either fluctuates or decreases over training, which is consistent with observations reported in recent studies (e.g., [1], [2]). Therefore, we do not necessarily expect $\\pi(y_w|x)$ to increase during training.\n\nFurthermore, determining the optimal number of training epochs for DPO is nontrivial. As far as we are aware, the number of epochs varies widely across implementations. In our experiments, we find that training for two epochs generally yields better performance than just one. Additionally, widely-used methods such as classic online DPO and DPO with curriculum learning typically involve multiple epochs of training. We believe this variability is closely tied to both the characteristics of the data and the compatibility of the base model.\n\n### **2. The increase of the chosen-rejected margin**\n\nWe fully agree that the chosen–rejected margin tends to increase as DPO training progresses. However, we do not think that this observation contradicts the fact that DPO can converge to the SFT solution under certain conditions. Convergence to the SFT solution inherently involves increasing the logits (or likelihood) of the positive samples. This, in turn, implicitly reduces the relative likelihood of responses not used for instruction, thereby increasing the chosen–rejected margin as well. Furthermore, in our numerical experiments under the online DPO setting, we also observe that the probability of generating the chosen response increases over training, which closely resembles the behavior typically associated with SFT.\n\n### **3. The collapse of online DPO**\n\nWe fully agree with the reviewer that online DPO can be prone to collapse. We would like to respond from two perspectives:\n\nFirst, the form of online DPO we study in Theorem 4.5 is a special case in which the chosen responses are of very high quality (e.g., written by human experts) and remain fixed throughout training, while only the rejected responses are updated and generated by the model. In contrast, standard online DPO typically generates both the chosen and rejected responses from the model itself, making it more likely to overfit and collapse. In our case, since the chosen responses are externally sourced and of consistently high quality, the risk of collapse is mitigated. We acknowledge that this distinction makes our setting narrower in scope, and we agree that your concern is highly relevant to the standard formulation of online DPO.\n\nSecond, we would like to point out that similar instability can also occur in SFT. There are numerous community reports indicating that SFT checkpoints sometimes perform worse than their pre-trained counterparts, even producing nonsensical or repetitive outputs. This issue has been observed in widely used models such as LLaMA and Gemma, including their most recent versions. These observations highlight that SFT, too, is vulnerable to overfitting and collapse under certain conditions.\n\n### **4. Numerical results**\n\nWe sincerely appreciate your great comments on the numerical results.  First, we present the results of SFT on the chosen response, and compare them with the online DPO we studied in Theorem 4.5.\n\n| **Dataset** | **Configuration** | **LC-AE2** | **MMLU** | **IFEval** | **TruthfulQA** | **GSM8K** |\n|:------------|:------------------|:-----------|:---------|:---------|:---------------|:----------|\n| N/A | SFT Baseline | 12.7 | 62.1 | 74.3 | 46.8 | 76.8 |\n| Open Assistant 2 | Continual SFT | 18.7 | 60.4 | 71.5 | 46.9 | 78.7 |\n| Open Assistant 2 | Online-DPO | 19.0 | 60.6 | 71.8 | 47.5 | 78.6 |\n| UltraFeedback | Continual SFT | 35.8 | 61.6 | 74.1 | 57.1 | 79.5 |\n| UltraFeedback | Online-DPO | 37.6 | 62.0 | 74.5 | 58.0 | 79.7 |\n\nThe table demonstrates that our online DPO setup yields performance comparable to that of SFT, consistent with the prediction of Theorem 4.5. Furthermore, by comparing the SFT results with those in the original Table 1, we observe that incorporating rejected responses does provide some performance benefit, though the improvement is relatively modest. We hope these additional results help address your concerns.\n\nAlso, we agree with your observation that, based on Table 2, comparing LG-HQ with SG-HQ (and similarly, LG-LQ with SG-LQ) may suggest that larger gap sizes lead to better performance. However, we would like to highlight that when comparing LG-HQ and SG-HQ, the difference is not solely in the gap size—the quality of the chosen responses in LG-HQ is also higher than that in SG-HQ.\n\nTo clarify this point, we present the revised Table 2 below. In the table, “Avg.Chs” and “Avg.Diff” denote the average quality score of the chosen responses and the average quality gap between the chosen and rejected responses, respectively. Δ Score represents the performance difference between two rows on the LC-AE2 benchmark.\n\n- **Part 1** of the table illustrates the comparisons we consider most informative—those that hold the gap approximately constant while varying the quality of the chosen response.\n\n- **Part 2** corresponds to the comparisons you suggested, where both the gap and the chosen response quality differ.\n\nFor example, the Avg.Chs values for LG-HQ and SG-HQ are –2.98 and –3.56, respectively, showing a notable difference in chosen response quality. This indicates that the observed performance differences may stem from both the chosen response quality and the gap size. As a result, we believe it is not safe to attribute the performance gains solely to the gap size based on LG-HQ vs. SG-HQ or LG-LQ vs. SG-LQ.\n\nTo further disentangle these effects, we constructed two additional datasets:\n\n- **LG-HQ-inverse**, which retains the high chosen response quality of LG-HQ but reduces the chosen–rejected gap.\n\n- **SG-HQ-inverse**, which maintains the chosen response quality of SG-HQ while increasing the gap.\n\nAs shown in Part 3 of the table, increasing the gap alone does not significantly affect DPO performance, thereby supporting our claim that the quality of the chosen response plays a more dominant role.\n\nWe hope these additional numerical results help clarify our conclusions and address your concerns.\n\n| **Configuration** | **Avg.Chs** | **Avg.Diff** | **LC-AE2** | **Δ Score** | **MMLU** | **IFEval** | **GSM8K** |\n|:------------------|:------------|:-------------|:-----------|:------------|:---------|:---------|:----------|\n| **Part 1: (Gap size is held constant)** ||||||| |\n| LG-HQ (High Quality) | -2.98 | 3.54 | 33.0 | **+8.7** | 64.9 | 76.9 | 81.2 |\n| LG-LQ (Low Quality)  | -5.15 | 3.45 | 24.3 |           | 61.9 | 73.8 | 80.5 |\n| LG-HQ (High Quality) | -3.56 | 1.34 | 28.4 | **+7.1** | 64.0 | 74.2 | 80.8 |\n| SG-LQ (Low Quality)  | -6.59 | 1.49 | 21.3 |           | 62.6 | 72.3 | 78.0 |\n| **Part 2:** ||||||| |\n| LG-HQ (Large Gap)    | -2.98 | 3.54 | 33.0 | **+4.6** | 64.9 | 76.9 | 81.2 |\n| SG-HQ (Small Gap)    | -3.56 | 1.34 | 28.4 |           | 64.0 | 74.2 | 80.8 |\n| LG-LQ (Large Gap)    | -5.15 | 3.45 | 24.3 | **+3.0** | 61.9 | 73.8 | 80.5 |\n| SG-LQ (Small Gap)    | -6.59 | 1.49 | 21.3 |           | 62.6 | 72.3 | 78.0 |\n| **Part 3: Effect of Gap (Chosen quality is identical)** ||||||| |\n| LG-HQ (Large Gap)        | -2.98 | 3.54 | 33.0 | **+1.4** | 64.9 | 76.9 | 81.2 |\n| LG-HQ-inv (Small Gap)    | -2.98 | 1.92 | 31.6 |           | 64.7 | 76.5 | 81.3 |\n| SG-HQ-inv (Large Gap)    | -3.56 | 4.51 | 29.2 | **+0.8** | 64.2 | 75.1 | 80.5 |\n| SG-HQ (Small Gap)        | -3.56 | 1.34 | 28.4 |           | 64.0 | 74.2 | 80.8 |\n\n### **Reference:**\n[1] Razin, N., Malladi, S., Bhaskar, A., Chen, D., Arora, S., & Hanin, B. (2025). Unintentional unalignment: Likelihood displacement in direct preference optimization. ICLR 2025\n\n[2] Ren, Y., & Sutherland, D. J. (2025). Learning dynamics of llm finetuning. ICLR2025"}]

Author rebuttal:

Meta-review: This paper investigates a fundamental question in alignment: which properties of preference data most influence DPO performance. The authors provide both theoretical analysis and comprehensive experiments. The central finding is that the absolute quality of the chosen responses dominates DPO outcomes, while the rejected responses and preference gap play only a secondary role once minimal contrastiveness is satisfied. The paper further shows that in an online variant with fixed chosen responses, DPO reduces effectively to SFT with mild regularization, helping interpret why online DPO often works.

Strengths: 
- A timely and important contribution given the growing adoption of DPO.
- Careful theoretical development, including Theorem 4.5, that connects DPO to SFT.
- Extensive empirical work, including additional experiments suggested during review (granular chosen/rejected quality analysis, disentangling gap vs quality effects, and online DPO vs SFT comparisons). These strengthen the robustness of the conclusions.
- Clear practical takeaways for dataset construction in alignment pipelines.

Weaknesses:
- The main insight, that chosen response quality matters most, is consistent with intuition and some prior practice. The novelty lies more in its rigorous articulation and empirical confirmation rather than a surprising or methodologically new contribution.
- The scope is limited to DPO and a particular online DPO variant; generalization to other preference-learning methods (e.g., IPO, ORPO, SimPO) remains speculative.
- Some reviewers felt the contribution was narrower than initially expected and that quantitative measures of relative importance could be stronger.

The authors responded constructively to reviewer concerns, adding substantial new experiments that directly addressed calls for more granular analysis and for empirical support of the theoretical claims. Two reviewers increased their scores to accept after these revisions. Another reviewer remained cautious, viewing the conclusions as intuitive and narrow in scope, but did not strongly contest the technical soundness.

Overall, the paper makes a clear, well-executed, and practically useful contribution to understanding DPO, even if the conceptual novelty is moderate. The theoretical clarity and expanded experimental support justify merits. The work will be of interest to practitioners building alignment datasets and to researchers studying preference-based learning.
Decision: Accept (poster)