3D-Properties: Identifying Challenges in DPO and Charting a Path Forward
Authors: Yuzi Yan
Yibo Miao
Jialian Li
YipinZhang
Jian Xie
Zhijie Deng
Dong Yan
Venue: ICLR 2025 Poster (2025)
Keywords: LLM
DPO
RLHF
Primary area: foundation or frontier models, including LLMs
TL;DR: understanding the degradation of reward-free alignment in LLMs

Abstract: Aligning large language models (LLMs) with human preferences has gained significant attention, with Proximal Policy Optimization (PPO) as a standard yet computationally expensive method and Direct Preference Optimization (DPO) as a more efficient alternative. While DPO offers simplicity, it remains underutilized in state-of-the-art LLMs, suggesting potential limitations. In this work, we revisit DPO, analyzing its theoretical foundations and empirical performance to bridge this gap. We identify three key properties—termed \textbf{3D}-properties—that emerge from DPO’s learning process: \textbf{D}rastic drop in rejected response likelihood, \textbf{D}egradation into response suppression, and \textbf{D}ispersion effect on unseen responses. We show that these issues arise from DPO’s optimization dynamics, where the interaction between chosen and rejected response gradients leads to instability. Our findings are supported by experiments on both a controlled toy model and real-world LLM tasks, including mathematical problem-solving and instruction following. To address these challenges, we propose simple regularization techniques that improve training stability and performance. Additionally, we examine how preference data distribution impacts DPO’s effectiveness, offering insights into how alignment models handle out-of-domain (OOD) data. Our work connects these observations to broader research and provides a theoretical explanation for DPO’s limitations. We hope these insights will guide future advancements in reward-model-free preference learning, bringing it closer to reward-model-based approaches.

Published as a conference paper at ICLR 2025
3D 3D-PROPERTIES: IDENTIFYING CHALLENGES IN DPO AND CHARTING A PATH FORWARD
Yuzi Yan 1.3: Yibo Miao Jialian Liº, Yipin Zhang", Jian Xie Zhijie Deng": Dong Yan "Department of Electronic Engineering, Tsinghua University
Shanghai Jiao Tong University Baichuan AI
van-yz17etsinghie org. (miacyibo, zhijied] jtj.u.ed. 1ijialian78166 com,
com, 2ypeyp665@gmal1 com, xiejian1990 9900gmailllllmm w.l.l..oooo .cm, peroobleem@gma.ee com
ABSTRACT
Aligning large language models (LLMs) with human preferences has gained sig- nificant attention, with Proximal Policy Optimization (PPO) as standard yet computationally expensive method and Direct Preference Optimization (DPO) as a more efficient alternative. While DPO offers simplicity, remains underuti- lized in state-of-the-art LLMs, suggesting potential limitations. In this work, we revisit DPO, analyzing its theoretical foundations and empirical performance to bridge this gap. We identify three key properties- termed 3D-properties--n that emerge from DPO's learning process: Drastic drop in rejected response likeli- hood, Degradation into response suppression, and Dispersion effeet on unseen responses. We show that these issues arise from DPO's optimization dynamics, where the interaction between chosen and rejected response gradients leads instability. Our findings are supported by experiments on both a controlled toy model and real-world LLM tasks, including mathematical problem-solving and instruction following. To address these challenges, we propose simple regular- ization techniques that improve training stability and performance. Additionally. we examine how preference data distribution impacts DPO's i effectiveness, of- fering insights into how alignment models handle out-of-domain (OOD) data. Our work connects these observations to broader research and provides theo- retical explanation for DPO's limitations. We hope these insights will guide fu- ture advancements in reward-model-free preference learning, bringing it closer reward-model- approaches.
INTRODUCTION
Large language models (LLMs) have demonstrated exceptional performance across wide range of tasks and domains Toovron et al. 2023) Chowdhery et al., 2023; Jiang et al. 2023 Zhang et al.) Several techniques have been developed for fine-tuning LLMs, most notably Supervised Fine- ToninT (SFT) and Reinforcement Learning from Human Feedback (RLHF) (Achiam et al.
2022).
2023
Touvron et al. 2023). SFT involves directly training LLMs on labeled data to tailor their responses for specific tasks, whereas RLHF refines L.MMs by incorporating feedback that aligns their outputs with human preferences. RLHF, in particular, has been instrumental in expanding the application of both closed-source (OpenAll 2022 Anthropic 2024 Team et al. (2033 and open-
source LLMs (Touvron al., 2023 Yang et | 2023). driven by the need to align foundational
models with human values and preferences (Ziegler et al. 2019 Stiennon et al. 2020: Ouyang
al. 2022).
Existing RLHF methods can be majorly categorized into two classes based on whether the re- ward signal explicitly modeled. Reward-model--aaved (RM-based) alignment pioneered by Ope- nAI (Ouyang et al.) 2022; Achiam et al.) 2023) Touvron et al., 2023) first trains Reward Model (RM) from user preferences, typically through Maximum Likelihood Estimation (MLE), and then
$ Equual contribution.
This work done during internship at Baichuan AL.
" *Corresponding authors.

---

Published as a conference paper at ICLR 2025
leverages actor-critic algorithms such Proximal Policy Optimization (PPO) (Schulman et al. 2017) to tune the SFT model to realize aligmment. This approach often requires substantial compu- tational resourees and suffers from sample inefficiency (Choshen et al. 2019.. Conversely, another class of methods, known as reward-model-free (RM-free) alignment, such as Direct Preference Op- timization (DPO) (Rafailov al., (024), Identity Preference Optimization (IPO) (Azar al. 2024). Sequence Likelihood Calibration (SLiC) (Zhao et al. 2023). DPO-positive (Pal et al., (024) and Simple Preference Optimization (SimPO) (Meng al. 2024), do not rely on an extra RM. These approaches offer a more resource-efficient alternatie by optimizing the policy directly from pref- erences, therefore attracting much attention from the academic community. where computational resources are often limited.
In this work, we begin our analysis by using the vanilla DPO as case study, subsequently extending our findings to encompass broader R.M-free alignment strategies. Despite its simplicity and promise, DPO has exhibited several perplexing phenomena that remain unclear or underexplained practice. One notable counter-intuitive observation is that the likelihood of both preferred and rejected re- sponses tends to decrease over the course of DPO training YYuan et al. 2024 Mitchell (023), while the likelihood of certain tokens diverging from the training data increases (Xu et al. (024a). Addi- tional observations are summarized in Section 2.1] Without a deeper theoretical exploration of these phenomena, purely empirical efforts apply improve DPO likely face inefficiencies.
Our work identifies the issues surrounding vanilla DPO and its variants from both theoretical and practical perspectives. The analysis reveals inherent instability in the DPO training process, which we encapsulate as the 3D-properties: Drastic drop in the likelihood of rejected responses, Degradation into response suppression, and Dispersion effect on unseen responses. Through our analytical framework, we show that these phenomena stem from the inherent features of DPO's op- timization objective, where the interaction between the gradients of chosen and rejected responses leads to instability and hinders overall performance. Furthermore, our findings confirm that the distribution of preference data critically influences DPO's effectiveness, with on-policy DPO per- forming better than off-policy DPO, which is consistent with concurrent empirical studies (Tang) ctt al. 2024 Guo et al. 2024).
To enhance DPO's stability and performance, propose several regularization methods, including the adaptive adjustment of weights on the gradients of chosen and rejected responses, as well as incorporating an SFT loss into the objective. Our results suggest a fundamental trade-off within the DPO algorithm: balancing the mitigation of the 3D Dr-prpperties while preventing LLMs from straying too far from the preference learning paradigm. Additionally, we compare DPO with the state-of-the- art RM-based method, RLHF-PPO, revealing that its superiority stem largely from avoiding the 3D- propprtessssss Our experimental approach begins with the design of toy model to quickly validate our hypotheses, followed by a rigorous test of the actual performance of real LLMs on tasks such as mathematical problem solving and instruction following.
As this topic has garnered significant attention recently, an increasing number of works are con- tributing the discussion. To highlight the contributions of our approach, we compare our findings with several of the most relevant concurrent studies in Section 2.2 A comprehensive review of related works is provided in Appendix A
PRELIMINARIES
Large Language Model (LLM). An LLM defines a 6-parameterized conditional distribution To(a(æ), which takes a prompt v as input and produces response a. More specifically, the sampling from LLMS performed in an auto-regressive manner, Ty (a(rr = II no (a. aa, (1:-1), where ae is the t-th token in the response and 01:1- are tokens in the response before a.
RM-based RLHE. Training LLMs typically involves three stages: Pretraining, SFT, and RLHF. We outline the standard PPO paradigm here, which is typical RM-based RLHF algorithm. Be- ginning with a well-trained SFT model, denoted as To, we proceed by sampling two responses from no for each instance in given prompt set. Subsequently, we compile preferece dataset D = {(x, at a }}. where at and a denote numan--referre and numan-dispreferred comple- tions, respectively. The distribution of the preference dataset is assumed follow the Bradley-Terry
2

---

Published as a conference paper at ICLR 2025
model (Bradley & Terry 1952) i.e., the probability of response a is better than a is given by:
exp(r(z,a*))
pa(at a a1)=)=----->))>>)))) exp(r(z,att) +xp((((xxx))) a ar((, r((,)))))
(I)
where represents the preference relation, and (() - Ite- is the sigmoid function. To train
RM r(.)), ) we maximize the log- likelihood of the observed preferences by minimizing the following loss function:
(n(r) =
Σ (
lo x a=
Σ
log logo(r(z, - r(x,a
(2)
(2,0 1,a
During the reinforcement learning phase, update the LLM to maximize the return from the learned RM using the following objective function:
no mo(())) max (0) max Σ Fanno (-Iw) (n(oza)) - Blog
To (o(a(a)
where mo is initialized as To and B3 controls the deviation from the original model. PPO (Schulman al.) 2017) is typically used to solve the problem in practice. Algorithms that optimize the policy using a separate RM are referred to as RM-based alignment.
DPO. Instead of learning a separate RM, DPO (Rafailov et al. 2024) directly optimizes the policy Ty over preference data. DPO implicitly leverages particular choice of RM parameterization that enables the extraction of its optimal policy in closed form, without reinforcement learning training loop:
DPO (0) =
Σ (z.at (-))
logo
Blog To [z) To [a)
3 log
TT
xz)
(4)
To To
As shown, DPO leverages logistic regression loss directly fine-tune the LLM preference data. This approach, along with its various variants (Zhao et al. 2023) Amini et al. 2024 Azar et al. 2024). referred to as RM-free alignment due to the elimination of an explicit RM.
2.1
UNDEREXPLORED OBSERVATIONS ABOUT DPO
Though the absence of the need for additional RM training makes DPO particularly attractive, sev- eral observations remain underexplored. The most concerning issue is, to the best our knowledge, few models using DPO (or other RM-free algorithm) have achieved performance comparable to the statc-of-the theth---- closed-source LLMs such as OpenAl's GPT-4o or Anthropic's ' Claude, which report- edly use PPO methods during training. Besides, many other phenomena have been reported but lack comprehensive theoretical explanations. Here we make a summary for clarity.
Observation During the vanilla DPO training, the likelihood of both the chosen and rejected responses in the preference datasets tends decrease, whereas the likelihood of unseen tokens not appearing in the preference pairs tends to increase (Mitchell 2023).
Observation Compared with RM-based alignment, the performance of DPO relatively unsta- ble and sub-optimal (Wang et al. 2024).
Observation 3. The performance DPO is significantly affected by the distribution shift between the model outputs and the preference dataset. In general, on-policy DPO, where both the chosen responses and the rejected responses are sampled from the policy model тæ. outperforms other scenarios (Tang al. 2024).
2.2 COMPARISON WITH RELATED CONTEMPORARY STUDIES
Several concurrent studies attempt explain these observations, and highlight our differences to underscore our contributions. For Observation Feng et al. (2024) shares similar points of view regarding the degradation gradients but offers limited analysis and lacks experimental validation on real L.Mss. In contrast, our work offers more rigorous and comprehensive analysis supported by theoretical insights and validation across toy models and large-scale real-world L.LM experiments.
3

---

Published as a conference paper at ICLR 2025
For Observation 2 Xu et al. (024a) point out that the policy minimizing the PPO loss is subset of that minimizing the DPO loss, which offers a partial explanation. However, their analysis focuses solely the endpoint of the optimization and does not examine the dynamic process by which the policy evolves. It leaves unaddressed how unexpected policies emerge during training. In contrast, our gradient analysis offers comprehensive understanding of the entire optimization trajectory, shedding light on how and why sub-optimal policies arise throughout the DPO training process. For Observation Tang al. (2024) investigate the performance gap between on-policy and off-policy alignment algorithms from an empirical perspective, while our insights are rooted in theoretical findings.
Our work advances the understanding of these observations, providing critical insights into the un- derlying mechanisms and reinforeing the findings of these concurrent studies. In the following section, we will present our theoretical explanations for these observations.
3 FUNDAMENTAL LIMITATIONS OF VANILLA DPO: 3D-PROPERTIES
We first identify a critical flaw inherent in vanilla DPO. At first glance, the loss function vanilla DPO, as defined Eq. 44. appears be composed of two parts: the term log
ma(a" (m) To ar) aims to
mo(a-1z) seeks to decrease the like- lihood of rejected responses. However, this seemingly straightforward interpretation overlooks sig- nificant underlying issues, which we characterize through the 3D-properties of vanilla DPO.
increase the likelihood of chosen responses, while the term log Te(a_(z)
Property (Drastic drop in rejected response likelihood). The likelihood of a rejected response tends to shift much more rapidly than that of a chosen response.
Property (Degradation into response suppression). As optimization progresses, DPO gradually loses its ability to steer the direction of optimizing chosen responses and instead devolves into merely suppressing the rejected responses.
Property 3 (Dispersion effect unseen responses). As DPO training progresses, the likelihood of both chosen and rejected responses gradually decreases, while the likelihood of generating out-of- distribution (OOD) responses increases.
These properties are non-trivial, as they reveal inherent challenges in DPO's optimization process that not immediately apparent from the loss function alone. These phenomena closely align with the empirical observations we discussed earlier, pointing to the structural limitations of DPO. In the following section, we delve into theoretical analysis to further explain the origins of these 3D- proppepesss and provide insights into their impact on the optimization trajectory.
3.1 THEORETICAL FOUNDATIONS
In this sections, we provide the theoretical foundations for the 3D-properties, followed by detailed explanations of the observations discussed in Section 2.1 The loss function for DPO as shown in
Eq. ( can be re- e-rritten by:
Ao(a" (x))
te [x:) 18 Te (x))
(0)
Σ log 1+
(5)
To(a (x)
(r,at,a-)
For given triple (2r, attaata) a let
To(a (xx) To(a lx))
To(a (x)
- = To(a+1x), क. = TT(aa
a =
Ta + (r)
π
Then we have
DEPRO
a Olog 1 + a2 dz
a
Oz
a8
8-1
Om
Oz
Ox
1 + a
Oh
1 +
o
(
DPP
logg( + az
Oz
a
dz
a
dz
14 +
oz8
1 az
with these simplified forms, we can obtain the following corollaries explain the properties above:
4

---

Published as a conference paper at ICLR 2025
Optimal Policy
Preference Dataset
10
10
05
04
0.2
02
Pesppnggo
chosen
rejected
other
paired preference data
Output Probabilities at Epoch
Output Probabilities at Epoch 500
032 0.22
032 0.32
0.01
0.30
0-
0.00
0.00
0:12 632
0.00
0:65
0.00
0.00
0.12 612 612 012 012 612 012 0.22
0.03
0.01
0.01
0:19
0:72
0.00
0:00 0.00 0.00 0.00
0.00
0.00
0.04
032 832 032 032
032
0.32
0.00
3-
0.35
0.00
0.00
0.00
Respornay
Paisposs y
Figure 1: Toy model setup. Top left: the optimal policy where the highlighted blocks represent optimal responses. Top right: preference dataset construction. Lower left: the initialization of the SFT model. Lower right: policy output after DPO training.
Corollary 1 (Explanation for Property [.. The ratio of the gradient with respect to the rejected response likelihood x to the gradient with respect to the chosen response likelihood n is equal to the ratio of n² to
DP
aeDRO
0z Ôग
0z dR
which indicates that as - increases and π decreases, the gradient with respect to π grows faster, leading to rapid decline in the likelihood of the rejected response.
Corollary (Explanation for Property
As π
+ O, we have z - and T+a2" + a,3. Thus,
aEPPO
@EDRO
Ont + -0;3( -
(T O,
(x ()++((=-((()) ) + oo;
given that B < and -- + O. This creates dynamic where the gradient for the rejected response grows exceedingly large, while the gradient for the chosen response diminishes significantly. As result, DPO progressively shifts its focus to suppressing the rejected responses and loses the ability to steer the direction of optimizing chosen responses.
Corollary (Explanation for Property 3. When π drastically drops to 0, the gradient on π fails and the likelihood of the chosen response is likely to decrease along with the rejected response as they often share many similar tokens and patterns. The constancy of the sum of probabilities implies that both æ and decrease, the likelihood will randomly disperse into other unseen responses out of the preference dataset.
Based on these theoretical insights, we can further explore and explain the observations discussed in Section 2.1 Observation is directly explained by Corollary 3 For Observation 2 we will show that 3D-properties do not manifest during the RM training process the RM-based alignment pipeline. Regarding Observation we will demonstrate that the distribution gap between the LLM's original outputs and the preference dataset plays crucial role in determining the influence of the 3D -r-properties. The impact of these properties is notably less pronounced in on-policy DPO, where the preference dataset is sampled directly from the policy model's outputs. In the following sections, will delve deeper into each of these statements and provide empirical validation.
3.2 SYNTHETIC VALIDATION WITH A TOY MODEL
In this section, introduce simplified toy model specifically designed to facilitate synthetic experiments, thereby enhancing the persuasiveness of our arguments from Corollary to Then conduct experiments on real L.LMs.
5

---

Published as a conference paper at ICLR 2025
Lide Change
Grodfort
Litethood crange flo asis)
30%
Average Choner Liae
Gredient
Jereage Creem sing
Maximam Teject Liadheed
Auerage inown kndioed
18
Muxmum Beject hand (ogt
Urseon Dogl
Epoch
Spoch
Epoch
Figure 2: Dynamic optimization process with vanilla DPO using the toy model. Left: likelihood dynamics over training epochs. The blue curve represents the average likelihood of chosen re- sponses, yellow shows the minimum for chosen responses, green represents the average for rejected responses, red shows the maximum for rejected responses, and purple represents the average for
unseen responses. Middle: dynamics of averaged agPro likelihood dynamics over training epochs on a log scale, highlighting the drastic drop in the likeli- hood of rejected responses.
0 and agorō dr. over training epochs. Right:
3.2. TOY MODEL SETUP
The diagram for the toy model is illustrated in Figure I We construct a discrete space consisting of 4 prompts and 10 responses. The policy Te, which simulates simplified version of an LLM, is imple- mented as a three-layer MLP that processes a one-hot vector and outputs a categorical distribution over the responses. The response space is organized such that the first dimensions correspond to chosen responses, dimensions 5 through 8 represent rejected responses, and the final 2 dimensions correspond to unseen responses not present in the preference dataset.
In this setup, each prompt has optimal response (e.g., response is optimal for prompt 1, as shown in the upper left figure). When constructing the preference dataset for DPO training, we adopt mini-batch sampling strategy to mimic real-world annotation processes. Specifically, assuming an ideal annotator, each input prompt perfectly matched with its optimal response- response-correspone corresponding to the diagonal elements of the matrix, illustrated in the upper right figure in Figure []] For each mini-batch, we then randomly select one other response within the batch to create preference data pairs. This approach ensures that gradient updates are computed from diverse mini-batch samples.
To simulate the Pretraining and SFT process, we manually assign output probabilities and use them as labels train Tg. Initially, as shown in the lower left figure, we the likelihood of both chosen and rejected responses at 0.12, treating both as on-policy. The constructed preference dataset is then used for DPO training, with the output after 500 epochs shown in the lower right. The code provided in the supplementary material.
3.2.2 RESULTS
Figure 2 illustrates the dynamic optimization process during DPO training. In the first figure, the likelihood of chosen responses (blue and yellow curves) increases, while the likelihood of rejected responses (green and red curves) decreases in the early phases of training. However, as training progresses, the likelihood of chosen responses begins decline in the longer run. During this degradation phase, as both chosen and rejected response likelihoods decrease, the probability is redistributed to unseen responses (purple curve).
The second and third figures reveal the underlying causes of this shift: as Te(a [z) approaches zero, the absolute value of /EDPO /a= increases sharply compared aeDoo The absolute value of aeDro /0 becomes progressively smaller, weakening its influence the optimization direction. These results align with the earlier theoretical analysis.
Moreover, this insight provides an explanation for the observed superiority of on-policy DPO (Ob- servation 3. contrast to off-policy DPO, on-policy DPO begins with a higher likelihood for rejected responses, thereby extending the duration before their likelihood significantly diminishes. To further validate the differing impacts of on-policy and off ooflpo DPO, we configure four scenar- ios by adjusting the initial distribution of outputs to simulate these conditions. A higher initialized likelihood (0.12) simulates responses sampled in an on-policy manner, while lower one (0.02) sim- ulates responses sampled off-policy. The initial state and the subsequent changes in the likelihood

---

Published as a conference paper at ICLR 2025
beak
Beck
Libe Sove
Joyt onl Changy
U Chr gt
Lep Change
will cinin Loyen buclhoo
Lin
Fairan Agpec Use
Figure 3: From left to right, the figures show the initial state and the likelihood dynamics for cho- sen/rtjected/unseen responses in Scenarios I to 4, similar to the left diagram in Figure 2 (1) both chosen and rejected responses are on-policy, (2) chosen off-policy and rejected on-policy, (3) chosen on-policy and rejected off- offlppllcy and (4) both off-policy.
of each response are illustrated in Figure 3 Notably, in Scenario 1, where both chosen and rejected responses are on-policy, the 3D proppettsss are relatively mild, as shown by the high peak probability of the optimal response (approximately 0.6) and the minimal dispersion effect on unseen responses. Additional detailed results and analyses for all four scenarios can be found in Appendix D
The intention of the toy model and its connection to real LLMs. The toy model serves as an abstract simulation that amplifies the effect of 3Dpprrpeesses, which are less pronounced and harder visualize in real-world experiments. While the toy model differs from real LLM training in several ways-such as sampling frequency-its design offers useful insights. In real-world settings, DPO typically trained over one epoch, with each data point used only a few times. In contrast, in the toy model, the same data points are sampled repeatedly. Conceptually, this is similar to treating each input/output as a token rather than complete prompt/response, where each token may be sampled multiple times during real-world training. Since both the chosen and rejected responses are generated from the same prompt, they often share common tokens. As a result, the decrease in the likelihood of rejected responses can impact the likelihood of chosen responses, leading to corresponding decline in their likelihood.
3.3 REGULARIZATION TECHNIQUES
It becomes evident that the rate at which ma (a |x) declines is crucial in determining the severity of the 3D-properties' impact. This observation leads to the following proposition:
Proposition 1. To lessen the severity of the 3D-properties, it is advantageous to moderate the rate which the likelihood of rejected responses declines.
introduce two straightforward regularization techniques. The first technique employs adaptive values of 3 to control the rate at which the likelihood of rejected re- sponses declines, referred as Flex-DPO. The second technique involves augmenting the DPO loss with an SFT loss, strategy that has been shown to significantly enhance the stability of DPO in previous studies (Hou al., [024]] Xu et al. 2024b). These regularization methods have shown promising results with our toy model and will be further validated in real LLMs in the following section. The theoretical analysis is similar to that of vanilla DPO thus deferred to Appendix B.2
Inspired by Proposition
3.4 INHERENT ABSENCE OF PROPREEERIEES IN
ALIGNMENT
In this section, we show that 3D-properties do not manifest in RM-based alignment methods, which may account for why DPO methods only achieve sub-optimal performance. Since DPO is closely related to RM training. and the Best-of-N performance of the RM can partially reflect the ultimate performance of the policy model (Gui et al. 2024). we focus on analyzing the RM's objective. For given (a,, at. a a), let + = r(a and r(a [2), the gradients with respect to r+ and r are:
aERM
@log(1 + el(T-rt)
e(r (r+)r))
Ox+
Or+
+ (rt-- + e(
7

---

Published as a conference paper at ICLR 2025
ae
a log(( + c(r--17) + c(r
e(
1
Ôr
e(rt-r-)
This indicates that the gradients for the chosen and rejected responses are balanced and not exhibit 3D- propeerriie. In Section 4.5 we will further discuss the relationship between DPO and R.M-baedd alignment in real LLMs.
EXPERIMENTS
In this section, we transition from theoretical analyses and toy model simulations to real-world experiments with LLMs to further validate our theoretical insights. We verify the existence of 3D- properties, the superiority of on-policy DPO over off-policy DPO, the superiority of RMM over DPO, and the effectiveness of the proposed regularization technique.
4.1 EXPERIMENTAL SETUP
Datasets. We chose mathematical reasoning and instruction following as our primary benchmarks because these tasks are easily quantifiable, providing clear metries for evaluating model perfor- mance. For mathematical reasoning, we used MATH (Hendrycks et al.) 2021) as the main dataset for both training and testing" [] To assess the model's aut-of-distribution (OOD) generalization capa- bilities, we selected SuperCLUE- Math (Xu et al.) 2020), another dataset which was used exclusively for testing. Additionally, we included two in-house datasets focused on poem and slogan generation to evaluate the model's s ability to handle creative tasks with strict structural and linguistic constraints. The poem dataset, for instance, which has rigid format and rhyme requirements, making a good test for evaluating the model's ability follow complex instructions. Further details and descrip- tions of the datasets used are provided in Appendix C
It is widely accepted in industry that preference datasets for model alignment should cover broad range of domains. Following this consensus, we further utilized a general dataset consisting of approximately 400,000 preference samples across diverse domains. These prompts were sourced from HH-rlhf (Bai et al. 2022) and UltraFedBBack (Cui et al., 2024 A detailed breakdown of the dataset sizes is provided in Table 6 in Appendix C
The LLMs of concern. We focus on Baichuan2- -33B and Baichuan2-338, an advanced bilingual (Chinese and English) LLM series. The 13B model is openly available (Yang et al., (023). and the 33B model extends the 13B architecture with increased parameters.
4.2 EFFECT OF TRAINING DATA DISTRIBUTION: ON-POLICY VS. OFF-POLICY
Building on the theoretical insights in Section we hypothesize that the performance of vanilla DPO is significantly influenced by the distribution gap between the training dataset and outputs of the policy model, specifically whether the algorithm is on-policy or off-policy. Off-policy DPO uses an external preference dataset, while on-policy DPO samples preferences directly from the policy model. On-policy DPO enjoys a smaller distribution gap compared with offlpoliy DPO.
To conduct on-policy DPO, used the policy model to produce 8 candidates for each prompt in the train set of MATH. The best and worst responses were selected by GPT-4 (Achiam et al.) 2023) to form preference pair, with the standard solutions given as the reference context. After filtering out uniformly good or bad responses, we compiled the MATH" dataset, which contains 5,826 pairs (x, at aa) }. We randomly selected 2,000 samples from the original test set for MATH', For off-policy DPO, we used the original solutions from the dataset as the chosen responses, and generated the rejected responses using Qwenl.5-7B (Bai al. 2023). relatively earlier model with limited capabilities.
serve as the test
To validate the hypothesis that the presence of off-policy data weakens performance, we imple- mented the four scenarios consistent with the toy model (Fi We evaluated the policy model using GPT-4 which assigned scores ranging from answer and the problem-soe sobleen--olvigg process, with also the standard solutions given as the reference
based on the accuracy of both the final
Only the prompts from the dataset were used to generate the preference dataset; further details are provided
in Section 4.2
8

---

Published as a conference paper at ICLR 2025
Table Results tested MATH and SuperCLUE-Math Scenario 1, with both chosen and rejected responses sampled on-policy, shows the best performance.
Baichuan2-13B
Baichuan2-33B
Setting
points
4&5 points
points
4&5 points
basemodel
32.237%
42.539 %
44.485%
53.229%
DPO in Scenario
37.132%
47.082%
47.465%
54.759 %
DPO in Scenario 2
32.860%
43.445%
44.216%
51.409%
DPO in Scenario 3
28.323%
41.576%
44.473% 46.618%
53.924%
DPO in Scenario 4 26.833%
37.685%
54.648%
Table Test results on the self-built Poem and Slogan datasets. All metrics are evaluated such that higher values indicate better performance.
Poem
Slogan Word Count
Row Number
Words per Row
Rhythm
Tone Pattern
Title
Content
Base
0.75
0.61
0.64
0.60
0.51
0.34
0.57
PPO
0.91
0.79
0.87
0.82
1
0.47
0.78
DPO
0.93
0.75
0.83
0.75
0.78
0.45
0.70
context. The scoring criteria are detailed Table The average performance on the MATH and SuperCLUE-Math datasets is reported in Table I with specific results in Table g Among the four scenarios, Scenario 1-where both chosen and rejected responses are on-policy ensured a more stable DPO training process and delivered the best performance.
Additionally, we report the log probabilities before and after training in Table in Appendix D.2 According to Proposition T the key factor affecting the impact of 3D-properties the decline rate of rejected responses' likelihood, log π (a-). Scenario shows the slowest decline in likelihood compared the other scenarios, effectively mitigating the adverse effects of 3D-properssss, which explains the superior performance of on-policy DPO in our tests.
We also plot the gradients during the DPO training process for Scenario 1, shown in Figure in the Appendix D.2 This visualization supports the analysis in Section 3 demonstrating that the gradients for rejected responses increase more rapidly during training. This excessive decline in the likelihood of generating rejected responses can ultimately lead to model degradation.
In addition to pure on-policy and off-policy DPO, Scenario where the chosen response off- policy and the rejected response is on-policy, is also prevalent in industry. For instance, in math problems, researchers often treat the correct dataset solution as the chosen response and the LLM- generated incorrect answer as the rejected one, which we demonstrate to be detrimental. Scenario a mirror experiment for Scenario These experiments confirm that incorporating off-policy data into the preference training set degrades DPO performance.
4.3 EXPERIMENTAL VALIDATION OF REGULARIZATION TECHNIQUES
Following Flex-DPO the regularization methods out- fixed 8+ and systematically de- creased 8 As indicated by the gradient analysis in Ap- pendix B.2.1 the gradient of rejected responses with re- spect to 3 follows a monotono trajectory, initially increasing and then decreasing. Reducing 3 on the left side of this extreme point can effectively reduce the gra- dient magnitude. However, indiscriminately minimizing the gradient is not always advantageous. As illustrated in Figure model performance does not consistently im- prove with excessively small 3 (see the trend with 3 < 0.08). Over-reduction of 3 risks causing the DPO algorithm to deviate from the preference learning
lined in Section 3.3
Sone
Figure 4: Performance poem gener- ation, 3 varying with 3+ 0.1.

---

Published as a conference paper at ICLR 2025
paradigm and regress toward behavior akin to the SFT algorithm, ultimately compromising its gen- eralization capabilities. This finding warrants further investigation, and while preliminary insights discussed in Appendix D deeper exploration is needed. This aspect will be addressed in future research.
Additionally, we tested other DPO variants, such IPO and SLiC, the MATH and SuperCLUE- Math datasets, with results presented in Table E Flex-DPO consistently outperforms vanilla DPO, IPO and SLiC, highlighting the effectiveness of the proposed regularization techniques.
4.4 RELATIVE INSTABILITY OF DPO TRAINING COMPARED TO RM TRAINING
To assess the stability and performance gap between DPO and RM training, we conducted parallel comparison us- ing identical datasets. Both models were built on the Baichuan2-33B architecture and trained on the HH-rlhf and UltraFeedback datasets, with evaluations conducted on the HH-rlhf and MATH datasets. The primary eval- uation metric was accuracy, defined the proportion of instances where the model correctly identified the chosen response as superior the rejected one.
HH-rihf eval set accuracy over training
Reverd Hocel
0.72
Zepech
0.70
Accuracy
0.68
0.66
0.64
As shown in Figure 5 RM training proved to be sig- nificantly more stable, whereas DPO training exhibited notable fluctuations. These findings are consistent with the theoretical results Section 3.4 which indicate that 31D-properties are absent in RM-based alignment meth- ods. Furthermore, as illustrated Figure S and Figure 9 in Appendix D.2] the DPO model demonstrated a higher tendency to overfit. Specifically, the sharp deceleration in accuracy improvement after the second epoch suggests that the model overfitting the training data, highlighting the more aggressive optimization dynamics of DPO.
0.52
0.60
1.5
Figure 5: Accuracy of RM and DPO on HH-rlhf eval set over the training pro- cess.
4.5
SUBOPTIMALITY OF DPO COMPARED TO RM-BASED ALIGNMENT
To further compare the performance of DPO and the end-to-end RM-based alignment (PPO), we tested both approaches on two datasets for poem and slogan generation. These datasets serve as ideal benchmarks for evaluating instruciion-folloving capabilities, given their explicit and struc- tured scoring criteria. For poem creation, the model must generate responses in accordance with specific text and tone formats based on the prompt. Evaluation metries include five key aspects: Row Number, Words per Row, Rhythm, Tone Pattern, and Title. For slogan creation, evaluation is based on Word Count and Content. Using Baichuan2-33B for our experiments, the results, shown in Table 2 demonstrate that DPO underperforms compared RLHF-PPO on both datasets.
CONCLUSIONS
In this study, conducted comprehensive theoretical analysis to elucidate why DPO does not perform as well as RM-based alignment algorithm. The principal challenge identified in DPO is summarized 3D-properties. We substantiated our theoretical framework through experimental results obtained from both toy model and real LLMs in practical applications, including mathe- matical reasoning and instruction following. Additionally, we assessed the effectiveness of specific regularization techniques. Furthermore, by contrasting DPO training with RM training, we high- lighted the inherent instability of DPO. We hope this work could offer research directions to narrow the gap between RM-free preference learning methods and RM-based ones. We leave the discussion of limiation to Appendix D.3
LECNONLESS
This work was supported by NSF of China (Nos. 92470118, 62306176) and Natural Science Foun- dation of Shanghai (No. 237R 1428700).
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 087744 2023.
Afra Amini, Tim Vieira, and Ryan Cotterell. Direct preference optimization with an offset. arXiv preprint arXiv:2402.10571, 2024.
Anthropic. Introducing claude. https:/ http://wwww..tttop....co comccludee 2024.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statittcs,, pp. 4447 4455. PMLR, 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 2.Xi:2309.16609, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2rXiv:2244005862 2022.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, pp. 324, 1952. doi: 10.2307/2334029, URL http://dx:
doi wrt.00.gg110000... 20070033372223000
Huayu Chen, Guande He, Hang Su, and Jun Zhu. Noise contrastive alignment of language models with explicit rewards. arXiv preprint arXiv:2402.05366, 2024.
Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. On the weaknesses of reinforcement learning for neural machine translation. arXiv preprint 1907v0190..017555 2019.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1-113, 2023.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao. Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultrafeedback: Boosting language models with sealed ai feedback. In Forty-first International Conference on Machine Learning, 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024.
Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, and Wenqiang Lei. Towards analyzing and understanding the limitations of dpo: A theoretical perspective. arXiv preprint arXiv:2404. 04626, 2024.
Lin Gui, Cristina Gårbacea, and Victor Veitch. Bonbon aligmment for large language models and the sweetness best-of-n sampling. arXiv preprint arXiv:2406.0088 00832, 2024.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792. 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 103.03874, 2021.
Zhenyu Hou, Yiin Niu, Zhengxiao Du, Xiaohan Zhang, Xiao Liu, Aohan Zeng, Qinkai Zheng, Minlie Huang, Hongning Wang, Jie Tang, et al. Chatglm-rlhf: Practices of aligning large language models with human feedback. arXiv preprint 2404:24044000344 2024.

---

Published as a conference paper at ICLR 2025
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.08825, 2023.
Ziniu Li, Tian Xu, Yushun Zhang, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effective, and efficient method for aligning large language models. arXiv preprint arXiv:2310.10505. 2023.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Salch, Peter Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.06657, 2023.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405. 14734, 2024.
Eric Mitchell. Online experimental results on dpo. 2023. https://wand..ai/eri.
anthony mi wwt.:mmitchheempeemmem atthooor_nicccceee///o/ooooo///r http:////wwnooooooooo/o//o/////s
mi mitchell
OpenAl. Introducing chatgpt, 2022. https://ppeea...co//ll//////
Last accessed
on 2023-05-09.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35:
27730-27744 2022.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.13228, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (0f.:::770.06344 2017.
Nisan Stienmon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, et al. Understanding the perfor- mance gap between online and offline alignment algorithms. arXiv preprint rxXx:22455084444 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud. Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. /805, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Xiang-Bo Bo Mao, Sitaram Asur, et al. A comprehensive survey of Ilm alignment tech- niques: Rlhf, rlaif, ppo, dpo and more. arXiv preprint arXiv:2407. 16216, 2024.
Wei Xiong, Hanze Dong. Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl- constraint. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023.
12

---

Published as a conference paper at ICLR 2025
Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. Clue: A chinese language understanding evaluation benchmark. arXiv preprint arXiv:2004.05986 2020.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for IIm alignment? a comprehensive study. arXiv preprint arXiv:2404.1077 10719, 2024a.
Yifan Xu, Xiao Liu, Xinghan Liu, Zhenyu Hou, Yueyan Li, Xiaohan Zhang, Zihan Wang, Aohan Zeng, Zhengxiao Du, Wenyi Zhao, et al. Chatglm-math: Improving math rroleem-solving in large language models with a self-critique pipeline. arXiv preprint arXiv:2404.0289. 2024b.
Aiyuan Yang, Bin Xiao, Bingming Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang. Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309. 2023.
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. Advancing IIm reasoning generalists with preference trees. arXiv preprint 2xXiv2404.000075 2024.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01065 2022.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Salch, and Peter Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint Xiv: 1909008999.999,,, 2019.
A DETAILED BACKGROUND AND RELATED WORKS
Large language models (LLMs) are profoundly transforming the way we work and live. Performing a three-stage process is the default practice for training LLMs: Pretraining, Supervised Tunnnn (SFT), and Reinforcement Learning from Human Feedback (RLHF). The roles of Pretraining and SFT are broadly understood: Pretraining encodes knowledge and SFT aligns question-answer for- mats. Relatively speaking. the understanding RLHF is relatively insufficient. Specifically, Direct Preference Optimization (DPO) and its variants, reward- model--ooo algorithmss, have garnered significant attention due to their elegant mathematical form and relatively low resource require- ments (Rafailov et al.] 2024 Pal et al. 2024 Guo et al.) 2024 Xiong et al., 2023). However, it has also sparked considerable debate because of its unstable performance practical applications
al. 2023
Xu al. 2024a).
A.1 ON-POLICY ALIGNMENT VS. OFFICY ALIGNMENT
The key inspiration for the DPO algorithm (Rafailov et al. 2024) is closed-form solution to the RL step RLHF, and thus an equivalent solution to the optimal policy for RLHF objective. The original DPO work is an off- oif-policy learning algorithm for it relies on an extra preference dataset (Helpful- and-Harssess (Bai et al. 2022), where the preference pairs are not generated by the policy LLM itself. On the other hand, there are bunch of on-policy learning algorithms developed, where the preference responses are sampled from the policy model. Guo et al. (2024) proposed the on-policy version of DPO. In on-policy DPO, all responses off between them the iterative DPO introduced by Xiong et al. (2023); Xu et al. (2024a). The algorithm begins by initializing with additional preference dataset. then iteratively trains policy using DPO, collects response pairs through exploration policies, obtains preference signals from human AI labelers, and updates the dataset with the newly labeled data.
sampled in batch-wise way. A natural trade-
13

---

Published as a conference paper at ICLR 2025
A.2 INSIGHTS INTO DPO
Though the RM-free algorithms are favored due to their lower computational overhead, if they can achieve on-performance with state-of-art RM-based methods such RLHF-PPO sparked lot of discussions. Liu et al. (2023) proves that the absence of RM in DPO constrains its ability to sample preference pairs from the optimal policy. Xu al. (2024a) show that DPO may have fundamental limitations that its optimal solution is a superset of the optimal solution of the PPO algorithm. This work also reports the empirical results that the performance of DPO is affected by the distribution shift between the model outputs and the preference dataset. Feng et al. (2024) discusses the limitations of DPO from the perspective of gradient numerical stability, and conducted experiments to preliminarily verify it. However, they did not conduct experiments on real LLM and illustrate the correlation.
A.3 OTHER RM-FRREE ALIGNMENT ALGORITHMS
A major limitation of the DPO objective is its reliance on the Bradley-Terry model convert pairwise preferences into point-wise rewards. To overcome this, [Azar et al. (2024) introduced d- preference optimization (&PO), where DPO special case of it that (P) log -p- Identity- preference optimization (IPO) is variant that replaces the @--fuccion by identity mapping func-
tion (PP) P.
Different from the DPO or IPO, the core idea of Sequence Likelihood Calibration (SLiC) ZZaao et al. 2023) is to calibrate the likelihood of ranked sequences sampled from the policy being trained. The SLiC loss function can be decomposed into two parts: the rank function to guarantee that the difference between log (a [<]]]] |æ)) and log To (x) is greater than 8 under the current policy Mo and the cross-entropy regularizer that to encourage the model to stay close the SFT model.
There are some other variants that tries to improve DPO, such as KTO Ethayarajh et al. 2024), NCA (Chen et al. 2024), ODPO (DPO with an offset) (Amini et al. 2024). KTO uses a Kahneman- Tversky model of human utility and proposes method that directly maximizes the utility of gen- erations instead of maximizing the log-likelihood of preferences. NCA leverages Noise Contrastive Estimation (NCE) to bridge the gap in handling reward datasets explicitly annotated with scalar evaluations. ODPO does not treat every preference pair equally during fine-tuning and requires the difference between the likelihood of the preferred and dispreferred response be greater than offset value.
B THEORETICAL FOUNDATIONS
B.
FUNDAMENTAL LIMITATION IN VANILLA DPO
Here revisit the theoretical findings in Section in detail. The loss function for vanilla DPO is given by
(a(a (x)) no (a (x) 8
EDP
Σ
log
To(a (rl(a_lz) To (a(aaa)
(2.a
For given (x, at ), let
Tola Ilaa+aaa (x)
z:= <a (a
x (ro(a(lr),
n == To(u [z))
Ta (a
xx)
n T
Then we have
Deo
Ô log(1 + az²)
Oz
a8
8- 8-1
dz
Oπt
dz
dπt
1 az
DPo
a log
+ xz
dz
a8
dz
dn
dz
+ az dm
Considering the case when मt - O, we get (a0)
)z + a,3 thus,
aEDPO
DEDRO
0m + a, 8- {
θπ + a,3( ) -8 )--1
14

---

Published as a conference paper at ICLR 2025
Be Da- is proportional to (F-) and tends O, while 2 0x- proportional
As O, since 3 <
(IT )8-1 and tends to infinity. Therefore, in this case, the gradient for the rejected action becomes extremely large, while the gradient for the chosen action becomes very small.
Then we want further explore the token-leve gradient. Here, x² and π are the likelihood of the sequences. Let x." be the current selection probability of the i-th token for the chosen sequence, and let `. be the current selection probability of the i-th token for the rejected sequence:
II = T.
T "
where we have are
ūt θm;
π n- Consider a softmax function, 8; E the corresponding gradients
et
0s; 0z;
0sj -.88t
s;(1 ss(1)
8s). Oz
i i
Let
C(x*, ,T)) aB (xx) (
then, consider the current selection probability ग.. of the i-th token for the chosen sequence, let the sampled token's index be c. The logit corresponding to this token e is denoted as vie then we have
aeo
OeDRO O+t dm, -C(r ① TT ztto)
Oxt 1
Oht θm 0x,,c
el Fc, we have
DEDRO dxt dmt Orr )axt
Oxt
Oπ+ θह. Oxt
Similarly, consider the current selection probability पं of the i-th token for the rejected sequence, let the sampled token's index be The logit corresponding this token is denoted as xe then we have
DE OEDRO' Ôn dz, + C(T , 7 xT,e): dr,
Ox- θm. dxi.c
èl # c, we have
debro DEDRO dn θ.; -C(r+ )aTTe'"
dxj On Ō.. Ox
We can see that the token-level gradients from the chosen response and the rejected response are the same scale level. This reflects that DPO may not cause gradient numerical instability in the generation of a single token. However, if the impact of the algorithm on the state transition proba- bility generated by autoregression is comprehensively considered, 3D-properties will still affect the performance of the algorithm.
B.2 ANALYSIS ON REGULARIZATION TECHNIQUES
In Section 3.3
we propose two straightforward regularization techniques. Here we provide theoret- ical analysis to why they can mitigate 3D-properties.
B.2.1
FLEXIBLE 3-DPO
The first technique employs variable values of / to control the rate at which the likelihood of rejected responses declines. Consider using different 1 and 3 for the chosen and rejected responses:
Te (aa
(a (x))
DFx
Σ
log
/+ log
B log
ma (a (z)
(a [z)
(I.a
The loss function can be re-written by:
log 1 (aa ot
(a(a (2:) B- Ta (x)
(ex-DDO
(a (x))
(z,a tiat
15

---

Published as a conference paper at ICLR 2025
For given (x, at aa), let
To 1z))
To(a (x)0
x = xaaaat)),, π == x<<aa<)))
2:=
To(a (z)0-
a (x) |
Then we have
DEDRO Ølog(1 + +a2)
0z
02
Out
Oz
Oxt
+ az dx
OEDPO' Olog(1 + az)
0z
02
Ōn
0z
Ōn
1 + az Ōn
Considering the case when π + O, we get a:((1 (1 + 02) + a, thus
OEDIO Ox+ a (ar++m+++))))))))))) 0+ (
Ôn aB ( (7-)
As shown by the expressions for the gradients of the DPO' objective with respect to the chosen and rejected response likelihoods, the magnitude of the gradients controlled by the parameters S+ and 3- In particular, increasing 3 strengthens the gradient for the chosen response, xt while reducing 3 dampens the gradient for the rejected response, π as it approaches zero. This effectively mitigate the 3D-properriess discussed in Section 3 Specifically, a large f ensures that the likelihood of the chosen responses remains sufficiently reinforced, while a small / prevents the likelihood of rejected responses from decreasing too rapidly, which would otherwise lead to the instability and degradation described earlier.
gradient behavior suggests that adjusting these parameters
By fine-tuning 3+ and 3_, it becomes possible to control the interaction between the gradients of the chosen and rejected responses, reducing the drastic drop in rejected response likelihood, the degradation into response suppression, and the dispersion effect on unseen responses. This strategy thus offers potential solution for improving the stability and performance of DPO by reducing the severity of the 3ID-properties.
B.2.2 SFT LOSS REGULARIZATION
The second technique involves augmenting the DPO loss with an SFT loss, a strategy that has been shown to significantly enhance the stability of DPO in previous studies. We can rewrite the loss function:
(SFT-DRO (A)=
Σ
logo
B
To(a+la) To(a+(z)
-Blog
To(a [x)] To(a [x)
y log no(atlo)) (6)
(æ,a
Similarly, we have,
DeSPI-DPO
2 log(1 0)
8z
0
9-1 dz
1
Oht
Oz
Ou
1
az3
DE
0log(1 + az
0z
a9
Oz
3
8-1
dz
Oπ
+ a
Om
OFTDDO
As n - O, the gradient for the chosen action of the chosen responses can continue to be optimized. This behavior is significant, as it indicates that even the likelihood of rejected responses л approaches zero, the chosen response л can still be improved. This is key advantage of SFT-DPO over the vanilla DPO, which suffers from gradient vanishing for + when allowing for more stable and effective optimization in the long run.
-7 -- 0. meaning that the likelihood
O. The negative impact of 3D-properties is thus reduced,
B.3 OTHER INVARIANTS OF DPO
B.3.1 DDENTITY-RREFERNNT OPTIMIZATION (IPO)
IPO, the loss function can written by,
To(a" ()) (x)
1 2
(P)
Σ
log
To(a ( 10000aaa000
2n
(0.at.a
16

---

Published as a conference paper at ICLR 2025
We directly give the gradients:
OAPO
TT (a 2 (o) (a [x)
1
-2 log
(8)
Oho (To(a-(z)
no ( (a
2n
To xx)
aEPO
To To (T) [x:
1
2
log
(9)
xx)
TT(aa (2) [x))
2m
Te(a (x)
B.3.2 SEQUENCE LIKELIHOOD CALIBRATION (SLIC)
In SLiC, the loss function be written by,
(S (8)
Σ
max [0, 5 (ogmo(attee (x) + log To(a 17 log (a (1112)
(10)
5 log Nola (x) mo(a-12) [z]
Sesic
1+n
Sesic
(11)
Ora(atl2)) Te(a+(z))
To(a (x)
else:
DeSLiC
aeSLic
@Te(a-12)) = 0
(12)
@тT(at|2) Ta(a+(z)*
B.3.3 SIMPLE PREFERENCE OPTIMIZATION (SIMPO)
In SimPO, the loss function is written by,
SimPO (0)
Σ
log
B [aat]
log (a (x)
B
log To (a- - -7)]
(13)
|a-1
where represents the length of the generated response and y is target reward margin term. Similarly, we directly give the gradients:
B
(-(12)
3 |a+|
log To (x)) (x)) (14)
log vo(at(z))
t (x)
la+|
No [x)
la-1
(-
log Te(a (2) (1)) -7)) (15)
log (a (x)
@mo(a-
xx)
a-
(a
(x))
a+
aa-
The ratio of the gradient
SimPO
@eSimPo
la Ta(a [x) no TT(a" xx)
(16)
which indicates that as ++ increases and π decreases, the gradient with respect to π grows faster and leads to rapid drop in the likelihood of the rejected response. However, different from DPO, as T - 0, we have SimPO /ō++ - 0. As for SimPO /ār the analysis non-trivial. We conclude it lemma.
Lemma 1. As Te(a (x:) + 0, the limit of the partial derivative regarding aa(a [1:] in SimPO is:
0,
if
> la
aeSiPo
lim za(a- (n) - 0.p (a (2:
+00,
if
a
(17)
BC
if
a-
where C = T log nola [x)+T is constant independent of mo(a (x).
17

---

Published as a conference paper at ICLR 2025
Table 3:
Data source of the responses in 4 scenarios.
chosen responses source
rejected responses source
Scenario
Baichuan2-33h
Baichuan2-33P
Scenario 2
Solutions from dataset
Baichuan2-33B
Scenario
Baichuan2-33B
Qwen-7B
Scenario
Solutions from dataset
Qwen-7B
Proof. Let - = =To(a (x) and π To(a+((x) The partial derivative becomes:
B o (-2),
f(
a 1
where 8 log 8 a
lat7 T
log -.
As π + 0, log - -00 thus z + too. The sigmoid function approximates to:
o(-2) ==== C
where == lat logx 1y By substituting back, we have:
3C
f(T)
(I (a-
la
Taking the limit as x +
0, if
1 1>> 08 la D) < (3 <
la-
lim
f(T
t00,
if
BC if = 0 (8 =
a
Note that practice, 33 is normally chosen less than Ja which means the drastic drop in rejected response will happen and 3D-properties still occur.
Remark 1. These variants all alleviate the 3D-properties problem of DPO, so the results math- ematical reasoning are partly improved (see (.. The invariants we tested do not perform well uniformly all the tasks. For example, instruction following tasks like poem generation, vanilla DPO outperforms SLiC and IPO. One hypothesis their solution forms diverge significantly from the Bradley-Terry model, leading loss of generalization in preference learning.
C DATASET DESCRIPTION
Table 3 shows the data source for the LLM experiments in Section 4 in 4 scenarios. For example, in Scenario I, the chosen and the rejected responses are both sampled from Baichuan2-33B and can be regarded given in the datasets while the rejected responses are sampled from another different LLM: Qwen- 7B. In the experiments regarding Baichuan2- we use the same data rather than re-sample the on-policy chosen responses. There are two reasons: 1. The 13B model is not as strong as the 33B model, therefore we can not sample enough high-quality responses as the chosen ones. 2. Models in the Baichuan2-series are all using the same dataset in Pretraining and SFT, therefore can approximately think that their outputs are identically distributed. The log probability for the base model in Table 7 confirms confirmss this fact.
on-policy learning. In Scenario 4, the chosen responses are exactly the solutions
Table 4 describes the evaluation criteria of the responses to the math questions. score of means both the process and the result are correct, and score of or 5 means the answer is correct. We use these two indicators to evaluate the mathematical reasoning ability of the model. GPT-4 is used as the Al evaluator. We provide the evaluation prompt in code in the supplementary material.
18

---

Published as a conference paper at ICLR 2025
Table Evaluation criteria of the responses the math questions.
5 points
Full-score answer, requiring correct response with the correct process, considering all possibilities, and being comprehensive.
4 points
For complex questions, the answer is correct but lacks the process; for simple questions, the answer correct but accompanied by very re- dundant and verbose reasoning process.
3 points
The answer is incorrect, but most of the process is correct, the answer is correct, but there are obvious errors in the process.
2 points
The answer is incorrect, and most of the process is incorrect.
1 point
The answer and the entire process and thought process are incorrect, or the answer doesn't 't process to final result.
Table 5:
Poem dataset test set.
Poem type
Quatrain
Song Ci
Ancient Poetry
Metrical poetry
Modern poetry
Test Count
138
518
93
173
85
Table shows the types of different poems in the poem dataset we used and their corresponding numbers in the test set. The language is all Chinese. Chinese poetry has strict format and rhyme requirements depending on the type. For example, for the quatrains, the row number must be 4, the number of words per row must be 5 or 7. The second and fourth sentences in the quatrain are required to rhyme, that is, the words at the end of the second and fourth sentences need to follow the prescribed tone. In this manner, design mile-based evaluation system to score each dimension of the generated answers. We selected the following characteristics as the basis for our evaluation:
Row Number: For quatrain, the row number must be 4. For metrical poetry, the number must be 8.
Words per Row: For quatrain and metrical poetry, the number of words per row must be
Rhythm: Every type of poetry has certain rhyme pattern requirement. Since it is bit complicated to describe case by case, we put the requirements in the form of code in the supplementary material.
Tone Pattern: For Song Ci, the tone pattern depends the brand name.
Title: Determined by the requirement in the prompt.
For the Slogan dataset, we evaluate the model's performance based on whether meets the word count requirements (Word Count) and the quality of the content (Content). We also provide the scoring and evaluation rule-based criteria our code in the supplementary material.
Tablee]jshos the amount of data in each dataset. MATH, SuperCLUE, UltrFeedBBcck and HH-rlfh are open-source datasets, while Poem, Slogan are in-house self-built datasets. We provide part of the in-house datasets in the supplementary material to clarify the format and the content. SuperCLUE only for cross-dataset testing, and COMMON only used for the comparison between the RM training and DPO training in Section 4.4
D EXPERIMENTS DETAILS
D.1 TRAINING SETTING
In this section, we provide detailed overview of our training settings. Following the implementa- tion of Rafailov al. (2024), we use the Adam optimizer with the learning rate set to 5e-7. The most sensitive parameter in the DPO algorithm is 3 (and learning rate but less significant). Here we use the default setting aligned with the original DPO paper, The , here is set be 0.1 and the learning
19

---

Published as a conference paper at ICLR 2025
Table 6: The statistic of used datasets.
MATH
SuperCLUB
Poem
Slogan
UltraFeedBack
HH--RHHF
train set test set 2,000
5,826
93,269
13,592
170,000
336,820
1,072
1,000
1,000
rate is set to be 5 10-6, which is the best hyperparameter set far explored. We the batch size to 80 and the number of gradient accumulation steps to 2. The training epoch was set to In IPO training, we set 17 be 0.1. In SLiC training, we set 5, 7 0.1. All experiments were conducted on a cluster consisting of 40 A100 GPUs.
D.2 SUPPLEMENTARY EXPERIMENTAL RESULTS
7 the log probabilities before and after training. Scenario exhibits the slowest decline in likelihood compared to other scenarios, effectively mitigating the adverse effects of 3D- -D-ropperties.
Table show the performance enhancement in MATH and SuperCLUE by vanilla DPO training respectively. It is easy to see that Scenario where both the chosen responses and the rejected responses are on-policy performs best.
Figure and and represet the additional results DPO variants and regularization techniques. can be seen that DPO variants can achieve on-par or better performance compared with vanilla DPO. Figure shows that as 3 decreases, the performance in poem generation initially improves, reaches the peak point at around / 0.08, and subsequently declines. The initial improvement intuitive. We conjecture that an excessively low 3 may cause DPO perform like SFT on the chosen responses, thereby reducing its generalization capabilities. For different tasks, the peak point of 3- be different. For example, in mathematical reasoning. we can set 3 be 0.01 and achieve better performance than vanilla DPO.
Figure 6 shows the change of the absolute value of gradient for the chosen and rejected responses TopDoOTTT and @eDPo /0 during the training process of DPO on MATH. It can be seen that jaeDPo /mm >> (eDDo /0m and the increasing rate of aeDPo /0m is much higher than that of DEo /rrt which is align with Property
Figure and Figure shoo the DPO convergence process with model trained on MATH and HH- rlhf respectively. In the second epoch, the accuracy growth of the model slows down sharply, which indicates that the model overfits the training data. The results further confirm that DPO an aggres- sive optimization strategy compared to RLHF-PPO, and makes the impact of -D-properties more prominent.
Among the four scenarios, Scenario 1, where both chosen and rejected responses on-policy, ensures a more stable DPO training process and delivers the best testing performance, as shown in Table I As shown in Table 7 Scenario 1 exhibits the slowest decline in likelihood compared to other scenarios, effectively mitigating the adverse effects of 3D-properties. This explains the superior performance of on-policy DPO in our tests.
In addition to pure on-policy and off-policy DPO, Scenario where the chosen response is off- policy and the rejected response is on-policy is also commonly in the industry. For example, in some math problems, researchers used to treat the accurate solution in the dataset to be the chosen response and the wrong answer generated by the LLM itself to the rejected response, which we show is harmful. Scenario 3 is mirror experiment to Scenario 2. These additional experimental results confirm that as long as the off-policy data mixed into the training preference dataset, the performance of DPO will be weakened.
D.3 LIMITATIONS
Despite the insights provided in this study, several limitations remain. First, while our theoretical analysis and experiments highlight the 3D-properties key factor in DPO's suboptimal perfor- mance, the complexity real-world LLMs may involve additional factors that were not fully ex- plored. Second, experimental evaluation, though spanning diverse tasks such mathematical
20

---

Published as a conference paper at ICLR 2025
350
log gradient of chosen log gradient
300
275
225
20
Figure 6: When training with on-policy data, the absolute value of the gradient for rejected responses increases, while the absolute value of the gradient for chosen responses remains almost unchanged.
HH-rtff eval set accuracy over training
MATH eval set accuracy over training
Sowar Nocal
Reward Nocel
D70 Leaceh
0.90
EPO Sepec
PPo Zealch
Sepect
0.70
0.85
0.68
Accuracy
0.80
0.66
0.75
0.65
0.60
1.5
2.0
0.5
1.0
1.5
Eooch
Epoch
(a) Accuracy of RM and DPO on HH-rlhf eval set over the training process.
(b) Accuracy of RM and DPO on MATH eval set over the training process.
Figure 7: Comparison of DPO and RM Training. DPO training shows significant fluctuations.
RM training demonstrates greater stability, while
MATH training lass over training
MATH ning
DR0
2.00
Teport Sazcch
DPO Zasech
3.2-
0.25
3.80
3.3-
0.87
0.00
3.3
2.50
1.25
1.50
0.50
1.50
(a) MATH train dataset loss.
(b) MATH train dataset accuracy.
Figure 8: Comparison between DPO and RM training on the training set of MATH. As can be seen, in the second epoch of DPO training, the loss is very small and the accuracy of distinguishing the chosen response from the rejected response is 100%, which indicates that the model overfits the training data.
21

---

Published as a conference paper at ICLR 2025
Table 7: Impact of on-policy training data on the results. lo* and log (aa)
represent the
average log probability per token for the chosen and rejected responses, respectively.
Baichuan2-13B
Baichuan2-333
log (aa))
log ((a
log (a+)
log T
basemodel
-0,9181
-0,9393
-0.3603
-0.3634
DPO in Scenario
-0.9681
-0,9982
-0.3629
-0.3670
basemodel
-1.6776
-0.9238
-1.4314
-0.3525
DPO in Scenario 2
-1.6265
-1.2293
-1.2734
-0.4254
basemodel
-0,9461
-1.7110
-0.3460
-1.1204
DPO in Scenario
-0,8786
-1.8848
-0.3439
-1.4333
basemodel
-1.7468
-1.7110
-1,2838
-1,1451
DPO in Scenario
-1.6617
-1.8265
-1,2273
-1.2234
Table 8: Vanilla DPO: Baichuan2-13B and Baichuan2-33B accuracy on MATH" and SuperCLUE.
Baichuan2-133
Baichuan2-333
MATH
SuperCLUE
MATH
SuperCLUE
5 points 4&5 points 5 points 4&5 points 5 points 4&5 points
points 4&5 points
basemodel
6.0%
12.2%
46.3%
58.8%
25.7%
36.5%
79.5%
84.4%
Scenario 7.9%
14.4%
52.8%
64.6%
29.9%
37.5%
80.2%
86.6%
Scenario 2
4.8%
9.2%
47.9%
61.8%
29.2%
36.6%
72.2%
79.0%
Scenario 4.3%
12.8%
41.2%
57.0%
28.2%
37.3%
74.8%
84.9%
Scenario 4
3.2%
9.3%
39.5%
52.9%
28.6%
38.1%
80.2%
85.8%
reasoning and instruction following, limited scope and may not generalize across all LLM applications. Additionally, the regularization techniques proposed, while effective in our controlled settings, require further validation in larger-scale models and more diverse datasets. Lastly, although contrasted DPO with RM-based alignment, study does not exhaustively address other poten- tial reward-free methods, leaving open questions for future exploration.
HH-rthff training
HH-rif training training
3.6
DP3 Zasoch
0.45
Sepzeh
335
0.40
3.4
0.82
0.00
0.5
1.35
3.35
1.50
spoch
spoch
(a) HH-rlhf train dataset loss.
(b) HH-rlhf train dataset accuracy.
Figure 9: Comparison between DPO and RM training the test set of HH-rlhf.
22

---

Published as a conference paper at ICLR 2025
Table 9: DPO and its variants/regullarized version performance mathematical reasoning. In Flex-
DPO, 3+ = 0.1, 8 0.08.
MATH
SuperCLUF
points
4&5 points
points
4&5 points
basemodel
25.7%
36.5%
79.5%
84.4%
DPO Flex-DPO
29.9%
37.5%
80.2%
86.6%
30.1%
38.0%
81.2%
86.7%
IPO
30.0%
37.7%
80.5%
85.9%
SLiC
29.3%
38.7%
79.7%
84.5%
23


Reviews:
[{"review_id": "ne9wyL1EZ4", "reviewer": "Reviewer_DBcQ", "summary": "The paper titled \"3D-Properties: Identifying Challenges in DPO and Charting a Path Forward\" presents a thorough analysis of the DPO method used for aligning LLMs with human preferences. The authors identify and term three critical properties of DPO's learning process the 3D-properties and propose regularization techniques to address the challenges these properties present. Theoretical analyses, toy model simulations and real-world experiments demonstrate the effectiveness of the proposed method.", "questions": "How to ensure that the initialization assumptions of parameter distribution can be applied to, or related to LLMs?\n\nThe detailed parameter adjustment strategy is only given in the toy experiment. What is the effect of different β values ​​in the real-world experiments?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "The paper is well-structured, where toy example can support their claims.\nThe paper offers a balanced mix of theoretical analysis and empirical evidence, which strengthens the claims made about the 3D-properties and their impact on DPO's performance.", "weaknesses": "The three observations have been widely studied by previous works. Besides, one of the proposed regularization methods, incorporating an SFT loss into the objective, has been widely used in existing preference learning approaches [1]. This limits the novelty of the paper.\nConsidering that there are many existing methods to solve the DPO problem proposed in this paper, there is a lack of comparison with them, such as [2] and others.\nConsidering the generality of the proposed constraint algorithm, some advanced preference learning algorithms, such as SimPO [3], should also be tested.\nMore and more general LLMs should be included for evaluation, such as Meta-Llama3.\n\nReference: \n[1] Pang R Y, Yuan W, Cho K, et al. Iterative reasoning preference optimization[J]. arXiv preprint arXiv:2404.19733, 2024.\n[2] Pal A, Karkhanis D, Dooley S, et al. Smaug: Fixing failure modes of preference optimisation with dpo-positive[J]. arXiv preprint arXiv:2402.13228, 2024.\n[3] Meng Y, Xia M, Chen D. Simpo: Simple preference optimization with a reference-free reward. NeurIPS, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tC7nA2pFhU", "reviewer": "Reviewer_x5ts", "summary": "The paper provides a comprehensive analysis of Direct Preference Optimization (DPO), examining its theoretical foundations and empirical performance to address current limitations. It identifies three perspectives—(1) Drastic drop in the likelihood of rejected responses, (2) Degradation into response suppression, and (3) Dispersion effect on unseen responses. The paper connects these observations to related research and offers a theoretical explanation for the underlying mechanisms. To improve DPO’s stability and performance, the authors propose regularization methods, including adaptive adjustment of gradient weights for chosen and rejected responses, as well as incorporating an SFT loss into the objective.", "questions": "For the toy model setup, which specific model is used in the paper?\n\nWhy does the paper focus primarily on math datasets rather than exploring a wider range of tasks?", "limitations": "", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The topic is interesting for RLHF. \n\nThe paper introduces effective regularization methods, including adaptive gradient weighting for chosen and rejected responses.\n\nThe experiments are well-conducted and thorough.", "weaknesses": "The study could benefit from using a wider range of LLMs.\n\nThe experiments can use more datasets except for math. \n\nThe code is not open source, which may limit reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SVwg22TrEr", "reviewer": "Reviewer_Dw57", "summary": "This paper presents an interesting theoretical and empirical analysis of Direct Preference Optimization (DPO) and identifies three main challenges in its optimization process, termed as “3D-properties”: Drastic drop in rejected response likelihood, Degradation into response suppression, and Dispersion effect on unseen responses. These limitations, which do not arise in RM-based approaches, impact the stability and effectiveness of DPO. To address these issues, the authors propose regularization techniques, including adaptive gradient weighting and SFT loss. They conduct experiments on toy examples as well as math reasoning and instruction-following tasks to validate the presence of the 3D-properties, the advantages of on-policy over off-policy DPO, the comparative superiority of RM-based methods, and the effectiveness of the proposed regularization technique.", "questions": "- In Section 4.2, it is mentioned that for the MATH dataset, the best and worst responses were selected by GPT-4. Why did the authors choose this method instead of directly verifying the answers? Given that GPT-4’s accuracy on MATH is only slightly above 50%, this approach seems potentially unreliable.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Significance: The paper addresses a crucial and interesting gap by analyzing the limitations of DPO\n- Theoretical Analysis and Empirical Validation: The paper provides a theoretical framework alongside empirical results to validate the presence of the 3D-properties in DPO. This combined approach strengthens the findings, offering clear insights into the mechanisms driving DPO’s limitations and supporting the proposed solutions.", "weaknesses": "- Presentation: The presentation could be improved to enhance readability. For example, the text size in Figures 2 and 3 is small, and the description of Scenarios 1-4, which is crucial for understanding the on-policy versus off-policy comparison, is currently only detailed in the appendix. Bringing this description to the main text would improve clarity.\n- Experimental Design for On-Policy vs. Off-Policy Comparison: The on-policy and off-policy experiments rely on different data sources, which introduces potential confounds in the comparison. Using a more direct on-policy and off-policy setup, such as comparing historical-only data with semi-on-policy DPO (e.g., iterative DPO), would make the findings more robust.\n- Parameter Tuning in Flex-DPO: Adjusting Flex-DPO requires tuning two parameters ( \\beta^+  and  \\beta^- ), and while Figure 4 provides some guidance, this approach may still present challenges for practical implementation due to a lack of clear tuning guidelines.\n\nIf the authors address these weaknesses, particularly by improving the clarity of presentation and by using a more controlled comparison between on-policy and off-policy data sources, I would raise my score. Addressing the Flex-DPO would also strengthen the work, though it is not essential for improving the overall contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wRslBr1CFR", "reviewer": "Reviewer_Ky9E", "summary": "This paper investigates the limitations of DPO in aligning large language models with human preferences, identifying three critical properties that hinder its performance: drastic drops in rejected response likelihood, degradation into response suppression, and dispersion effects on unseen responses. The authors provide theoretical explanations for these properties and demonstrate how they arise from DPO's objective. To address these challenges, the paper proposes regularization techniques and validates their effectiveness through experiments on both toy models and real-world language model tasks.", "questions": "- The probability distributions in the bottom-right figure don't seem to match with the leftmost figure in Figure 2. In Figure 2, the unseen probability at 500 epochs approaches 1, but in Figure 1 it's all zeros. The chosen probabilities also don't quite align.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Studying DPO degradation phenomena is important due to its widespread use. This paper originally summarizes and theoretically analyzes several degradation phenomena of DPO discovered in previous work.\n\n- Novel comparative analysis between on-policy and off-policy DPO on toy models.\n\n- The paper is well-written and easy to follow.", "weaknesses": "- I think the Explanation for Property 3 is inadequate. Firstly, compared to the previous two explanations, it lacks mathematical formulation and seems to merely restate empirical phenomena. Secondly, since the optimization process is conducted in mini-batches, while the model may ensure that overflow probability won't disperse to recently seen samples, I suspect it could also disperse to samples from the preference dataset that were encountered earlier, rather than necessarily dispersing to unseen samples outside the preference dataset.\n\n- Following the previous point, the toy model setup, as mentioned by the authors in lines 344-353, is closer to treating each input/output as a token rather than a complete prompt/response, which is not a good toy model approximation of the real situation. One possible improvement would be to maintain other settings unchanged while increasing the sample size to enable mini-batch optimization that better resembles real-world conditions, with fewer epoch repetitions.\n\n- While the authors used self-built Poem and Slogan datasets to evaluate the model's instruction following ability and acknowledged their limited scope, these datasets are insufficient to assess the model's general instruction following capabilities. The paper lacks evaluation on widely-used benchmarks in preference optimization work, such as AlpacaEval2, Arena-Hard, and MT-Bench, which are designed to test models' general instruction following ability.\n\n- The proposed regularization techniques lack substantial significance. The first technique, which independently adjusts beta for reject responses, shows effectiveness in the poem task, but the optimal reject beta is merely 0.02 lower than the chosen beta. Without showing gradient comparisons for this technique, it's unclear whether it actually improves performance by addressing the large gap demonstrated in Figure 2. Moreover, the second technique, SFT loss, is already a widely established regularization technique.\n\n- I am not quite convinced by the claims in section 3.4. Although existing works are cited to establish conceptual connections between RM and DPO, the subsequent gradient analysis focuses on r, creating a gap with the previous gradient analysis that focused on $\\pi$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)