Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization
Authors: Junkang Wu
Yuexiang Xie
Zhengyi Yang
Jiancan Wu
Jiawei Chen
Jinyang Gao
Bolin Ding
Xiang Wang
Xiangnan He
Venue: ICLR 2025 Poster (2025)
Keywords: Direct Preference Optimization
LLM's alignment
Distributionally Robust Optimization
Primary area: foundation or frontier models, including LLMs

Abstract: This study addresses the challenge of noise in training datasets for Direct Preference Optimization (DPO), a method for aligning Large Language Models (LLMs) with human preferences. We categorize noise into pointwise noise, which includes low-quality data points, and pairwise noise, which encompasses erroneous data pair associations that affect preference rankings. Utilizing Distributionally Robust Optimization (DRO), we enhance DPO's resilience to these types of noise. Our theoretical insights reveal that DPO inherently embeds DRO principles, conferring robustness to pointwise noise, with the regularization coefficient $\beta$ playing a critical role in its noise resistance. Extending this framework, we introduce Distributionally Robustifying DPO (Dr. DPO), which integrates pairwise robustness by optimizing against worst-case pairwise scenarios. The novel hyperparameter $\beta'$ in Dr. DPO allows for fine-tuned control over data pair reliability, providing a strategic balance between exploration and exploitation in noisy training environments. Empirical evaluations demonstrate that Dr. DPO substantially improves the quality of generated text and response accuracy in preference datasets, showcasing enhanced performance in both noisy and noise-free settings.

Published as a conference paper at ICLR 2025
TOWARDS ROBUST ALIGNMENT OF LANGUAGE MODELS: DISTRIBUTIONALLY ROBUSTIFYING
DIRECT PREFERENCE OPTIMIZATION
Junkang Tuu'xiannnn Xie Zhengyi Yang' Jiancan Wu
Jiawei Chen Jinyang Gao² Bolin Ding² Xiang Wang' Xiangnan He
University of Science and Technology of China Allibaba Group ZZejjanng University
'MoE Key Lab of BIPC, University of Science and Technology of China (jkwu0909, wujcan, xiangnanhe gm com
ABSTRACT
This study addresses the challenge of noise in training datasets for Direct Prefer- ence Optimization (DPO), a method for aligning Large Language Models (LLMs) with human preferences. We categorize noise into pointwise noise, which includes low-quality data points, and pairwise noise, which encompasses erroneous data pair associations that affect preference rankings. Utilizing Distributionally Robust Optimization (DRO), we enhance DPO's 's resilience to these types of noise. Our theoretical insights reveal that DPO inherently embeds DRO principles, confer- ring robustness to pointwise noise, with the regularization coefficient ,3 playing a critical role in its noise resistance. Extending this framework, we introduce Dis- tributionally Robustifying DPO (Dr. DPO), which integrates pairwise robustness by optimizing against worst-case pairwise scenarios. The novel hyperparameter 3' in Dr. DPO allows for fine-tuned control over data pair reliability, providing a strategic balance between exploration and exploitation in noisy training environ- ments. Empirical evaluations demonstrate that Dr. DPO substantially improves the quality of generated text and response accuracy in preference datasets, showcasing enhanced performance in both noisy and noise-free settings. The code is available at https::/gttthtii http://ttt...oooon//ao.oo.oa/eee
com/ junkanggt/Dr_DDDP
INTRODUCTION
Aligning Large Language Models (LLMs) (OpenAI, 2023; Touvron et al., 2023; Anil et al., 2023; Bubeck et al., 2023) with human preferences is critical for their implementation in real-world scenarios. Central to the alignment is the fine-tuning of LLMs using human feedback (Ouyang et al., 2022), ensuring they adhere to human values and mitigate safety risks. Among the aligmment methods, Reinforcement Learning from Human Feedback (RLHF) (Ouyang al., 2022) is becoming widely adopted technology. It initially learns a reward model on pairwise preference data, and optimizes LLMs using the Proximal Policy Optimization (PPO) (Schulman al., 2017) method. However, its inherent reinforcement learning nature poses significant challenges to computational efficiency and training stability (Rafailov al., 2023a; Zhao al., 2023). Addressing these, Direct Preference Optimization (DPO) (Rafailov et al., 2023a) eschews the explicit reward model learning, using human preferences train the LLMs directly. achieves the same objectives (Azar al., 2023) as RLHF by learning an optimal proxy for each pointwise instance and simultancously ranking preferences in a pairwise manner, offering greater simplicity and training stability (Ivison al., 2023).
While offering an effective solution by directly learning policy from collected data, DPO inevitably heightens the dependency on the data quality (Liu al., 2023). However, training data frequently marred by noise, potentially posing significant challenge to DPO. Here we delineate two primary noise categories based on their origins:
"Work done at Alibaba Group.
Jiancan Wu and Xiangnan He the corresponding authors.

---

Published as a conference paper at ICLR 2025
high quality review
low quality review
mistaken preferred review
correct preferred review
DPO
DP0()'=0.5)
Review A- "Fantustic experience with delicious food and excellenn service!"
"Fantastic, uh, something food, the service W00 yeahs
| yill
Dr. DPO(8"-1.0)
Dr. DPO($"=2.0)
pointwise noise
Prompt: restourant
your experience:
Review
amazing culinory
quite good, and stoff
odventure: flavors extroordinary!"
friendly."
Low (Noisy)
High ( Clean) )
ranking
Incormect
pairwise noise
Quality of pairwise preference
Figure 1: Left: An example illustrating pointwise and pairwise noise. Right: Comparison of gradients
between DPO and Dr. DPO under varying levels of pairwise noise.
Pointwise noise (Gunasekar et al., 2023) refers to low-quality data points containing irrelevant or incoherent information. Taking the movie reviews in Figure (Left) as an example, it might manifest reviews filled with meaningless chatter, thus rendering them uninformative.
Pairwise noise (Sharma al., 2023; Cui al., 2023), the other hand, arises from erroneous associations between data pairs, leading to misjudged preference rankings. Revisiting the movie reviews in Figure (Left), it is evident in misranked reviews where an inferior review (yr) is incorrectly rated higher than a superior one (Vw).
The presence of noisy preferences naturally raises critical question: How robust is DPO against pointwise and pairwise noise? To answer this, we examine DPO through the lens of Distributionally Robust Optimization (DRO) (Namkoong & Duchi, 2017: Duchi & Namkoong, 2018). At the core of DRO is training model across a distributional family, which is determined by an empirical distribution within robust radius 7j. As result, DRO endows the model with enhanced robustness w.r.t distributional uncertainty, usually caused by the data noise. By incorporating DRO principles, we can assess the resilience of DPO the pointwise and pairwise noise. Specifically, our DRO lens on DPO offers insightful findings as follows:
DPO is equivalent to applying DRO on the reward function. The principal contribution of DPO is deriving the optimal policy for PPO in closed-form expression. This achievement facilitates the implicit determination of a worst-case distribution for optimization, guided by the Kullback-Leible (KL) divergence criterion. Such an approach endows DPO with intrinsic pointwise robustness, enabling it to explore a better policy model rather than relying solely on the reference model.
The DPO's and DRO's 1) share an inverse relationship, highlighting noise levels in the reference model. Through DRO theory, we establish that higher noise in the reference model necessitates larger search radius, corresponding to a larger 7 (or equivalently, a smaller (3). This inverse relationship provides clear measure of the noise level in the reference model.
These findings elucidate the strengths of DPO in ensuring pointwise robustness. Recent effort (Chowdhury et al., 2024) has started addressing pairwise noise DPO frameworks; however, this method relies on explicit noise estimation, process that is computationally intensive and may not fully capture noise complexities. Building on these insights, we introduce the Distributionally Robustifying DPO (Dr. DPO) framework, aiming to incorporate pairwise robustness within the DPO paradigm. The core idea is optimizing against the worst-case pairwise scenarios, enabling the models to implicitly adjust the importance of data pairs in the gradient space and eliminate the explicit noise estimation. Towards the adjustment, Dr. DPO introduces a simple hyperparameter 3' E (0, +o0) to modulate the loss function, balancing between exploration and exploitation of pairwise preferences. 3' serves as a pivotal "knob", allowing the navigation from a conservative strategy that diminishes the influence of potentially noisy pairs (e.g.. G 0.5) risk- solerttt stance that leverages such pairs (e.g., 8 = 2). Consequently. Dr. DPO fosters a more resilient optimization process that effectively mitigates the influence of both pointwise and pairwise noise.
'The abbreviation DD. DPO" not only encapsulates "Distributionalyy Robustifying DPO" but is playfully intended ccho the abbreviation for "Doctor," adding a quirky element the naming.
2

---

Published as a conference paper at ICLR 2025
In nutshell, our contribution is the development of Dr. DPO, which robustifies DPO with just single additional line of code. Empirical evaluations reveal that Dr. DPO significantly enhances performance across diverse settings, such controlling the sentiment in generated text and improving the response quality in single-turn dialogues, under both noisy and noise-free conditions.
PRELIMINARIES
Bradley-Terry Model. Given context z within finite space of contexts X, we employ the policy x (y]ær to independently generate a pair of actions (y1, These actions are presented to human raters, who then indicate their preference, with the preferred action labeled as the and the less preferred as yt, satisfying the = yı. Although we cannot directly observe the latent reward model r"( y) that underlies these preferences, the Bradley-Terry (BT) model (Bradley & Terry, 1952) offers well-established approach for modeling pairwise comparisons, which is given
exp (r" V1))
p" p1(11 V2(x) exp(r* 91) + exp(r* (z. y2))
(I)
Given the dataset O (a(2,y) y) and estimate the parameters by optimizing the following logistic regression loss:
(xx(1 you 1ii
)
(0) = sampled from p", we can parametrize reward model
CR(Ts, O) --:=---pp (. [log (r yw) - rz( 3)]],
(2)
where (.) is the sigmoid function. As the size of dataset O grows, the empirical distribution of the dataset O converges to the underlying distribution p", and the reward model rs converges the true reward model r".
Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022). The standard RLHF paradigm is composed of three phases: i) supervised fine-tumin tne-tuning, ii) reward modeling, and iii RL fine-tuning, Using the reward model rs learned from the reward modeling, we can then fine-tune the policy by optimizing the following objective:
max Eew.O. yorn- (x)=)) w)] [To (v|=)) Ter(())))))))
In practice, both the language model policy To and the reference policy Ner are typically initialized to the same supervised fine-tuning (SFT) model TSFT. Here, / is parameter that controls the strength of the regularization term, and DKi represents the KL divergence penalty used to regularize the policy my to be close to Tref-
Directed Preference Optimization (DPO) (Rafailov et al., 2023a). DPO offers an alternative approach to the RI. paradigm described above. It establishes functional mapping between the reward model and the optimal policy under a KL divergence constraint with the following formulation:
r(z, v) = B log no(ylæ) + 3 log Z (xx)
where Z(x)= = E, (z)()) exp(r(2,, v)/ is the partition function. By incorporating this reward into the BT model, the DPO objective enables the comparison of response pairs, facilitating the discrimination between preferred and dispreferred actions, given by:
Coro (Tg; Teat) -B(..p...) (3 log [o yue x)
To (g/
x)) x))
3 log
)}
(5)
Tee Tee (Yer
x))
Tee (yr
Distributionally Robust Optimization (DRO) (Namkoong Duchi, 2017: Duchi Namkoong, 2018). DRO provides strategic framework to effectively mitigate the uncertainty inherent in training data. It achieves this by optimizing for the worst-case expected loss across a set of potential distributions Q. These distributions are confined within robustness radius 7 anchored around the empirical training distribution Qu. and are bounded by a prescribed divergence metric Dz. The formal formulation of DRO can be succinctly expressed follows:
LORO 2 maxBg][(r;(0]]
s.t. (Q, Qu) <n.
(6)
where L(x; 8) represents the training loss for an input æ. Intuitively, models employing DRO exhibit increased robustness due to the presence of Q that acts as an "adversary". optimizing the model under distribution set with adversarial perturbations instead of a single training distribution.
3

---

Published as a conference paper at ICLR 2025
3 ANALYZING DPO'S POINTWISE ROBUSTNESS
In this section, we explore DPO's robustness to pointwise noise, analyzing its response to noise to identify key strengths and vulnerabilities. We assess how noise degrades performance and leverage insights from DRO to understand DPO's 's underlying resilience mechanisms.
3.1 POINTWISE NOISE IMPAIRS DPO PERFORMANCE
We begin by investigating the impact of pointwise noise DPO through experiments on the IMDB sentiment dataset (Maas et al., 2011). Following the setup in (Havrilla al., 2023), we fine-tune the GPT-2-large (Radford et al., 2019) model and use SiEBERT (Hart- mann et al., 2023), specialized variant of RoBERTa-large (Liu et al,, 2019), for reward calculation. Pointwise noise is introduced exclusively during the SFT stage by incorporating responses gener- ated by the unrefined GPT-2-large model, resulting in lower quality data for this stage, while the data used in the DPO stage remains unchanged. To assess DPO's robustness to this pointwise noise, we evaluate each algorithm by examining the trade-off between the achieved reward and the KL divergence from the reference policy.
data quality (DPO)
16
KLind Rad
Figure 2: Impact of pointwise noise on the expected reward frontier and KI. divergence in DPO (3 0.1).
Figure 2 reveals that beyond a (T Tar) threshold of 10.0, both models converge in terms of reward. Notably, the DPO model trained with high-quality data (blue points) significantly outperforms its low-quality data counterpart (orange points), highlighting the critical impact of data quality on optimizing model performance.
3.2 POINTWISE ROBUSTNESS REWARD MODELING
In Section 3.1, we explore how pointwise noise negatively affects individual instance rewards. To address this issue and enhance the robustness of LLMs, we propose integrating DRO during the reward modeling stage. We define the Reward Modeling DRO (RM-DRO) objective, which optimizes the expected reward under the worst-case noise distribution within specified ambiguity set:
max Taww:.yreellllll (r, v)] me
s.f. DD (y)ar) ())()))) S n7-
The direct consequence of pointwise noise is the resultant umreliability of the reference model (SFT). By adopting RM-DRO, we aim maximize a surrogate objective that accounts for various potential distributions within robustness radius around the reference distribution (z)()) measured by the distance metric IDg With this formulation, we provide a fresh perspective on DPO.
A. DPO is Implicitly Pointwise DRO.
Theorem 3.1 (Optimal Reward Function under KL. Divergence). Let the Kullback-Leibler (KL) divergence between policy ao and reference policy Tee be defined as: (oo(TT)TTe))
(T( log Ter(s)
<a(a) dx. Optimizing the RM-DRO objective as defined in Equation (7) yields
optimal reward TKI (x, v) given by:
To To(y)x)
(xI v) = (n) log
(8)
Here, a are Lagrange multipliers, (n) denotes the optimal value of 3 that minimizes Equation (7), acting as the regularization coefficient in DPO. By deriving the optimal value of a. given by:
-1 log Ez~O.y=zaf exxp( ro(y)
)
3
Equation 8 can be re-expressed to match the ultimate form of the reward function in Equation 4.
Please refer to Appendix B. for detailed proofs and Appendix B.2 for the formal proof. For a broader discussion optimal reward functions under general o-divergences, see Appendix C.I.
Consistent with the reward function formulation in Rafailov et al. (2023a), Theorem 3.1 not only reaffirms established results but also introduces several novel insights, outlined below:
4

---

Published as a conference paper at ICLR 2025
1.0
nolse ratio-10%
64
noise ratio=20%
0.9-
Win Rate (75)
62
nöise 4atio=40%
noise rat
0.8
nois 4atio=400%
nokse rafio-20%
600
noise ratlo=60%
58
ratio-80%
0.7-
noise 100%==100%
56
0.6
10
10
10
10 10
10"H
10 10
(a) Performance w/ different IMDB.
(b) Performance w/ different B HH.
Figure 3: (a) Comparative analysis of the effect of pointwise noise on the expected reward frontier for different 3 values IMDB dataset. (b) Comparative analysis of the effect of pointwise noise the win rate for different 3 values on HH dataset. The star (+, *, *) indicates the optimal / selection for the corresponding pointwise noise ratio.
Why DPO is Robust to Pointwise Noise. We propose that the reference distribution closely mirrors the empirical training distribution, given the pre-training step (SFT) common to both RLHF and DPO methods. This ensures the reference distribution in the DPO phase accurately reflects the training data noise. In terms of DRO, while the reference model nee may not be entirely reliable, the implicit robust framework of DPO counters data perturbations effectively. Specifically, the "worst-case distribution" defined the distribution that maximizes risk within established divergence constraints, analogous to an adversarial noise model in DRO. Varying 3 enables DPO to exhibit varying search space for better iTy. leading improved performance. For more discussion about the connection between DPO and DRO, please refer to Appendix C.2.
Moreover, the incorporation of DRO provides a new interpretation of the coefficient 3 in DPO, transforming it from mere heuristic design into "noise reflector". We provide Lemma 3.2 to disclose the relationship between 3 and 17.
B. The Optimal Value of 3 Reflects the Noise within the SFT Model.
Lemma 3.2. (Faury et al., 2020, Lemma 5) The optimal (*)) in DPO monotonically decreasing with respect to n7 and obeys the following relationship:
8 (n) = / Vnn v)l/2n.
(10)
where V Ir v)r = E,, Txy(2, y) (r(()=))) E, ( 2))² denotes the variance of the reward model r(x,y) y) under the reference distribution Reef
Lemma 3.2 elucidates the inverse correlation between the parameter 33 and the robustness radius Specifically, noise within the model increases, the required search space expands, necessitating larger and consequently a smaller optimal B.
To empirically validate this relationship, we conducted experiments on the IMDB dataset, as outlined in Section 3.1. In these experiments, the noise ratio is controlled by the proportion of low-quality pairs (ye, 31) introduced into the training data, generated by the unrefined GPT-2 model. Figure 3a shows that models trained with lower 3 values (e.g., 0.01) outperform those with higher 3 values (e.g., 0.1) when trained on 100% low-quality data. This is because a lower 3 allows for a larger search space to counteract significant pointwise noise in the SFT model.
We also conducted experiments on the HH dataset, injecting pointwise noise during the SFT phase by incorporating rejected responses into the training samples. Importantly, during the DPO phase, the positive and negative samples remained consistent, ensuring noise was introduced only during SFT. The noise ratio is determined by the proportion of rejected responses used training samples during SFT. As shown in Figure 3b, the optimal value of 33 decreases as the noise ratio increases, indicating that higher noise levels in SFT require smaller /3 for optimal performance.
For detailed experimental settings and procedures for both datasets, please refer to Appendix C.3, where more comprehensive explanations provided.
5

---

Published as a conference paper at ICLR 2025
1.0
1.0
0.9
0.9
Reward
0.8
Reward
0.8
0.7
0%
Flipped pairs
0.7-
Filpped pairs
10% plipped pairs
0.6
30% Flipped pairs
0.6-
30% Filpped pairs
Pilpped pairs
0.5
40% Filpped pairs
14
10
10
10
Figure 4: Left: Impact of pairwise noise on the expected reward frontier and KL divergence in DPO (3 = 0.1). Right: Comparative analysis of the effect of pairwise noise on the expected reward frontier for different / values.
4 DR. DPO: TOWARD PAIRWISE ROBUSTNESS
In this section, we investigate the impact of pairwise noise and introduce Dr. DPO as a mitigation strategy. We conclude with theoretical examination of its robustness against such noise.
4.1 PAIRWISE NOISE IMPAIRS DPO CONVERGENCE AND PERFORMANCE
We previously explored DPO's pointwise robustness, while recent work (Chowdhury et al., 2024) has examined its resilience to pairwise noise. However, methods that rely on explicit noise estimation may overlook complex noise behaviors. We thus empirically examine how pairwise noise affects DPO's performance. Similar to the experimental settings in Section 3.1, we corrupt the dataset with pairwise
noise by randomly swapping pairs (ywe B1) in the preference dataset O = {a(()) Vie yi Ji=1 As illustrated in Figure (Left), when exposed to elevated levels of pairwise noise (specifically, 10% and 30% label flipping), DPO exhibits diminished rate of increase in (L(TT|| number training epochs (epoch = 1). This pattern is indicative of a decelcrated convergence rate in terms of reward value, especially when compared to the trend of the model trained on noise-free data (0% flipped pairs). Thus the overall performance of DPO drops significantly.
(4) (6)
over an equivalent
Despite its effectiveness in mitigating pointwise noise through the adjustment of (3, DPO still suffers from the issue of pairwise noise. As shown in Figure 4 (Right), the model trained with 40% flipped data exhibits significant performance degradation compared to the model trained with nrise-free data (0% flipped). Even when the model is trained with a lower 3 value, still fails to achieve the same level of performance as the model trained with noise-free data. This observation reveals DPO's vulnerability to pairwise noise, motivating enhancements to its robustness against such interference.
4.2 DISTRIBUTIONALLY ROBUSTIFYING DPO
Building upon the principles DRO, we introduce the Distributionally Robustifying DPO (Dr. DPO) framework, designed to enhance DPO's resilience to pairwise noise while preserving its inherent robustness to pointwise noise. The Dr. DPO objective formulated as follows:
max ta [1( Uw. w)] s.t. (O', O) < nn
(11)
Here, h(,, yoe wi) log (r, Voo) T (2, yt)) denotes the log-likelinn likelihood objective of dataset point. The gidivergeece, denoted as (O', O), quantifies the discrepancy between the hypothetical distribution O' and the dataset distribution O. Additionally, vy' signifies the robustness radius, which quantifies the degree to which the model can withstand perturbations.
Theorem 4.1. Consider the scenario where the KL divergence is employed to measure the discrepancy between the hypothetical distribution O' and dataset distribution O derive the ultimate loss function for Dr. DPO as follows:
hppo x. yw: 31) )].
LDr ( Twd) -3' log B. (exp(
(12)
G'

---

Published as a conference paper at ICLR 2025
where horo represents the log-likelihood in the DPO framework, defined as:
hDpolæ, yw, we) log a (3 log ( xx)
B log (w xx) Ter 2))
(13)
VeelWw x)
with 3 and 3' being regularization coefficient respectively.
Please check Appendix B.3 for detailed proofs. In contrast the objective function described in Equation 7, our Dr. DPO method specifically targets the pairwise noise present within the dataset. Rather than assigning a uniform weight of TV to each instance (xx(1,, Yiu Dr. DPO seeks to reweight these instances by an optimal distribution O'. This approach is driven by the intention capture the incorrect pairs in the data, thereby enhancing the robustness of the resulting policy.
(+) ())) (Rafailov et al., 2023a),
Theorem 4.2 (Upper Bound for Dr. DPO), Let hPpP E [a, b] and L DPO represents the Dr. DPO loss on N samples. Given hypothetical distribution O' satisfying (xa S n/' dataset distribution O we have that with probability at least 1 8:
Lo S L.. DPO B,
(14)
where:
2b 2bexp ((b a)/3") N 1+ + exp a)/3 V
N
B(6 N'))
In
(15)
Please check Appendix B.4 for detailed proofs. In scenarios involving pairwise noise, consider O' as the "ideal" distribution that discerns the correct ranking between pairwise instances accurately. Theorem 4.2 suggests the ideal loss relative O' is upper bounded by the proposed Dr. DPO loss. This bound is achieved when B(,, NN 8¹) approaches zero, or in other words, the number of samples N approaches infinity. Furthermore, this upper bound can offer guidance in real-world applications. For instance, in conjunction with Lemma 3.2, we infer that increasing the robustness radius results in a decrease in 8", consequently increasing (5,5 N, f"). In such case, ensure tight upper bound, it becomes necessary to enlarge the sample size N. This relationship between robustness radius, B", and N provides insights for guiding the training LLM models to achieve desired performance.
4.3 WIYY IS DR. DPO ROBUST TO PAIRWISE NOISE?
Our approach extends the analysis presented in Rafailov et al. (2023a). To understand the resilience of Dr. DPO to pairwise noise, we examine the gradient of its loss function, denoted VaL DPO:
VaL (TT meet) = -9E ww.. w(x, ywe y1) a (fa(2, mn) fo(x, yw:) (V f.y
Von. ),
Boost mismatched pair gradients.
Reduce incorrect pair's impact.
where (z y) = B log )arlyla) represents the reward function implicitly learned by the policy me- relative to a reference policy Tee- In this framework, Vo.v= and Ve.yp are gradients increasing the probability of the "chosen" action yhe and decreasing for the "rejected" action yı, respectively. The factor (f, v)) - fx yw)) serves to amplify the gradient contributions from mismatched action pairs, which is principal aspect of the Dr. DPO's design aimed at enhancing learning from compara-
T0(p)
exp(M(x. 3/33') tive feedback. Conversely, the function w(x, Yar 3t), defined as w(x,, Bw. yr) = Eo (elppe (h. 3737] (ef. Appendix B.5), acts to mitigate the influence of these incorrect pairings. It achieves this by preferentially weighting correct action pairs over incorrect ones, thus refining the policy update mechanism. Moreover, the parameter 3' does not require intensive tuning; setting it to default value of typically yields stable enhancements. Remarkably, Dr. DPO straightforward to implement, requiring only an additional line of code with negligible computational overhead.
Toy Example of How Dr. DPO Works. Consider a training set that contains two samples from both the unperturbed and perturbed datasets. Their corresponding values are assumed to be [2( V1.w: 12,1), h (x2, w2 v2.)] = [-0.1, -1.0]. According to the formula h(x, yw y1 =
log or - -r"). the label flip in the second sample leads to h(#2, V/, V2,) < (00) -0.9331.
(1) In the case of Empirical Risk Minimization (ERM), the sum is: -0.1 + (-1.0) = 1.11
(2) Under Distributionally Robust Optimization (DRO), when 3' 0.1, have the weights
7

---

Published as a conference paper at ICLR 2025
Table 1: Preference accuracy and win-rate comparison on the Anthropic HH dataset with various levels of noise. The columns indicate the performance on both the noise-free dataset and the datasets with inverted response labels, denoted by their respective flip ratios. The performance improvements of cDPO, IPO, and Dr. DPO over DPO are also presented.
Preference Accuracy 20% Flipped
Win Rate
Models
0% Flipped
10% Flipped
30% Flipped
40% Flipped
0% Flipped
40% Flipped
DPO
63.63
62.27
61.28
58.54
55.23
54.36
49.00
<DPO
63.48
48.28--.25% 0.25%
62.67 6..64466
61.48 8.88
58.69 69-00.2%
55.31 -..15%
51.46
61.46655.88% 5.33%
45.68 6.78%
IPO
65.95+
65.95
-3.64
61.8224411%
63.52
61.45
4.9%%
56.64
2.65%
50.81 -6533
54.82 54.82+11.88
112-11.88%
+DPO
64.37
62.72
62.72+0.72%
62.53-2.045
62.53
22004%
60.56
3.43%
57.11
-3.40%
52.15
4..68%
53.54+9 9.226%
Dr. DPO
66.22 +- 4.00%
65.38+5.00%
65.38
+5.00%
64.19 +4.49%
62.65+7.022%
62.65
+7.02%
58.83+66.558
58.83
+6.52%
56.67
+1.26%%
61.65 +25.81%
[2.0, 0.0]. Therefore, the outcome is: (-0.1 2.0) (- 0.0) = 0.2, which is greater than 1.11. w(x 3) places more emphasis on the unperturbed sample,
[ wr M1,1), 2(22 32.w: V2.1)]
making the weight allocation more reasonable and also qualifying as a worst-case distribution.
5 EXPERIMENTS
In this section, we conduct an empirical assessment of Dr. DPO to evaluate its ability to mitigate noise impacts in preference datasets and to improve performance in noise-free environments. We outline our experimental design, including the datasets used, evaluation metrics, and comparative benchmarks. Our results underscore Dr. DPO's effectiveness, supporting its utility in relevant scenarios.
5.1 How WELL. CAN DR. DPO RESIST THE PAIRWISE NOISE?
Datasets and Setup. We conduct experiments on two datasets: IMDB (Maas al., 2011) and Anthropic HH (Bai et al., 2022). The IMDB dataset is widely utilized for sentiment analysis tasks. The Anthropic HH dataset consists of approximately 170,000 dialogues between humans and automated assistants. The objectives of these experiments were twofold: firstly, to evaluate the robustness of the proposed Dr. DPO against pairwise noise; and secondly, to investigate whether Dr. DPO exhibits superior performance on poise-free datasets. To achieve the first objective, we introduce random inversions between selected and rejected responses in the training data at varying noise levels--seciffcallly with probabilities of 10%, 20%, 30%, and 40%. For the second objective, we benchmark Dr. DPO against other DPO-related baselines to discern its relative advantages. Unless stated otherwise, all experiments in this study were conducted using the Pythia-2.83 model (Biderman et al., 2023). Please refer to Appendix for more details.
Baselines. We compare Dr. DPO with four baseline methods: (i) The standard DPO, which is the of-t-of-thh-art (SOTA) method for directly optimizing the policy and reward model; (ii) Conservative DPO (cDPO (Rafailov et al., 2023b)), variant introduced by the authors of DPO address scenarios with probabilistically flipped labels by incorporating a binary cross-entropy (BCE) loss; and (iii) IPO (Azar al., 2023), innovative approach that enables learning directly from preferences, bypassing both the reward modeling phase and reliance on the BT model. (iv) rDPO (Chowdhury et al., 2024) a variant of DPO that diabis the effect of preference noise and makes the policy robust.
Metrics. We adopt two metrics, Preference Accuracy, and Win-Rate, in the experiments. The Preference Accuracy measures the proportion of test instances from the Anthropic HH dataset where the model's predicted reward for the preferred response, r(x:, Vw). exceeds that of the less preferred alternative, r(z, 31). We further draw on the evaluation framework introduced by Rafailov al. (2023a) to calculate the Win Rate. This metric assesses how often the GPT-4 model selects a response generated by our model rather than the chosen response within the dataset. The Win-Rate computation specifically designed for the single-turn dialogue portion of HH dataset's test subset.
Dr. DPO Outperforms Across Noise Levels. Table I illustrates the effect of noise on preference accuracy, showing a decrease from 63.63% to 55.23% the label flip ratio rises from 0% to 40%. This indicates a deteriorated ability to distinguish between responses with increasing noise. In a noise-free environment, cDPO achieves preference accuracy of 63.48%, which falls to 55.31% % at a 40% noise level, highlighting the limited impact of BCE loss adjustments. In comparison, the IPO method consistently outperforms DPO by an average of 3.78% in accuracy, proving its efficacy.
8

---

Published as a conference paper at ICLR 2025
0% Flipped
40% Flipped
12 8%
6200
:0FO
5.63%
8.13%
0.55
0.54
9.50
Dr. 500 (1-
Wia Rata
D70
180
68.8%
14.4%
vs DPG
I80
76.0%
12.5%
0.60
Oh Flooo
9.50
D:: DPO
19 D20
70.4%
Dr. DRO
va DPG
71 116
0/45-
0.56
Papel
Lom 150
64
2-03
D-0.7
3.0
2.0
3.5
50
112 21
Lows
102
Divergenoe
Figure 5: MT-Bench evaluates DPO and its variants using GPT-4 showing Win, Tie, and Loss rates at 0% and 40% pairwise noise levels in Figures 1-2. Figure 3 illustrates Dr. DPO's win rate across different g-divergences, while Figure 4 presents its Preference accuracy for varying 3 values.
10% Flipped
Flipped
30% Pllpped
Fliyped
Proference Accuracy
0.02
8.30-
0.50
0.44
IP0
0.54
0.54
0.54
0.5
25
Training Progress (96)
Trainling Progiress (%6)
Tralning Prograss (56)
Training Progreas (96
Figure 6: Preference accuracy across training steps for different levels of pairwise noise.
rDPO reports a 3% accuracy gain over DPO. Notably, Dr. DPO exhibits outstanding noise resistance, achieving the highest preference accuracies of 66.22% in the absence of noise and 58.83% with 40% noise, superior to DPO, cDPO, IPO, and rDPO.
Dr. DPO Surpasses Other Methods. We evaluated the response quality of various models DPO, cDPO, IPO, rDPO, and Dr. DPO-across across different noise levels. Table shows that in a noiseless setting, DPO leads with a 56.67% win rate. However, as noise increases, both DPO and cDPO's performance declines, whereas Dr. DPO excels, achieving 61.65% win rate.
Dr. DPO's Performance the 0% Flipped Case. The 0% flipped case indicates no intentional label flips were introduced, representing the original dataset. Nevertheless, this does not mean the dataset free of label noise. Dr. DPO continues to outperform DPO and IPO in this scenario, suggesting the presence of inherent label noise in the dataset. This observation aligns with rDPO's findings (Chowdhury et al., 2024), where a default flip rate of 0.1 improved performance on the HH dataset, further justifying the design of Dr. DPO.
5.2 COMPARING DR. DPO WITH BASELINES ON MT-BENCH
To evaluate the generation quality of DPO and its variants, we conduct pairwise comparisons using the MT-Bench framework (Zheng et al., 2023). This framework, grounded in GPT-4, reliably aligns with human evaluative preferences, exhibiting an agreement rate exceeding 80% on the quality of outputs from LLMs. Adhering to the established MT-Bench guidelines (Zheng et al., 2023) 2, our approach involves generating model responses at a controlled temperature of 0.7 and restricting the token count to maximum of 1024. We systematically compare the outputs from the base DPO model and its variants, which have been fine-tuned with 0% and 40% flipped pairs on the HH dataset.
Figure 5 (1,2) shows that Dr. DPO consistently outperforms DPO in both noise-free and noisy datasets, becoming the only method to exceed DPO's performance in the MT-Bench evaluation. While IPO slightly improves over DPO in the noise-free dataset, it underperforms in the noisy dataset where DPO prevails. In contrast, Dr. DPO demonstrates robust, significant enhancements in both conditions, highlighting its superior ability generate high-quality responses.
5.3 ABLATION STUDIES ON DR. DPO
We conduct ablation studies to investigate the impact of the d-divergence and 3 on the performance of Dr. DPO, and provide the convergence analysis. More ablation studies listed in Appendix D.
ht com http::////aaaaaaaooooooo/////aaa http://tt.bbaaaaaaaaaaaaaaaaaaae wtwpiihatttinnnnnnneeeeee gee__ model Lodel_answer.pp

---

Published as a conference paper at ICLR 2025
Table 2: Comparison of KL Divergence and Win Table 3: Comparison of Model Performance via
Rate of Model Performance at Different Noise Ratios Win Rate on the LLMA2213BB
Noise Ratio
Loss Function
Win Rate
KL
Loss Function
Win Rate
0%
DPO
54.36
19.68
0.1
DPO
59.05
Dr.DPO
56.67
20.01
0.1
cDPO
53.52
Llama2-135 1-
DPO
49.00
8.32
rDPO
60.54
40 %
Dr.DPO
61.65
7.96
0.1
Dr.POO
63.67
Evaluating the Impact of ø-divergence Dr. DPO. Figure 5 (3) explores Dr. DPO's performance with various ø-divergences, including Jensen-Shannon (JS) and a-divergence. Demonstrated results indicate that Dr. DPO consistently outperforms the baseline DPO when 3 is a viable default without requiring further adjustments. This is in contrast to the baseline / = 0.1 setting, where although improvements can be realized by manually tuning 3*, the process becomes lime-consuming and impractical for regular use.
1.0, serving as
Evaluating the Impact of B'. Figure 5 (4) illustrates how varying 3' across different noise levels affects preference accuracy. The experimental results reveal a trend wherein increased noise levels correspond reduced optimal value for 3", which is consistent with our theoretical analysis provided in Section 3. Consequently, we propose a default setting of 3' = 1.0 for balancing accuracy and robustness the presence of noise.
Convergence Analysis. Figure 6 shows that Dr. DPO not only converges faster but also surpasses DPO in the early training stage, attributed to its superior management of flipped noisy pairs. This enhancement meets our goal of boosting DPO's robustness in noisy environments.
Dr. DPO does not increase KL divergence. A potential concern is that Dr. DPO might improve performance by increasing the KL divergence. To address this, we computed the KL divergence of models trained under two noise ratios: 0% and 40%. As shown in Table 2. the KL divergence does not increase with DRDPO for the same 8 settings. This comparison demonstrates that DRDPO maintains a stable KL divergence, effectively mitigating the concern that performance gains are achieved through increased divergence.
Comparing Dr. DPO with Baselines on LLaMA2-13B, We further evaluate the performance of Dr. DPO the LLaMA2-13B dataset, which is a large-scale language model. Table shows that Dr. DPO outperforms DPO, cDPO, and rDPO, achieving a win rate of 63.67% compared to 59.05% for DPO, 53.52% for eDPO, and 60.54% for rDPO. These results demonstrate the superior performance of Dr. DPO in generating high high-quality responses on the LLaMA2- 13B dataset.
DISCUSSION
Conclusion. In this study, we analyze DPO's robustness from a DRO perspective, highlighting its resilience pointwise noise. We establish link between DPO's regularization and DRO's robustness, showing that a smaller regularization parameter 3 enhances stability against uncertain data. Our experiments confirm the crucial role of 99 in noise resistance but uncover DPO's weakness against pairwise noise. To address this, we introduce novel Distributionally Robustifying DPO framework with an additional parameter 3' that balances data pair importance in training to enhance model robustness. The Dr. DPO's fine-tuning of exploration and exploitation could markedly improve the alignment of language models, assuring reliable performance in the presence of real-world noise.
Limitations and future work. The current work introduces Dr. DPO, an enhancement to DPO that addresses label flipping noise in training datasets through an additional hyperparameter H'. Despite the robust performance indicated by empirical results with default 3' value of 1.0, the need for parameter tuning in different applications remains. The sensitivity of 3' to data and task specifics may require search process to fully leverage Dr. DPO's potential. Additionally, we aim to explore adaptive mechanisms that allow Dr. DPO to adjust robustness dynamically during training, reducing the need for manual parameter tning and enhancing its practical applicability across diverse tasks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research is supported by the National Science and Technology Major Project (20252D0121102), National Natural Science Foundation of China (92270114, 14, U24B20180 62121002, 62302321). This research was also supported by the advanced computing resourees provided by the Supercomputing Center of the USTC.
REFERENCES
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez. Misha Khalman, Jakub Sygnowski, and al. Gemini: A family of highly capable multimodal models. CoRR, abs/2312. 1805, 2023.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. CoRR, 0../2310.12033 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer EI Showk. Nelson Elhage. Zac Saffield-Dodde Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback. CoRR, abs/2204.05862. 2022.
Aharon Ben-Tal and Mare Teboulle. An old-new concept of convex risk measures: the optimized certainty equivalent. Mathematical Finance, 2007.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In ICML, 2023.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-35,, 1952.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4. CoRR, abs/2303. 12712, 2023.
Xi Chen, Simai He, Bo Jiang, Christopher Thomas Ryan, and Teng Zhang. The discrete moment problem with nonconvex shape constraints. Oper. Res., 69(1):77--296, 2021.
Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust DPO: aligning language models with noisy feedback. In /CML, 2024.
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martie, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In NeurlPS, 2017.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao. Wei Zhu, Yuan Ni. Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ulrafeedback: Boosting language models with high-quality feedback. CoRR, abs/2310.01377, 2023.
11

---

Published as a conference paper at ICLR 2025
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. RAFT: reward ranked finetuning for generative foundation model alignment. CoRR, abs/2304.06767, 2023.
John C. Duchi and Hongseok Namkoong. Learning models with uniform performance via distribu- tionally robust optimization. CoRR, abs//8800007555 2018.
Louis Faury, Ugo Tanielian, Elvis Dohmatob, Elena Smirnova, and Flavian Vasile. Distributionally robust counterfactual risk minimization. In AAAI, 2020.
Suriya Gunasekar, Yi Zhang. Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giomo, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need. CoRR, abs/2306. 1644, 2023.
Jochen Hartmann, Mark Heitmann, Christian Siebert, and Christina Schamp. More than a feeling: Accuracy and application of sentiment analysis. International Journal of Research in Marketing. 2023.
Alexander Havrilla, Maksym Zhuravinskyi, Duy Phung, Aman Tiwari, Jonathan Tow, Stella Biderman, Quentin Anthony, and Louis Castricato. trix: A framework for large scale reinforcement learning from human feedback. In EMNLP, 2023.
Jean-Baptiste Hiriart-Urrut and Claude Lemaréchal. Fundamentals of convex analysis. Springer Science & Business Media, 2004.
Ruomin Huang, Jiawei Huang, Wenjie Liu, and Hu Ding. Coresets for wasserstein distributionally robust optimization problems. In NeurIPS, 2022.
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. Camels in a changing climate: Enhancing LM adaptation with tulu CoRR, abs/2311. 2023.
Binbin Jin, Defu Lian, Zheng Liu, Qi Liu, Jianhui Ma, Xing Xie, and Enhong Chen. Sampling- decomposable generative adversarial recommender. NeurIPS, 2020.
Henry Lam, Zhenyuan Liu, and Xinyu Zhang. Orthounimodal distributionally robust optimiza- tion: Representation, computation and multivariate extreme event applications. arXiv preprint arXiv:2111.07894, 2021.
Defu Lian, Qi Liu, and Enhong Chen. Personalized ranking with importance sampling. In WWW, 2020.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. CoRR, abs/2309.06657, 2023.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint
2019.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. ACL, 2011.
Paul Michel, Tatsunori Hashimoto, and Graham Neubig. Modeling the second player in distribution- ally robust optimization. In ICLR, 2021.
Paul Michel, Tatsunori Hashimoto, and Graham Neubig. Distributionally robust models with parametric likelihood ratios. In ICLR, 2022.
Hongseok Namkoong and John C. Duchi. Variance-based regularization with convex objectives. In NeurIPS, 2017.
XuanLong Nguyen, Martin J. Wainwright, and Michael I. Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEFE Trans. Inf. Theory, 2010.
12

---

Published as a conference paper at ICLR 2025
OpenAl. GPT-4 technical report. CoRR, abs/2303.08774, 2023.
Yonatan Oren, Shiori Sagawa, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust language modeling. In EMNLP/IJCNLP (1), pp. 4226-4236. Association for Computational Linguisties, 2019.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurIPS, 2022.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In NeurlPS, 2023a.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. A note on dpo with noisy preferences and relationship ipo. 2023b. URL ericmitchen se1/cdpo... pdf.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs//707 .6346, 2017.
Soroosh Shafiezzadeh-Abadee Peyman Mohajerin Esfahani, and Daniel Kuhn. Distributionally robust logistic regression. In NeurlPS, 2015.
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Mi- randa Zhang, and Ethan Perez. Towards understanding sycophancy in language models. CoRR, abs/2310. 13448, 2023.
Aman Sinha, Hongseok Namkoong, and John C. Duchi. Certifying some distributional robustness with principled adversarial training. In ICLR, 2018.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne I Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307. 092888, 2023.
Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse KL: generaliz- ing direct preference optimization with diverse divergence constraints. ICLR, 2024.
Chenwang Wu, Defu Lian, Yong Ge, Zhihao Zhu, and Enhong Chen. Influence-driven data poisoning for robust recommender systems. TPAMI, 2023.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. RRHF: rank responses to align language models with human feedback without tears. In NewrIPS, 2023.
Runtian Zhai, Chen Dan, J. Zico Kolter, and Pradeep Ravikumar. DORO: distributional and outlier robust optimization. In ICML, 2021.
13

---

Published as a conference paper at ICLR 2025
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Salch, and Peter J. Liu. Slic-hf: Sequence likelihood calibration with human feedback. CoRR, abs/2305.10425, 2023.
Lianmin Zheng. Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging ammaee--judge with mt-bench and chatbot arena. CoRR, Abss/2306.05685, 2023.
Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, and Chenguang Zhu. WPO: enhancing RLHF with weighted preference optimiza- tion. In EMNLP, pp. 8328-8340. Association for Computational Linguistics, 2024.
Dixian Zhu, Yiming Ying, and Tianbao Yang. Label distributionally robust losses for multi-class classification: Consistency, robustness and adaptivity. In ICML, 2023.
A RELATED WORK
Reinforcement Learning from Human Feedback. RLHF (Christiano al., 2017; Bai et al., 2022; Touvron et al., 2023; Ouyang al., 2022) has emerged as a key method for aligning language models with human values and preferences, mitigating the generation of biased or factually incorrect outputs. Compared supervised learning, RLHF is rather complex, less stable, and requires more memory resources. These challenges have motivated the development of alternatives to the RLHF pipeline. For example, RAFT (Dong et al., 2023) uses existing reward model to select the best set of training samples based on the model outputs, while RRHF (Yuan et al., 2023) leverages a much simpler ranking loss to align human preferences and retain the performance of PPO. DPO (Rafailov et al., 2023a) is another alternative to RLHF that uses a preference loss function to directly optimize the LLMs, and has been shown be more stable and less computationally intensive than RLHF. Despite these efforts, all the methods ignore the noise in the training data, which can lead to suboptimal performance. In recent advancement, WPO has been introduced as novel strategy to simulate on on-policy learning with off off-policy preference data, effectively addressing the distributional gap problem and enhancing the optimization process without incurring additional costs (Zhou et al., 2024).
Distributionally Robust Optimization. DRO differs from traditional robust optimization methods (Jin et al., 2020; Lian et al., 2020; Wu et al., 2023) by minimizing the worst-case error within an uncertainty set defined by constraints like d-divergence (Namkoong & Duchi, 2017; Duchi & Namkoong, 2018), Wasserstein distance (Shaficzzadeh-Abatee
et al., 2015; Sinha al., 2018; Huang et al., 2022), and shape (Lam et al., 2021; Chen et al., 2021). (Michel et al., 2021; 2022) introduced parametrization the uncertainty set for greater architectural flexibility. Separately, (Zhai al., 2021) addressed sensitivity to outliers in DRO, diverging from the other studies. And (Oren et al., 2019) introduced novel approach language modeling that enhances robustness by optimizing against the worst-case topic mixture, aligning with the overarching theme of minimizing maximum losses within predefined uncertainty sets.
B APPENDIX OF PROOFS
B. PROOF OF THEOREM 3.1
Theorem 3.1 (Optimal Reward Function under KL Divergence). Let the Kullback-Leibler (KL) divergence between policy ao and reference policy Tef be defined as: Dxa(Tol
I (T(()) log Turla))
(a(w)) dx. Optimizing the RM-DRO objective as defined in Equation (7) yields
optimal reward Fxu
(x. v) given by:
To(y
(KI( v) (n) log
(
TT)((g
Here, 3 are Lagrange multipliers, (*) denotes the optimal value of B that minimizes Equation (7), acting as the regularization coefficient in DPO. By deriving the optimal value of a, given by:
To(y (y(r)
a= = =-B Ex yrrke lexp(
}
(9)
1
14

---

(20) Published as a conference paper at ICLR 2025
Equation 8 can be re-expressed to match the ultimate form of the reward function in Equation 4.
Proof.
Definition B.1 (d-divergence (Nguyen et al., 2010)). For any convex function with (11) the d-divergence between Q and Qu is:
TT
(TO; Tret)
(16)
Wyre
where Do(Q,Q0) = 00 if Q is not absolutely continuous with respect to Qo- Specially, when <(x) = x log x - x: + o-divergence degenerates to the well-known KL divergence.
Definition B.2 (Convex conjugate (Hiriart-Unie Urruty & Lemaréchal, 2004)), We consider pair (A, B) of topological vector spaces and a bilinear form (..-) + R such that (A, B, (..-)) form a dual pair. For convex function / : R IR, domB {i IR : [(x) 6e) the effective domain of f. The convex conjugate, also known as the Legendre- transform, of f - IR is the function
f : B + R defined as
(b) = sup(ab-f(a)) f(a)}, bEB
(17)
Theorem B.3 (Interchange of minimization and integration Ben-Tal& Teboulle, 2007)). Let (Ω, F) be a measurable space equipped with o-algebra F. LP F, P) be the linear space of measurable real valued functions I + with II/llpp < so, and let X IP F, P), p E [1, +00). R x Ω2 + R be normal integrand, and define on X. Then,
Let
min / s(x(()).)) = min ((s ((s SER
(18)
ZEX
To ease the derivation, we denote the likelihood ratio L(y|z) d-divergence between and Tre is constrained, and thus L(.) is well-defined. For brevity, we usually short L(ylxz) as L. And in terms of Definition B.1 of d--divergence, the expression of RM-DRO (cf. Equation equation 7) becomes:
Note that the
RM--RR max B. (ro
s.t. ( S
(19)
Note that Em, ()) and E.... [+((()]z)) are both convex in L. We use the Lagrangian function solver:
ce RM = min max (Ex~ In ( -
-Bll:>>))BBB__aaaa 1l - 7) + a [L(v|=]] 1)}
$20,a L(v(z)
ro ro(y)a)) a L(yl - <((()))))1
{1 a max (ExnDyynee a20,8
min
-(
ro x2) + L(ylz) +((y)|)))1}
min {Bn a + BE $20.a
max L(g
B
To
(x) + +
min { - a + BEy $20,a
10
B
The first equality holds due to the strong duality (Boyd & Vandenberghe, 2004). The second equality is re-arrangement for optimizing Lyy ar). The third equation follows by the Theorem B.3. The last equality is established based on the definition of convex conjugate B.2. When we choose KL- divergence, we have ΦKL( x log x + 1. It can be deduced that && (x) e - 1. Then, have:
(o(y) +
CKL CRM-DRO = min {3n- - a + F. an ( $20,a
}}}
3
(21)
To To x)) +
min {3n- a 3E u (exp( $20.a
11}
and the maximum of term L(y|z) in Equation equation 20 is achieved when
ro +
L(y|x) exp( B
(22)
15

---

Published as a conference paper at ICLR 2025
We differentiate the Lagrangian function w.r.f. and set it to zero:
da (Bn + BBB.DD y. lexp(
re re(y)
+
-11 =
(23)
B
Then, we have: o" = - log BawD.yra
ra (y)(r)
exp(
)]
(24)
Substituting the optimal a" into the Lagrangian function equation 21, we have:
re (y/a) a)) a - 1]}
CKL DRO
min (3y - a + F. eep( $20,a
3
To (y((r)
min {Bn log B. eexp( /20
}}}
(25)
B
PDPo (2. y)
= 3" (n) log E.wD. (exp(
) + C,
(*)
where (1) signifies the optimal value of 3 that minimizes the Lagrangian function and C' =31]
Besides, if we plug the optimal a² into the optimal Lyy(x), have:
T' ( (r) +
(y)a)) = exp(
(26)
To (r))
exp(
g
((x)
EznD,y D.ym (exp( [o(yl=) ]]. Here, rearrange Equation equation 26 and obtain the expression of rkL (x, y):
where ((x)
"x.(2,,) v) 8* log I.- (y)x) + B log ((x) 3 log
+ 3 log ((x)
(27)
The theorem is proven. In comparison to the proofs presented in DPO (Rafailov et al., 2023a), our proof comprehensive and direct, applicable any d-divergence constraints in the general PPO objective. DPO (Rafailov et al., 2023a) represents a specific case that employs strategies to construct an objective in the form of KL-divergence.
16

---

Published as a conference paper at ICLR 2025
B.2
FORMAL PROOF
Step 1: Definition of o-divergence
The d-divergence between and Tief is defined as:
To (1)() [re((y) (2) )
(T( Feet) = Eroe
where Φ is convex function with ((1) = 0. For KL-diiergence,, x(t) = flog - t 1.
Step Reformulating the RM-DRO Objective Using the Lagrangian dual form incorporate
the robustness constraint, the RM-DRO objective becomes:
min max (ro 820,0 L(v(z)
[[[()=) + ( - 11)
where 8, a are the Lagrange multipliers associated with the robustness constraint, L(y|z) = (
Step 3: Solving for Optimal Policy function with Kl.-divergence yields:
According to the proof of Theorem 3.1, solving this Lagrange
To(()
(a 8) -llog
a.
(rr((y/z)
where is normalization constant ensuring that wo is valid probability distribution:
( 8)
-3 log By
exp
Step 4: Connection to DPO The reward function derived above matches the closed-form expression for the reward function in DPO:
To(() ())()))
r(z, y) = 3 log
+ 3 log Z (xx,
where ((x) the partition function defined as:
(r(x,)) v)
((x) Σ mer(y)a exp 3
Step 5: Robustness to Pointwise Noise The DRO formulation inherently considers the worst-case distribution within the robustness radius n. By solving the constrained optimization problem under Kl--diergence, DPO implicitly optimizes for the worst-case distributional perturbation. Thus, DPO achieves robustness to pointwise noise, as the DRO mechanism mitigates the impact of noisy data points by optimizing over a family of perturbed distributions.
By aligning the RM-DRO objective with the reward function formulation DPO, we establish that DPO implicitly operates pointwise DRO framework under KL-divergence, This demonstrates its inherent robustness to pointwise noise.
17

---

Published as a conference paper at ICLR 2025
B.3 PROOF OF THEOREM 4.1
Theorem 4.1. Consider the scenario where the KL divergence is employed to measure the discrepancy
between the hypothetical distribution O' and dataset distribution O we derive the ultimate loss function for Dr. DPO as follows:
LDr (To Try) 3' log Re (exp( ( Moo W1) }}-
(12)
g'
where hDpo represents the log-likelin log-likelihood in the DPO framework, defined as:
hpro(z, yu. vi) log (3 log TT xx)
log (a( (ed x)) (ir x))
(13)
with 3 and 3' being regularization coefficient respectively.
To solve the optimization problem in Equation equation 11, we first introduce the Lagrangian function:
Proof.
010',3', a') Ex 31) )~ [1(,,, ywe w:)
+ B'D'D 0) r/) + a (Bo O'
(28)
1).
Then, we can obtain the optimal distribution O'* * by solving the following saddle-point problem:
O arg max min O 8" a').
(29)
O'
D'
Specifically, when the KL divergence is selected the measure of o-divergence, that KL(O', ΣN Oo the optimal distribution Oki can be derived as follows:
(1x Yur U1)
O'i = Z exp
(30)
8
where Z = [z (e (x,, yw B)) denotes the partition function. In this case, we can derive closed-form expression of the ultimate objective O KL. X') follows:
((2, Dwn 31),
O kL B¹) 33' log Eo (exp
(31)
3'
In order to attain a Distributionally Robustifying DRO objective that encompasses both pointwise and pairwise robustness, we consider the previously established fact that the DPO approach confers pointwise robustness. Consequently, by substituting the term (ax, Nor Vt) in Equation equation 3 with the DPO objective from Equation equation 5, we can derive a comprehensive objective that integrates the strengths of both methods:
CDr (TO Tret) = -8' log Eooeex((Dpro(o Bwr 8')]
(32)
Here AprO = log (31 log
no(ywlx) malgu (x) - 3 log (o(wi denotes the optimal policy using in DPO.
18

---

Published as a conference paper at ICLR 2025
B.4 PROOF OF THEOREM 4.2
Theorem 4.2 (Upper Bound for Dr. DPO), Let InPP E (a, bj and L DPO represents the Dr. DPO loss on N samples. Given hypothetical distribution O' satisfying Dxi(O', ,O',O) S n/ dataset distribution O we have that with probability at least 8:
Lo S Lp. DPO N, 8'),
(14)
where:
2b exexp (b /)/3") - exp ((b /)/31) V
N 1
B(6,N,8))
In 8"
(15)
Proof. Firstly, we assume that the optimal policy O' satisfies the following constraint:
O' {Q DKI (0', O) S n
(33)
Under this assumption, the loss function Lo can be bounded as:
Lor = Boy [h( yw: sn)
max Eor [h(2, Mi 31)] (N.(0'00)S
h(,,, Vw y1
(34)
B' log Eo
exp
Dx. DPO
We now introduce McDiarmid's inequality foundational result:
Theorem B.4 (McDiarmid's Inequality). Let X1,..,, X be set of N 2 independent random variables and assume that there exists 01, ... CN such that f XNN R satisfies:
If(. Di, .... ØN) - fI1,,... L,,..,, IN)I S Cs. For all E 1.N and any points 21. II.,..IN,2 2t E X. Let f(S) denote f(X1,..., XN), for all c <> 0, the following inequalities hold:
(35)
22
P[f(S) - E[f(s)) 2 S exp
(36)
ΣN cg
Given dataset with N samples, for any pair of samples: (x,, yoor m)), (x', V'n 1/), have: wxx Mws y1) (n)(((2; yes yt)) w(a', Var (/) (/., v\))
2 sup w(x ywu yt h (z, yw 3r))
O
(37)
2b exp ((b a)/B')
N + exp ((b 33)
where the second inequality holds as w(x, Mwr 31)
exp(h(2.pw...1/8)
and AIPPO E [a, b).
to exp(h(
By applying McDiarmid's inequality, we obtain:
28 NN-12 - -
exp ((b (0000-a889) 2
(EDz DPO C.. DPO E) exp
2bexp ((b 03/ (10-))1)))
(38)
N
Setting:
20a N
N - 1 + exp ((b
a)/8") exp((b-c)18))
= exp
(39)
2bexp ((b (b-o)/3)
we can solve for e as:
2b exp ((b a)/3")
N
N - exp ((b - a)/3')
V 2 J
(40)
Thus, for any 8 E (0, 1), we conclude that with probability at least - d:
2b 2bexp ((b a)/3')
IN
1
Lo z CDr DPO
C... DPO + N N- exp ((b a)/
In
5
(41)
19

---

Published as a conference paper at ICLR 2025
B.5 PROOF OF w(2. Ywr Mt)
In this section, we present the derivation of the gradient for the Dr. DPO objective function as follows:
TIDPO (2t, Mery Vt) ]
V DD (T0 Wret)
Vo3 3.33 log Bo exp
(42)
The right-hand side of Equation equation 42 can be rewritten as:
NIDPO Your yt) ]
Vo log Eo
exp
g
(43)
hNDPO (2t, Merr 311)
-Vheoo (2
V
log Eo
exp
PohpPo-
Br
Considering the gradient with respect to the function ippo (z, Ows u1) yields:
knm 2igten
8
/nPP (at, you ut)
)[
exp
(a
log Eo
exp
(44)
3'
Eo
exp
Appo (2. (a.
Focusing the gradient with respect to 0, we have:
To To x)) Ke (yol)))
To(2)(z)) (x) net (1 (x) (u)
d'u))
VoliDRO = Vo logo (Blgg log
3 log
yuu
(45)
where 3 log Norlsw (x)
Korlsn (ar) Leveraging the properties of the sigmoid function, where a'(x) = o (1 - (xr)) and o' o'(-x) = (w)), we derive the final gradient expression:
[0(wu|z) 3 Blog
ne (x)
(DDPO Mrr M1) }
v log Eo exp
gr
Aueo (ww.23
exp
Gr
To(yi (x) (ullz))
To(ywl
(46)
Bo
(Blog
Blog
E
exp
Aneo (ar
Tret
Tedt (Bwe xx)
Ve log Te o (vwlzz) Ve log [[(]]==]
Here, a crucial indicator that distinguishes Dr. DPO from traditional DPO is encapsulated by the weight term:
Auro (ho
exp
w(r, yhe, 31)
(47)
Eo
exp
Apeo (at, (t. u0)
which gravitates towards a uniform distribution as the parameter 3 approaches infinity. In such scenario, the gradient of Dr. DPO aligns with that of the standard DPO. This relationship furnishes deeper insight into how Dr. DPO can be linked and differentiated from DPO through the incorporation of dynamic tuning parameter 3', enhancing the adaptability of policy optimization in varied environments.
20

---

Published as a conference paper at ICLR 2025
C ANALYSIS
C.I ANALYSIS ABOUT GENERAL G-DIVERGENCE.
Lemma C.1 (Optimal Reward Function under General (-Divergence). (Wang al., 2024, Theorem 1) Given a o-divergence D& with corresponding derivative ø', the optimal reward function T under the RM-DRO framework is defined by:
To(ı,y) = (1)) To To(y|x), a,
(48)
TT(()/z)
where o is Lagrange multiplier.
Proof. To determine the optimal expression for (r, y). one must identify the optimal L(y(z). As
established in Theorem 3.1, the optimal L(yl [x) is given by:
ro (o (.) + L(yla) -
3 arg max E. Myyla)
B
ro(u)x) + L(glz)) - (((n)))(((()x))) 6(E())=))
D.
arg max L(yix)
(49)
B
(o(()2)) + +
-BEg-D.ye Dyy
To find this maximum, we differentiate the convex function w.r.t L(ylæ) and equate the derivative to zero:
a ro ro(yya) + L(yl=) -())))(((((())))))))----)
(50)
aL
Solving for ro(y)æ) yields the optimal expression:
ro(ylæ) 36' (I(y)=)) a.
(51)
Given that is a constant, the critical component of the expression is (Lyl))) While this result aligns with Theorem 1 from (Wang et al., 2024), our approach is grounded in a comprehensive DRO framework, providing more direct and complete theoretical justification. Thus, the lemma is substantiated.
Comparison with Wang et al. (2024). Since a is a constant that does not affect the optimization process, Lemma C.1 reveals that the reward function r(x. is influenced not only by the choice of ø-divergence but also by the parameter B. This observation suggests an intuitive understanding that various d-divergences enforce constraints with unique geometric characteristics, which, in turn, dictate the optimal value of (T)) within DRO. Interestingly, our findings contradict the conclusions presented in Wang al. (2024), which suggest that various ø-divergences lead to different alignment accuracy. Instead, our results demonstrate that by fine-tuning the parameter 3, comparable performance can achieved across different divergences (ef. Table 4). This underscores the critical role of the robust radius in determining the efficacy of DRO frameworks. For detailed experimental settings, please refer to Section 5.1. Moreover, a thorough analysis of the behavior of diverse g-divergences can be found in Appendix D.
We compute the gradients
=e(ual)) and the rejected ratios, the chosen likelihood ratios, (x)
T=(y)l)) As depicted in Figure the choice of @-divergence influences the gradient update rules Tar(sn [)
Table 4: Comparison of win rates across various d-divergences with adjustments to 3.
é-divergence
win rate
win rate
B
KL
54.36
0.1
55.40
0.15
JSD
54.36
0.1
54.75
5e-2
45.02
0.1
54.59
le-4
0.5
44.06
0.1
56.60
lc-6
0.7
41.17
0.1
58.45
1e-6
21

---

Published as a conference paper at ICLR 2025
in a consistent manner, though the performance varies with the scaling parameter B. With an optimal selection 3, the disparities between the performances different é-divergences diminish significantly (refer to Table 4 for a comparative analysis).
Gradient chosen logratios
Gradient lograttoos
K1.66 0.1)
FK - le 3)
]S - 0.05)
alpha-B.7(g) 0.01)
-3
0.82
-4
0.03
600
Figure 7: Gradient analysis with respective to different d-divergence.
C.2 COMPARISON BETWEEN DPO AND DRO
The fundamental principle of the DRO framework is to guard against the worst possible distribution within a defined ambiguity set. Notably, the "min" operation manifests implicitly outside the loss function LDPO- To provide to trace DPO's genesis back to its roots in RLHE A two-phase process within RLHF elucidates this mechanism:
insightful examination of the min-max component, it is instructive
1. Reward Modeling Phase: In this phase, we integrate human preferences via a negative log- likelihood loss:
min minC(r: O) min
(og (ro aor mw) r (ar, yr))
F
It optimizes for a reward function re by minimizing the predicted disparity in preferences between winning and losing outcomes you and yt. as determined through interaction instances (2; Dos y1) gathered from the training set O.
Reinforcement Learning (RL) Fine-Tuning Phase: This stage leverages the learned reward function to generate feedback for the language model, adopting policy improvement step that embodies our "max' operation perceived during the fine-tuning phase:
max (r, (ro v)] - 3lxx (TT (()]))]())))())) 750
As elucidated in the introduction of DPO in preliminaries,
This allows for the direct optimization of the policy by reparameterizing the reward function using the policy (i.e., the language model) in supervised manner.
Subsequently, the closed-form solution from the RL phase is substituted into the Reward Modeling Phase, and the reparameterization of ræ into no yields:
No To xx)
Te Te(: xx)
min (To; Ther) = min E -n (3 log
-Blog
)}.
(52)
(Bar x) Tee V1 x))
Comparing DPO and DRO enhances understanding:
DPO
Motivation: Suboptimal initial distribution of SFT model (reference model).
Max Part: Explore maximization criterion around the reference model, here aiming for maximal reward.
22

---

Published as a conference paper at ICLR 2025
Min Part: Optimize the BT model the novel reward function.
DRO
Motivation: Suboptimal initial training set distribution.
Max Part: Explore maximization criterion around the initial training set distribution, traditionally loss, but varies in different applications.
Min Part: Optimize the model this novel distribution.
In conclusion, our methodology faithfully embodies the essence of DRO by instituting a protective mechanism against the distribution determined by select ambiguity set and specific criterion. The "min" operation, though indirectly represented in Equation equation 5, is an integral part of our model's s optimization process, fitting well within the DRO framework's 's intent.
C.3 SETUP IN POINTWISE NOISE
To introduce the concept of pointwise noise, we first clarify the stages of DPO training:
SFT Stage: Prior to alignment, both DPO and RLHF require supervised fine-tuning (SFT). Typically, in the SFT stage, a prompt is paired with chosen response Yu--
DPO Stage: DPO uses the model trained in the SFT stage both the initialization and reference model. Training samples consist of a prompt x paired with a chosen response yor and rejected response yt. In most cases, the chosen response remains consistent with that of the SFT stage.
The term "pointwise noise" in this context refers to the impact of poor data quality. the IMDB dataset, generate training data of varying quality by using samples from fine-tuned GPT-2 Gargee-large (Radford et al., 2019) as high-quality samples and from an unfine-tune sunne- GPT-2-large as low-quality samples. By adjusting the ratio of these sets, create different noise ratios (e.g., 20% noise ratio indicates that 20% of the training data consists of low-quality samples generated by the unfine-tuned GPT-2-large).
In the HH scenario, positive and negative samples are predetermined. To create an unreliable reference model Tee. we replace the chosen responses in the SFT stage with their corresponding rejected responses. The DPO stage pairs remain unchanged, affecting only the SFT distribution, aligning with the intent described Section 3.1.
SFT
DPO
Chosen response Yw
rejected response y
Prompt x
Prompt
IMD3
Response generated fine-tuned GPT-2-large
Response generated unfine-tuned GPT-2-large
response in SFT
Prompt
Chosen response
Rejected response
response in SFT
Figure 8: Pointwise Noise Data Construction IMDB and HH 23

---

Published as a conference paper at ICLR 2025
Note: In our experiments, ensure that the data used during the DPO phase remains consistent across both datasets. This consistency ensures that the only variable influencing the outcomes the noise introduced the SFT model. By maintaining the DPO data unchanged, isolate the effect of noise on the SFT model's performance, allowing for a clear analysis of its impact.
D APPENDIX OF EXPERIMENTS
pi _i_gssps
policy logprebe, shape (B,)
ref ref_1oge logps
reference model loeprobs, shape (B,)
yw idxs
preferred completion indices in [0, B 11, shape (2,)
yl_idxs
dispreferred completion indices in I0, B-1l, shape (T,)
beta
regularization coefficient
beta_1
regularization coefficient for pairwise robustness
pi_yw_logps, pi_yl_logss pi_logps(yw_idke]) (y
ref ref_yw_loge logps, ret_yl_ ogps low__ ret_logps (y
[ [ixxs]
10
pi ___logratios - pp pi_.w__oopps
1i_yp_oopps
11
ref ref_logratios
re ref_yx_logps - ref_yl_lopps
12
losses -7. www.ogsigooi.............o...... beta + pi pi_logratios - ref ref_logratios))
13
14 #DPO
DPO_loss losses.meatt
#Dr. DPO
18
DrDso_loss beta_l
torch. log (torch.meet mean (torch. exp
losses / beta_l)))
Figure Pseudocode for our proposed Dr. DPO, as well as the original DPO objective.
Figure 9 presents PyTorch-style pseudocode comparison between the standard objective and our proposed Dr. DPO objective. The implementation simplicity of the Dr. DPO loss highlighted, as it necessitates no additional lines of code beyond what is required for the standard objective. This ease of integration underscores the practicality of adopting DPO in existing machine learning workflows without the need for extensive code modifications.
D.1 EXPERIMENTS SETUP ON HH
For our preliminary research, we conducted experiments on the Anthropic HH dataset (Bai al., 2022), which comprises 170,000 human human-automated assistant dialogues. Each dialogue concludes with two large language model-generated responses and an accompanying preference label denoting the human's 's favored choice. Our training regimen was in line with the DPO esttabisbll protocol (Rafailov et al., 2023a). We built upon the Pythia 2.8B model, as described in (Biderman al., 2023), to develop our Supervised Fine-Tuuing (SFT) model. The SFT model was fine-tuned on the Anthropic HH dataset over the course of one epuch, employing a batch size of 64 and a learning rate of 10-7 In addition, loss function (or other bascline approaches) through an additional epoch of fine-tuning. To test the model's resilience noise, introduced random inversions between selected and rejected responses in the training data with probabilities of 10%. 20%, 30%, and 40%. Throughout these experiments, we consistently set the 3 parameter to 0.1 and adopted the Kullback-Leibler (KL) divergence the metric for ø-divergence. We carried out all computational tasks on suite of four 80GB A100 GPUs.
further refined the model using the Anthropic HH dataset and the DPO
D.2 EXPERIMENTS ON REDDIT TL:DR DATASET
For fair comparison with DPO, maintained the parameters 3 0.5 and Ir 1e 6, and chose 3' = 1.0 without extensive tuning. This approach ensures that our evaluation of the proposed Dr. DPO framework is consistent and comparable to the existing baseline.
Finally, the table below presents the win-rate comparison on the TL:DR dataset under various sampling temperatures, further supporting claims:
24

---

Published as a conference paper at ICLR 2025
Table 5: Comparison of DPO and Dr. DPO across various sampling temperatures.
Sampling Temperature
0.0
0.25
0.5
0.75
1.0
DPO
45,06
46.74
46.70
39.50
21.28
Dr. DPO
62.36
67.34
71.29
63.91
27.75
As evidenced by Table 5, Dr. DPO consistently outperforms DPO across different sampling tempera- tures, particularly at lower temperatures which are crucial for complex tasks such as summarization.
D.3 EXPERIMENTS ON AMBIGUOUS DATASETS
To incorporate the feedback regarding the evaluation of our approach on datasets with ambiguity- induced noise, we conducted additional experiments. These were aimed at understanding how performs under varying conditions of data perturbation, specifically through token masking and substitution. The comparative analysis between the traditional DPO and Dr. DPO was carried out under consistent experimental conditions to ensure the validity and reliability of the results.
Experimental Setup. To simulate ambiguous datasets, we introduced randomness in the form of token masking and substitution at different ratios, thereby increasing the difficulty of the dataset. The intention
assess the resilience and adaptability of our Dr. DPO method under challenging conditions that are akin to real-world scenarios. The experiments were conducted using the HH dataset, known for its complexity and relevance in evaluating data processing algorithms.
The configurations for both DPO and Dr. DPO were kept consistent with previous experiments maintain comparability. Specifically, set 0.1 and the learning rate Ir 5e for both approaches. For Dr. DPO, an additional hyperparameter, , was introduced and set to 1.0. Notably, we did not undertake extensive hyperparameter tuning, opting instead for straightforward comparison.
Experimental Results. The results of our experiments are summarized in the table below, illustrating preference accuracies under varying noise conditions:
Table 6: Preference Accuracy on the HH Dataset with Varying Noise Ratios
Masking Ratio
0.05
0.10
0.15
Replacing Ratio
0.05
0.10
0.15
DPO
58.77
56.64
54.39
DPO
57.84
54.43
52,40
Dr. DPO
59.21
58.44
56.10
Dr. DPO
58.45
55.60
53.37
Discussion.
The results indicate that Dr. DPO consistently outperforms DPO across different levels of induced noise, whether through masking or replacing tokens. Notably, the improvement in preference accuracy becomes more pronounced as the noise ratio increases, suggesting that Dr. DPO is more robust ambiguity-indueed noise compared DPO. These findings validate our hypothesis that Dr. DPO can better handle the complexities and uncertainties inherent in real-world datasets.
is also important note that these results were obtained without extensive tuning of the Dr. DPO- specific hyperparameter, 33. Future work could involve a more detailed exploration of hyperparameter settings to potentially unlock further improvements performance.
In conclusion, the additional experiments conducted in response to feedback have not only rein- forced the effectiveness our Dr. DPO approach but also opened avenues for future research into optimization strategies for processing ambiguous datasets.
D.4 REWARD
Reward. The reward metric computed on the IMDB dataset, which is selected for the availability of ground-truth reward function provided by a sentiment classifier. Figure 10 demonstrates that the Dr. DPO algorithm achieves enhanced stability and superior reward performance under varying pairwise noise and different 3. Additionally, by setting 3 fixed value of 1. we address the issue of DPO's sensitivity to the parameter 8. This consistent setting of 3r eliminates the need for extensive parameter tuning, which significantly benefits practical applications.
25

---

Published as a conference paper at ICLR 2025
30% Flipped
40% Flipped
0.9
0.75
Reward
0.70
0.8
32%
30% Improv.
Improv.
DPO
0.7
Dr. DPO
0.60
B
B
Figure 10: Evaluation of IMDB and 40% flipped pair ratios.
Fllooed
20% Plypol
0.82
10ts Fliyped
Fliqued
2.60
0.50
0.02
0.50
0.0
Dr. DPO
0.18
0.54
Figure 11: Preference accuracy across varying 3 for different levels of pairwise noise on the Anthropic HH dataset.
D.5 IMPACT OF VARYING 3
Concomitantly, we have carried out an evaluation of the performance enhancement of Dr. DPO relative DPO under diverse beta values, shown in Figure 11. The aforementioned Dr. DPO assures a stable performance augmentation regardless of beta selection, further attesting to the efficacy of the Dr. DPO methodology. Notably, the attained results based the default value of 3', 1.0, negating the necessity of additional parameter adjustments to 8' for steady performance uplift.
D.6 IMPACT OF DIFFERENT TEMPERATURE
Ultimately, we ventured experiment with the temperature coefficient evaluation for GPT-4 (cf. Table 7), where, at a value of 0.7, the Dr. DPO method consistently outperforms its baseline counterparts, henceforth, reaffirming the method's validity and effectiveness.
Table 7: Comparison of Win Rate Performance on the Anthropic HH Dataset at 0% and 40% Flipped Pair Ratios (Temperature=0.7).
Models
0% Flipped
40 Flipped
Improv.
DPO
48.30
48.49
+0.39
cDPO
47.96
46.14
-3.79
IPO
51.20
52.39
+2.32
Dr. DPO
53.62
56.71
+5.76
D.7 DISCUSSION
Comparison with LDR Zhu al. (2023): Unlike LDR, which applies DRO for robust multi- class classification, Dr. DPO is tailored for preference learning tasks. While LDR offers pointwise robustness by adjusting weights for individual class labels per instance x, Dr. DPO provides pairwise robustness by optimizing weights for each pair of responses (Vw, 91) within dataset O. Furthermore, LDR seeks to reduce overfitting by decreasing the weights selected instances, whereas Dr. DPO counters mismatched pair effects by up-weighting chosen response pairs.
26

---

Published as a conference paper at ICLR 2025
E GENERATED SAMPLES FROM DPO-TRAINED MODELS AND DR. MODELS
In this section, present examples showcase the pairwise comparison between DPO- trained models and Dr. DPO-trained ones. Some comparisons on Anthropic HH and Reddit TL;DR summarization dataset come with reference answers, to save space, omit the reference answers here.
Table 8: A qualitative pairwise comparison of samples generated by DPO and Dr. DPO, where Dr. DPO wins.
Prompt
What some easy to make Christmas decorations I can make for inside the house?
DPO
There are many easy Christmas decorations make, here are some ideas ornaments to hang on the Christmas tree, lights to hang on the light fixtures in the house, ornaments to put on the doors and windows. You can make these decorations from a selection of holiday-themed objects, such as silver Christmas balls, crystal Christmas balls, mini glass Christmas figures, Christmas ornaments, paper Christmas trees, Christmas banners, silver tinsel, mini silver lights.
Some ideas for decorations to put up on the Christmas tree Christmas trees with mini Christmas lights, Christmas ornaments mini Christmas trees, mini Christmas lights, Christmas banners, paper Christmas trees, mini tinsel garlands. If you add Christmas tree your house decorations, there also many different types of Christmas trees to choose from, you can choose from Christmas trees with lighted branches, Christmas trees with hanging lights, Christmas trees with mini lights on the top, Christmas trees decorated with tiny ornaments.
Dr. DPO
You want to make holiday decorations inside the house, for Christmas? Would these be OK for you? What have available would make for fun experience: garland candles pine cones Christmas lights can also offer to help you choose what kind of decoration you like best, to give you specific ideas for what you want to make. If you have any questions about materials or decorating, or about the process, feel free ask. will help you make the decorations, but also provide suggestions and ideas for you.
pine cone garland easy and fun decoration, hang on your door, or inside your windows, and include some small pieces for each guest, or you can make them from whole pinecones and some string to attach them to. Decorate the sides of your home with green garland on a wooden stick. For the garland, you could:
make pine cone garland from whole pinecone and string. To make each pinecone:
Cut pinecone in half.
Cut each half in half.
Take the center of one half, and put the other half on top of it. Repeat with all the pinecones. If the pinecones are large, you can stack them, then use extra pinecones to "tic" them all together.
Cut a few small circles out of the center of each pinecone for heads and arms. I recommend using pinewood sticks, such Christmas tree ornaments. Attach them to the pinecone by sticking them into the center of each cone. You may need to add string attach each stick the pinecone by wrapping around and through the center of the cone. Practice making garland before you start, to get used to how the pinecones and sticks are arranged, to make sure you have the hang of it. Cut green grass cutouts to put on the garland, to decorate your doors.
Judgment
Response of DPO provides a couple of simple decoration ideas, while Response of Dr. DPO offers detailed step-by-step guide how to make specific decoration, along with offering further assistance.
Verdict results
DR. DPO wins.
27

---

Published as a conference paper at ICLR 2025
Table 9: A qualitative pairwise comparison of samples generated by DPO and Dr. DPO, where Dr. DPO wins.
Prompt
SUBREDDIT: r/personalfinace TITLE: Helping empty nesting parents, what to do? POST: xpost from /t/legaladvice they suggest /ripersonalfinacee would have good insights
I'm phase in life where I'm helping take care of many matters for my parents. I need to find who would be the right set of people to consult with (tax attorney? real estate attornyy cfa?) in Chicago, IL. Regarding my parents situation:
Only social security income (addl: help with everything can) 50-75k in credit card debt Owe 300k on primary home (unable to sell, been trying) Inherited 175k property (rented it out in past no remaining mortgage very sellable)
need understand the best way for them to sell both properties (lax and other implications) and protect $ from rental property sale to use towards general debt + retirement. Thank you so much!!!
DPO
Parents empty nesters with lot of debt and property sell. Need to understand what the best way to proceed is.
Dr. DPO
parents are struggling with huge debt, need to understand best way to utilize their assets and income streams to pay off debt. They are empty nesters and have no retirement savings. What should they do?
Judgment
Dr. DPO does better job of summarizing the post because it includes more details about the parents' financial situation and the need for advice on how to manage their assets and debts.
Verdict results
DR. DPO wins.
28


Reviews:
[{"review_id": "M1Mf5fFqlp", "reviewer": "Reviewer_dGhR", "summary": "The paper addresses the noisy data problem in DPO alignment. Specifically, the authors start with pointwise noise data and show that the SFTed model on these data harms DPO performance. Then, the authors build a connection between DPO and DRO and point out that the smaller $\\beta$ in DPO implicitly enlarges the search space of DRO, therefore suitable for larger noise.\n\nAuthors then argue that DPO cannot handle the pairwise noise and show pairwise noise harm the DPO performance. To tackle the problem, authors derive DrDPO from DRO. DRO can be understood as a dynamic-reweighting DPO that assigns more weight to less likely noisy data.\n\nDuring experiments, authors show how small $\\beta$ can mitigate pointwise noisy data and DrDPO achieve higher rewards than DPO with faster convergence and similar KL.", "questions": "Typo: line 194: \\pi_\\theta", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* Clear problem formulation and motivation. Each motivation has introductory supporting experiments.\n\n* Concise writing is easy for readers to understand even though readers do not know DRO much in advance. The running example is very helpful to understand the algorithm.\n\n* Extensive experiments and ablation study to support motivation, main claim (DrDPO), and method's characteristics (convergence, KL, etc).", "weaknesses": "* DrDPO requires the estimation of E_O.  In practice, this is often achieved within a batch or pre-computed on the whole dataset. Which implementation is used in the paper? How does the batch size affect the optimization results if using the former one?\n\n* Experiments on more models and sizes will make the results more convincing.\n\n* There are several papers discussing reweighting DPO, such as WPO (https://arxiv.org/pdf/2406.11827v1), which could be discussed in related work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AylY62ofFp", "reviewer": "Reviewer_eoeA", "summary": "This paper addresses the challenge of noisy training data in Direct Preference Optimization (DPO) for aligning Large Language Models (LLMs) with human preferences. The authors identify two types of noise: pointwise noise (low-quality data points) and pairwise noise (incorrect preference rankings). They enhance DPO's robustness using Distributionally Robust Optimization (DRO). They reveal that DPO inherently provides robustness to pointwise noise. To tackle pairwise noise, they introduce Distributionally Robustifying DPO (Dr. DPO), which optimizes against worst-case pairwise scenarios and introduces a new hyperparameter to control data pair reliability. Empirical results show that Dr. DPO significantly improves text quality and response accuracy in both noisy and clean datasets, offering a more robust method for LLM alignment in the presence of data noise.", "questions": "I have included my questions in the weaknesses part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and easy to follow.\n2. Addressing the issue of noisy data in preference learning for LLMs is both important and interesting.\n3. The paper has extensive experiments and theoretical analysis to prove the effectiveness of their proposed method.", "weaknesses": "1. The authors include IMDB and Anthrophic HH for experiments. It’s better to provide more datasets like TruthfulQA, GSM8k and other datasets.\n2. It’s better for authors to conduct experiments on more open-source LLMs to further verify the effectiveness of their algorithms.\n3. The authors can provide some more case studies to better understand the performance of their method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KICxhw1KTa", "reviewer": "Reviewer_Te5v", "summary": "In this paper, the authors explore the noisy data challenges when using Direct preference optimization (DPO) methods in large language models (LLMs). In this paper, noise is divided into pointwise noise and pairwise noise, and a method called Dr. DPO based on DRO is proposed to enhance DPO's resistance to these two kinds of noise. By introducing a new hyperparameter $\\beta'$, Dr. DPO is able to achieve a better balance between exploration and utilization in noisy environments. The experimental results show that Dr. DPO not only performs well on noisy data sets, but also shows stronger performance on noiseless data sets.", "questions": "1. Despite how funny this question might sound, can I understand Dr. DPO as simply adding an exponential function ($e^{(\\cdot)}$)to the objective function of DPO?\n\n2. Could you explain the weakness-3?  IPO learns the $h(x,y_w,y_l)$ to a specific value, but Dr. DPO maximize the $h(x,y_w, y_l)$, are you be in contradiction ?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The Dr. DPO framework proposed in this paper is an important improvement over the existing DPO methods, which effectively solves the noise problem in training data and improves the robustness and accuracy of the model.\n\n2. The authors provide a solid theoretical basis to explain why DPO essentially incorporates the principles of DRO and how to improve the model's resistance to point noise by adjusting the regularization coefficient $\\beta$.\n\n3. This paper verifies the effectiveness of Dr. DPO through a large number of experiments, and the experimental results show that the method can significantly improve the quality of generated text and the accuracy of response under different types of noise environments.\n\n4.  The authors provide links to the code, which helps other researchers reproduce the experimental results and further promote the research progress in related fields.\ndefect.\n\n5. The authors provide a Toy Example of How Dr. DPO Works, which is helpful to understand the proposed method.", "weaknesses": "1. The evaluation could be compared with more baselines, please refers to https://arxiv.org/pdf/2409.02795.\n\n2. These evaluation tasks are too simple; some methods may be effective on benchmarks like IMDB but may not generalize well to more complex tasks.\n\n3. Although the authors provide beautiful theory proof, the objective of Eq (12) seems to be in contradiction with IPO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wFrKeVaxM2", "reviewer": "Reviewer_7Q2Q", "summary": "This paper addresses the problem of noise in training datasets for Direct Preference Optimization (DPO). It first classifies noise into pointwise (low-quality data points) and pairwise (erroneous data pair labeling). Then, it analyzes DPO from the perspective of Distributionally Robust Optimization (DRO) to evaluate its resistance to both types of noise. Finally, it presents the Distributionally Robustifying Direct Preference Optimization (Dr. DPO) framework, which incorporates pairwise robustness by optimizing for worst-case pairwise situations and features a new hyperparameter β' to manage data pair reliability. Experiments reveal that Dr. DPO enhances the quality of the generated text within preference datasets.", "questions": "1. In Figure 4 (Left), there seem to be similar trends of trading off between Reward and KL, and if a small KL value is required under the scenario of RLHF (e.g. less than 2), are similar rewards gained from different rates of flipped pairs?\n\n2. The noise added to the pointwise dataset and the pairwise dataset in the paper cannot be regarded as the same type of noise. The noise added to the pointwise dataset by generating samples from the un-tuned GPT-2 only weakens the extent of a sample being good, similar to reducing $P(y\\ \\text{is\\ positive})$ rather than setting $P(y\\ \\text{is\\ positive})=0$. So a more corresponding method of adding noise to the pairwise dataset would be reducing $P(y_w > y_l)$, rather than reversing the order to make $P(y_w > y_l)=0$. Therefore, it seems that the different impacts on DPO caused by adding noise to the pointwise and pairwise datasets in Sections 3.1 and 4.1 are not really comparable.\n\n3. The toy example in Section 4.3 is a little hard to follow. The function $h$ that has been clearly defined in the previous text is used, but an inconsistent combination of parameters is passed in, which is confusing.\n\n4. During the training process, the estimations of both $L_{Dr.DPO}$ and $\\omega$ rely on a batch of data. Will the batch size become a crucial factor affecting the effectiveness of the policy? There is a lack of ablation experiments regarding the batch size in the paper.\n\nPS. There is a typo \"DRDPO\" in Section 5.3.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- DPO is inherently a form of Distributionally Robust Optimization (DRO) that confers pointwise robustness, with the regularization coefficient $\\beta$ inversely related to the robustness radius $\\eta$.\n- The authors introduce Distributionally Robustifying DPO (Dr. DPO), which incorporates pairwise robustness by optimizing against the worst-case pairwise scenarios, without the need for explicit noise estimation.\n- Dr. DPO demonstrates enhanced performance compared to DPO and other baselines in both noisy and noise-free environments, showcasing its effectiveness in aligning language models with human preferences.", "weaknesses": "- The assumptions about pointwise and pairwise noise in Subsection 3.1 and Subsection 4.1 are not quite comparable, which may affect the conclusion in the paper that DPO is robust to pointwise noise but not robust to pairwise noise. Details in Question 2.\n- There is a lack of ablation experiments regarding the batch size in the paper. Details in Question 4.\n- Some declarations in the paper is not so clear, which make it a little hard to follow. For example, 1) It is declared a topic \"DPO is Implicitly a Pointwise DRO\" in subsection A on line 192. But apart from a standalone Theorem 3.1 and some \"insights\" statements, there are lacking of a formal proof to connect DPO and Pointwise RPO. 2) When discussing \"Why DPO is Robust to Pointwise Noise\" on line 212, it is directly stated that \"the implicit robust framework of DPO counters data perturbations effectively\" on line 215, rising the question again \"why is DPO robust to pointwise perturbations?\". As a comparison, Section 4.3 is presented much better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jTkcd7OTJz", "reviewer": "Reviewer_SaBc", "summary": "The paper examines the robustness of Direct Preference Optimization (DPO) in aligning large language models (LLMs) with human preferences and introduces \"Distributionally Robustifying DPO\" (Dr. DPO) as a framework to enhance this robustness. The authors identify two types of noise in training data: pointwise noise (low-quality data in supervised fine-tuning) and pairwise noise (incorrect preference rankings in preference training). For pointwise noise, they show that the original DPO framework inherently applies DRO principles, where the parameter $\\beta$ correlates with the robustness radius $\\eta$ in the DRO framework, reflecting tolerance to pointwise errors in the data. To address pairwise noise, they extend DPO with Dr. DPO by introducing a new hyperparameter that adjusts the impact of noisy pairs, resulting in improved performance in challenging, noisy environments. Empirical results validate Dr. DPO’s effectiveness in producing high-quality responses and sustaining accuracy across various noise levels.", "questions": "I look forward to discussing these questions further in the weaknesses section, as they have the potential to either increase or decrease the score.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\n- The idea of applying Distributionally Robust Optimization to LLM alignment is novel and interesting.\n\nClarity:\n- The paper is well writen and easy to follow.\n\nSignificance:\n- The proposed Dr. DPO method has natual interpretation and easy to compute. \n- Experimental results are promising. Auther compres the performance of Dr.DPO with various alignment methods and provide detailed ablation studies and parameter $\\beta'$.", "weaknesses": "Major concerns:\n- The discussion on pointwise noise and DRO lacks clarity. Equation 7 appears to be a dual form of the PPO objective (Equation 3). Typically, DRO aims to optimize a target function over distributions close to the empirical sample distribution. In reward modeling, the \"sample\" refers to the input prompt and the answer generated by the target model, $\\pi_{\\theta}$. Following this logic, the DRO objective would optimize over distributions $Q$ close to $(x, \\pi_{\\theta}(y|x))$. However, in Equation 7, the authors approach the problem as optimizing objectives over distributions $\\pi_{\\theta}(\\cdot|x)$ near $\\pi_{ref}(\\cdot|x)$. I don't see a clear connection between equaiton 7 and DRO. Providing an interpretation of Equation 7 and discussing its relation to Equation 6 would help clarify this connection.\n\n\nMinor concerns:\n- The model used in pairwise experiments is not explicitly stated in the main text. The Pythia 2.8B model is mentioned only in the appendix, which reduces confidence in the experimental results.\n- The authors have omitted relevant literature on applying Distributionally Robust Optimization to LLMs, specifically, Oren, Yonatan, et al.'s \"Distributionally Robust Language Modeling.\"\n- Setting $\\beta' = 1$ results in $L_{Dr.DPO} = -\\log E[\\exp(h_{DPO})]$, which differs slightly from the original DPO formula. A more detailed interpretation of this form would be insightful.\n- In Figure 5 (left), Dr. DPO shows higher win and loss rates at 0 flips. Does this imply that incorporating Dr. DPO leads to a trade-off between performance and robustness?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)