ROPO: Robust Preference Optimization for Large Language Models
Authors: Xize Liang
Chao Chen
Shuang Qiu
Jie Wang
Yue Wu
Zhihang Fu
Hanzhu Chen
Feng Wu
Jieping Ye
Venue: ICML 2025 poster (2025)
Keywords: noise tolerance
large language models
preference optimization
Primary area: deep_learning->large_language_models
TL;DR: We propose an iterative alignment framework that mitigates the impact of preference noise by effectively identifying and filtering noisy samples.

Abstract: The prevalent noise in the preference data unavoidably poses significant challenges to the preference alignment of large language models (LLMs). Existing efforts for this problem either marginally alleviate the impact of noise without noise reduction, or rely on external LLMs that incur substantial computational costs. To address these challenges, we propose **RO**bust **P**reference **O**ptimization (**ROPO**), an iterative alignment approach that integrates *noise-tolerance* and *noise filtering* without the aid of external models. Specifically, ROPO first formulates the training process with adaptive noise reduction as an optimization problem, which can be efficiently solved in an iterative paradigm. Then, to equip this solving process with noise-tolerance and noise-identification capabilities, we derive a robust loss that suppresses the gradients from samples with high uncertainty. We demonstrate both empirically and theoretically that the derived loss is key to the noise-tolerance and effective filtering of noisy samples. The derived loss further inspires a robustness-guided rejection sampling technique to compensate for the potential important information in discarded queries. Extensive experiments on several widely-used datasets and model architectures demonstrate that ROPO significantly outperforms all baselines under **four** practical noise settings and the random symmetric noise, with its advantage increasing as the noise rate increases.

ROPO: Robust Preference Optimization for Large Language Models
Xize Liang *1 Chao Chen * Shuang Qiu *3 Jie Wang' Yue Wu² 2
Zhihang Fu? Hanzhu Chen Feng Wu Jieping Ye?
Abstract
1. Introduction
Recent research indicates that the significant achievements of Large Language Models (LLMs) understanding vari- ous queries and providing helpful responses (Achiam et al., 2023) rely on the preference alignment, which aligns LLMs' responses with human values and expectations (Wang et al., 2023c; Bubeck et al., 2023; Lin et al., 2023). A typical preference aligmment approach is Reinforcement Learning from Human Feedback (RLHF) (Casper et al., 2023; Ziegler et al., 2019), which first trains a reward model to fit human preferences and subsequently employs (Schulman et al., 2017) to guide LLMs to generate high- reward responses. However, due to the potential risks of misgeneralized reward modeling (Casper et al., 2023) and the unstable training (Liu et al., 2023a; Shen et al., 2023) of RLHF, various ranking- methods represented by Di- rect Preference Optimization (DPO) (Rafailov al., 2023) bypass the explicit reward modeling stage and eschew RL techniques via directly optimizing the implicit reward mar- gins between preferred and dis-preferred responses (Yuan et al., 2023; Wang et al., 2023a; Song et al., 2023). Owing to the stable and computationally lightweight supervised learning paradigm, ranking- baased methods have emerged as competitive alternatives to RLHE, thus drawing increasing attention recently (Shen al., 2023; Wang et al., 2023c).
The prevalent noise in the preference data un- avoidably poses significant challenges to the preference alignment of large language models (LLMs). Existing efforts for this problem either marginally alleviate the impact of noise without noise reduction, or rely on external LLMs that incur substantial computational costs. To address these challenges, we propose RObust Preference Optimization (ROPO), an iterative alignment ap- proach that integrates noise-tolerance and noise filtering without the aid of external models. Specifically, ROPO first formulates the training process with adaptive noise reduction as opti- mization problem, which can be efficiently solved in an iterative paradigm. Then, to equip this solving process with noise-tolerance and noise- identification capabilities, we derive robust loss that suppresses the gradients from samples with high uncertainty. We demonstrate both empiri- cally and theoretically that the derived loss is key to the noise-tolerance and effective filtering of noisy samples. The derived loss further inspires a robustness-guin guided rejection sampling technique to compensate for the potential important informa- tion in discarded queries. Extensive experiments several widely-used datasets and model archi- tectures demonstrate that ROPO significantly out- performs all baselines under four practical noise settings and the random symmetric noise, with its advantage increasing as the noise rate increases.
RL algorithm
Despite their impressive performance preference align- ment, ranking-based methods heavily rely on high-quality preference data, which is costly and limited in practice (Kim et al., 2023; Chen et al., 2024). First, the noise (e.g., incor- rect or ambiguous preferences) in the preference data unavoidable (Wang et al., 2024a). Many recent studies have observed the presence of preference noise levels of 20% 40% across various scenarios (Gao et al., 2024; Lee et al., 2023; Zheng et al., 2024; Touvron et al., 2023; Cui et al., 2023; Zhao et al., 2023; Munos et al., 2023), whether the annotators are humans or LLMs. Second, the performance of LLMs will significantly deteriorate when trained with noisy preferences (Chowdhury al., 2024; Gao et al., 2024; Lee et al., 2023). For instance, a 10% increase in the noise rate may lead to 30% decrease in the performance of DPO in terms of win rate (Gao et al., 2024). Therefore, it is highly desirable to develop noise-robust preference alignment tech- niques.
'Equal contribution "<xizeliang @ mail.ustc.edu.com 'MoE Key Laboratory of Brain-ing irrain-inspired Intelligent Perception and Cognition, University of Science and Technology of China 'Independent Re- scarcher City University of Hong Kong. Correspondence to: Wang <jiewangx @ wwwuuseceuuu.. edu....d...n..
Proceedings of the 4 International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

ROPO: Robust Preference Optimization for Large Language Models
To address these problems, some recent studies have explored the label smoothing (Chowdhury et al., 2024; Mitchell, 2023) and regularization (Gao et al., 2024) tech- niques to alleviate the impact of preference noise. However, these methods can only marginally mitigate the side effects of noise, as the noisy samples are still involved in the train- ing phase. Besides, (Gao et al., 2024) also attempts to filter out noisy samples but requires another teacher LLM (i.e., reward model serving the proxy of the Bradley-Temy model (Bradley& & Terry, 1952)) to assign confidence values to samples, which introduces additional computational costs. Moreover, the teacher LLM may not necessarily provide the correct preference direction (Casper et al., 2023), and this method is shown to be ineffective at reducing random symmetric noise (Gao et al., 2024).
2. Preliminaries and Problem Settings
Given a query x = [ En.. an LLM To (with pa- rameters 8) generates response y = .y....... ymm, where the tokens (z4){=1 and (8y)}= come from a predefined vocabulary, in autoregressive paradigm. Specifically, the model samples yj from the conditional probability distribution Te(- | X, where y1. is null and Ylg-1 = [,,,...,, -/5-1 for j = m. Finally, we can decompose the conditional probability oo | x) into mo x) [] mo(yj x. y1.3-1))
m.
2.1. Alignment of Large Language Models
Most of the existing LLM aligmment frameworks first fine- tune a pre-trained model on high-quality datasets of down- stream tasks (e.g., dialogue and post-summarization) via maximum likelihood, in order teach the model to respond to queries. We denote the supervised fine-tuned model Tatt. Then, we train the model wo (initialized by T.ftt based on human preference data. Specifically, a preference sample contains a query x, responses y1 and y2. and a ranking label e provided by annotators. We use c = to indicate that yı is preferred to y2 (denoted y1 > Y2 x) and use 1 to indicate the opposite. We assume that the preference data (x, V2, e) is sampled from distribution D.
In this paper, we propose RObust Preference Optimization (ROPO), an iterative alignment approach that unifies noise- tolerance and filtering of noisy samples without the aid of external models. We first provide a general formulation of learning from noisy preference data as a constrained opti- mization problem, where we dynamically assign a quality- aware weight for each sample (see Section 3.1). Then, we solve the problem through provably convergent iterative paradigm, consisting of two alternating steps: noise-tolerant model training and noisy sample filtering.
We propose a robust pref- erence alignment framework that unifies noise-tolerance and filtering of noisy samples. Without the need for any external LLM, the model's 's robustness and discrimination ability against noisy samples gradually improve as the al- ternating iterative training proceeds. (2) We derive robust loss function by suppressing the gradients of samples with high uncertainty. The loss contains a noise-aware term, which not only prevents the model from over-fitting to noisy samples but also facilitates identifying noisy samples ver- sus clean samples' (see Section 3.2). (3) We propose a robustness-guite guideed rejection sampling technique to compen- sate for the potential important information in discarded queries (see Section 3.3), which improves the data quality and thus leads to further improvement in alignment per- formance. (4) We conduct extensive experiments on three widely-used datasets (i.e., UltraFeedback Binarized, Alpaca Comparison, and TL:DR) with Mistral-7B, Llama-2-7B, Llama-3-8B, Llama-2-13B, and Llama-3-70B. Evaluation results on AlpacaEval, Arena-Hard, and MT-Bench show that the performance of ROPO remains stable in both practi- cal and artificial noisy scenarios.
Our Main Contributions. (1)
A popular formulation of the generation of preferences is the Bradley-Tery (BT) model (Bradley & Terry, 1952), i.e.. P' [y1 > Y2 | x) = ar" (y1, - r (y2, x)). where o is the sigmoid function, and r" is a latent and inaccessible reward function. The key to existing preference learning methods is to explicitly or implicitly approximate r' or P*. RLHF (Ouyang et al., 2022) approximates r" by training parameterized reward model rs via maximum likelihood on preference data, then uses the well-trained To to provide signals for the reinforcement learning of To.
Due to the complexity and instability of RLHF, some recent works (Rafailov al., 2023; Azar et al., 2023; Wang et al., 2023a) directly learn preferences from offline collected re- sponse pairs by optimizing the implicit reward margins be- tween preferred and dis- dis-preferee responses. For example, the objectives of DPO (Rafailov et al., 2023) is given by ]a(y)|x) kapo = log or log Tea x - B log [xe[(y=|x]] ). where Y1 > Y2 x, 3 is a hyperparameter, and &eef is fixed refer- ence model (usually the SFT model). Ranking-based meth- ods are more computationally lightweight and stable than RLHF, thus drawing increasing attention recently. Thus, we mainly focus on ranking-basee methods in this paper.
ma =e(ya)x)]
In Section 3.2, we demonstrate that the cross-entropy loss (i.e., DPO loss) cannot distinguish between noisy samples and clean samples in the context of preference learning, even though is widely used for learning from noisy data in other scenarios such as image classification (Jiang et al., 2018: Liu et al., 2020).
2.2. Preference Learning with Noisy Data
Preferences are unavoidably noisy due to the cognitive bias among annotators (see Appendix C for detailed dis- cussion). Thus, we have no access to the clean dataset
2

---

ROPO: Robust Preference Optimization for Large Language Models
New samples
Online Rejection
leginns, innkeppttt
Model
Noisy samples
Sampling
Preference ranking label
Sippress
Training samples
Other samples
x)
x(y,x) Blog
Fzzl
Tolerance
Identification
Robust
Mapp
Loss
Model Training
Sample Filtering
f(y,,))
*(y,,))
A 2(22, 1(
Implicil reward comparrioo
Weighting function
ROPO Framework
Gradient Weight Strategy
Figure 1. Framework of ROPO and comparison between the gradient weighting strategies of ROPO and DPO (Rafailov et al., 2023). Left: ROPO alternates between noise-tolerant model training and noisy sample filtering and integrates the online rejection sampling paradigm to further improve the data quality. Please see Appendix A for the detailed description and pseudocode of the framework. Right: Unlike wapo. which increases with respect to A = f(y2, x) f(y1, x). Wrope decreases when A is large. Given a noisy sample (x, y1, ya y2 x), whose preference label contradicts the comparison of implicit rewards, ROPO suppresses its gradient. A larger a implies a stronger suppressive effect.
D = ((x(),)) (1)
.22 , ,,()) ~ D and can only obtain (2) (1)
(Section 3.2), Further, we propose a robustness-guice re- jection sampling technique to compensate for the important information in discarded queries and thus improve the data quality (Section 3.3). For detailed proofs of the theorems in this section, please refer to Appendix F.
a noisy dataset D, = ((x(9, yi .22 2 N Dz where z(0) c( with probability 1 77 and all) 11 cl with probability 77.
Remark. (1) We assume the random symmetric noise in our theoretical analysis because it is the standard assumption for learning from noisy data (Liu & Guo, 2020; Zhang & Sabuncu, 2018) and existing research on LLM alignment has indicated the challenges posed by this kind of noise (Gao al., 2024). Besides, in the context of preference alignment, the symmetric and asymmetric (or class-conditional) noise is equivalent, as the ground truth label is changed if we swap the positions of y1 and y2. (2) In addition to this artificially introduced random noise, our experiments also include four types of practical noise settings, covering a variety of unavoidable noises from human and LLM annotations. For more details, please refer to Section 4 and Appendix E.3.
3.1. A General Formulation
Given N preference samples (xx(), .y 1112222 (.) ) we hope that the weights of noisy samples in the preference op- timization are smaller than those of others, thereby reducing the impact of noise on the alignment performance. Without prior knowledge of which samples are noisy, a natural ap- proach would be to assign a dynamic quality-aware weight to each sample and constrain the sum of these weights to a constant, which can also prevent the weights from tending toward zero. Therefore, we formulate learning from noisy preference samples as the following constrained optimiza- tion problem:
3. Robust Preference Optimization
N
min f.w
(6) 200 To); wrl(O; x ,y .22
(,
(1)
We propose ROPO, an iterative preference alignment frame- work. ROPO alternates between noise-tolerant model train- ing and noisy sumple filtering, as shown in Figure 1, which mathematically equivalent to iteratively solving a con- strained optimization problem (Section 3.1). In the model training step, we introduce robust loss function by sup- pressing the gradients of samples with high uncertainty, which prevents the model from over-fitting to the noisy preference. In the sample filtering step, we filter out noisy samples based on the magnitude of their training losses. The key to ROPO is that our proposed loss contains noise- aware term, which not only features noise-tolerance, but also facilitates identifying noisy samples versus clean samples
N
1=1
s.t.
A θ
W; E 10, 1]. i I,...,N N.
N
Σ
w; N, P [(1 - -)NJ,
where Wi..... WN are dynamic weights, θ is compact, and P [0, 1] is the proportion of the samples aim to filter out. Please note that we minimize Problem (1) with respect to both 0 and w, resulting in a training process that learns the weights adaptively. Hence, we expect that Problem (1) will gradually lead to much smaller weights for noisy samples than those for others. To achieve this, we first analyze the
3

---

negative and its value does not increase during the iteration, ROPO: Robust Preference Optimization for Large Language Models
properties of the optimal solution to Problem (I). As shown in Theorem 3.1, Problem (1) admits an optimal solution and the elements in its minimizer w" are either 0 or 1.
Theorem 3.2. Consider Eapo and the corresponding min- imizer 0g to Problem (3). Given a query x and responses Y2), the relationship between the preference probability given by the optimal model, i.e., Po: (y1 > Y2 x), and that
Theorem 3.1. Assume {(0) is continuous on a compact pa- rameter space O then Problem (1) admits an optimal solu- tion (f", ,w") Suppose that E(0"; ;(0)x((i),,
given by the BT model, i.e., P* >- Y2 xx is Po,(Y1 /
(é1) -g--) <
Y2 x) = P (y1 > Y2 x) + (1 - 21" (y1 > Y2 x)) .7
.Y2
hence we have Po: (Y1 > Y2 x) - Pa- > Y2 x)
< 1(0";x .1 .Y2 TT ). then w 1 for 1 1 k S Np and w for Np < N.
(bx) (1x)
27| Pn(P > 82 x) 1/2
As shown in Theorem 3.2, the impact of noise on the opti- mal solution corresponding to kapo increases as the noise rate increases. Specifically, the difference between the op- timal probabilities under noisy and clean distributions, i.e. Po; (y1 > Y2 x) Pg- (y1 Y2 x){, is proportional to the label flipping probability v).
We solve Problem (1) in an iterative paradigm, which con- sists of two alternating steps: model training and sample filtering. In the step of model training, we fix w and learn model parameters 9. In the step of sample filtering, we fix 0 and assign weights w...... WN for samples based on their loss values. Because the objective in Problem (1) is non-
Finding 2: DPO faces challenges in distinguishing be- tween noisy and clean samples.
the iterative solving process is guaranteed to converge.
Theorem 3.3. For samples (x(1),, (1)
.82 ((())) = ))
3.2. A Noise- Tolerant Loss
and (x(2), y(2) .22 c(2)
(2)
1 c22),, suppose that
is not 6r but satisfies max (Poly (8)
> y2 I x))
(i)
To guarantee the effectiveness of the iterative solving pro- cess within the preference alignment framework, we delve into identifying additional conditions that should be im- posed on &. Here, we discuss the properties of in the context of minimizing its expected risks under distributions of noisy and clean preference data, i.e., finding the optimal solutions 0* and 6 by solving
1=1,2
Po; (yl)) > -2 (5)
[x(1)|| < 8. then we want to ensure that
Eápo (x(1), yI .
(1) (1) 2(1) < (apo (x(2), (2)
.22
(2)
8
must satisfy P-2n (P"(c()) + P* (cl2)
2
As shown in Theorem 3.3, the distance between Ta and Th; we need for lapo to differentiate between clean and noisy samples decreases as the BT probability approaches
8" = arg min (l Aiee
(2)
50% and the noise rate increases. Specifically, Theorem
3.3 shows that the upper bound of 5 is proportional to
" = arg min (xx,,,a,0--,,,,000xx.xx1 m²) (3)
(1 - 222 and (P"(el)) - -1 1122 + PT - 1 Due to the intrinsic diversity and stochastic nature of human preferences, the BT distribution is usually not "hard" dis- tribution with probabilities close to O or 1, but rather a "soft" one (Swamy et al., 2024; Strobl et al., 201 This brings difficulties to unconverged DPO-trained model in identi- fying noisy samples. For example, when 7) = 30% and pa (c()) = P* (d(2)) = 60%, we need < 4%, which is a challenging requirement for a model that has not yet converged.
dee
Requirement 1: Noise-tolerance. It cannot be guaranteed that the sample filtering stage will eliminate all noise sam- ples (e.g., when p is less than the actual noise proportion in the preference data). Consequently, it is crucial that the presence of noisy preferences does not significantly impact the model training stage, i.e., f is noise-tolerant.
Requirement 2: Distinguishable losses for clean and noisy samples. As noisy samples generally exhibit larger loss values (Liu et al., 2020), in the sample filtering step, we filter out the N- - N₂ samples with the largest losses. is noteworthy that this step takes place midway through training, hence f needs to exhibit distinguishable loss values for clean and noisy samples prior to the convergence of the model.
The gradient weighting strategy of DPO may amplify the impact of noise. Given a sample (x, C = 0). according to (Rafailov et al., 2023), the gradient of Edpo is given by
Tee (y
xx)
Volapo = =330 (f(y), x) f(y1,)))
. Vlog No (yz
x)
As DPO is one of the most popular alignment methods, it is natural for us to explore the effectiveness of the DPO loss lapo within our iterative solving process. However, our findings show that Edpo does not satisfy the aforementioned requirements. In this section, we assume that 17 1/2.
Mápo
(4)
where f(y, x) = log xx(y))) is the implicit reward func- 7 [
AAs ()) and c(2) are clean labels, we have P' and P" (c(2)) > 1/2.
(c > 1/2
Finding 1: DPO is not noise-tolerant.
4

---

ROPO: Robust Preference Optimization for Large Language Models
tion of DPO. Intuitively, the greater the discrepancy between the reward function's 's comparison of y1 and y2 and the label the greater the weight Wapo (x, 1 Y2) of the DPO gradient becomes. This aggressive weighting strategy can be risky if the label is incorrect, as the model may imply a high uncertainty about the sample by giving a higher re- ward to y2 than to yı, increasing Wapo and thus amplifying the impact of the noise.
0.06
clean noisy
0.06
clean noisy
0.05
0.04
Y1 >> Y2
0.04
0.03
0.03
0.02
0.02
0.01
0.01
0.00
0.00
0.0
0.5
1.0
15
2.0
2.5
3.0
3.5
0.0
0.5
1.0
1.5
2.0
2.5
(a) DPO (epl). 7 0.48.
(b) DPO (ep2). 7 = 0.34.
Conservative gradient weighting strategy. A simple and straightforward idea is that when the implicit reward margin 2 A f(y2, x) - f(y1.x) is excessively posi- tive, we should assign a conservative weight to the gradient. Based on this idea, we propose the conservative gradient weight
0.10
0.10
clean noisy
clean noisy
0.08
0.08
0.06
0.04
0.04
0.02
0.02
0.00
0.00
1
2
3
2
4
6
7
Wropo
(c) ROPO (epl). 7 = 0.58.
(d) ROPO (ep2). 7 0.60.
4a
(1 a1) X1,)) (1 + -o
(5) where >> 2 controls the conservatism of weighting and 4a/(1 + a)²² is used to normalize the maximum value of Wropo (see Appendix F.9). As illustrated in Figure I, unlike the monotonous increase of Wdpo, Wropo decreases when A y1, x) is large. Then, the corresponding loss function can be decomposed as
Loss distributions of Llama-2-7B trained with DPO and ROPO different training epochs (epl and ep2) TL:DR. We denote y as the proportion of noisy samples in the 20% of samples that are filtered out. Larger y indicates better discrimination be- tween clean and noisy samples.
Figure 2.
Theorem 3.5. For samples (x(2),, (1) (1) ,(()) = clo)
,y .Y2 ,22,6(0)
and (x(2), y1 (2) (2) e(2)
1 ((2),, suppose that 0
/ V Volrapa do (1 a)+
4a² lna +
4a
Propo
Edpo:
is not 07 but satisfies max (Polr > y I x())
(1 + a22
1=1,2
(6)
Pez (y1)) Y2 (6)
x(1) < 6, then if we want ensure
that l (xxx(11))) vi y2 (00) lna (xl2) y() y ((l)).
(1) (1)
(2) (2)
where ln = (Blog Nest (y - 3 log [xx)(y and we omit the constant term of the primitive function (see Ap-
ma (x)
we must have 8 < 2.
pendix F.4 for details). The introduced loss consists of lapo and noise-aware term Lns whose weight is a times that of Edpo- We claim that Lna has the following advantages.
As shown in Theorem 3.5, contrary to the challenging re- guirement Edpo places on unconverged model in Theorem 3.3, we can expect that en yields a larger value for noisy samples than for others as long as the difference between the preference probability given by an unconverged model and that of the optimal model is less than 50%. We verify our theoretical analysis in experiments, as shown in Figure 2. For more details, please refer to Section 4.1.
Advantage 1: lna is noise-tolerant.
Theorem 3.4. Consider &na and the corresponding mini- mizer 0r, to Problem (3). Given a query x and responses (y1. Y2), the relationship between the preference probabil- ity given by the optimal model, i.e., Poz (y1 > Y2 x), and that given by the BT model, i.e., P*(y1 > y2 x) is
Discussion. Ena is capable of improving noise-tolerance and separating noisy samples from clean samples. However, compared with lua. Edpo leads to "softer" optimal pref- erence probability, which could potentially avoid discrim- ination against minorities by LLMs. Besides, the aggres- sive weighting strategy may be useful for clean preference datasets (although they are rare). Thus, it is considered necessary to incorporate a minor component of Eao into the final loss. From this perspective, the hyperparameter plays an important role in trading-off between aggressive (Lápa) and conservative (Cna) gradient weighting strategy. Given that the weight of lua is a > 2 times greater than that of dpo (in our experiments and ablations, a 2 6), lyg dom-
Po: (y1 > Y2 x) 1(P > y2 x) > 4), hence we
have Poz (y1 > y2 x) Po- (y1 > Y2 x).
As shown in Theorem 3.4, contrary to the conclusion in Theorem 3.2 that the optimal solution corresponding to lapo affected by the noise, the optimal preference probabil- ity corresponding to Lua. i.e., Po-(y1 Y2 x), remains unchanged when the label flipping probability
< 1/2. Specifically, 3.4 shows that Po. (y: > y2 x) is an indica- tor function of Pa (y1- > y2 x) x 1/2.
Advantage 2: lua can distinguish noisy samples from clean ones.
5

---

ROPO: Robust Preference Optimization for Large Language Models
Reddit TL;DR (Völske al., 2017: Stiennon et al., 2020)).
Sample
Robustnss guided Rejection Sampling
Epoch
Traiiin
Fillaring
For details about the datasets, please refer to D.1.
Ncise Ieleratt
Noiy Sample
Robustnesi guided Rejection Sampling
Fpuch 2
Noise Settings, Our experiments include four types of practical noise settings. As stated in Section 1, original datasets unavoidably contain noise introduced by annotators (see Appendix E.3 for details about the two related practi- cal noise settings). To further explore the performance of ROPO and baselines under noise, we randomly alter prefer- ence labels at different proportions (20% and 40%) within the datasets to produce more challenging symmetric noise (Gao et al., 2024). Besides, in Appendices E.3.1 and E.3.2, we supplement experiments in another two practical settings, where the noise comes from annotators' trust in larger mod- els over smaller models and L.LM preference comparisons. Please refer to the supplementary material for more details.
Traising
Fillering
Veisc-tnlerant
Noney Sample Filtering
Robendncergridle Rejection Sampling
Fiuch M-1
Training
Neiee-tokemntt
Epoch
Training
Figure 3. The iterative process of ROPO.
inates the optimization process. Thus, the incorporation of Edpo does not hurt the noise-tolerance and noise filtering too much.
Baselines, Models, and Hy Hyperparameters. Our baselines are DPO (Rafailov et al., 2023), IPO (Azar et al., 2023), and two approaches that use the label smoothing technique to alleviate the impact of noise, i.e., rDPO (Chowdhury et al., 2024) and cDPO (Mitchell, 2023). Besides, we supplement experiments on reward modeling in Appendix E.2.
3.3. Robustness-guided Rejection Sampling
The sample filtering step effectively reduces the proportion of noise but may also discard some important queries. For example, a query designed to eliminate the occupational dis- crimination in LLMs may be filtered out because the ranking label of its associated responses is incorrect. Thus, inspired by the sample distinguishing ability of our proposed Loopo we propose a rejection sampling technique to compensate for the essential but discarded information and thus im- prove the robustness of our ROPO framework. Specifically, we sample K responses .., .YK to x for each sample V1) that is filtered out and generate 2K candidate samples { (xx, VR. y1 > Y& x) K k= (((( Yz. Y2 > ys x)) Then, we compute their loss values and add the sample with the minimum loss to the dataset. Note that we treat the model's responses as dis-preferred ones compared to the original responses, which suppresses the potential unsatisfactory or even harmful information in the model's 's outputs.
We Mistral-7B (Jiang et al., 2023) and Llama-2-7B (Touvron et al., 2023) as base models for all baselines and datasets in the main text. For experiments on Llama-2-131 2-3BB and Llama-3-70B, please refer to Appendix E. On UFB, we use Zephyr-7B-SFT. (Tunstall et al., 2023) as the SFT model for experiments with Mistral-7B, and adopt the re- sult of Zephyr-7BBBi (Tunstall et al., 2023) on AlpacaEval (90.60) as the performance of DPO under no artificial noise. In other cases, we fine-tune base models on the preferred responses (SFT targets) to form the SFT models. For details about our baselines, models, and hyperparamters, please re- fer to Appendix D.2. We run all experiments on 16 NVIDIA A100 GPUs (80 GB).
Evaluation. For models trained on UFB and Alpaca Com- parison, evaluate them on the AlpacaEval benchmark (Li et al., 2023a) by comparing their outputs with those of text-davinci-003 (recommended by the benchmark for comparison). For models trained on TL:DR, we evalu- ate them by comparing their outputs with the SFT targets (chosen responses) on the test split of TL:DR. Following (Rafailov al., 2023; Tunstall al., 2023), we employ GPT-4 as the referee for head-to-head comparisons, using the win rate as the metric. The win rate can be computed by # # #CComparisona) where # #(Wi)). # #(Tic), and # (Comparisons) are the numbers of wins, ties, and com- parisons, respectively. For evaluation details, experiments on more benchmarks, and human evaluation, please refer to Appendices D.3. E.4, and E.6, respectively.
3.4. ROPO Framework and Complexity Analysis
As shown in Figure 3, ROPO iteratively carries out three stages: noise-tolerant training, noisy sample filtering, and robustness-guiit guided rejection sampling. In the last epoch, we only perform the noise-tolerant training stage and then get the final model. For the detailed pseudocode and complexity analysis of ROPO, please refer to Appendix A.
4. Experiments
Tasks and Datasets. We focus on two dialogue datasets (i.e., Ulraaeedback Binarized (UFB) and Alpaca Comparison (Peng al., 2023)) and one post-summarization dataset (i.e.,
hhitps::/huggiigggfac. htt.giinnnaacccccaaaattttttt /otasettsts/////// SangginngPaceee/tttrraae hwtp///eaaaaaaaaaaeccaaaa.. biiriiiizdddd
6

---

ROPO: Robust Preference Optimization for Large Language Models
Table Win rates (%) of different methods vs SFT targets under different proportions (i.e.. O, 20% and 40%) of artificial noise. evaluated by GPT-4. The bold font indicates the best result and an underline indicates the second-best result. Please note that 0% represents no artificial noise, which does not mean that the dataset is clean.
Dataset
UFB
Alpaca Comparison
TL:DR
Model
Method
0%
20%
40%
0%
20%
40%
0%
20%
40%
DPO
90.60
86.21
82.67
73.66
70.19
65.84
63.00
56,80
49.60
IPO
88.45
87.32
82.86
72.92
70.81
67.33
62.00
57.00
48.80
Mistral-7B
rDPO
88.07
87.45
84.72
72.55
72.05
70.31
62.40
58.20
52.60
cDPO
88.82
86.96
83.35
73.04
71.30
69.94
59.40
57.40
53.00
ROPO
91.06
88.63
87.70
75.40
76.27
74.04
79.00
77.80
75.80
DPO
68.57
66.71
62.36
53.42
50.68
48.20
56.80
42.40
35.20
IPO
67.70
66.09
64.35
53.54
50.56
49.19
54.20
50.80
51.60
Llama-2-7B
rDPO
68.07
67.83
65.59
52.80
51.18
50.31
54.80
54.00
50,40
cDPO
68.20
67.33
65.09
53.79
50.81
49.81
52.20
52.00
49.80
ROPO
68.94
69.44
66.71
55.90
54.41
54.53
78.80
78.00
79.20
Table 2. Win rates (%) of ROPO and DPO vs SFT targets under different proportions (i.e., 0, 20% and 40%) of artificial noise at different training epochs on TL:DR, evaluated by GPT-4.
0%
20%
40%
Model
Method
epl
cp2
cp3
cpl
cp2
cp3
epl
cp2
cp3
DPO
62.60
60.20
63.00
56.80
51.00
48.60
49.60
44.40
44.60
Mistral-7B
ROPO
75.40
75.60
79.00
68.80
76.40
77.80
61.60
70.80
75.80
DPO
49.00
53.60
56.80
42.40
38.40
39.20
32.00
35.20
33.60
Llama-2-7B
ROPO
74.00
82.00
78.80
58.40
76.40
78.00
46.00
70.80
79.20
4.1. Main Results
without the priors on the noise proportion and iteratively reduces the noise proportion as the training proceeds, thus leading to superior performance to rDPO and cDPO.
ROPO robust to noisy preferences. We present the win rates of different methods SFT targets under different proportions of artificial noise in Table 1. From the table, we have several interesting observations: (1) For all prefer- ence alignment methods, their win rates show a decreasing trend as the noise rate increases. (2) Compared to the com- petitors, our proposed ROPO demonstrates a more stable performance under noisy preference data. (3) ROPO consis- tently outperforms the baselines under different proportions of artificial noise in all the three datasets. Even without artificial noise, ROPO still outperforms DPO by 16.0% TL.DR and 2.5% on Alpaca Comparison. which indicates that the datasets inherently contain noise. (4) Baselines that use the label smoothing technique (i.e., rDPO and cDPO) mostly outperform other baselines under 20% and 40% arti- ficial noise, but underperform ROPO. We speculate that the reasons for their limited effectiveness are as follows. First, iDPO and cDPO are noise-tolerant only when the hyperpa- rameter & exactly equals the proportion of noise and when e = 0.5, respectively (see Appendix F.7), which is difficult to achieve in practice, as we have no prior knowledge of the exact noise proportion. Second, they do not reduce the pres- ence of noise and thus can only marginally mitigate the side effects of noise. In contrast, ROPO exhibits noise-tolerance
ROPO distinguishes noisy samples from clean samples.
In Section 3.2, we have theoretically shown that ena can distinguish noisy samples from clean ones, while Edp can- not. Besides, we also claim that the minor incorporation of Eapo in éropo does not hurt the noise filtering ability. To support our analysis, we report the loss distributions for Llama-2-7B trained with ROPO and DPO on TL;DR in Fig- ure 2. Specifically, for models trained for one (two) epoch, we use the SFT model (the model trained for one epoch) as the reference model and compute the losses for all noisy and clean samples. The results in Figure demonstrate three important observations: (1) Cropo can distinguish be- tween noisy and clean samples by yielding larger values for noisy samples than for others. (2) The distributions of fapo on noisy and clean samples are similar and the gap be- tween them narrows as training progresses. (3) ROPO has a stronger capability for filtering out noisy samples compared to DPO. Specifically, in the top 20% of samples with the largest Cropo- noisy samples make up 60%; whereas in the top 20% of samples with the largest lapo- noisy samples account for about 34%.
ROPO gradually improves the performance. In Table 2,
7

---

ROPO: Robust Preference Optimization for Large Language Models
work, we compare the performance of our proposal with and without: (a) noise-aware term laa (b) noisy sample filtering stage, and (c) rejection sampling stage. As shown in Table 3, all components improve ROPO's performance, validating the rationale of our robust framework. Compared to the aggressive DPO loss, our proposed noise-aware term na consistently improves the performance, which indicates that a proper trade-off between aggressive and conservative gradient weighting strategy effectively prevents the model from over-fitting to noise. Besides, the results also show that the noisy sample filtering is the most effective part of our method, which also makes ROPO significantly superior to other label smoothing- based methods (Chowdhury et al., 2024; Mitchell, 2023).
Table Ablations different components of ROPO for Mistral- 7B on UFB. NSF and RS stand for the noisy sample filtering and rejection sampling stages, respectively.
Method
0%
20%
40%
DPO
90.60
86.21
82.67
ROPO (fma)
89.19
87.58
86.34
ROPO (fna + NSF)
89.44
88.20
88.07
ROPO (Ena NSF RS)
91.06
88.63
87.70
95
75-
90
70-
85
80
How many noisy samples should we filter out? The sam- ple filtering ratio p is key factor to the data filtering stage. In the main experiments, we only report the results with -= 0.2. Here, we also present the results of filtering 10% and 30% samples with larger loss values. The results in Figures 4(a) and 4(b) show that better performance could achieved when filtering 20% or 30% samples. We at- tribute the reason for this result to the noise ratio in the preference data, which is generally between 20%-30% (Gao et al., 2024). There's substantial risk of eliminating considerable amount of high-quality data if we set larger ratio Thus, we recommend using 0.2 in practice.
75
70
0%
20%
40%
55
0%
20%
40%
(a) Mistral-7B
(b) Llama-2-7B
80
74.
75
72
04
20%
20%
(c) UFB
(d) TL;DR
Sensitivity to hyperparameters a. The trade-off hyper- parameter a controls the importance of the conservative noise-aware term. A larger a indicates a more conservative gradient weighting strategy. As C A lim Wropo (A) 4a (a+1)) we search the best in the range of {6, 14, 30), which corresponds to C' E {1/2, 1/4, 1/8]. Then, we use a = 14 in our main experiments (see Appendix D.2 for the settings of hyperparameters). To explore the effect of o, we provide ablations on a in Figures 4(d) and 4(c), As observed, the model's performance remains largely unaf- fected for within an appropriate range, as the loss scale does not change significantly (note that aC E 22.44, 3. 3.75] for E (6)30)) Besides, for the dialogue task, smaller a results in better performance, as a smaller a will lead to more diverse answers. In contrast, a larger a results in better performance in the summarization task. As the summariza- tion task more objective than the dialogue task, the results are more sensitive to noise, and hence need model that is more robust the noise.
Figure 4. Ablations on p and a. (a) and (b) respectively show the performance of ROPO-trained Mistral-7B and Llama-2-7B on UFB with different proportions of artificial noise and sample filtering ratio p. (c) and (d) respectively show the performance of HOPO-trained Mistral-7B 7B on UFB and TL:DR with different proportions of artificial noise and a.
report the win rates of ROPO and DPO SFT targets under different proportions of artificial noise at different training epochs on TL.DR. From the results, we find that the performance of ROPO gradually improves as training progresses most cases, while DPO does not exhibit the same trend. Specifically, the performance of DPO at the second and third epochs is generally lower than that at the first epoch under 20% and 40% artificial noise. As a compar- ison, the second epoch training of ROPO brings an 8%%24% increase in the win rate, and the third epoch also leads to a 5%-9% improvement under 40% artificial noise. These results demonstrate that the iterative training of ROPO ef- fectively reduces the impact of noise and thus consistently improves the alignment performance.
5. Related Work
Preference Alignment of LLMs. The most representative paradigm of preference alignment is RLHF (Ziegler et al., 2019; Ouyang et al., 2022), which involves training a re- ward model to capture human preferences and then steering LLMs towards producing high-reward responses through RL algorithms (Schulman al., 2017). However, in real
4.2. Ablations
Effectiveness of components in ROPO. To evaluate the effectiveness of different components of our ROPO frame-
8

---

ROPO: Robust Preference Optimization for Large Language Models
applications, RL-based methods are complex and prone to instability during training (Rafailov et al., 2023; Wu al., 2023; Yuan et al., 2023). Therefore, many recent studies have explored more straightforward and stable alternatives for RLHF (Yuan et al., 2023; Rafailov et al., 2023; Song et al., 2023; Wang et al., 2023b; Lin et al., 2023; Li et al., 2023b; Wang et al., 2023c; Zhao et al., 2022). Among these studies, the most promising direction is to use a contrastive or ranking loss to calibrate the likelihood of the output se- quence. Specifically, DPO (Rafailov et al., 2023) implicitly optimizes the same objective as existing R[-FF-aased meth- ods and enables human preference ali alignment directly with a simple cross- eross-entropy loss. In addition to the aforementioned methods using data in the form of (x, Y1, c), where is the preference label, some recent studies (Duan et al., 2024; Ethayarajh et al., 2024; Chen et al., 2024) have also used data in the form of (x.y y, c). where is annotation of the response y, for preference alignment.
the scalar rewards, while (Yang et al., 2024b) augments samples near the Pareto front of multi-dimensional rewards, leading to strong multi-objective alignment performance. Compared to the aforementioned methods, which all rely on rewards provided by external models, our robustness- guided rejection sampling technique selects new samples based on loss values that reflect the quality of the samples. Moreover, our technique benefits from being independent of external LLMs, thus leading to computational and memory efficiency.
Robust Alignment of LLMs. Many efforts have been made from various perspectives to achieve robust preference align- ment (Chowdhury et al., 2024; Mitchell, 2023; Choi et al., 2024; Bukharin et al.; Yan et al., 2024; Wu et al., 2024; Ramesh et al., 2024; Yang et al., 2022; Yang et al.; Xu et al., 2025). Specifically. (Choi et al., 2024) improves the model's ' adaptability to different preference distributions and enables iterative output refinement by jointly optimizing a self-improvement policy and a generative policy. (Bukharin et al.) models potentially corrupted preference labels as sparse outliers and solves an f1-regularized maximum like- lihood estimation problem, thereby consistently learning the true underlying reward. (Yan et al., 2024) introduces multi-head reward model (RM) that reflects each head's s confidence in the output reward using the standard deviation of a Gaussian distribution, effectively addresses the chal- lenge of RM imperfections in R.M-based RLHF. (Wu et al., 2024) focuses on different forms of noise and enhances DPO's 's resilience to both pointwise and pairwise noise in LLM alignment by leveraging Distributionally Robust Op- timization (DRO). (Ramesh et al., 2024) robustly aligns LLMs to the preferences of diverse individual groups by incorporating group information into the LLM context and optimizing against the worst-case alignment performance across all groups Compared to them, our method integrates noise-tolerance and noise-identificatinn capabilities with- out external models, offering a novel paradigm for robust preference alignment.
Learning from Noisy Data. Studies in learning from noisy data primarily fall into three categories. The first category is sample-selection based methods (Swayamdipta et al., 2020; Pleiss et al., 2020; Paul et al., 2021; Sorscher et al., 2022),
which identify high-quality samples before training and filter out noisy samples. For example, (Swayamdipta et al., 2020) uses the training dynamics to identify valuable samples. The second category is weighting- beighting-bbsed methods, which assign greater weights for important samples and lesser weights for noisy samples (Ren et al., 2018; Han et al., 2022; Shu et al.. 2019). Besides, another important area of research is dedicated to the design of loss functions that are robust to noise (Ghosh et al., 2017: Wang et al., 2019; Zhang & Sabuncu, 2018). The findings in (Ghosh et al., 2017) indicate that the traditional cross-entropy loss is sensitive to the label noise, while symmetric loss functions are robust to such noise. Furthermore, recent advances in LLMs have also underscored the essential role of data quality in both pre-training and supervised fine-tuning (SFT) phases of LLMs (Marion al., 2023; Zhou et al., 2023; Korbak et al., 2023).
6. Conclusion
Rejection Sampling. The rejection sampling is popular approach of data augmentation to improve the data quality and performance in existing preference alignment meth- ods. Specifically, (Dong et al., 2023) ranks newly-collected responses based on their rewards and selects the highest ranked one to add to the dataset. To address the issue of the excessively high rejection rate and thus improve the effectiveness of rejection sampling, (Xiong et al., 2023) proposes a multi-step sampling technique, which also re- quires an external reward model. Besides, (Wang et al., 2024b) and (Yang et al., 2024b) consider rejection sam- pling for the multi-objective preference alignment, where (Wang et al., 2024b) projects multi-objective reward vec- tors onto one dimension and then selects samples based
Robust preference optimization is critical for the LLM align- ment, as noisy preferences are inevitable in practical scenar- ios. Unlike existing methods, which rely on label smoothing or external LLMs for the sample selection, we propose a robust preference alignment framework that unifies noise- tolerant model training and effective filtering of noisy sam- ples. Specifically, we incorporate a noise-aware loss term to prevent the model from over-fitting to noise. Further, we propose a pooustness-guidedd rejection sampling technique to compensate for the potential information reduction caused by the filtering stage. We provide extensive theoretical and empirical evidence demonstrate the effectiveness of our proposed ROPO framework.

---

ROPO: Robust Preference Optimization for Large Language Models
Acknowledgments
Chen, Z., Deng, Y., Yuan, H., Ji, K., and Gu, Q. Self-play fine-tuning converts weak language models to strong lan- guage models. arXiv preprint arXiv:2401.012 001355, 2024.
This work was supported in part by National Key R&D Program of China under contract 2022ZD01 19801, National Nature Science Foundations of China grants U23A20388 and 62021001. We would like to thank all the anonymous reviewers for their insightful comments.
Chiang, W.-L.,, Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonza- lez, J. E., et al. Chatbot arena: An open platform for evaluating Ilms by human preference. arXiv preprint arXiv:2403.04132, 2024.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Choi, E., Ahmadian, A., Geist, M., Pietquin, O., and Azar, M. G. Self-improving robust preference optimization. arXiv preprint arXiv:2406.01660, 2024.
Chowdhury, S. R., Kini, A., and Natarajan, N. Provably ro- bust dpo: Aligning language models with noisy feedback. arXiv preprint arXiv:2403.00440 2024.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I,, Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni. Y., Xie, G., Liu, Z., and Sun, M. Ultrfeedback: Boosting lan- guage models with high-quality feedback. arXiv preprint arXiv:2310.01377, 2023.
Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., and Munos, R. A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310.111. 12036, 2023.
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T. Raft: Re- ward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06766 2023.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F,, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Duan, S., Yi, X., Zhang, P., Lu, T., Xie, X., and Gu, N. Negating negatives: Alignment without human posi- tive samples via distributional dispreference optimization. arXiv preprint arXiv:2403.03419 2024.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.05862 2022.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint utXiv2240..21782 2024.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024.
Bubeck. S.. Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. Sparks of artificial general intel- ligence: Early experiments with gpt-4. arXiv preprint 2Xiv:2303.12212. 2023.
Gao, Y., Alon, D.,, and Metzler, Impact of preference noise the alignment performance of generative lan- guage models. arXiv preprint arXiv:2404.09824 2024.
Bukharin, A., Hong, I., Jiang, H.. Li, Z., Zhang, Q., Zhang, Z., and Zhao, T. Robust reinforcement learning from corrupted human feedback. In The Tiihth- Annual Conference on Neural Information Processing Systems.
Ghosh, A., Kumar, H., and Sastry, P. S. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D.,, Freire, P., et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15211, 2023.
Han, Y., Pu, Y., Lai, 7., Wang, C., Song. S., Cao, J., Huang, W., Deng, C., and Huang, G. Learning to weight sam- ples for dynamic early-exiting networks. In European Conference on Computer Vision, pp. 362-378. Springer, 2022.
10

---

ROPO: Robust Preference Optimization for Large Language Models
Jiang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C.,
Liu, S., Niles-Weeds J., Razavian, N., and Fernandez- Granda, C. Early-learning regularization prevents rization of noisy labels. Advances in neural information processing systems, 33:20331 10931120032,, 2020.
Chaplot, D. S., Casas, D. d. I., Bressand, F,, Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b, arXiv preprint arXiv:2310.06825, 2023.
Jiang, L., Zhou, Z., Leung. T., Li, L.-J,, and Fei- Fei, L. MentorNet: Learning data-driven curricu- lum for very deep neural networks on corrupted la- bels. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2304-2313. PMLR, 10-15 Jul 2018. URL https: / v80 v80/jiangl$0 .e..ml....mm html.
Liu, T., Zhao, Y., Joshi, R., Khalman, M., Saleh, M.,
Liu, P. J., and Liu, J. Statistical rejection pling improves preference optimization. arXiv preprint arXiv:2309.0660 2023b.
Liu, Y. and Guo, H. Peer loss functions: Learning from noisy labels without knowing noise rates. In International conference machine learning, pp. 2020.
tttesttss///poeeeeee /ptp//eeooongsssseoeeeeep.p...se mlr malr.press.
PMLR,
Kim, S., Bae, S., Shin, J., Kang, S., Kwak, D., Yoo, M., and Sco, M. Aligning large language models through synthetic feedback. arXiv preprint 1rX1v:2305.13735, 2023.
Lou, X., Zhang, J., Xie, J., Liu, L.,, Yan, D., and Huang, K. Spo: Multi-dimensional preference sequential align- ment with implicit reward modeling. arXiv preprint arXiv:2405. 12739. 2024.
Marion, M., Üstün, A., Pozzobon, L.., Wang, A., Fadaee, M., and Hooker, S. When less is more: Investigating data pruning for pretraining ilms at scale. arXiv preprint MxXiv:230.04564 2023.
Korbak, T., Shi, K., Chen, A., Bhalerao, R. V., Buckley, C., Phang, J., Bowman, S. R., and Perez, E. Pretraining lan- guage models with human preferences. In International Conference on Machine Learning, pp. 17506-17533. PMLR, 2023.
Mitchell, E. A note on dpo with noisy preferences & relationship to ipo, 2023. URL https://
Lee, H., Phatale, S., Mansoor, H., Lu, K., Mesnard, T., Bishop, C., Carbune, V., and Rastogi, A. Rlaif: Scal- ing reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309. .00267, 2023.
ericmitched ai htti/cpp.ppp pdf.
Munos, R., Valko, M., Calandriello, D., Azar, M. G., Row- land, M., Guo, Z. D., Tang, Y., Geist, M., Mesnard, T., Michi, A., et al. Nash learning from human feedback. arXiv preprint arXiv:3312.0088:
Lee, S., Park, S. H., Kim, S., and Seo, M. Aligning to thou- sands of preferences via system message generalization. arXiv preprint arXiv:2405.1799 2024.
2023.
Ouyang, L.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., al. Training language models follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-2777 2022.
Li, T., Chiang, W.-L., Frick, E., Dunlap, L., Wu, T., Zhu, B.,
Gonzalez, J. E., and Stoica, I. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arx:::2406. 1939, 2024.
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacae- val: An automatic evaluator of instruction-follooinng models. https:/ //g/ttttbb////// wwt../cwtttttae..eeeeeeeeo /ab/ alpaca_evas 2023a.
Paul, M., Ganguli, S., and Dziugaite, G. K. Deep learning on a data diet: Finding important examples early in training. Advances in Neural Information Processing Systems, 34: 20596-20668 2021.
Peng, B., Li, C., He, P., Galley, M., and Gao, J. Instruc- tion tniing with gpt-4, arXiv preprint arXiv:2304.03277, 2023.
Li, Y., Wei, F., Zhao, J., Zhang, C., and Zhang, H. Rain: Your language models can align themselves without fine- tuning. arXiv preprint arXiv:2309.00 0.0124, 2023b.
Pereyra, G., Tucker, G., Chorowski, J., Kaiser, E., and Hinton, G. Regularizing neural networks by penal- izing confident output distributions. arXiv preprint arXiv: (tv:170). 1.1.:1700.06548. 2017.
Lin, B. Y., Ravichander, A., Lu, X., Dziri, N., Sclar, M., Chandu, K., Bhagavatula, C.,, and Choi, Y. The unlocking spell on base Ilms: Rethinking alignment via in-context learning. arXiv preprint arX2v:2312.01552, 2023.
Pleiss, G., Zhang, T., Elenberg, E., and Weinberger, K. Q. Identifying mislabeled data using the area under the mar- gin ranking. Advances in Neural Information Processing Systems, 33:470444117 17056, 2020.
Liu, H., Sferrazza, C., and Abbeel, P. Chain of hindsight aligns language models with feedback. arXiv preprint arXiv:230..0267. 2023a.

---

ROPO: Robust Preference Optimization for Large Language Models
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. Direct preference optimization: Your language model is secretly reward model. arXiv preprint uxiv:2305.18225 2023.
Swayamdipta, S., Schwartz, R., Lourie, N., Wang, Y., Ha- jishirzi, H., Smith, N. A., and Choi, Y. Dataset car- tography: Mapping and diagnosing datasets with train- ing dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9275-9292 2020.
Ramesh, S. S., Hu, Y., Chaimalas, I. Mehta, V., Sessa, G., Bou Ammar, H., and Bogunovic, I. Group robust preference optimization in reward-free rlhf. Advances in Neural Information Processing Systems, 37:37100 37137, 2024.
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction model. https: /gi thub wwt.c/co///t/eeeeeee stanford stanford_alpaca 2023.
llama
Ren, M., Zeng, W., Yang, B., and Urtasun, R. Learn- ing to reweight examples for robust deep learning. In International conference machine learning, pp. 4334- 4343. PMLR, 2018
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhos- ale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint -xXiv:2307.09288 2023.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347 2017.
Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Ra- sul, K. Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310. 16944, 2023.
Shen, T., Jin, R., Huang, Y., Liu, C., Dong, W., Guo. Z., Wu, X., Liu, Y., and Xiong, D. Large language model alignment: A survey. arXiv preprint urXiv:2309.5502 2023.
Völske, M., Potthast, M., Syed, S., and Stein, B. TI; dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pp. 63, 2017.
Shu, J., Xie, Q., Yi, L., Zhao, Q.. Zhou, S., Xu, Z., and Meng, D. Meta- weighttetttt Learning an explicit mapping for sample weighting. Advances neural information processing systems, 32, 2019.
Wang. B., Zheng, R., Chen, L., Liu, Y., Dou, S., Huang, C., Shen, W., Jin, S., Zhou, E., Shi, C., et al. Secrets of rlhf in large language models part ii: Reward modeling. arxiv preprint arxi:22401 ,60000, 2024a.
Song, F., Yu, B., Li, M., Yu, H., Huang, F., Li, Y., and Wang, H. Preference ranking optimization for human alignment. arXiv preprint arXiv:2306.1 17492, 2023.
Song, H., Kim, M.. Park, D., Shin, Y., and Lee, J.-G. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Wang, C., Jiang, Y., Yang, C., Liu, H., and Chen, Y. Be- yond reverse kl: Generalizing direct preference optimiza- tion with diverse divergence constraints. arXiv preprint arXiv:2309. 16240, 2023a.
Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., and Mor- cos, A. Beyond neural scaling laws: beating power law scaling via data pruming. Advances in Neural Information Processing Systems, 35:19523-19558 2022.
Wang. H., Lin, Y., Xiong, W., Yang. R., Diao, S., Qiu, S., Zhao, H., and Zhang, T. Arithmetic control of lims for diverse user preferences: Directional preference alignment with multi-objective rewards. arXiv preprint arXiv.2402.18571. 2024b.
Stiennon, N., Ouyang, L., Wu. J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33: 3008-3022 2020.
Wang. J., Wang, H., Sun, S., and Li, W. Aligning language models with human preferences via a bayesian approach. arXiv preprint arXiv:2310.05788 2023b.
Strobl, C., Wickelmaier, F., and Zeileis, A. Accounting for individual differences in bradley-terry models by means of recursive partitioning. Journal of Educational and Behavioral Statistics, 36(2):135-153: 2011.
Wang. Y., Ma, X., Chen, Z., Luo, Y., Yi, J., and Bailey, J. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF international conference computer vision, pp. 322-330, 2019.
Swamy, G., Dann, C., Kidambi, R., Wu, Z. S., and Agarwal, A. A minimaximalist approach to reinforcement learning from human feedback. arXiv preprint 3XXi:2441.000555 2024.
Wang. Y., Zhong, W., Li, L., Mi, F. Zeng X., Huang, W., Shang, L., Jiang, X., and Liu, Q. Aligning large language models with human: A survey, arXiv preprint arXiv:2307.12966, 2023c.
12

---

ROPO: Robust Preference Optimization for Large Language Models
Wu, J., Xie, Y., Yang, Z., Wu, Chen, J., Gao, J., Ding, B., Wang, X., and He, X. Towards robust alignment of lan- guage models: Distributionally robustifying direct pref- erence optimization. arXiv preprint arXiv:2407.07880, 2024.
Yuan, Z., Yuan, H., Tan, C., Wang, W., Huang. S., and Huang, F. Rrhf: Rank responses align language mod- els with human feedback without tears. arXiv preprint arXiv:2304.052 0.00000 2023.
Zhang, Z. and Sabuncu, M. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
Wu, Z., Hu, Y., Shi, W., Dziri, N., Suhr, A., Ammanabrolu, P. Smith, N. A., Ostendorf, M., and Hajishirzi, H. Fine- grained human feedback gives better rewards for language model training. arXiv preprint arXiv:2306.0166 6933, 2023.
Zhao, Y. Khalman, M., Joshi, R., Narayan, S., Salch, M., and Liu, P. J. Calibrating sequence likelihood im- proves conditional language generation. arXiv preprint DxXiv:2100000455, 2022.
Xiong, W., Dong, H., Ye, C., Wang, Z., Zhong, H., Iterative prefer- ence learning from human feedback: Bridging the- ory and practice for rlhf under kl-constraint. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023.
Ji, H., Jiang, N., and Zhang,
Zhao, Y.. Joshi, R., Liu, T., Khalman. M., Saleh, M., and Liu, P. J. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint 23X.:23335.1022. 2023.
Xu, J., Yang. R., Qiu, S., Luo, F., Fang. M., Wang, B., and Han, L. Tackling data corruption in offline reinforce- ment learning via sequence modeling. In The Thirteenth International Conference on Learning Representations, 2025.
Zheng, L.,, Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z.. Zhuang, Y., Lin, Z., Li, Z., Li, D.. Xing, E., al. Judging Ilm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36: 46595-46623, 2023.
Yan, Y., Lou, X., Li, J., Zhang, Y., Xie, J., Yu, C., Wang, Y., Yan, D., and Shen, Y. Reward-robust rihf in Ilms. arXiv preprint 2509 2024.
Zheng, L., Chiang, W.-L.,, Sheng. Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D.. Xing, E., al. Judging lm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024a.
Zhou, C.,, Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L.,, al. Lima: Less is more for alignment, arXiv preprint arXiv:2305. 1206, 2023.
Yang, R., Zhong, H., Xu, J., Zhang, A., Zhang, C., Han, L., and Zhang, T. Towards robust offline reinforcement learning under diverse data corruption. In The Twelfth International Conference on Learning Representations.
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv: 1909 005993, 2019.
Yang, R., Bai, C., Ma, X., Wang. Z., Zhang, C., and Han, L. Rorl: Robust offline reinforeement learning via conservative smoothing. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K. and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 23851- Curran Associates, Inc.,
2022. URL htttps:/pproceedingg. mee..eee cc/paper_files//peeeeeeeeeeceeep filees/paper/2222//lll ootp:aaafeeaattta--aa---eooorrre
pdf.
Yang, R., Pan, X., Luo, F. Qiu, S., Zhong, H., Yu, D., and Multi-objective alignment
Chen, J. of foundation models with dynamic preference adjust- ment. arXiv preprint arXiv:2402.10200 2024b.
Ye, X., Li, X., Liu, T., Sun, Y., Tong, W., et al. Active negative loss functions for learning with noisy labels. Advances in Neural Information Processing Systems, 36: 6917-6940, 2023.
13

---

(M) ROPO: Robust Preference Optimization for Large Language Models
A. ROPO Framework
this section, we describe the overall iterative framework, provide the pseudocode, and analyze the computational cost for ROPO.
Algorithm ROPO
Input: dataset D, B, a, p, K, number of epochs M, SFT model saft.
Initialization:
(( <-
(0)
Tg -- Tift
for m 1..... M do
(m)
<- " (m-1) with frozen parameters
" ref
Noise-tolerant Training:
(m) (m-1) on D(m-1) with "ref and Topo in Eq. (6) for one epoch Obtain (m)) by training %g
Noisy Sample Filtering:
(m) Compute Trppa with m(mr) and re for D
Diop
Poop-p <- samples with top-p ROPO loss value D
D(m)
bot (1-p) + samples with bottom-(1 p) ROPO loss value in
Robustness-guidel Rejection Sampling:
Dnew + Ø
(m) for (x. (2)11.2) in top-p do
Sample responses YI.... YK to x using T8 (m) Deand + (1, Vk, > Y& x)] U ((x Y2 Yz x))
Dnew f Duew U (arg minzeDDann
Eropo (2, mm)))
end for D(m)= f (m)
bot-(1-p) U Unew
end for (M)
+ (M- with frozen parameters 0
i
n
Obtain T (M) by training
To (M-1) on D(M-1) with π Ne and Propo in Eq. (6) for one epoch Output: (M)
As shown Figure 3 and Algorithm 1, ROPO iteratively carries out three stages: noise-tolerant training, noisy sample filtering, and aabustness-guider rejection sampling. Specifically, in the Ist (M 1)th epochs, we first train the model using the ROPO loss fropo- After training for epoch, compute the value of ROPO loss for all samples in the original dataset and divide them into two subsets (i.e., Drop-p and Dbot-(1-p)) according to their loss values. Then, the vobustness-guidde rejection sampling stage generates new samples Dnew based on Dtop-p- The new samples are used together with Dbot-(1-p) as training samples for the next epoch. In the last epoch, we only perform the noise-tolerant training stage and then get the final model.
Computational Cost Analysis. ROPO introduces additional costs for the noisy sample filtering and robustness-guit guided rejection sampling stages compared with non-iterative methods. However, these additional costs acceptable compared to the training cost and almost negligible in the entire chain of real-world large-scale LLM training. The notations in the computational cost analysis is shown in Table 4. We have
CROPO
MC + (M- 11)( - Crs)
Cnon-it
MC
Since the main cost of the noisy sample filtering stage per epoch is compute the loss of N samples, we have Cn &e NCoss &e 4N Cforward- As for the rejection sampling stage, the main costs per epoch come from pNK response generation and 2p K loss computations, hence Cps R3 pNKC gen + 22NNKKK. K F pN KC + 8pN K Corward = 9pN K Ctowward- Because the training process mainly involves loss computation for two models (i.c., the reference model and the model being trained) and gradient propagation, we have Ct Rs N(Cross + Obackward) N(4Cforward + Chackward).
14

---

ROPO: Robust Preference Optimization for Large Language Models
Table 4. Notations in the computational cost analysis.
Notation
Description
N,M, K
Please see Algorithm 1
CROPO
Cost of ROPO
non-iit
Cost of non-iterative methods
Ctr
Cost of (noise-tolerant) training per epoch
Cfl
Cost of noisy sample filtering per epoch
Crs
Cost of coanstness-guider rejection sampling per epoch
Closs
Cost of computing the loss for a sample (x, y1, without gradient propagation
Cgen
Cost of generating a response y for a query x
C forward
Cost of computing the log-likelihood for query-response pair (x,
C backward
Cost of computing the gradient and updating parameters for a sample (x, Y1,
Table 5.
The estimated noise rate in commonly-seen datasets. This table is from (Gao et al., 2024).
Dataset
Noise rate (%)
Reference
MT-Bench
15,0-37.0
(Zheng et al., 2023)
TL;DR
21.3-27.0
(Lee et al., 2023)
CBArena
22.0-36.0
(Zheng et al., 2023)
AntHH 27.9-30.9
(Lee et al., 2023)
SHP
35.5-41.9
(Cui et al., 2023)
WebGPT
34.8
(Cui et al., 2023)
Therefore, Eq. (7) leads
CROPO Cnou-it
M (4Crorwwrd + Cbackward) + (M - 1) 9pk))torwaad
forowardd + backward)
(4 (M 1)
Clorward
= 1+
M
4Crorward Cbackward
=1 (4 9pK) 11
(8)
M
4 Obackward/ Ctorward
where the ratio Cbackwardd Corwaard is approximately 3 for LLMs. Take p 0.2, = M = 3 example, without considering inference acceleration, we can estimate that CROPO 1.6Cnon-itt In practice, we can use inference acceleration methods to increase Obackward/ Cforward thereby further reducing the additional cost of ROPO. Compared with the computational cost of the entire chain of real-world LLM training (including continual pre-training and SFT), the additional cost is almost negligible.
B. Related Work
Preference Alignment of LLMss. The most representative paradigm of preference alignment is RLHF (Ziegler et al., 2019; Ouyang et al., 2022), which involves training a reward model to capture human preferences and then steering LLMs towards producing high-reward responses through RL algorithms (Schulman et al., 2017). However, in real applications, RL-based methods are complex and prone to instability during training (Rafailov et al., 2023; Wu et al., 2023; Yuan et al., 2023). Therefore, many recent studies have explored more straightforward and stable alternatives for RLHF (Yuan al., 2023:
15

---

ROPO: Robust Preference Optimization for Large Language Models
100
Human-3
66.0
57.0
62.5
62.5
75.0
73.0
100.0
95
Human-2 68.0
58.0
58.5
66.5
76.0
100.0
73.0
90
Human-1
67.0
59.0
62.5
64.5
100.0
76.0
75.0
85
80
Qwen
66.8
61.7
75.9
100.0
64.5
66.5
62.5
75
Llama
60.3
60.4
100.0
75.9
62.5
58.5
62.5
70
GPT-4
59.1
100.0
60.4
61.7
59.0
58.0
57.0
65
Label
100.0
59.1
60.3
66.8
67.0
68.0
66.0
-60
Label
GPT-4
Liama
Qwen
Human-1
Human-2
Human-3
Figure The inter-ande annettaontttoor agreement heap map on the TL;DR dataset. The "Label", "Llama", and "Qwen" refer to original labels in the dataset, 1lama-2-70B, and Qwen-Max, respectively. We assess the preferences of human annotators across 200 randomly selected samples extend evaluation to 1,000 samples L.Mss, which include initial 200 samples.
Rafailov et al., 2023; Song et al., 2023; Wang et al., 2023b; Lin et al., 2023; Li et al., 2023b; Wang et al., 2023c; Zhao et al., 2022). Among these studies, the most promising direction is to use a contrastive or ranking loss to calibrate the likelihood of the output sequence. Specifically, RRHF (Yuan et al., 2023) introduces a ranking loss to encourage larger likelihoods for better responses and smaller likelihoods for worse responses. Besides, another important work is DPO (Rafailov et al., 2023), which implicitly optimizes the same objective existing RLHF-based methods and enables human preference alignment directly with simple cross-entropy loss. In addition to the aforementioned methods using data in the form of (x. (x, 12,2,,,,44 c), where is the preference label, some recent studies (Duan et al., 2024; Ethayarajh et al., 2024; Chen et al., 2024) have also used data in the form of yx,y. y.y. c). where e is an annotation of the response y, for preference alignment.
Learning from Noisy Data. In the era of deep learning, there is an urgent demand for large-scale training samples, and the cost of manually annotating or filtering data is prohibitively expensive most circumstances (Song et al., 2022). Therefore, learning from noisy data has become increasingly important, which primarily falls into three categories. The first category sample-selection based methods (Swayamdipta et al., 2020; Pleiss al., 2020; Paul al., 2021; Sorscher al., 2022), which identify high-quality samples before training and filter out noisy samples. For example, (Swayamdipta al., 2020) uses the training dynamics to identify valuable samples. The second category is weighting-basee methods, which assign greater weights for important samples and lesser weights for noisy samples (Ren et al., 2018; Han et al., 2022: Shu et al., 2019). Besides, another important area research is dedicated to the design of loss functions that are robust to noise (Ghosh et al., 2017: Wang et al., 2019; Zhang & Sabuncu, 2018). The findings in (Ghosh et al., 2017) indicate that the traditional cross-entropy loss is sensitive to the label noise, while symmetric loss functions are robust to such noise. Furthermore, recent advances LLMs have also underscored the essential role of data quality in both pre-training and supervised fine-tuning (SFT) phases of LLMs (Marion et al., 2023; Zhou al., 2023; Korbak et al., 2023).
Rejection Sampling. The rejection sampling is popular approach of data augmentation to improve the data quality and performance in existing preference alignment methods (Dong et al., 2023; Liu et al., 2023b; Xiong et al., 2023; Yang et al., 2024b; Wang et al., 2024b). Specifically, (Dong et al., 2023) ranks newly-collected responses based on their rewards and selects the highest ranked one to add to the dataset. To address the issue the excessively high rejection rate and thus improve the effectiveness of rejection sampling, (Xiong et al., 2023) proposes multi-step sampling technique, which also
16

---

ROPO: Robust Preference Optimization for Large Language Models
requires external reward model. Besides, (Wang al., 2024b) and (Yang al., 2024b) consider rejection sampling for the multi-objective preference alignment, where (Wang al., 2024b) projects multi-objective reward vectors onto one dimension and then selects samples based the scalar rewards, while (Yang et al., 2024b) augments samples near the Pareto front of multi-dimensional rewards, leading to a strong multi-objective alignment performance. Compared to the aforementioned methods, which all rely on rewards provided by external models, our robustness- guided rejection sampling technique selects new samples based on loss values that reflect the quality of the samples. Moreover, our technique benefits from being independent external LLMs, thus leading to computational and memory efficiency.
C. Discussion on Preference Noise
Due to the inherent differences in annotators' preferences, the preference noise is usually unavoidable. In this section, we discuss the definition and identification of preference noise.
Before giving the definition of preference noise, we invite our readers to pay attention to the following two points.
1.
This paper focuses on noisy preferences rather than the more general noisy preference data. The former refers specifically to the noise in preference labels, while the noise corresponding the latter may come from multiple factors such as preference labels, text quality, and the matching degree between queries and responses. It is interesting and meaningful to study a wider range of noisy data, but it is beyond the scope of our paper and related work (Mitchell, 2023; Chowdhury et al., 2024; Gao et al., 2024).
2. Like related work (Mitchell, 2023; Chowdhury et al., 2024; Gao et al., 2024), this paper is based on the Bradley-Terry (BT) model. The BT model assumes the existence of a "gold", latent, and inaccessible reward model r". Then, we can express the BT preference probability (y) > Y2 x) for a sample (x, y1. using the reward model r*. Intuitively, the BT model assumes that there are mainstream preferences human society that reflect values such as peace, friendliness, honesty, etc. Differently, there are also studies on multifaceted or multidimensional preferences (Lou et al., 2024; Lee et al., 2024), but defining noise for them is challenging because is difficult have "ground truth" label. Therefore, our following discussion is based on the assumption of the BT model.
Definition of preference noise. For a sample yI, é). if P* (2) > 0.5, then the sample is clean; otherwise, the sample is noisy. Because the BT model usually represents preferences that are consistent with mainstream values of human socicty, so the formation of such noise is usually caused by the personal preferences or cognitive biases of annotators. Please note that the annotators can be humans or LLMs.
Identification and detection of preference noise. As mentioned above, the definition of preference noise is based on the inaceessible reward model, so we can never identify preference noise accurately. However, we can estimate the noise rate by using advanced LLMs as the proxy for the BT model or computing the inter-annotator agreement.
Using advanced LLMs as the proxy for the BT model (Gao et al., 2024). Given dataset, we can prompt advanced LLMs (e.g., GPT-4, Llama-3-708B -aae------B---ttree (Dubey et al., 2024), and Qwen-2-72BB-Inttree Innstrcct (Yang et al., 2024a)) to identify the noise. For example, we can provide them with rules and ask them to rate or rank the responses in the dataset. If a sample's s new label is different from its original label, it is identified as noisy. The stronger the proxy LLM, the more reliable the noise identification.
Computing the different annotators (humans or LLMs) to relabel the dataset and calculate the agreement between them. For this approach, should try to ensure that all annotators have the same criteria, and similar cognition and ability. Suppose that we have n. annotators and the agreement between annotators i and j is O I aij S 1, then the estimated noise rate is w(n-1) Eij= (1 - a:y). Take the TL;DR dataset as an example. We employ GPT-4, Llama-2-706 (Touvron et al., 2023), Qwen-Max (Bai et al., 2023), and three human annotators to relabel the TL:DR dataset. The human annotators are three of the four volunteers mentioned in Appendix E.6. The inter-annotato agreement heat map is shown in Figure 5, which indicates an estimated noise rate of 17.6%.
agreement (Ouyang et al., 2022; Wang et al., 2024a; Bai et al., 2022). We can employ
1
Besides, Table 5 from (Gao et al., 2024) summarizes the estimated noise rate in some commonly-seen datasets. As can be seen, the existence of preference noise is ubiquitous and cannot be ignored, which highlights the importance of studying robust preference optimization approaches.
17

---

ROPO: Robust Preference Optimization for Large Language Models
D. More Details about Experiments
D.I. Tasks and Datasets
We run experiments on two dialogue datasets (i.e., UltraFeedback Binarized and Alpaca Comparison) and one post summarization dataset (i.e., TL;DR).
The Ulrraeeddbck Binarized dataser' is pre-processed version of the UltraFeebbcck dataset (Cui et al., 2023), which contains 64,000 prompts and each prompt has four model responses from various LLMs. Based on the score assigned by GPT-4, (Tunstall et al., 2023) selects two responses for each prompt and construct UltrFeedback Binarized for the preference alignment.
The Alpaca Comparison dataset contains 52,000 queries from the widely-used Stanford Alpaca dataset (Taori et al., 2023). (Peng et al., 2023) generates several responses using GPT-4 and other LLMs including text-davinci-003 to each query and employs GPT-4 to assign score for each response.
In the TL;DR dataset, each prompt is forum from Reddit, and the model is required to summarize the given forum Following (Rafailov et al., 2023), we use the Reddit TL;DR summarization dataset (Völske et al., 2017) along with human preferences collected by (Stiennon et al., 2020).
D.2. Baselines, Models, and Hyperparameters
Baselines, Our baselines are DPO (Rafailov et al., 2023), IPO (Azar et al., 2023), and two approaches that use the label smoothing technique to alleviate the impact of noise, i.c., rDPO (Chowdhury et al., 2024) and cDPO (Mitchell, 2023).
Specifically, given preference data (x, V2) with the ranking label y1 > Y2 x, the objectives of our baselines are
T | x) Teaf
Ty x)
Lapn = llog
log
-Blog
(9)
(v1 x)
Teef (y2 xx)
T (y1
x)
log Ty x)
1
2
lipo
log
(10)
Te (y1
x)
Teel (y2 x)
2/9
1- E
(y1 x)
TT x)
lddpo
logo
Blog
|3 3log
1 - 2e
Tref (y x)
Teel (y2 x)
(y² |x)
To T x)
log a Blo
3 log
(11)
2e
Tre! Tre( |x)
zee (y1 xx)
To x)
To xx)
ledpo
(1 <) log
log
-Blog
Trel (y1 x)
Teel (y2 x)
To
x)
T x)
E log a
Bog
3 log
(12)
Feol (yz x)
Neel (y1 x)
where (0, 1) and |3 E (0, 1) are hyperparameters.
Models. We use Mistral-7B (Jiang et al., 2023) and Llama-2-7B (Touvron et al., 2023) as base models for all baselines and datasets. On UFB, we use Zephyr-7B-SFT-B (Tunstall et al., 2023) as the SFT model for experiments with Mistral-7B, and adopt the result of Zephyr-7B-8 (Tunstall et al., 2023) on AlpacaEval (90.60) as the performance of DPO under no artificial noise. In other cases, we fine-tune base models on the preferred responses (SFT targets) form the SFT models.
Hyyperparameters We run all experiments on 16 NVIDIA A100 GPUs (80 GB). Unless otherwise noted, batch size of 512 to train all models. For all hyperparameters except for of label smoothing, search for the best one on each dataset without artificial noise and use the same setting for 20% and 40% % artificial noise.
global
For all methods, we search the best learning rate in (le-5, 5e-6, le-6, 5e-7, le-7) and the best 33 in 10.1, 0.5). We find that the best performing learning rate is le-6, and the best /3 for dialogue and post summarization are 0.1 and 0.5, respectively. This conclusion is consistent with that in (Rafailov et al., 2023).
18

---

ROPO: Robust Preference Optimization for Large Language Models
For ROPO,
14 and p 0.2 in the main experiments. In ablations (Section 4.2), we a in (6, 14, 30),
4a which makes (I+a}2 be around 1 respectively, and tune p in {0.1, 0.2, 0.3). We set K = 3 for the rejection sampling. For rDPO and cDPO, we search the best E in {0,05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45} for each dataset and each proportion of artificial noise.
D.3. Evaluation
For models trained on UFB and Alpaca Comparison, we evaluate them on the AlpacaEval benchmark (Li et al., 2023a) a widely used dialogue benchmark- comparing their outputs with those text-davinci-003 (recommended by the benchmark for comparison). AlpacaEval contains 805 queries in various domains and exhibit strong concordance with ground truth human annotators. For TL;DR, randomly select baselines by comparing their outputs with the chosen responses (SFT targets) for the queries.
500 500 queries from the test split of it and evaluate ROPO and
Following existing studies (Rafailov et al., 2023; Tunstall et al., 2023), we employ GPT-4 as the referee to conduct head- to-head comparisons, using the win rate as the metric. On AlpacaEval, we conduct evaluations using the API provided by AlpacaEval. On TL;DR, we use the following prompt, which is similar to that used by AlpacaEval, to conduct GPT-4 evaluation.
You are helpful assistant that ranks models by the quality of their summaries of given forum posts.
I want you to create leaderboard of different of large-language models. To do so, will give you the instructions (forum posts) given to the models, and the responses of two models. Please rank the models based on which responses would be preferred by humans.
Here is the post:
<Forum Post>
Here are the outputs of the models:
Model 1: <Summary 1>
Model 2: <Summary 2>
Now please rank the models by the quality their answers, so that the model with rank 1 has the best output. Please provide the ranking that the majority of humans would give. Your response should use the format: Better: <Model 1 or Model 2>
E. More Experiments
E.1. Experiments on Llama-2-13B-Ban and Llama-3-70B-Bas
To evaluate ROPO and baselines on models larger than 7B, supplement experiments Lama-2-11B--Bas and Llama-3- 70B-Base.
Experiments on Llama-2-13B-Bas We run SFT on Ultrahhat-200k for one epoch with the learning rate of 1e-5, the global batch size of 128, the weight decay of 0.1, and a cosine-type learning rate scheduler. Then, we fine-tune the SFT model with ROPO and baselines for two epochs on UFB (under artificial noise ratio of and 20%) with the learning rate of le-6 and the global batch size of 512. In the experiments, tune B in 10.1, 0.5, 1.0] for IPO and tune in 10.1, 0.2, 0.3, 0.4] for cDPO and rDPO. The results shown in Table 6.
fix 14 and p = 0.2 for ROPO without tuning them, and
Experiments on Jlama-3-7BB-Rase We run SFT on UltraChat-2006 for one epoch with the learning rate of 1e-5, the global batch size of 128, the weight decay of 0, and a cosine-type learning rate scheduler. Then, we fine-tune the SFT model with ROPO and DPO for two epochs on UFB (under artificial noise ratio 0 and 20%) with the learning rate of 5e-7 and the global batch size of 512. We fix a = 14 and p = 0.2 for ROPO without tuning them. The results are shown Table From the results we can conclude that: (1) 70B models outperform 7B/13B models in terms of win rate. However, the
19

---

ROPO: Robust Preference Optimization for Large Language Models
Table rates (%) of different methods vs SFT targets under different proportions (i.e., and 20%) artificial noise, evaluated by GPT-4 on AlpacaEval.
Dataset
UFB
Model
Method
0%
20%
DPO
82.98
80.50
IPO
81.99
79.75
Llama-2-11 2-23B
rDPO
81.37
80.87
cDPO
82.36
80.50
ROPO
83.23
82.98
Table 7. Win rates (%) of ROPO/DPO vs SFT targets under different proportions (i.e., 0 and 20%) of artificial noise, evaluated by GPT-4 on AlpacaEval.
Dataset
UFB
Model
Method
0%
20%
DPO
94.29
88.70
Llama-2-70B
ROPO
95.53
94.04
Table rates (%) of different methods vs SFT targets under different proportions (i.e., GPT-4 on AlpacaEval.
and 20%) artificial noise, evaluated by
Dataset
UFB
Model
Method
0%
20%
DPO-RM
69.69
68.32
cPPO-RM
68.45
67.95
Mistral-7B
:PPO-RM
68. 70
67.33
ROPO-RM
69.94
70.43
performance of the models trained with DPO still has non-negligible drop under 20% artificial noise. (2) Our ROPO still significantly exceeds DPO on the scale of 70B.
E.2. Experiments reward modeling
In the main text of our paper, the baselines reward-free. Considering the reward modeling (RM) still plays important role in many real-world LLM applications, although RM is not our focus, we supplement experiments on RM with Mistral- 7B-Base to test the potential of ROPO in scenarios including reward modeling. Given sample (x, y1, 0), if we denote P (fr(x,y1) r(x, y2)), then the RM-training losses of ROPO and our basclines are as follows.
DPO-RM: log
cPPO-RM: -(1 - log eloog1 - P)
rPPO-RM: -- log + 1-25 log(1 - P)
ROPO-RM (Ours): - -44r/ + a)z) P (4a2/(1+a)3) (1 (1-P)
We train 7B-vral-7B-v0.1 on UFB for two epochs with the aforementioned losses to obtain reward models. Then, we use Best N Sampling (N = 16) to generate responses based on RMs and Mistral-7B- Se--SF--Bel (SFT model). We use the learning rate of 5e-6, the batch size of 512, and cosine-type learning rate scheduler. The results are shown in Table
20

---

ROPO: Robust Preference Optimization for Large Language Models
Table rates (%) of different methods vs SFT targets under noise coming from annotators' trust larger models over smaller ones, evaluated by GPT-4 on AlpacaEval.
Dataset
UFB
Model
Method
DPO
75.16
IPO
72.55
Mistral-7B
cDPO
76.27
rDPO
78.26
ROPO
80.50
Table 10. Win rates (%) of different methods SFT targets under noise coming from LLM preference comparisons, evaluated by
GPT-4 on AlpacaEval.
Dataset
UFB
Model
Method
DPO
84.22
IPO
84.84
Mistral-7B
cDPO
85.22
rDPO
86.21
ROPO
88.07
F.3. More practical noise settings
The experiments in the main text cover two types of practical noise as follows.
1. Practical noise coming from human comparisons. In the original TL:DR dataset, the preferences labeled by human annotators who compare the post-summaries generated by different models in pairs. This leads to unavoidable noise due to the diversity of human preferences.
2. Practical noise coming from LLM (GPT-4) rating, Each query (instruction) in the original UltraFeedback dataset has four responses coming from different models, GPT-4 scores them based on criteria such as citructtoonfollowinne honesty, helpfulness, etc. Then, for each query, the highest ranked response is selected "preferred", and one the remaining responses is randomly selected "dis-preferred", This leads to unavoidable noise due to the bias of GPT-4.
In this section, we explore another two practical noise settings Appendices E.3.1 and E.3.2.
E. EXPERIMENTS UNDER NOISE COMING FROM ANNOTATORS' TRUST LARGER MODELS OVER SMALLER ONES
is common practice to treat the response from a larger model the chosen (preferred) one and the response from a smaller model as the rejected (dis-preferred) one. Therefore, obtain new noisy preferences from UFB (each of query has four I.M responses) based the sizes of models that generate the responses. As shown in Table 9, under this practical noise setting, ROPO still significantly outperforms DPO and other baselines.
E.3.2. EXPERIMENTS UNDER NOISE COMING FROM LLM PREFERENCE COMPARISONS
We use Llama-3-70B 1---3-7---sttruue (Dubey et al., 2024), which one the most advanced open source LLM, to relabel the preferences in UFB. To make the labels as reliable as possible, we instruct the model to list the advantages each response. The prompt we use is as follows.
21

---

ROPO: Robust Preference Optimization for Large Language Models
For the given instruction and two responses and B), please answer: (1) which response is better overall, (2) the aspects in which is superior to B, and (3) the aspects in which B is superior to A.
Strictly ly adhere to the following rules:
1. Answer in bullet points, with each point starting with a gerund or adjective, excluding the words 'response A'' and "response B''
2. If a response has no superior aspects over another, output NONE.
Instruction:
(instruction)
Response A
[responseA}
Response
[responseB]
Your answer MUST STRICTLY follow the format as follows: Better <Choose A or B>
** better than B**
<First aspect for which is superior to B>
<Continue with other points if any>
*Why B is better than A**
<First aspect for which is superior to A>
<Continue with other points if any>
However, observe that about 30% of the labels different from those in the original UFB dataset. This shows that noise unavoidable due to the diversity in LLM preferences. Then, train Mistral-7B with different methods the new noisy dataset. As shown in Table 10, under this practical noise setting, ROPO still significantly outperforms DPO and other baselines.
Table I1. Performance of difference methods on Arena-Hard and MT-Bench. The hold font indicates the best result and an underline indicates second-hest result.
Benchmark
Arena-Hard
MT-Bench
Model
Method
0%
20%
40%
0%
20%
40%
DPO
10.7
8.5
6.3
7.3
5.7
4.3
IPO
9.2
7.9
7.3
7.2
5.9
4.9
Mistral-7B
rDPO
9.8
9.2
8.9
7.1
6.4
5.8
cDPO
10.3
9.0
8.4
7.2
6.2
5.2
ROPO
13.1
12.6
11.8
7.3
6.9
6.5
DPO
17.9
15.3
14.1
7.8
6.1
4.6
IPO
18.6
16.8
16.0
7.4
6.3
5.0
Llama-3-8B
rDP
18.3
17.5
17.1
7.5
6.9
6.1
cDPO
17.5
16.4
15.3
7.7
6.7
5.8
ROPO
20.5
19.6
18.5
7.7
7.0
6.7
22

---

ROPO: Robust Preference Optimization for Large Language Models
Table Win rates (%) different variants of DPO vs SFT targets under 0% and 20% artificial noise, evaluated by GPT-4 on AlpacaEval. The base model is Mistral-7B and the training dataset is UFB.
0%
20%
DPO DPO + NSF
90.60
86.21
90.06
85.09
DPO + NSF + RS
90.31
84.97
E.4. Experiments more benchmarks
To comprehensively explore the performance of ROPO and baseline methods, evaluate them another two widely-used benchmarks, i.e., Arena-Hard (Li et al., 2024) and MT-Bench (Zheng et al., 2023). The details of the benchmarks are as follows.
MT-Bench (Zheng et al., 2023) contains 80 two-turn conversations, each which has an open-ended instruction and a corresponding follow-up question. Due to the well-designed questions and the wide coverage of topics, MTT-ennch has become a widely-used benchmark to evaluate the multi-turn conversational and instruction-folloving abilities of AI models.
Arena-Haie Hard (Li et al., 2024) is challenging benchmark containing 500 single-turn conversations. Compared to AlpacaEval and MT-Bench,, Arena-Hard features better model separability, tighter confidence intervals, and achieves correlation of 98.6% with Chatbot Arena rankings (Chiang et al., 2024).
We evaluate ROPO and baseline methods using Mistral-7B and Llama-3-88 (Dubey et al., 2024). For Mistral-7B, we use the same models as evaluated on AlpacaEval in the main experiments. For Llama-3-8B, we first train a Llama-3-8B-Ban on UltraChat-2000 decay of 1, and consine-type learning rate scheduler), and then continue training with ROPO and baseline methods. The results shown in Table 11. As observed, under various artificial noise levels, ROPO consistently outperforms baseline methods in most cases and demonstrates superior robustness in noisy scenarios.
obtain an SFT model (one epoch with the learning rate of 1e-5, global batch size of 128, weight
E.5. Experiments of combining DPO with noisy samples filtering and rejection sampling
As shown in Figure 2, the distributions of the DPO loss on clean and noisy samples very similar, and the difference gradually decreases the training proceeds. This shows that the DPO loss is prone to overfitting to noise, hence cannot serve as a reliable measure of model uncertainty in noisy scenarios. In this section, to further support our claim, we conduct experiments of combining DPO with noisy samples filtering (NSF) and rejection sampling (RS) using Mistral-7B as the base model and UFB as the training dataset. Please note that our proposed robustness-guid guided RS only works on the filtered samples, so we not conduct experiments combining DPO and RS alone. The results are shown Table 12. As can be seen, the incorporation noisy samples filtering and rejection sampling degrades the performance of DPO, especially at 20% artificial noise.
E.6. Human evaluation
We invite four lab members with no conflicts of interest this paper to serve as volunteers to conduct human evaluations. Two of them are PhDs and the other two doctoral students, believe that they have the ability understand the evaluation rules and make reliable judgments.
We randomly select 200 queries from the AlpacaEval benchmark. Then, we pair the corresponding responses of ROPO, DPO, and rDPO under 0% and 20% artificial noise to form four groups: (1) ROPO vs DPO under 0% artificial noise, (2) ROPO vs rDPO under 0% artificial noise, (3) ROPO vs DPO under 20% artificial noise, and (4) ROPO vs rDPO under 20% artificial noise.
'https: www.ac..c.....ooooooooo...oooo.. wtt.aaacc.cc/ccaaaaaaaammaaaaaaa 1ama /ate/Metaaait ama - 3-88 hottts://huggin wgfacinnnacc.... cttp htt.aa/eeeeaaaaaaaaeaaaaaaaaaaee trachat_200k
23

---

ROPO: Robust Preference Optimization for Large Language Models
Table 13. Human evaluation of ROPO DPO and ROPO vs rDPO on AlpacaFval. The base model is Mistral-7B and the training dataset is UFB. The #(Win), #(Ti)), and #(ose) are the numbers of ROPO's wins, ties, and ROPO's losses.
Artificial Noise Ratio
0%
20%
#( #(Wn))
#(Te)
# #(Lose)
WR (%)
#(Wn)
#(Tie)
#(Lose)
WR (%)
ROPO vs DPO ROPO vs rDPO
77
69
54
55.8
103
59
38
66.5
84
63
53
57.8
89
64
47
60.5
Table 14, Win rates (%) of DPO with confidence penalty vs SFT targets under 20% and 40% artificial noise, evaluated by GPT-4 on AlpacaEval. The base model Mistral-7B and the training dataset is UFB.
20%
40%
DPO DPO + CP
86.21
82.67
86.96
81.86
For each group, randomly shufile the order of the queries and the order of responses in each pair. Each volunteer is in charge of one group. None of the volunteers know which method corresponds to each response. They are asked to compare the responses in 200 pairs and choose the better one. If they are unsure about which response is better, they can choose "Tie". During the evaluation process, we allow the volunteers to use translation tools and search engines.
We count the number of ROPO's 's wins, ties, and losses, and compute the win rate of ROPO by Ω = # The 200
results are shown in Table 13. We have the following interesting observations from the table: (1) The win rate of ROPO against DPO and rDPO is consistently over 55%, demonstrating ROPO's advantages over the baselines. (2) As the artificial noise rate increases, the win rate of ROPO increases to more than 60%, which shows the superiority of ROPO in noisy scenarios. (3) All four volunteers give least 29% tie judgments, indicating the limitations of human evaluation: is challenging for most human evaluators make reliable evaluations difficult tasks such as long-contex reasoning, coding, mathematics, etc. This highlights the importance of developing automated LLM evaluation tools.
E.7. Experiments of applying regularization strategies to DPO
In experiments in the main text, we have evaluate the performance of label smoothing (i.e., cDPO and rDPO) under noisy scenarios. The label smoothing techniques can be seen as regularization strategies applied to DPO. As shown in the experiments, they bring performance improvements over DPO under 20% and 40% artificial noise, but underperform ROPO. Their limited effectiveness might be attributed to the fact that rDPO and cDPO are noise-tolerant only under specific conditions: when the hyperparameter E exactly matches the noise proportion for rDPO, and when E = 0.5 for cDPO. Achieving these conditions in practice challenging due to the lack of prior knowledge about the exact noise proportion.
In this section, we explore another two widely-used types of regularization strategies in noisy scenarios, i.e. the normalized negative loss and confidence penalty.
Normalized negative loss (NNL) (Ye et al., 2023), such as normalized negative cross entropy (NNCE) and normalized negative focal loss (NNFL), are shown to be effective when combined with the cross-entropy loss (i.e., the DPO loss in preference optimization). However, when the problem is binary classification like preference comparison, NNCE and NNFL degenerate into constant terms. Specifically, for a sample (x, Y1. > Y2 | x), if we denote P y1) r(x, Y2)), then we have
log mimmiinPpp P) + log P 2 Mogmin(P.1) P) + log P + log(1 P)
Lnnce
1, if P < 0.5,
0, if P > 0.5,
24

---

ROPO: Robust Preference Optimization for Large Language Models
and
(1 min(P,1 P)) log (Pn(PP P) P)7 log
annf
-2(1 - min( 7 log min P) + P)r log P + P log(1 P)
1, if P.0.5
0, if P> 0.5.
Therefore, NNL does not work for DPO.
Confidence penalty (CP) (Pereyra al., 2017) is an entropy-aware regularizer for the cross-entropy loss, which prevents the model from making overconfident inferences. Specifically, for a sample (2. > Y2 x). if we denote Po (ro( - y2)), CP computes the entropy by
Hg = Po log Po (1 Pog( - Po).
Then, the CP regularizer is
cp = -Amax(0, Ho)-
We combine DPO with CP and tune the hyperparameters À and the range of X E (0.01,0.1) and 0.25, 0.5}. As shown in Table 14, we do not observe significant improvement over DPO in noisy scenarios. We speculate that the limited effectiveness of CP is because CP has no guaranteed noise-tolerance.
F. Mathematical Derivations and Theoretical Analysis
F.1. Proof of Theorem 3.1
Proof. As Eis W; Np is a hyperplane and wy E [0, 1] for = 1,...., N, {w: : w; E 10, 1), Σ
==1 wE = Np} is
compact. Because is compact, x S is compact. Therefore, the continuous & E wif (0:x(0, y 82 c(() T
(1)
admits an optimal solution (0", w²) on θ x
Assume that (0*;)(1),, yfi) .22 TT- ... < (0(xx)()))) (iN), (6x) Np. Then, we have
(i1) (61)
y22 ((x) -000 but with wi, < 1 for some 1 كSJS ,
i < (N, - 1) Np:
(13)
hence there exists ii > 0 for some N, < < N. By letting w/, = 1, w// = no, + w - 1, and w/, = w for k + j.l, N we have Σx= w/ and
will (0":x
(1)
20
Σ w/o (0);x())) yy .92
(kx) (is) glii),, nT8
111 :22 T00
N
ktil
+ w/, l (0";
(.66)) To
+ w/, l (0); (5c)
T
w
aloh), m8.
wi,
00";;0) (0xx((1))
(5g) (15) (.66)) no-
.2
+ m 8 (0;x((1)
(4) 260) T8-
/ x0;x (6)
2(1) Tg.
(14)
which leads contradiction. Therefore, we must have w for < & N, and 1 for N, k S N.
25

---

ROPO: Robust Preference Optimization for Large Language Models
F.2. Proof of Theorem 3.2 Proof. For é lápor we have
((xy1112)
(P* y2 x) 1) (1 Py1 > y2 x))n) y1,
+ (P y2 x) + PM y1 y2 xx)) 11)) 112
(x.1.32)
(P' y2 x) + - - (y1 y2 x)n) log Po > y2 /21x) x)
-(PP y2 x)n +1 PT (y1 y2 x)) 17) log Po(y1 > y2 x2|x))
(15)
Consider
f()) = -p" + +1 2p" 2p'n)) logp (2p
11 + 1- p" - -11) log(1 p),
(16)
we have
f'(p) = p² 11 2pn' 2p" 1) 1 pp 1)
(17)
1 p
From ['(p) we know that f decrease when S p² + 1j - 2p"n and increases when p 2 p" + - 2p"1), which means that f reaches its minimum at Po p" + (1 2p')) )n-
Therefore, Eq. (15) reaches its minimum when
Por (y1 > y2 x) = (y1 > y2 x) + (1 - (1-21"(6 > y2 x))n
(18)
for any Y2). Specifically, for 17
have Po- (y1 Y2 x) P P* > y2 x). which leas to
Po; (y1 > y2 x) Po- (y1 > y2 x)] 2m| PM (y) > y2 x) 1/2/-
(19)
F3. Proof of Theorem 3.3
Proof. For samples (x(1) yi y2 201) c) and (x(2), (2)
(1)
.22 ((2)
(2))) c())). according to Eq. (18), we have
Po; (x(), y( ,0
(1) a0a)) = Po; (x(1) y11 .92 ((a))
(1)
+ (1 ())
2P (c (() )))
(20)
and
Po:
(x(2) .1 .2
(2) (2) 2(2)
Por [ (2) (2) (2)
V1 Y2 (20)
(21)
= P' - (2) + (1 2P' (1 - (2)))
- 1)m 1- 11-P + (2 (2P(())
(22)
Therefore, to ensure that
Edpo (x()) y .y2
(1) 12, cwu) Eupo x22)
(2) ,y ((2)) 0,
(23)
we must have
log (P'(c(1) + +(1-2P") 122P"((()))m= 2) <-log (1 (1-P"(c(2) + (2P" (c 1)n + +8).
(24)
26

---

ROPO: Robust Preference Optimization for Large Language Models
which is equivalent
1-2n
(P*(c())) PM (=((c())-1
8
(25)
2
F.4. Detailed Derivation of Eq. (6)
From the definition of Wropo we have
4a
4a
Wropo
(1 + a)2
(1 (1+ (((22.111)
+
(26)
According to Eq. (4) we know that
- B (1 + a) a))
4a
T& Yy x)
4a
(A. log
d0
(1 + a) ldpo
(27)
Ti Y2
x)
Beside, note that for (xx) Iter we have
cx
e5 6d)) - E" e
E²
1
d'(x)=
(
(x))((=2)
(28)
(1+
e1+
I+e Itc
and
d'(-x) -a(r)((-1))
(29)
Letting
Te x)
Te x)
t(0) = Blog
Bllgg
(30)
Fro (y x)
Trat (V2 x)
have
To (y xx)
Vot(O) ,V log
(31)
(y² x)
Hence,
4a²
/ [o0A((y...1.3)))(A(11...... log
Te (y1 x)
(1 + a)²
na
ry(
x)
4a²
(-o((10))0(-10011 (sV
T
x)
d0
(1 a)²
T
Teg
x)
4a²
V(())(-t((0) vot(O) d0
(1+a)²
4a2
Voo(-t(0)) do
(1 (1+
4a²
(1 a)+ (-1(())
4a2
To x) Fret (y2
To x)
Blog
3 log
(32)
(1 + a)2
x)
Tee (y1 x)
where we omit the constant term of the primitive function.
27

---

ROPO: Robust Preference Optimization for Large Language Models
F.5. Proof of Theorem 3.4 Proof. For & Cua, have
B(x.yiyy B www.::/weee..................... (lx mb)
(x.
(P' x) (1 P" Y2 x))n) y1.
+ (P y2 x)7 Pd (y1 y2 x1) - x(0::00.1.1. .T))
(P" > y2 xx) - 2P (y y2 x)m) (1 Poly1 Y2 x))
+2P y2
y2 x) Po > y2 x21x)
(33)
Consider
- - 22-
-1)
= (1 - 2n) 2p)) (p" -
(34)
Therefore, when p" 1/2. (p) reaches its minimum p 1; when p" 1/2, (p) reaches its minimum p 0. This
means that the optimal point of (p) is po 1(p² > 1/2).
Therefore, Eq. (33) reaches its minimum when
> > y2 /2 = P > y2 x) ! 2
(35)
for any (x, Y2) Obviously, we have
Pop Py: > y2 12|x0 x) >22
Py- > y2 xx)
(36)
F.6. Proof of Theorem 3.5
Proof. For samples (x(1), yf (1) ((2) = =) and (x(2), .y .y (2) (2(2,((2)
(1)
1 - -1)) Without loss generality, we only
need consider two cases: (1) ) c(2) and (2) c()) c For the first case, have
Cn (1) vi .2 2(1) = Poly) > yi x) (0,e)
(1)
(37)
and
L Ln
(2) y .y2
(2) (2) (2)
Po(y > p² x) (1 --,,1]]
(38)
For the second case, we have
En (1)
.22 11",, ,1,y222 c01) = Po > y (1)
.
x) E (0,e)
(39)
and
L Lns
(2 vi .22 202)
(2)
22)
Po(y > (²) x) (1 -,
(40)
Therefore, to ensure that
Eng (x(1) y 23z , c(1) < lua yx . 612)
.22
(41)
we must have 1.
28

---

ROPO: Robust Preference Optimization for Large Language Models
E.7. rDPO and cDPO Are Not Noise-Tolerant In Most Cases
Proof. According to Lemma 3.2 in (Chowdhury et al., 2024), the noise-tolerance of rDPO only guaranteed when the proportion of noise, i.e., fJo, exactly equals the hyperparameter E.
Next we show that eddo is not noise-tolerant for (0, 2)) Let
Te)l, edpo (0) B(x.y1.32.)) ced y1, y1, mo]];
Cedpo (0)
Cho
and assume that 0* and 4% are the minimizers of Ledpo and C have
cdpo respectively. For any the space of parameters, we
- cdppo (f)
www. (8:x, V2. CTT]] B(x.y1.2.)) - y1, c, T&) (f:: c, To)
mo) Cedpo + 7m (x. Lndpo (0; y1, V2, To)].
(42)
Next, we give a counter-example to show that ledpo is not noise-tolerant. Suppose that
P (x. (x
V1 00)
Y2
(0))
and yi (0)
(0)
(9)
(43)
where x! is a fixed input and (yl .22 is a fixed pair of responses. Hence Eq. (42) becomes
cdpo (
To ( (0)
xx)
Te (y²
x x0)
= (2emo 170 E) logo
Blog
BBlog
Tzel (y
xx)))
Teel T
(0)
xx(1))
To (y
x()))
Te (y (0)
+ (110 + - -25100 - 1) logo
B log
3 log
(44)
Teal (y
x(1))
Tref (y1 x()))
Let
To(y (0) x0)))
no ry( x(1))
A() = Bog
B bog
(45)
Ty x(01)
Tree (y xx(0))
then Eq. (44) becomes
C cdpo ( (0) =2277o 170 log (A(o) + ( +E 2Eno 1) 1) (llgol-A(()))
(46)
We have
gr arg min Ledpo lEG
arg min - -e elogo(A(0)) - - logooloo (A(-))) dEe
E 0 (00: A(0) log
E
(47)
and
0
arg min Cedpo
AEG
arg min (2e1jo 1/0 8) log (A(())) (no
-28np 1) log ()(())(()))
dEe
0 0 A()) log
Nio
-6770
(48)
1/0 - 5 2s1
29

---

ROPO: Robust Preference Optimization for Large Language Models
Hence 0" 0% if and only if
1/0 + 2E170 E
(49)
1 which means that 2. However, E E (0, f1) Therefore, H" # t and thus ledpo is not noise-tolerant.
28710
F88. IPO Is Not Noise-Tolerant
Proof. Let
(0) = bxx
m.)]
the (0) = sxx. Kipo (0:x, X1. V2, mo)): ip
and assume that 0 and 0% are the minimizers of Lipo and Lip respectively. For any 0 in the space of parameters, we have
www. y1 Y2, [,
_) 70) Cipo (0: y1, C, me) + nooipp (0; y2, TT)]
ys.) (x Pipo y1. 1 Next, we give a counter-example to show that li is not noise-tolerant. Suppose that
+ Lipo
To)]-
(50)
P ((X = (x (0)
(0)
and y1 (0)
Y2
(0)
x (0)
(51)
where (0) is a fixed input and (yl .y2 ) is a fixed pair of responses. Hence Eq. (50) becomes ipp (0)
log To (y( (0)
x))))
T (y
x
x()))
(1 10)
log
Frof (yr (0)
x(0))
Tref (y2
(0)
x()))
2,3
To (y²2 (0) x()))
T (0)
|x(0))
1
+ 170
log
log
(52)
Tref (y) (0)
(0)
ne
(y (0)
(0)
2,3
Let
(y
(0) xx
Te Ty (0) |x
A() log
log
(53)
Teol (y (0)
x0)
Teel (y
(0)
xx())
then Eq. (52) becomes
1
2
1
ch (0) (1 -To)
((0))
+ 170
AA(0)
23
23
2110
1
= (A(())
(8)
+
(54)
3
4
which is quadratic function. Hence
170
gr E
(55)
t0
233
3
However,
8" arg min Lipo de
(A(0)
1
arg min
AE
28
1
0 e :
A(0)
(56)
29
which means that 0r + 0r Therefore, li is not noise-tolerant.
30

---

ROPO: Robust Preference Optimization for Large Language Models
E9. The normalization of Wropo
In Eq. (5), use (1+a)7
4cx
scale the maximum value of Wropo to 1, Here, we provide the details about it. Let
g(t) (1) + ao(())) e2t + (1+
(1+
where > 2, then we have
g'(t) (2c²t 2t + +a (c² + 26" + 1) (2c + 2e2t) + + ()ee))
(1+
et(1 + etys ((2 - a) + + + (a+ 2))
ee(1 (1 e2) a)e' + a 2)
e + et)3 ((2 a)e' e +a 2
Hence, g(t) increases if and only if (2 - a'e' + a + + 0.
Since 2, g() increases when < log 2+2 and decreases when t log a+2 a-2
Therefore, we have
g()) 9 (l0g a+2 (1+ a)2
())
a 2)
4a
31


Reviews:
[{"review_id": "92tRpadc53", "reviewer": "Reviewer_smYA", "summary": "LLM model alignment has shown great potential for several applications. However, popular techniques such as DPO are highly sensitive towards positive vs negative samples, and therefore any noise in the training preference data can significantly impact the performance. To alleviate this issue, the paper proposed an optimization framework for selecting noisy samples and then develop an augmented DPO loss function that is noise tolerant and can distinguish the noisy samples. Experimental results on several academic benchmark dataset demonstrate that proposed ROPO technique can almost always outperform base DPO and its variant.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer smYA,\n\nThank you for your valuable review. We respond to each comment as follows and sincerely hope that our response can properly address your concerns.\n\nFigures and Tables can be found in **smYA.md** in **https://anonymous.4open.science/r/ICML25-ROPO-F6CD**\n\n---\n\n# Other Strengths And Weaknesses\n\n> W1: It is not clear how ROPO performance generalizes to models larger than 7B.\n\n**Res:** In **Appendix E.1** of our initial submission, we show the performance of ROPO on Llama-2-13B and Llama-3-70B (trained on UFB and evaluated on AlpacaEval). We will make references to Appendix E.1 in the main text more obvious. For you convenience, we also provide the results in **Table smYA-1** in the anonymous link. As can be seen from the table, ROPO significantly outperforms the baselines at the scales of 13B and 70B.\n\n---\n\n> W2: It would be good to add the cost comparison between ROPO, IPO, and DPO.\n\n**Res:** In **Appendix A** of our initial submission, we provide the analysis of the computational cost for ROPO and **non-iterative methods (e.g., DPO, IPO, rDPO, and cDPO)**. We will make references to Appendix A in the main text more obvious. For your convenience, we quote the important content as follows.\n\nROPO introduces additional costs for the noisy sample filtering and robustness-guided rejection sampling stages compared with non-iterative methods. We estimate that **the cost of ROPO is approximately 1.6 times that of non-iterative methods.** The additional costs of ROPO mainly come from forward computations, which are acceptable compared to the training (backward) cost and almost negligible in the entire chain of real-world large-scale LLM training. For details, please refer to Appendix A.\n\n---\n\n> W3: Any discussion or insights on where ROPO shines over DPO (like on TL;DR) and vice versa would be a good addition.\n\n**Ans:** We speculate that the extent of ROPO's advantage over DPO depends on whether the task is subjective.\n- For tasks that are relatively more objective (such as the TL;DR summarization task), the groundtruth preference ranking labels are usually more deterministic, as the criteria for evaluating the quality of a summary are typically objective and quantifiable, such as whether it contains complete information and whether the numbers and other details are accurate. In such tasks, flipping preference labels can easily **provide the model with incorrect information**. Therefore, ROPO has greater potential compared to DPO in such tasks.\n- For tasks that are relatively more subjective (such as dialogue generation), it is often difficult to definitively say that one response is better than another for most questions, as the conclusions of preference comparisons can be influenced by factors such as the evaluator's cultural background, education level, age, etc. In such tasks, flipping preference labels **does not necessarily introduce \"incorrect\" information** to the model. Therefore, the advantage of ROPO over DPO may not be as significant in these tasks.\n\nAdditionally, we would like to share our observation that may help explain why ROPO still demonstrates a significant advantage on TL;DR with 0% artificial noise. According to the estimation in Table 5 of Appendix C, the original TL;DR dataset **inherently contain 21.3%-27.0% noise**. We also observe that due to the use of different annotators to label preferences for TL;DR, 5.8% of the posts exhibit \"cyclic preferences\" among multiple summaries. That is, for a given post $x$, the preference ranking among three summaries is $y_1 \\succ y_2, y_2 \\succ y_3, y_3 \\succ y_1$, which is evidently a form of noise. In such cases, ROPO naturally outperforms DPO.\n\n---\n\n# Other Comments Or Suggestions\n\n> C1: Algorithm 1 and Related Work should go into main text.\n\n**Res:** We will put them in the main text by reorganizing the presentation.\n\n---\n\n# Questions For Authors\n\n> Q1: Do you have performance analysis on larger models (except for 7B)?\n\n**Res:** Please refer to our response to W1.\n\n---\n\n> Q2: What is the computational complexity of ROPO?\n\n**Res:** Please refer to our response to W2.", "claims_and_evidence": "The claims about noisy data filtering and robust framework for LLM preference optimization have been validated both theoretically and experimentally.", "methods_and_evaluation_criteria": "Experiments are conducted on 3 well-known academic benchmark datasets and the results were compared against SOTA baseline DPO methods. Evaluation metrics and overall experimental settings make sense to me.", "theoretical_claims": "The claim that ROPO is noise-tolerant as opposed to DPO has been proved theoretically using Theorem 3.1-3.5.", "experimental_designs_or_analyses": "Experiments are carefully designed. The performance is tested on Alpaca, UFB and TLDR dataset and compared against DPO, IPO, cDPO and rDPO. Table 1 demonstrate that ROPO always outperforms other baselines (albeit by a small margin). More interestingly, the performance of ROPO increases monotonically over the iterations. Ablation studies show the importance of different steps in the ROPO framework.", "relation_to_broader_scientific_literature": "Preference selection and domain alignment of LLMs is an important problem and will have broader interest in the community. This work improves the performance of traditional DPO methods and will be of interest for both academic and industry audience.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Overall, ROPO is an interesting framework for improving performance over DPO. Concept of noisy data filtering and rejection sampling in DPO is novel. Experimental and theoretical results are also solid. Having said that, there are some concerns:\n1. Experiments are only conducted on 2 7B models and therefore, it is not clear how ROPO performance generalizes to other bigger models.\n2. ROPO is an iterative method, therefore, in terms of computational complexity and cost, it should be much higher than DPO. It would be good to add the cost comparison between ROPO, IPO and DPO.\n3. Experimental results demonstrate that the performance gain on 2 datasets (UFO and Alpaca) is only 1-2%, but ROPO seems more cost inefficient over DPO. Results on TLDR dataset seem very promising, so any discussion or insights on where ROPO shines over DPO and vice versa would be a good addition.", "other_comments_or_suggestions": "Structure of the paper requires a lot of changes. Algorithm 1 and related work section should go into main text.", "questions_for_authors": "1. Do you have performance analysis on larger models (except for 7B)?\n2. What is the computational complexity of ROPO?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NRFHQXsNPf", "reviewer": "Reviewer_mD6s", "summary": "This paper tackles the important problem of learning from noisy offline preference data. Motivated by the observation that previous noise-aware preference optimization methods either only partially mitigate the noise problem or requires costly invocation of a separate LLM during the training process, the authors proposed an iterative noise-aware preference alignment method RObust Preference Optimization (ROPO). ROPO combines a robust loss, a noisy sample filtering process, and rejection sampling. On common preference tuning benchmark datasets UFB, Alpaca, and TL;DR, the authors demonstrate that ROPO consistently outperforms previous methods, establishing a practical method for handling preference noise.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer mD6s,\n\nThank you for your valuable review. We respond to each comment as follows and sincerely hope that our response can properly address your concerns.\n\nFigures and Tables can be found in **mD6s.md** in **https://anonymous.4open.science/r/ICML25-ROPO-F6CD**\n\n---\n\n> Q1: How would ROPO perform in scenarios with non-uniform or clustered noise patterns?\n\n**Res:** In addition to the artificial uniform noise, our experiments also contain **two practical settings** that include **non-uniform or clustered noise patterns**. ROPO still outperforms all baselines under these two following settings.\n\n1. (*Appendix E.3.1*) Practical noise coming from human annotators' trust in larger models over smaller ones. It is common practice to treat the response from a larger model as the preferred one and the response from a smaller model as the dis-preferred one. This leads to **non-uniform and clustered noise patterns**, as preference noise tends to appear in problems that large models do not handle well but small models handle effectively, rather than being uniformly distributed.\n2. (*Appendix E.3.2*) Practical noise coming from LLM comparisons. We use Llama3-70B-Instruct to relabel the preferences in UFB, where the original preference label in the UFB dataset comes from GPT-4 rating. Then, we observe that about 30% of the labels are different from the original ones. The noise here is **non-uniform and clustered** rather than uniform, as differing preference labels only arise in cases where Llama-3-70B-Instruct and GPT-4 fail to reach a consensus.\n\n> Q2: Could the robustness-guided rejection sampling be extended to incorporate more diverse negative examples beyond the model's own generations (e.g., samples from additional sources)?\n\n**Res:** We have added the experiment of training Mistral-7B on UFB with 0% and 20% artificial noise, where the rejection sampling phase uses outputs from Llama-2-7B to obtain negative examples. However, as shown in **Table mD6s-1** in the anonymous link, we observe a decrease in the performance. We speculate the reasons as follows.\n\n1. Recent studies [1,2] suggests that on-policy training, where responses are sampled from the model’s distribution, generally outperforms off-policy training, where responses are sampled from other distributions. Therefore, the on-policy training paradigm of standard ROPO is naturally superior to the off-policy training paradigm that uses outputs from other models (e.g., Llama-2-7B) as dispreferred responses.\n2. Another line of research [3,4] indicates that the relationship between \"the value of preference samples (x,y1,y2)\" and \"the reward margin between y1 and y2\" remains inconclusive. It is unclear which reward margins are effective for preference alignment. We speculate that this is also one of the reasons why using generations from other models as dispreferred responses is often ineffective---because we lack a clear understanding of the distribution and impact of reward margins in this scenario.\n\n> Q3: How does ROPO handle cases where legitimate preferences might appear contradictory due to subjective differences rather than noise? It might be good to include a discussion on relationship with pluralistic alignment.\n\n**Res:** We will include the following discussion on pluralistic alignment in Appendix C.\n\nHuman preferences in the real world are often multi-dimensional and vary significantly due to differences in cultural background, education level, age, and region. This diversity in preferences has prompted the study of pluralistic alignment [5,6].\n\nHowever, defining \"noise\" in pluralistic alignment is challenging. When preference dimensions exceed one, there is no \"gold\" and latent reward model to rely on, and preference modeling based on the Bradley-Terry model becomes infeasible. Therefore, no groundtruth preference label exists between two responses, so we cannot define \"preference noise\". To address this challenge, two promising directions can be considered: (1) Inject multiple preference dimensions into the prompt, enabling alignment conditioned on specific preference dimensions [7]. (2) Introduce an additional explanatory text in samples (x, y1>y2) to describe in what sense y1 is superior to y2. Although pluralistic alignment is beyond the scope of the paper, we look forward to engaging in interesting discussions with readers on this topic.\n\n---\n\n[1] Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study\n\n[2] RLHF Workflow: From Reward Modeling to Online RLHF\n\n[3] Larger or Smaller Reward Margins to Select Preferences for Alignment?\n\n[4] Not All Preference Pairs Are Created Equal: A Recipe for Annotation-Efficient Iterative Preference Learning\n\n[5] Group Robust Preference Optimization in Reward-Free RLHF\n\n[6] Aligning to Thousands of Preferences via System Message Generalization\n\n[7] Rewards-in-Context: Multi-Objective Alignment of Foundation Models with Dynamic Preference Adjustment", "claims_and_evidence": "The authors claim that noise in preference data is prevalent and properly handling the noisy samples is critical for preference alignment. The experiments designed in this study adequately support this claim and the proposed methods that mitigate the noisy preference data issue led to a clear performance improvement.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are sound and aligns with widely adopted evaluation processes adopted by the alignment research community.", "theoretical_claims": "I did not check the proof of the theoretical claims.", "experimental_designs_or_analyses": "I carefully reviewed the experiment design and ablation study. The experiment designs are sound.", "relation_to_broader_scientific_literature": "This paper this tackles the important problem of handling noisy preference samples.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The paper is well-written and structured. The theoretical analyses are clearly presented with appropriate mathematical formulations. The experimental methodology is thoroughly explained, though some sections could be more concise.\n\nThe technical quality is high, with rigorous proofs and comprehensive experiments. The ablation studies effectively isolate the contributions of different components.\n\nThe approach is novel and represents a significant advancement over existing methods. While it builds on DPO, the integration of noise-tolerance, filtering, and rejection sampling is innovative and well-executed.", "other_comments_or_suggestions": "I don't have additional comments/suggestions beyond those provided in other review sections.", "questions_for_authors": "1. How would ROPO perform in scenarios with non-uniform or clustered noise patterns (e.g., where certain types of queries are more prone to noisy preferences)?\n2. Could the robustness-guided rejection sampling be extended to incorporate more diverse negative examples beyond the model's own generations? Would it be beneficial to include negative samples from additional sources? \n3. How does ROPO handle cases where legitimate preferences might appear contradictory due to subjective differences rather than noise? It might be good to include a discussion on relationship with pluralistic alignment.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "POgFuyvZFt", "reviewer": "Reviewer_tkDe", "summary": "This paper addresses the problem of robustly learning preference from noisy preference data. It proposed the ROPO framework which iteratively filtering noisy preference data and aligning the LLM with the filtered data. The ROPO framework consists three key modules, 1) noise-aware DPO loss for preference alignment, 2) noisy preference filtering based on the noise-aware loss, and 3) Response resampling based on the noise-aware loss.\n\nThe empirical results show that ROPO is better than other noisy preference learning baseline on extensive datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer tkDe,\n\nThank you for your valuable review. We respond to each comment as follows and sincerely hope that our response can properly address your concerns.\n\nFigures and Tables can be found in **tkDe.md** in **https://anonymous.4open.science/r/ICML25-ROPO-F6CD**\n\n---\n\n# Methods And Evaluation Criteria\n\n> M1: Would the following cases undermine the effectiveness of the framework? (1) If the LLM is aligned to the noisy preference in the beginning, would the error be carried and enlarged throughout the iteration? (2) If the target preference is very different from the initial model's judgement, would the preference learning be very inefficient?\n\n**Res:** These two cases will undermine the effectiveness of **various preference optimization methods**, not just ROPO.\n\nPlease note that in the *standard practice* of DPO-like methods, preference optimization typically requires the initial model to have undergone *supervised fine-tuning (SFT)* on data that is in-distribution for the DPO algorithm beforehand [1,2]. In other words, **the model should possess a basic ability to judge preferences before preference optimization is performed**. Based on this prerequisite, we derive the gradient and corresponding loss function for ROPO.\n\nIf the initial SFT model lacks basic capability, a reasonable suggestion is to perform SFT first to enhance its fundamental preference judgment ability, rather than proceeding directly with preference training. Otherwise, both DPO and ROPO would be ineffective.\n\n---\n\n# Experimental Designs Or Analyses\n\n> E1: Is there any noisy data filtering baseline that can be compared?\n\n**Res:** Our experiments in **Appendix E.5** have tested a confidence-based data filtering method and we find that it underperforms the standard DPO, and thus underperforms ROPO. This further supports our claim that the widely used cross-entropy loss (i.e., DPO loss) **cannot serve as a reliable measure of model confidence** in scenarios containing noisy preferences. The details are as follows.\n\nAccording to [3], confidence-based data filtering method is a popular approach to combat noisy preferences. Since our paper focuses on DPO-like methods that leverage implicit rewards, a natural choice is to use implicit rewards and the corresponding loss to reflect confidence. To this end, we conduct experiments of combining DPO with noisy samples filtering (NSF) and rejection sampling (RS) using Mistral-7B as the base model and UFB as the training dataset. For your convenience, the results are shown in **Table tkDe-1** in the anonymous link. As can be seen, the incorporation of noisy samples filtering and rejection sampling degrades the performance of DPO, especially at 20% artificial noise.\n\n---\n\n# Other Strengths And Weaknesses\n\n> W1: The related work section shouldn't be put into appendix. There isn't clear description of the overall framework in the main paper. The overall algorithm is put in Appendix A.\n\n**Res:** We will put the related work section and the overall algorithm in the main text by reorganizing the presentation.\n\n---\n\n# Questions For Authors\n\n> Q1: Is $\\ell_{\\rm na}$ in Line 245 a typo? What's the difference between $\\ell_{\\rm na}$ and $\\ell_{\\rm dpo}$?\n\n**Res:** This is not a typo. If we denote $P = \\sigma(\\beta\\log\\frac{\\pi_\\theta(y_1 \\mid x)}{\\pi_{\\rm ref}(y_1\\mid x)} - \\beta \\log\\frac{\\pi_\\theta(y_2\\mid x)}{\\pi_{\\rm ref}(y_2\\mid x)})$, then $\\ell_{\\rm dpo} = -\\log P$ and $\\ell_{\\rm na} = 1-P$.\n\n---\n\n[1] Direct Preference Optimization: Your Language Model is Secretly a Reward Model\n\n[2] https://huggingface.co/docs/trl/dpo_trainer\n\n[3] Impact Preference Noise on the Alignment Performance of Generative Language Models", "claims_and_evidence": "yes, as far as I can see, the claims are well supported.", "methods_and_evaluation_criteria": "The proposed iterative framework does make sense to robustly refine and learn the preference. The empirical results also support this. \nHowever, I have one concern, as the framework is heavily enhancing its internal preference judgement, would the following cases undermine the effectiveness of the framework?\n\n1. If the LLM is aligned to the noisy preference in the beginning, would the error be carried and enlarged throughout the iteration?\n2. If the target preference is very different from the initial model's judgement, would the preference learning be very inefficient?", "theoretical_claims": "yes, as far as I can see, the claims are accurate.", "experimental_designs_or_analyses": "The experiment setup is extensive, especially glad to see the ablation study on noise ratio $\\rho$ in Figure 3 and different component in Table 3.\n\nOne potential concern is all compared baselines are noisy preference learning methods, is there any noisy data filtering baseline that can be compared?", "relation_to_broader_scientific_literature": "As far as I can see, this paper fits into the literature well. There hasn't been paper on iterative framework for robust preference learning. The noisy-aware loss also make sense.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n- The content of this paper is substantial and the experiments and ablation study are extensive.\n\nWeakness:\n- The presentation of this paper is arguable.  The related work section shouldn't be put into appendix. There isn't clear description of the overall framework in the main paper. The overall algorithm is put in Appendix A.", "other_comments_or_suggestions": "N/a", "questions_for_authors": "Line 245, is this a typo? What's the difference between $l_{na}$ and $l_{dpo}$?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k45wMiLtrw", "reviewer": "Reviewer_GZqo", "summary": "The paper considers the alignment problem of large language models (LLMs) trained on noisy preference data, where human preferences are flipped with a certain probability $\\eta$. To align the LLM in a robust manner and mitigate performance degradation due to noisy data, the authors propose an iterative alignment framework that alternates between training the model in a noise-tolerant manner and filtering out noisy samples.\n\nThey begin by formulating this as a constrained optimization problem, aiming to minimize a weighted loss over the samples, where the weights are dynamic and intended to be smaller for noisy samples. Moreover, they introduce a constraint ensuring that the sum of these weights remains a fraction of the total number of samples, thereby reducing the effect of noisy data.\n\nNext, they analyze the noise tolerance of the Direct Preference Optimization (DPO) loss and demonstrate that it is ineffective in distinguishing between noisy and clean samples. Specifically, DPO aggressively updates the model parameters to fit noisy samples during gradient descent. To address this issue, the authors propose an alternative loss function, $l_{na}$, which exhibits lower tolerance to noise and better differentiates between clean and noisy samples. Furthermore, they introduce the ROPO loss, $l_{ropo}$, which combines $l_{na}$ and $l_{dpo}$ with a trade-off parameter $\\alpha$. A detailed discussion is provided on the choice of this parameter and its practical implications.\n\nFinally, they propose a rejection sampling strategy that generates new responses for samples identified as noisy and creates candidate samples using both the generated responses and the original responses from the dataset. They then select the candidate sample with minimal loss and include it in the next stage of training.\n\n\nIn the experimental section, they evaluate the effectiveness of their proposed method against baselines across three datasets and two base models under different levels of artificial noise injected into the data. Additionally, they conduct ablation studies to examine the impact of different framework components and hyperparameters on overall performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer GZqo,\n\nThank you for your valuable review. We respond to each comment as follows and sincerely hope that our response can properly address your concerns.\n\nTables can be found in **GZqo.md** in **https://anonymous.4open.science/r/ICML25-ROPO-F6CD**\n\n# Claims And Evidence\n\n> C1: A formal justification and theoretical analysis for the effectiveness and robustness of the rejection sampling (RS) technique would strengthen the validity of this approach.\n\n**Res:** The reliability of RS can be guaranteed by the noise identification capability of our loss function, similar to Theorem 3.5. Specifically, since both our RS and noisy sample filtering use loss values as the criterion for sample selection/filtering, their effectiveness is guaranteed similarly.\n\nDue to the rebuttal length limit, we will include the detailed analysis in the paper. We would greatly appreciate your understanding.\n\n# Methods And Evaluation Criteria\n\n> M1: The noise has been artificially injected in the dataset to demonstrate the utility the approach (under 0% noise injected, the performance difference is minimal except for TL;DR dataset).\n\n**Res:** We understand your concern that artificial noise may not align well with real-world scenarios. For this, please see experiments in *Appendix E.3.1, E.3.2* under practical noise coming from annotators' trust in larger models over smaller ones and LLM comparisons. For your convenience, results are also provided in **Tables GZqo-1 and GZqo-2** in the anonymous link. As shown, ROPO significantly outperforms baselines in both settings.\n\nWhile ROPO's improvement may not always be substantial at 0% artificial noise, we humbly believe that experiments across several practical and artificial noise settings sufficiently demonstrate its advantages over baselines and its contributions to preference alignment.\n\n# Experimental Designs Or Analyses\n\n> E1: In Tables 2 and 3, cDPO and rDPO are missing. In particular, ... rDPO against different components of ROPO.\n\n**Res:** We have added them into Tables 2 and 3. Please see **Tables GZqo-3 and GZqo-4** in the anonymous link.\n\n# Essential References Not Discussed\n\n> R1: A few references [3-7] on robust preference optimization of LLMs are missing.\n\n**Res:** We will expend Related Work as follows.\n\n**Robust Preference Alignment of LLMs.** Many efforts have been made from various perspectives to achieve robust preference alignment [1-7]. Specifically, [1,2] uses label smoothing to mitigate the impact of preference noise. [3] improves the model’s adaptability to different preference distributions and enables iterative output refinement by jointly optimizing a self-improvement policy and a generative policy. [4] models potentially corrupted preference labels as sparse outliers and solves an $\\ell_1$-regularized maximum likelihood estimation problem, thereby consistently learning the true underlying reward. [5] introduces a multi-head reward model (RM) that reflects each head’s confidence in the output reward using the standard deviation of a Gaussian distribution, effectively addresses the challenge of RM imperfections in RM-based RLHF. [6] focuses on different forms of noise and enhances DPO’s resilience to both pointwise and pairwise noise in LLM alignment by leveraging Distributionally Robust Optimization (DRO). [7] robustly aligns LLMs to the preferences of diverse individual groups by incorporating group information into the LLM context and optimizing against the worst-case alignment performance across all groups. Compared to them, our method integrates noise-tolerance and noise-identification capabilities without external models, offering a novel paradigm for robust preference alignment.\n\n# Other Strengths And Weaknesses\n\n> W1: Please see C1.\n\n> W2: ROPO introduces a hyperparameter $\\alpha$ and requires estimating $\\rho$, which may vary across different datasets and applications.\n\n**Res:** Our ablations show that ROPO is insensitive to $\\alpha$ and $\\rho$ within our recommended range and does not require extensive tuning for different tasks.\n\nAs stated in Section 4.2, we fix $\\alpha=14$ and $\\rho=0.2$ without tuning them in most of experiments after we observe that ROPO is insensitive to $\\alpha$ and $\\rho$. For readers, we also recommend $\\rho=0.2$ in practice. As for $\\alpha$, we recommend $\\alpha=14$ or $30$ on relatively objective tasks (e.g., summarization), and $\\alpha=6$ or $14$ on relatively subjective tasks (e.g., dialogue).\n\n# Other Comments Or Suggestions\n\n> S1: Please see E1.\n\n---\n\n[1] Provably Robust DPO: Aligning Language Models with Noisy Feedback\n\n[2] A Note on DPO with Noisy Preferences & Relationship to IPO\n\n[3] Self-Improving Robust Preference Optimization\n\n[4] Robust Reinforcement Learning from Corrupted Human Feedback\n\n[5] Reward-Robust RLHF in LLMs\n\n[6] Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization\n\n[7] Group Robust Preference Optimization in Reward-Free RLHF", "claims_and_evidence": "All theorems stated in the paper are accompanied by proper proofs in the appendix. However, the robustness-guided rejection sampling strategy presented in Section 3.3 lacks a rigorous theoretical foundation and is primarily heuristic in nature. Specifically, the authors sample multiple responses from the model and generate candidate samples using both the newly generated responses and the original responses from the dataset. They then select the candidate sample with minimal loss and include it in the next stage of training. \n\nHowever, a formal justification for the effectiveness and robustness of this strategy in practice is lacking. In particular, there is no guarantee that this strategy would not introduce noisy samples into the data. A more thorough theoretical analysis would strengthen the validity of this approach.", "methods_and_evaluation_criteria": "The proposed method has been properly evaluated on multiple datasets and models. Further, ablation studies have been performed to study the utility of the different components of the method. Although, one limitation would be that the noise has been artificially injected in the dataset, to demonstrate the utility of the approach (In the 0% noise injection case, the performance difference is minimal except for TL;DR dataset).", "theoretical_claims": "All theorems stated in the paper have proper proofs in the appendix.", "experimental_designs_or_analyses": "The proposed method has been properly evaluated on multiple datasets and models. Further, ablation studies have been performed to study the utility of the different components of the method. However, in Table 2 and 3, other baselines like cDPO and rDPO are missing. In particular, it would be interesting to compare the performance of rDPO against different components of their proposed ROPO framework in the ablation studies.", "relation_to_broader_scientific_literature": "The paper analyzes the effect of noisy preference data in the alignment of large language models (LLMs). Prior work on this problem primarily focuses on alternative loss functions that are robust to noise and often require prior knowledge of the percentage of noisy data. In this work, the authors first propose a noise-tolerant loss function that facilitates the identification of noisy samples while preventing overfitting to noisy data. Building on this, they introduce an effective noise-filtering strategy to remove noisy samples from the dataset. Furthermore, they propose a robustness-guided rejection sampling technique to introduce new clean samples into the data.", "essential_references_not_discussed": "A few references in the area of robust preference optimization of LLMs are missing. Namely,\n\n1. Choi, Eugene, Arash Ahmadian, Matthieu Geist, Oilvier Pietquin, and Mohammad Gheshlaghi Azar. \"Self-improving robust preference optimization.\" arXiv preprint arXiv:2406.01660 (2024).\n2. Bukharin, Alexander, Ilgee Hong, Haoming Jiang, Zichong Li, Qingru Zhang, Zixuan Zhang, and Tuo Zhao. \"Robust reinforcement learning from corrupted human feedback.\" arXiv preprint arXiv:2406.15568 (2024).\n3. Yan, Yuzi, Xingzhou Lou, Jialian Li, Yiping Zhang, Jian Xie, Chao Yu, Yu Wang, Dong Yan, and Yuan Shen. \"Reward-robust rlhf in llms.\" arXiv preprint arXiv:2409.15360 (2024).\n4. Wu, Junkang, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jiawei Chen, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. \"Towards robust alignment of language models: Distributionally robustifying direct preference optimization.\" arXiv preprint arXiv:2407.07880 (2024).\n5. Ramesh, Shyam Sundhar, Yifan Hu, Iason Chaimalas, Viraj Mehta, Pier Giuseppe Sessa, Haitham Bou Ammar, and Ilija Bogunovic. \"Group robust preference optimization in reward-free rlhf.\" Advances in Neural Information Processing Systems 37 (2024): 37100-37137.", "other_strengths_and_weaknesses": "Strengths\n\nThe paper analyzes the effect of noisy preference data in LLM alignment, addressing an important problem in this domain. The authors propose multiple strategies to tackle this issue in an iterative manner, collectively forming a novel framework in this area. Specifically, they first introduce a noise-tolerant loss function that facilitates the identification of noisy samples while preventing overfitting to noisy data. Building on this, they propose an effective noise-filtering strategy to remove noisy samples from the dataset. Furthermore, they introduce a robustness-guided rejection sampling technique to incorporate new clean samples into the data.\n\nWeaknesses\n\n1. The robustness-guided rejection sampling strategy in Section 3.3 lacks a rigorous theoretical foundation and is primarily heuristic in nature. Specifically, the authors sample multiple responses from the model and generate candidate samples using both the newly generated responses and the original responses from the dataset. They then select the candidate sample with the minimal loss and include it in the next stage of training. However, a formal justification for the effectiveness and robustness of this strategy in practice is lacking. . In particular, is there any guarantee that this strategy would not introduce noisy samples into the data?\n    \n2. Unlike prior noise-robust approaches that require knowledge of the percentage of noisy data, the proposed method iteratively estimates and filters noisy samples. However, the approach introduces a new hyperparameter, $\\alpha$, which governs the trade-off between $l_{na}$ and $l_{dpo}$, and must be either estimated or predefined. Additionally, the method requires estimating the filtering ratio $\\rho$, which may vary across different datasets and applications.", "other_comments_or_suggestions": "Consider adding the baseline methods' performance in Table 2&3.", "questions_for_authors": "Kindly refer to weaknesses above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: LLM alignment techniques such as DPO can be highly sensitive to noisy preference data. To alleviate this problem, the paper proposes ROPO (RObust Preference Optimization), an iterative framework that combines a noise-aware loss, dynamic sample weighting, and rejection sampling to effectively filter out and learn from noisy preferences. Extensive experiments on several benchmarks show that ROPO consistently outperforms DPO and other baselines.

Reviewers initially were concerned about the size of the models (7B), the lack of cost comparison, and some perceived lack of rigorous theoretical foundation, but these concerns have been mostly addressed. During the reviewer-AC discussion, one of the two reviewers who recommended “weak accept” finally acknowledged this paper offers a good contribution. Considering the reviewers’ consensus, I recommend accepting the paper. As discussed with the reviewers, the authors should probably improve the paper to avoid the confusions that had to be sorted out during the discussions.
Decision: Accept (poster)