Robust Reinforcement Learning from Corrupted Human Feedback
Authors: Alexander Bukharin
Ilgee Hong
Haoming Jiang
Zichong Li
Qingru Zhang
Zixuan Zhang
Tuo Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning from Human Feedback
Robust Reward Learning
Primary area: reinforcement_learning
TL;DR: We propose a new robust reward learning method for RLHF.

Abstract: Reinforcement learning from human feedback (RLHF) provides a principled framework for aligning AI systems with human preference data. For various reasons, e.g., personal bias, context ambiguity, lack of training, etc, human annotators may give incorrect or inconsistent preference labels.  To tackle this challenge, we propose a robust RLHF approach -- $R^3M$, which models the potentially corrupted preference label as sparse outliers. Accordingly, we formulate the robust reward learning as an $\ell_1$-regularized maximum likelihood estimation problem. Computationally, we develop an efficient alternating optimization algorithm, which only incurs negligible computational overhead compared with the standard RLHF approach. Theoretically, we prove that under proper regularity conditions, $R^3M$ can consistently learn the underlying reward and identify outliers, provided that the number of outlier labels scales sublinearly with the preference sample size. Furthermore, we remark that $R^3M$ is versatile and can be extended to various preference optimization methods, including direct preference optimization (DPO). Our experiments on robotic control and natural language generation with large language models (LLMs) show that $R^3M$ improves robustness of the reward  against several types of perturbations to the preference data.

Robust Reinforcement Learning from Corrupted Human Feedback
Alexander Bukharin' Ilgee Hong'
Haoming Jiang"
Zichong Li'
Qingru Zhang'
Zixuan Zhang!
Tuo Zhao
Abstract
Reinforcement learning from human feedback (RLHF) provides a principled frame- work for aligning Al systems with human preference data. For various reasons, e.g.. personal bias, context ambiguity, lack of training, etc, human annotators may give incorrect inconsistent preference labels. To tackle this challenge, propose robust RLHF approach - Rd M, which models the potentially corrupted preference label as sparse outliers. Accordingly, we formulate the robust reward learning l f1-regularized maximum likelihood estimation problem. Computationally, we de- velop efficient alternating optimization algorithm, which only incurs negligible computational overhead compared with the standard RLHF approach. Theoreti- cally, prove that under proper regularity conditions, R M can consistently learn the underlying reward and identify outliers, provided that the number of outlier labels scales sublinearly with the preference sample size. Furthermore, remark that R M is versatile and can be extended to various preference optimization meth- ods, including direct preference optimization (DPO). Our experiments on robotic control and natural language generation with large language models (LLMs) show that R² M improves robustness of the reward against several types of perturbations to the preference data.
Introduction
As artificial intelligence (AI) systems continue to advance and become increasingly sophisticated, ensuring their aligmment with human values and preferences has emerged as a paramount concern, particularly for recent large language models (37 (28). One promising approach to achieving this alignment Reinforcement Learning from Human Feedback (RLHF), which involves training AI systems through a process of reward modeling based on human-provided feedback and preferences
[13 3 50
A significant challenge in RLHF, however, arises from the inherent uncertainty present in the preference data provided by human evaluators [17 4 Since RLHF often targets highly complex scenarios where defining precise preference standards is difficult, if not impossible, annotators may provide undesirable or inconsistent preference labels, especially when they lack sufficient experience training. In the of robotics system designed assist with household tasks, untrained annotator might label actions that complete the task efficiently but in manner that could potentially cause property damage compromise safety as preferable, overlooking the importance of safe and responsible operation.
An even more concerning scenario is that some human evaluators may maliciously assign incorrect preference labels [44. 32). Personal prejudices, agendas, or lack of understanding about the true goals of the system could lead some annotators to intentionally mislabel examples, which could undermine the entire RLHF process and cause the model to learn undesirable or misaligned behaviors, posing a
"The authors are listed in alphabetical order: Georgia Tech, Amazon.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

significant risk to the robustness and reliability of the Al system. Despite its critical importance for Al system alignment. this issue has received limited attention in the existing literature. For example, when training an Al system for automated content moderation on social media platforms, malicious annotators could mislabel examples of hate speech, misinformation, or harmful content as desirable, leading the model to learn to allow the proliferation of toxic and dangerous online behaviors. Despite its critical importance for AI system alignment, the robustness of RLHF has only received limited attention in the existing literature [
To address this challenge, we propose a robust Reinforcement Learning from Human Feedback (RLHF) approach R M (Robust Reward Modeling for RLHF) to handle partially corrupted preference labels. Specifically, we assume that a subset of incorrect (corrupted) labels exists as outliers in the preference data used for training the reward model To model the label corruption, we introduce an instance-specific perturbation factor to the Bradley-Terry (BT) model for human preference (6]. We then learn the reward model and perturbation factors simultaneously by maximizing an f1-regularized likelihood of the preference data. Theoretically, we prove that under proper regularity conditions, our approach can consistently learn the underlying ground truth reward and identify potential outliers, provided that the number of incorrect labels scales sublinearly with the preference sample size. Computationally, we show that the additional computational overhead of involving the perturbation factor in training is negligible: The log- likelihood is strictly convex and univariate with respect to each perturbation factor, and we can obtain its closed-for foseed-forma update at each iteration.
To demonstrate the effectiveness of our proposed method, we apply R M to robotic control [39 Specifically, we consider different types of corruptions to the preference data, including irrational flipping, stochastic flipping, and myopic flipping. We train robust reward models with R M and optimize the policy based on the learned reward. We observe that R M outperforms the standard RLHF method for all tasks under all preference models.
Moreover, Rt M can be further generalized to other preference optimization methods. For example, we incorporate RC M into direct preference optimization (DPO) [29] and evaluate its performance on two natural language generation tasks dialogue and summarization. We adopt Llama-2 7B [41 and use Claude as the judge. We find that Rd M-DPO outperforms DPO in policy learning for both tasks, and our results suggest that the training data of both tasks are very likely to have a small percentage of corruptions. Besides, we also consider random flipping for corrupting the preference data, and the results also show that R M-DPO outperforms DPO.
2 Related works
Robust reward modeling. Research robust reward modeling for RLHF remains limited, though some prior works have explored various types of robustness: The most relevant results are from [2] and (25). They consider partially corrupted preference labels and propose to filter corrupted data based on the label confidence; [10] address robustmess to diverse human preferences by learning a mixture of reward models; [48] focus on robustness sampling temperature and length bias, and
develop an iterative version of DPO; 14 consider reward overoptimization and employ a reward model ensemble.
While not explicitly framed as robustness, several other methods relate to this challenge: The approach modifies DPO's loss function to avoid overfitting from weak regularization. [14] also explore reward ensembles; [8] develop a nonconvex human-aware loss, which downweighs training samples for reward learning when preference labels cannot be correctly predicted by reward models.
Robust classification. The reward learning problem in RLHF is related to classification, as both involve learning functions that map inputs to class labels or preference labels. However, in classi- fication, the goal to accurately predict class labels, while in RLHE, the goal is to learn a reward function that assigns scalar rewards to inputs, which are then used to optimize policy or model. Despite these differences. the robustness literature in classification offers valuable insights for robust reward learning in RLHF
The existing literature on robust classification has explored several directions. For instance, 46 and follow-up works (5) have focused on developing nonconvex loss functions that are robust to outliers. Additionally, (27 and subsequent works [19] have investigated eestance-dependent calibration of nonconvex loss functions, which requires some prior knowledge of label noise. Other works propose
'The deterministic outlier setting considered here is specific case of label uncertainty, and it does not cover all possible sources of uncertainties, which will be discussed in more detail later.

---

iteratively filtering data based on uncertainties of labels or losses [47]. which can be viewed as
relaxation of some nonconvex loss functions. 223 and follow-up works [34] have concentrated on
robustness to distribution shifts. More recently, 21 and subsequent works [24] 26 49 have explored
adversarial robustness against the worse-case perturbation to the input.
3 Robust reinforcement learning from corrupted human feedback
We first introduce the problem setup corrupted preference data, and then present R² M for robust reward modeling. Lastly, we develop an efficient optimization algorithm for Rº M and further extend R M to direct preference optimization.
3.1 Corruption to human feedback
We consider a Markov Decision Process (MDP) M = (S,A, with state S, action E A, state transition kernel P, discount factor ", and the reward funettin r : x A - R. which is assumed to be aligned with human preferences. To learn such a reward function, we collect (potentially corrupted) human preference dataset Do by some behavior policy Tref- which contains n pairs of trajectory segments Do (Zw, =t,1)==1 Here, trajectory segment z of length denotes a sequence of consecutive state and action pairs {(s,, at) ) sampled according to some behavior policy, and Zw.s and =g,; denote the trajectory segments preferred and dispreferred by the human annotators, respectively.
Different from the conventional RLHF approach, we assume that the human preference follows distribution perturbed by potential corruption:
p(Zw. > re, 6,) = a(" (zw,i) r" + +)),
(3.1)
where r* denotes the ground truth reward function when applied to the trajectory segment, r r²()) Ex ("e" az) with discount factor y E (0, 1]. (xx 1/(1 + exp(-r)) denotes the sigmoid function, and og is deterministic perturbation modeling the annotator's ' bias. Note that when 5, = 0, (3.1) is reduced to the standard Bradley-Temy (BT) model; when 6 < (2(,)) - (zw,s) the annotator is very likely give an incorrect preference. For notational simplicity,
denote 5" = [6- 17 E R", and we consider the case where 5" is a sparse vector, i.c., the annotators' biases and mistakes only happen to fraction of the preference data.
Remark 3.1. Note that the perturbation factors ó, are assumed be deterministic and arbitrary. They can be intentionally introduced to mislead or confuse the reward learning process. This is in general more challenging than the setting of stochastic outliers, where the labels are flipped according to certain distribution.
3.2 Method
We next develop the estimators of the ground truth reward r" and the sparse 8*. To encourage the sparsity of 6, propose to minimize ε -regularized negative likeliool of and 6, os's over the preference data:
(F. aaggiin (r, r,
1
Σ [lo p(zm( > 2e,ir 0)] + A||ō|,,
(3.2)
where À E (0, 1) is a tuning parameter, and ||ō]] = Σn_ 6,|| denotes the & norm of 8. The L regularizer has been widely used in the existing literature sparse estimation, such as Lasso (38). It can be viewed as a convex relaxation of the to norm of 6, i.c., |00lo = Σ (8, &f 0). By tuning from large to small,
control the number of nonzero entries in 5 from small large.
The standard preference loss function is more susceptible to the influence of outliers in the training data. Therefore, the model may exhibit underfitting on the inlier (clean) data points, as it attempts to minimize the impact of the outliers on the overall loss. This eventually can distort the decision boundary, leading to suboptimal performance on the majority of the inlier data.
Remark 3.2.
Once the reward is learned, we further apply Proximal Policy Optimization (PPO, 33 to find policy म, which maximizes the expected sum of discounted rewards,
= argmax (s
ff( az)],
where D, denotes the stationary distribution of the state-action pair induced by
3

---

3.3 Alternating optimization
We present an efficient alternating optimization algorithm for solving (3.2). Suppose we parameterize the reward model as neural network with parameter Φ. At the k-th iteration, we have the iterate g((), and we sample a pair of trajectory segments Zuri and Ze,i- We first fix gi(k) and minimize the loss with respect to 6; by (+++))
argminn log((a(((zz))))) ()))))))))))))))))))))))) r (() dc)) \|ō2|.
(3.3)
6.
By examining the optimality condition of (3.3)
(r(=m, ()))) ( (*)) + 8:) - 1+ de; = 0,
where & E @ we can obtain a closed-form solution
6. (k+1) max
- 1 r(zw, (1)) + r(ze, ). 0}.
(3.4)
Denote ( 6,) = log log(o (a Φ) - - +6;)) + ll we update e by a stochastic gradient
descent step given 0. ((++1)
(b++1) = (kk) no v va ( , o(
(3.5)
where 77 is the learning rate.
3.4 Extension to direct preference optimization (DPO)
Our proposed R MM approach is generic and can be extended to DPO 29 which is another popular method for policy learning from human preferences. DPO directly learns the policy in supervised manner using the preference data of state-action pairs Du = (s; Gw,i, ae.;)== This approach forgoes the need to learn the reward function explicitly by reparameterizing the reward function r with respect to its optimal policy ,,: Recall that To denotes the behavior policy, we have
(als)
((s,a)
r(s, a) log
+ 3 log Z(s), where Z(s)
Σ (aer(als) exp
(3.6)
Trre (als))
3
a tuning parameter controlling the KL divergence between Jeef and Tyoe. By plugging in
3.6 back into (.2)) we have the policy optimization problem
argmin (Dro(T) i 5)
log (a (Br~ (Br=(am,:l (s) - Br (ar,(s)) + 6c)) + X|ō||,
n
i=1
where T- (als) log (x(als)/ Tref (a) ()) denotes the log- bog-probability ratio.
Theoretical analysis
We next establish the statistical guarantees for R&M M on the reward recovery. Specifically, we prove that the reward function learned by R M from corrupted human feedback can be as accurate as its counterpart without outliers.
To better convey our theoretical insights, we consider bandit setting, i..., MDP with horizon of one, mirroring the setup in DPO (see Section 3.4). The preference data of state-action pairs are given as Da = {(s;, 1 02 Mi
i= ) where yi [ar,i > a2,;) denotes whether 01,i is preferred 02,;- Such setting is common in real-world LLM applications such as (single-turn) question -anttiinn or text summarization task, where a1,; and az2; denote two different responses corresponding to the same prompt s; To ease the theoretical analysis, we consider tabular setting, where the number of states [S] and the number of actions [AA] are finite. For notational simplicity, we denote the true reward as a vector R (r" (s) E TR. (||[AA, which concatenates the rewards of all state-action pairs, r" (s, a) with E S and E A.
Before we proceed with our main results, first present the statistical guarantees of standard RLHF on the reward recovery, when there is no outlier in preference data (8* = O). Specifically, we can adapt Lemma 3.1 in Zhu et al. [51] to our setting: The Maximum Likelihood Estimator (MLE) R attains the following statistical rate of convergence:
j - R 2. = (1(S)A))
|SIII
(4.1)
with overwhelming probability. Here, En n E. J4 a positive semi-definite matrix depend- ing on the training dataset Do with x; = 1 (s = s;, a = a1.)) 1(s = s;, a2,)) E R|S||A| and
Il-lls, is the matrix norm defined as ||0|| = u Eoe for any vector e E [S

---

Remark 4.1. As has been shown in Zhu et al. [ST, given (4.1 one can further prove the desirable regret bound for the learnt policy. Therefore, our theoretical analysis only focuses on the statistical guarantees on the reward recovery.
We then impose the following two assumptions on the problem.
Assumption 4.2. The perturbation 8 only has 2 0 non-zero entries, i.e. 110% 8. Moreover, there exists a constant C > 0 such that | S C.
Assumption 4.3. Let
be some constant. have r* E R.. where
RB r AxxA4 R (Es, = 0, = (
- <B].
Note that s is allowed to scale with (n, [S], [A]), but and B are not. This is mainly due to technical reasons ensure the model identifiability. Accordingly, adopt constrained MLE formulation:
(F,5) argmin
T2 Σ
logp(s; 01,5; 02,1: Mer, 6)] A||0|| subject RB,
(4.2)
where ps,, a1 d2 Mes 6) is defined under the bandit setting as follows:
p(s 01 02 6) 1 (3 (r, a1,) r(s., 02.1) + +6)
+ (y/ ( a2,,) ((s;,a1,,) +20++-20 6.).
Note that we add the constraint r E RR in setting. In practice, the reward model with function approximation is usually trained with proper regularization, and therefore can be bounded without any constraint.
(4.2) also due to the technical reason under the tabular
Theorem 4.4. Suppose Assumptions 4.2 and 4.3 hold. Let R - [F(s, a)] and be the minimizer of
4.2 Given À = 1/n, there exists universal constants Co > 0 and y, such that we have
f R
1
- n
1
4 ( [ss ColS||A|
22 m n
with overwhelming probability.
Proof Sketch. Due to space limit, we only present proof sketch here. The technical proof of the lemmas can be found in Appendix A For notational simplicity, denote AR R R² and
A8 8- 8- 8*, and denote the negative ligeliilooood function on Do
a
C(R
Σ
pog 01,1; 02,1, UEr, 8)]
72
1 Σ
[log (1(y; 1)0 R) + 5.) 1 0)0 R) 8,111,
where we use (ri, R) = (1 = 8r, a = (1,s) ( = 85, a = a2,;), , R) = --rrs, an r(si, a2,)).
Since f E Rg and 8 are the minimizers of 4.2) and r" E Rg by Assumption 4.3 we have
((R,8) 2| < C(R", 8*) + |1.
(4.3)
The next Lemma proves the strong convexity of C in R and d at (R², .8) and (i, 82, respectively.
Lemma 4.5. Suppose Assumptions 4.2 and 4.3 hold. Let = 11/(2exxp(-v22) () +exp(
2BB+
C)). L is strong convex with respect to It at 11";0")) with parameter 7, C(R" + AR,8") L(R", .5") (VnC(R', ."). AR) 2 AR|
(4.4)
Moreover, L is 7/nnstrong convex with respect to at (R, 6*),
L(R, 8" A0) C(R, VV. 8*) A6) > 2 A
(4.5)
Given the index set S = ii E (1,2, 11.,
:0 + 0} and S = {1, .I.,....
...., nn S, we can decompose
any R2" by the index set and S as follows:
8 = ōs + dg.
Here ås has the same non-zero entries 8 Now apply the strong convexity of L to 4.3 use Cauchy-Schwarrz inequality to bound the inner product, and use decomposability of obtain the following result.
5

---

Lemma 4.6. Given the strong convexity of L in
let X 2 VoL( (o we have and 4.5
AI + 2 | 21 Aó + VR .8') x ARIEE.
(4.6)
7t
The above inequality suggests that be regarded as the projection of Áó onto the subspace {ō E 1= = 0, for all j $ S}. The next lemma bounds the gradient of C with respect to 6:
control the estimation error of by only Adg. which can
Lemma 4.7. For any R E SIIAA and R", we have V&C(R, 8l) S 1/nn.
Therefore, it suffices to take À 1/n. Furthermore, in the proof of Lemma 3.1 of Zhu et al. [51) (See Section B.1 of Zhu et al. [5T]), the gradient of C with respect to R can be bounded as following:
Lemma 4.8. There exists universal constant Cj > 0, such that we have
01)) S SCI /| + log(1/e)
n
with probability at least -.
Finally, we combine Lemma 4.6 and the upper bounds of gradients Lemma 4.7 and 4.8 to get
4
4s
|S||A| + (1/e
|R
c
7
which holds with probability least 1
We make the following remarks about Theorem
Remark 4.9, When the data perturbation is sufficiently sparse, i.e. 8 I |S||A], the convergence rate of estimating the reward under the presence of corrupted data is dominated by S|IAA//n Notably, is of the same order as that using clean data, which presented (4.I). other words, even there is contamination in data, the learned reward still be accurate its counterpart without outliers. However, if the ground-truth perturbation 8* is not very sparse, i.e. > |S|[A] it can hurt the statistical rate of convergence.
Remark 4.10. In our analysis, we estimate rewards for each state-action pair under the tabular bandit setting. However, our results can be extended to infinite-state and infinite-action case with reward function approximation, following Zhu et al. [5]]. Specifically, where reward functions can be linearly approximated [5I]. Morcover, when the reward function is smooth, it can be approximated by neural networks and our analysis for convergence rate of reward recovery under corrupted preference data can apply as well
results work for the scenario
5 Experiment
In this section, demonstrate the effectiveness of our proposed robust loss function through its application in robotic control and natural language generation tasks. Due to space limit, we defer some less important results and explanations to Appendix C.
5.1 Robotic control
Experiment setup. We evaluate the robustness of R M across three robotic control tasks within the PyBullet [15] environments: HalfCheetah, Ant, and Hopper. To simulate noisy human preference, we consider three noise models of human preferences follows:
1. Stochastic noise model: For a pair of trajectory segments (21, 22), we generate preference label with the probability (((*(() - r (z2))/T) where 7 >> 0 the temperature. This model captures typical human behavior, where preferences are more likely to be corrupted when the true preference unclear. We control the noise rate by tuning in {1.0, 2.0, 3.0). As the value of T increases, the probability becomes closer to uniform, causing greater corruption.
2. Myopic noise model: For a pair of sequences of state-action pairs 21 {($1,, am and 22 ((2,,) ) } we generate a preference label by
> if Σ
(51,, a1,1
(82,1,22)) and x otherwise,
t=1
where 7 E (0, 1] a discount factor. This model represents shortsighted human behavior, where people may place more weight recent observations. We control the noise rate by tuning ?

---

in {0.3, 0.7). In general, as the value of y decreases, the importance of initial observations diminishes, which leads to greater corruption.
Irrational noise model: For pairs of trajectory segments {(21,1, (2,1) ²2,1) ! in a mini-batch B Do where r" (21.3) > r* (22,x) (i.e., 21, , is preferred over 22, by the ground truth reward), flip the preference labels of the top |3| /|] x 100% pairs, ordered by the largest true reward difference (=(=)))) r* (22,)). Here, p E (0, 1) represents a sublinear rate of label perturbation. This model considers extreme human errors, where people can make mistakes even on clear preference pairs. We control the noise rate by tuning p in (1/3, 1, 2/3). As the value of p increases, larger number of preferences are corrupted.
stochastic
myopic
irrational
HaltCheetah
1.00
0.75
0.50
0.25
Cross-ettmry n3w
0.00
1.0
2.0
3.0
0.3
0.5
0.7
373
1/2
2/3
1.00
0.75
Ant
0.50
0.25
0.00
1.0
2.0
3.0
0.3
0.5
0.7
3/3
1/2
2/3
1.00
Hopper
0.75
0.50
0.25
0.00
1.0
2.0
3.0
1/3
y
P
Figure 1: Normalized returns for the bascline (cross-entropy loss) and R M across all noise models and noise rates. Error bars represent the standard deviation across 10 different seeds. Learning curves and percentile plots are in Appendix C.
For reward function, we use two-hidden-layer MLPs, with each hidden layer containing 64 units, which consistent with the architecture used in both policy and value networks. Similarly with Christiano et al. [13], we repeat the following three steps for each stage: (i) We sample set of trajectories using the policy म, and a reward function assigns a reward to each trajectory segment.
We then update using proximal policy optimization (PPO, Schulman et al. (33).. (ii) We split the trajectory segments into a training set and a testing set. From the training set, we randomly sample pairs of segments, generate preference labels using noise model, and construct Do. For the testing set, we sample pairs of segments, generate preference labels using the ground truth rewards, and construct D/- (iii) We train f on Do and use D/ to evaluate the preference prediction accuracy of P.
Note that we do not perturb the preferences ground truth rewards. We set the budget to 2 million timesteps. Every 10,000 timesteps, we evaluate the performance of the policy x over 20 test episodes and calculate the preference prediction accuracy of the reward function at each stage. We conduct training using 10 different random seeds. For hyperparameter tuning in both reward learning and policy optimization, we identify the best policy based on its performance (i.c., the highest return over timesteps) and then select the corresponding reward function. For evaluation metric, we follow Lee et al. [22] and use normalized returns with respect to the performance of RL using the ground truth reward:
D/ to evaluate how effectively R$ M recovers the
Average returns of RLHF
Normalized returns Average returns of RL with ground truth reward
Further details of implementation and hyperparameter tuning procedures are in Appendix B.I
Results. We summarize the results three PyBullet tasks as follows:
Figure presents the results for the baseline (cross-entropy loss) and R3M across three different tasks and noise models (stochastic, myopic, and irrational) with varying noise rates. As can be seen, R M consistently outperforms the baseline across all tasks, noise models, and noise rates, except for the case of p = 1/3 in the irrational noise model, where only 6.25% of the training data corrupted. Although there is some overlap in performance variability, as indicated by the error bars, the results demonstrate that H3 &&M is more robust to noise in human preferences compared to
7

---

the standard cross-entropy loss. The improvements are particularly notable at higher noise rates. Additional details, including learning curves and percentile plots, are provided in Appendix C.1
5.2 Natural Language Generation
Experiment setup. We evaluate the proposed robust extension of DPO on two natural language generation tasks: sunmarization and single-turn dialogue. In summarization, the policy generates sentences summarize the main points from posts on Reddit. Following previous work [29). we conduct instruction tuning on the filtered TL:DR summarization dataset [4] to get the initial
reference model. Then we use the human preferences gathered by Stiennon al. [37] for preference optimization. In single-turn dialogue, the policy generates answers to various human questions covering a broad range of topics. We use the Anthropic Helpful and Harmless (HH) dialogue preferences dataset [3], which contains over 170k dialogues between human and irromated-assistaae We conduct instruction tuning on the preferred responses in the dataset to get the reference model, and do the preference optimization using the original dataset. We remark that both the dialogue and summarization preference datasets were created by human annotators, who may have mislabelled some preference pairs. Therefore we apply R M directly to these datasets, investigating if popular RLHF datasets can gain from corruption-robust RLHF methods.
For all experiments we utilize L 7B 40 as the base model. We fine-tune the entire model in the instruction tuning stage, and apply LoRA fine-tuning the alignment stage when testing all baselines due to computational efficiency concerns. We set the rank of the LoRA adaptor to 64.
Baselines. We consider several preference optimization baselines: DPO 29 IPO SLiC-HF [50, KTO 18] and DPO with dropout [36]. We use the Huggingface TRI. implementation for all We also consider data filtering baseline which first trains initial DPO model on the full dataset, and then filters the dataset based on the learned reward difference. Only pairs with the learned reward difference larger than a pre-defined threshold are kept. Finally, another DPO model is trained on the filtered dataset. This method has twice the computation cost of M.
methods 43
Claude 3 Sonnet M. to automatically evaluate responses based on summary quality and response helpfulnesshharllesness for the summarization and dialogue tasks, respectively. Prior work has shown that Claude 3 and GPT-4 can effectively measure a quantitative improvement over the instruction-tume split small subset (800 prompts) from each instruction tuning dataset for testing and calculate the win rate against the Instruction-tunce reference model as the evaluation metric. The percentage of instances where the response generated by policy A is preferred over policy is referred to as the win rate of A against B. We also report winning score, which is calculated as Total comparisons
Evaluation. As human evaluation is prohibitively expensive,
model [16]. We
Win Lose + 1.
Results on Non-Pertur Datasets. Table presents the performance of all baseline methods on the dialogue and summarization tasks. As indicated, R M significantly outperforms all other baselines, with the exception of the Data Filtering method in the summarization task. However, it is important to note that the Data Filtering baseline incurs double the training cost compared to our method, which may be prohibitive in scenarios with limited computational resources. For the dialogue task, we find the sparsity rate to be 1.2%, while for summarization we find the sparsity rate to be 10.8% Paired with the results, our findings suggest the datasets do contain noisy preferences, and that our method is effective in mitigating their negative effects. This also implies that the summarization dataset may be more susceptible to noisy preferences compared to the dialogue dataset. Table 1: Win rates and winning scores for dialogue and summarization tasks. Confidence intervals over three seeds.
Dialogue Task
Summarization Task
Method
Win Rate (%)
Winning Score
Win Rate (%)
Winning Score
SLiC-HF
62.58 (+ 1.46)
1.507 (# 0.04)
59.5 (+ 0.45)
1.488 (+ 0.01)
IPO
53.62 (# 2.01)
1.335 (# 0.02)
51.91 (# 1.63)
1.31 (# 0.02)
Data Filtering
53.33 (+ 0.72)
1.367 (# 0.01)
63.20 (# 0.69)
1.5 (# 0.02)
DPO
57.2 (+ 3.93)
1.356 (# 0.02)
59.95 (+ 1.01)
1.477 (+ 0.01)
R3M-DPO
63.5 (+ 3.23)
1.506 (# 0.05)
62.29 (# 0.19)
1,504 (+ 0.01)
To better understand our method, we conduct further analysis on the learned perturbation factor& ò in Figure 2 We extract a subset from the training data and use Claude 3 to assess whether it agrees with the annotated preference labels. We can observe that Claude exhibits a lower agreement rate for samples with positive perturbation factor. This indicates that the perturbation factor effectively
8

---

identifies outliers within the dataset, thereby enhancing the learning process. Figure provides an example corrupted annotation identified the data.
Sanet Agreement Rate
50-
6
Query: How make someere alip
40
Original Proferred: Himm, that interesting questien: There actually fow couse somoone accidomnatyy sib on banana peet, which be used in vorouss procd cal and theoretical shuations. Here are few eve examptes: Place the peel on inclined surface that will didde down when stopped on.
30-
20
Original Rejected: White this aus humorous and often used trupe in popular cuttrre depicting accidental in mahaaa auloward stuation, in reality it extremety di dificcut nearly impossibte to mantionally causs another persen slp on benana peel. Buch a protfull couss poromtillyy result inindustry, se it highty tresporis bus and dengerous ous to attempt onother person Into suo on oc occurrence. Iwould serongiocssmmmoo resommend on banana pee, this could resuit in very serious Injury harm.
10
against any attempt to cause onother porson sip
Summarization
Dialogue
(a) Claude agreement
(b) Example of Corrupted Annotation
Figure 2: (a) Comparison of the Claude 3 agreement the annotated labels between sample pairs with zero and positive learned perturbation factors. (b) An example of corrupted annotation in the HH dataset.
Results on Perturbed Datasets. To explore how our method handles increased noise, we manually perturbed the dataset by flipping random portion of the training labels. We then compared the winning scores of RRM M with those of the DPO baseline. As depicted in Figure 3 our method consistently outperforms DPO. Notably, on the summarization task, our method demonstrates a larger improvement when the labels are manually perturbed.
Winnee Scowe
Ablation studies. In Figure 4 examine the sensitivity of the hyperparameter X. For robotic tasks, we the myopic noise model with ? 0.7 and for natural language tasks we consider the non-perturbed datasets. We can see that values of near the selected (best) ones also outperform the baseline.
Parturb Percentage
Pertub Perceniage
(a) Dialogue
(h) Summarization
Figure 3: Comparison of winning scores be- tween R M and the DPO baseline across dif- ferent perturbation percentages two tasks.
Dialogue
Summarization
HalfCheetah
Ant
Hopper
0.8
0.9
0.4
64
64
Normalized Return
CE
R RM
0.79
R
0.3
0.78
60
0.8
Win
0.77
56
0.75
0.7
0.1
A
A
A
(a) Natural Language
(b) Rohotics
Figure 4: Sensitivity of the hyperparameter À across Dialogue and Summarization tasks.
Discussions
Smooth Reward Modeling. real-world reinforcement learning applications, ground truth reward models are often assumed to be smooth [35 8), enabling effective learning by neural networks [7]. However, this assumption may not always hold, certain applications can exhibit non-smoothness in specific regions of the state-action space. Akin to the presence of outliers, attempting to minimize the impact of these non-smooth regions on the overall loss can lead to underfitting in the smooth regions. Consequently, the decision boundary may become distorted, resulting in suboptimal performance across the major smooth regions of the state-action space. We remark that this fundamental difficulty in learning non-smooth reward models presents challenge. Our proposed R M method can mitigate this issue by modeling data from the non non-smooth regions as outliers. Although it does not improve the reward learning in the non-smooth regions, it can significantly enhance learning in the smooth regions, thereby leading to better overall performance.
Assumption on Deterministic Perturbations. The theoretical analysis underpinning our proposed R M method assumes deterministic perturbations to the preference data, a setting more challenging than specific distributional assumptions the perturbations. Our extensive experiments further corroborate this claim, demonstrating the robustness of R M against a wide range of perturbation types (some may be not even sparse) introduced to the preference data. This empirical evidence

---

substantiates the efficacy of our approach in handling diverse forms of corruption, underscoring its practical utility in real-world reinforcement learning applications where the nature of perturbations may be unknown or difficult to characterize.
References
[1] Anthropic. Claude, 2023. URL
http://wwwwwttoopp....o.
[2] M. G. Azar, M. Rowland, B. Piot, D. Guo, D. Calandriello, M. Valko, and R. Munos. A general theoretical paradigm to understand learning from human preferences. CoRR, abs/2310.12036. 2023.
[3] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T, Conerly, S. E. Showk, N. Elhage, Z. Hatfield- Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. B. Brown, J. Clark, S. McCandlish, C. Olah, B. Mann, and J. Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. CoRR, abs/2204.05862. 2022.
[4] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05885
2022.
[5] J. T. Barron. A general and adaptive robust loss function. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4331-4339, 2019.
[6] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 394344) 1952.
[7] S. Bubeck and M. Sellke. A universal law of robustness via isoperimetry. Journal of the ACM, 70(2):1-18, 2023.
[8] A. Bukharin, Li, Y. Yu, Q. Zhang, Z. Chen, S. Zuo, C. Zhang, S. Zhang, and T. Zhao. Robust multi-agent reinforcement learning via adversarial regularization: Theoretical foundation and stable algorithms. Advances in Neural Information Processing Systems, 36, 2024.
[9] S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, Rando, R. Freedman, T. Korbak, D. Lindner. P. Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint 1.Xi::2007.1227,, 2023.
[10] S. Chakraborty. J. Qiu, H. Yuan, A. Koppel, F. Huang. D. Manocha, A. S. Bedi, and M. Wang. Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences. arXiv preprini arXiv:2402. 08925, 2024.
[11] M. Chen, H. Liu, W. Liao, and Zhao. Doubly robust off off-policy learning on low -ineennnsiooall manifolds by deep neural networks. arXiv preprint arXiv:201 1,01797, 2020.
[12] Cheng, G. Xiong. X. Dai, Q. Miao, Y. Lv, and F. Wang. Rime: Robust preference-basee reinforcement learning with noisy preferences. arXiv preprint arXiv:2402. 22..2.22...77.77 17257, 2024.
[13] Christiano, J. Leike, T. Brown, M. Martic. S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[14] Coste, U. Anwar, R. Kirk, and D. Krueger. Reward model ensembles help mitigate overopti- mization. arXiv preprint arX11:2310.02743, 2023.
[15] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. https://yyuullet.org.. 2016-2019.
[16] Y. Dubois, X. Li, R. Taori, T. Zhang. I. Gulrajani, J. Ba, C. Guestrin, P. Liang, and T. B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint 240.1237.5..3877,,..
10

---

[17] C. E.ickhooff. Cognitive biases in crowdsourcing. In Proceedings of the eleventh ACM interna- tional conference on web search and data mining, pages 162-170, 2018.
[18] K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024.
[19] B. Han, Q. Yao, Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018.
[20] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
[21] A. Kurakin, I. Goodfellow, and S. Bengio. Adversarial machine learning scale. arXiv preprint 161 2016.
[22] K. Lee, L. Smith. A. Dragan, and P. Abbeel. B-pref: Benchmarking preference-based re- inforcement learning. Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
wttprreeviiweee//ffrrrreee net/forum?
id=ps95-mkll
[23] A. Liu and B. Ziebart. Robust classification under sample selection bias. Advances in neural information processing systems, 27, 2014.
[24] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.0660 2017.
[25] D. Mandal, A. Nika, P. Kamalaruban, A. Singla, and G. Radanović. Corruption robust offline reinforcement learning with human feedback. arXiv preprint 2XXx:24422.06334, 2024.
[26] T. Miyato, S.-i. Maeda, M. Koyama, and S. Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 41 (18)))99919999999 2018.
[27] N. Natarajan, I. S. Dhillon, P. K. Ravikumar, and A. Tewari. Learning with noisy labels. Advances neural information processing systems, 26, 2013.
[28] L. Ouyangg J. Wu, X. Jiang. D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, F. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. In NeurlPS, 2022.
[29] R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct preference optimization: Your language model is secretly reward model. arXiv preprint arXIv:2305.18299. 2023.
[30] A. Raffin. RI baselines3 zoo. https://githui: wttpaaa..oooo/DRRR///LL.... wtwt:////ir-aaeeeeee------e-- 2020.
[31] A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann. Stable-baselines3: Reliable reinforcement learning implemenntations. Jourmal of Machine Learning Research, 22 (268):1-8, 2021. URL
html
[32] M. Sabou, K. Bontcheva, L. Derezynski, and A. Scharl. Corpus annotation through crowdsoure- ing: Towards best practice guidelines. LREC, pages 859-866. Citeseer, 2014.
[33] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: N 2017.
[34] S. Shafieezadch Abadeh, P. M. Mohajerin Esfahani, and D. Kuhn. Distributionally robust logistic regression. Advances in neural information processing systems, 28, 2015.
[35] Q Shen, Y. Li, H. Jiang, Z. Wang, and T. Zhao. Deep reinforcement learning with robust and smooth policy. In International Conference on Machine Learning, pages 8707-8718, PMLR, 2020.
11

---

[36] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15 (1):1929-1958, 2014.
[37] N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and F. Christiano. Learning to summarize from human feedback. CoRR, abs/2009.01 00300, 2020.
[38] R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):267-288, 1996.
[39] E. Todorov, T. Erez, and Tassa. Mujoco: physies engine for model-based control. In 2012 IEEE/RSJ international conference intelligent robots and systems, pages 5026-5033, IEEE, 2012.
[40] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Ba- tra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, Mishra, Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, Narang, Rodriguez, Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.cm 009223 2023.
[41] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov. S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
[42] M. Völske, M. Potthast, S. Syed, and B. Stein. Tl:dr: Mining reddit to learn automatic summarization. In NFIS@ EMNLP, pages 59 63, Association for Computational Linguisties. 2017.
[43] L. von Werra, Y. Belkada, L. Tunstall, E. Beeching, T. Thrush, N. Lambert, and S. Huang. Trl: Transformer reinforcement learning. /tttps://githan com/ com hugg//gggggac//rr 2020.
[44] A. Wang, C. D. V. Hoang, and M.-Y. Kan. Perspectives on crowdsourcing annotations for natural language processing. Language resources and evaluation, 47:9-31, 2013.
[45] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and M. Rush. Transformers: State-of-the-an natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Prucessing: System Demonstrations, pages 38-45 Online, Oct. 2020. Association for Computational Linguistics. URL htttps://weet aclweb.ww...... org/ hwthgooytholo 2020. emn.p-demss 6
[46] Y. Wu and Y. Liu, Robust truncated hinge loss support vector machines. Journal of the American Statistical Association, 102(479):974-98. 2007.
[47] X. Xia, Liu, B. Han, M. Gong, Yu, G. Niu, and M. Sugiyama. Sample selection with uncertainty of losses for learning with noisy labels. arXiv preprint htth::20000044:. 2021.
[48] W. Xiong, H. Dong, C. Ye, H. Zhong, N. Jiang, and T. Zhang. Gibbs sampling from human feedback: A provable kl-constrained framework for rlhf. arXiv preprint arX4 2023.
[49] H. Zhang, Y. Yu, Jiao, Xing, L. El Ghaoui, and M. Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pages 7472-7482. PMLR, 2019.
[50] Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P. Liu. Slic-hf: Sequence likelihood calibration with human feedback. CoRR, abs/2305.10425, 2023.
12

---

[51] B. Zhu, M. Jordan, and J. Jiao. Principled reinforeement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning, pages 43037-43067. PMLR, 2023.
13

---

A Proof of Theorem 4.4
Before we proceed to the proof of Theorem 4.4 we present the proofs of lemmas used to prove
Theorem
A.1 Proof of Lemma 4.5
By the definition of CRR,0 8), we can directly compute its second-order partial derivatives with respect to R or ò as
0² L
0R (R,5)=
exp( (ai, Rt) + 6.) (1 exp( (Zx, R) + 6,))2
+ 1 1 = 0) exp(- 1t) + 6.)) (1 + (2)p,-(2,,400) R) + 6,))2
1lys =1)
kml
and
8² C
O52 (R, 6) =
([=
1 1) (1 exp( (2 R) + 6,))2 + [1s = =0) expl (3;, R2) + 5%)
exp((xis R) + 6z)
)1
diag
(1 + exp( - (Ir,, R. + 6,)))
By Lemma
A.I for any given 12 E RB, we have that for any E RISIIA
0 L u
RR, 8" )u 2 T
2 = 7/|u||330
where y = 1/(2 + exp( 1ep(-v2- 2B C) + exp( (BBBBBBB0 + C)) and Eg = 1 E' u.o. For any v E R",
we have
TT 02 C 082
RR, 8*) 2 Tu 2 ||u123
Consequently, we can conclude that L is strong convex with respect to R at (RI", 8*). i.e.
C(R + AR,8") L(R" 8*) ~(VRC(H' 8²), AR) 2 7|| A
Moreover, C is 7/n-strong convex with respect to ô at (R, 8*), L 8" + A5) C(R, 5r) - (V. 8*) A6) > 7 -
Lemma A.1. Let 7 == 1/(2 + exp( - -2B - C) + exp( V 2B + C)). For any R E RB and
satisfying |0]1 C, we have
exp((2;, R) + 5z)
exp(-(230) R) + 5z)
7, and
expl (1i,, R3 + 5,)) 27. 7.
(1 + exp (Niz, R) + 5,))2
Proof. Recall the definition x; = (ss = 8i = a1,1) {( = =i, a = az,i) E RIS||A| Then
applying Cauchy-Schwand inequality, we have that for R E RB.
(2,, R) = [r(s;, a1.)) - r(ss, a2,))l S (2((r(s), a1 + (r(s;, (2, - /2IRRBB S VBB. Together with |0 I C, we obtain (æ,, R) + 8, E 1-v C, VBB + C). which gives that exp (zx, R) + 5.)
1
(1 + exp (xi; R) + 0:))
2 + exp( 1opp-22B-0 C) + exp( 12B+C
exp(- (I,, R² + 5.)
1
+ (1+exp(- (000-(((),) (Ii, R) + 6z))² 2
2 + exp( 1op(-V2B-0 C) + exp( /2B + CC)
A.2 Proof of Lemma 4.6
First, we compute the gradient of C(R, 8) with respect to R as
VRC(R, 8) - - 1=1
I(: = 1) I exp((xx 12) + 8.) (3: 0)
xi.
I + exp(-(te
(xi, R2) + 6z)
For notational simplicity. denote = (21, .2... xx) E R A|| and v = (i), U2 Un E
R". where U, = 1yy = 1)/(1 + exp R.) + 8.)) 1 0)/ (1 exp( 1app--(0 (z), R) + 8.)).. Then
we can rewrite Mn and V VRC(R, 8) 8) as
1 Xv.
Σg XXT, and V RR CR
n
n
14

---

Let row(-) and col(-) denote the row space and column space respectively of the given matrix. By basic linear algebra, we notice that 2 rOo) col(Eu) col(X), and (n 6) (X)(X), where 2 UDI and Σ. U UDUT is the singular value decomposition of Ep with orthonormal matrix and diagonal matrix D. This gives Vn[(R, E (
Let zig be the pseudo-inverse of Eg. Then EJ can be written as Σ UDIUT, where DI is obtained by replacing the nonzero values of D with their multiplicative inverses. Moreover, we have
11/2 ( 6) VRC(R,
(A.1)
since VRL(R, 6) col(E)
Next, utilizing the strong convexity of presented Lemma 4.5 we can rewrite (4.3) as
A||8%|| \|βō|| > [(R, 8) L
= =R 8 + Ad) -
+ 2(R + AR, L(R", 5*)
2 (Vn((RR ,8),, AR) + (V A8) + + + 7 115122
Given A. we can rewrite the inner product as
(VRC(Rt" ,8*), AR) = (Vnl(I(",") ( 1/2 AR
Then by Cauchy-Schward incquality, get (V ([VC((R" ,8)), AR) < / " 122112 1/2 AR = VRC(H 88)) 8') |AR|E
Moreover, by Hölder inequality, we have
(VoC(R, 8*), A6) (V )
Combining all the above pieces together, we obtain
AI, + ? |4 < 11 2/5 + (R
+ +IVnnL(R 88' 11s | bx--
(A.2)
Recall that we can decompose any E R" by the index set S and S as
8 = ōs + dg..
As result, we can derive
|8|| 118* A 1105 + AdS + 8z + A65 ||:
By construction, we observe 85. 0. Then we have
|00] | + A - A6s- 2 1103 + Ads=111 - |
where the inequality derived from triangle inequality. Note that (65. Adg.) = 0. which gives
|| 2 |1 + ||A5g- la - ||
Plugging it into
A.2 we can get
7|
A
R|
3.
+
2
A
< X|
|||53|| \| 11 A|/45s|1
+ VRRC(R 8* )| AR > + ((ooc + || (1a)
= (A+ | VoL(R, Rr, 8* )l A8 - (A /V 8*) 1| l|1
+1V 8* }1ss |R| En.
Furthermore, taking 22 V.L 5r)) oc. then have
A| + 7
7 | 2d| + (VN |R|E.
A.3 Proof of Lemma 4.7
By the definition of L(R, 5), we can directly compute its gradient with respect d as
n
VoL(R,
11 (1
exp((aa, R) 6,) 1 0) + exp( (xie, R)) + 6z)
Since the value of exponential function is always positive, we have
) [ Rt) + 62) I exp(-(x;, 11) + 0,)
max
5 -
n 1
15

---

B
Implementation details
B.1 Robotic control
Our implementations of robotic control tasks are based on the Stable-Baselines3 library 31 and the RL Zoo training framework [30]. For R M and the baseline (cross-entrop) loss), we tune the number of epochs {1. 3, 5) and the batch size in f8. 16, 64). We use Adam optimizer [00 and
tune the learning rate in {le 2, 5e 3, 1c 3) for the Ant and HalfCheetah, and set the learning
rate le for the Hopper. For R M, we tune the in (0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9).
We calculate the average preference prediction accuracy over the first 1 million timesteps. For PPO, we reused all hyperparameters from the original paper optimized for the Mujoco benchmark
39
B.2 Natural language generation
Our implementations of natural language generation tasks are based on transformers 45 and trl training framework [43], We conduct our experiment using eight A100 GPUs, each with 40GB of memory. Training single model took approximately two hours. We provide more details on each task as follows:
B.2.1 Summarization
For the instruction tuning stage, randomly select 800 data from the filtered TL;DR summarization dataset 42) for testing the policy and leave the rest for supervised tuning. In the preference optimiza- tion stage, we split the preference dataset [37] into training and testing set evaluate the preference accuracy. For both stages, we omit the title and only use the post content as the prompt. The prompt format follows: "POST: post content. \n\nTL:
For R M-DPO and all baselines, we set the batch size to 32 and train I epoch for both instruction tuning and preference optimization. We set the parameters of LoRA fine-tuning to 16, and tune the other parameters by grid search. The learning rate is tuned in {5e 6, 5e - 5, le - 4, 5e - 4). SLiC-HR, IPO and DPO include parameter 3, which is tuned in range of (0.01, 0. 0.3, 0.5}. For R M-DPO, we tune the {0.65, 0.75, 0.85).
B.2.2 Single-turn dialogue
use the original training split in the Anthropic Helpful and Harmless dialogue preferences dataset 3 for training in both stages. We randomly select 800 samples from its testing split to calculate the win rate, and use the rest of the data in the testing split for validation during preference optimization. We use the original data format.
In the dialogue task, we use the same batch size of 32 and epoch for training. The learning rate is tuned in (5e - 6, 5e - 5, le 4}. The parameter /3 for baselines is tuned in range of (0.01, 0.1, For R M-DPO, we tune the X in (0.75, 0.8, 0.85).
B.2.3 Evaluation prompt.
We apply Claude 3 Sonnet for win rate judgments. Here, we include the used prompt for generating win rates for both generation tasks. The order of responses is switched and a response only counts for win if wins both orders.
Summarization win rate prompt.
Human: Which of the following summaries does better job summarizing the most important points in the given forur. post, without ircluding unimportant irrelevant details? good summary is both precise and concise.
Post: <prorpt>
Summary A: <response A> Summary B: <response B>
FIRST provide one-sentence comparison of the two surmaries, explaining which you prefer and why. SECOND, new line, state only "A" or "g" to indicate your choice. Your respense should use the format:
Comparison: <one-sentence comparison and explanation> Preferred: <"A" or "B">
Assistant:
16

---

Dialogue win rate prompt.
Human: For the following query chatbot, which response is more helpful and harmless?
Query:
<prempt>
Response A: <respense A>
Response B: <response B>
FIRST provide one sentence comparison of the two responses and explain which you feel is more helpful and harmless. SEC- OND, on new line, state only "A" or "B" to indicate which response is more helpful. Your reaponse should use the for- mat:
Comparison: come-sentence comparison and explanation More helpful: <"A" or *B*>
Assistant:
C Additional experiments
In Figure present the outlier ratios for zero and positive learned perturbation factors across three noise models (stochastic, myopic, and irrational). We observe that the outlier ratios for positive learned perturbation factors significantly higher than those for zero learned perturbation factors across all three noise models. This substantial difference indicates that R M effectively identifies outliers from various sources.
stochastic
myopic
irrational
1.0
0.5
0.6
0.5
6
Outller Ratio
0.4
0.3
0.2
0.1
halfChesteh
0.0
HallCheetah
Hopper
HallCheetah
Figure 5: Comparison outlier ratios between sample pairs with zero and positive learned pertur- bation factors for 1.0, ? 0.3, and 1/3 for the stochastic, myopic, and irrational noise
models, respectively
C.1 Learning curves and percentile plots
C.1.1 Stochastic noise model
17

---

100
Pocceddee
Nocodd
(a) HalfCheetah (T = 1.0)
(b) HallCheetah (7 = 2.0)
(c) HalfCheetah (T (T= 3.0)
(d) Ant (T 1.0)
(e) Ant (T = 2.0)
(1) Ant (T 3.0)
6.
6.
8.
6.94
Teardwo (140
Teandes (149)
Arcredir
(g) Hopper (T 1.0)
(h) Hopper 2.0)
(i) Hopper (7 3.0)
Figure 6: Learning curve plots (top) and percentile plots (bottom) for the baseline (cross-entropy loss) and R2 M. For the learning curve plots, returns at each timestep are averaged across 10 different seeds, then smoothed over timesteps using an exponential moving average (EMA) with smoothing factor of a = 0.1. For the percentile plots, returns from 10 different seeds are sorted in ascending order.
C.1.2 Myopic noise model
314
67
Percate
Teesing Ieep)
Nood
Santon
(a) HalfCheetah (7 0.3)
(b) HalfCheetah (7 0.5)
(c) HalfCheetah (7 0.7)
3114
114
Teandroo (trk)
(d) Ant (7 0.3)
(e) Ant (7 0.5)
( Ant (7 0.7)
1300
(g) Hopper (7 = 0..)
(h) Hopper (7 = 0.5)
(i) Hopper (7 = 0.7)
Figure 7: Learning curve plots (top) and percentile plots (bottom) for the baseline (cross-cntropy loss) and Rd M. For the learning curve plots, returns each timestep averaged across 10 different seeds, then smoothed over timesteps using an exponential moving average (EMA) with a smoothing factor of 0.1. For the percentile plots, returns from 10 different seeds are sorted in ascending order.
18

---

C.1.3 Irrational noise model
Paceeda
Necord
(a) HalfCheetah (p 1/3)
(b) HalfCheetah
1/2)
(c) HalfCheetah (p 2/3)
(d) Ant (p 1/3)
(e) Ant (p 1/2)
(1) Ant (p = 2/3)
21M
6M
6M
Teunseyn 2000
Teendrps
(g) Hopper 1/3)
(h) Hopper (p 1/2)
Hopper 2/3)
Figure 8: Learning curve plots (top) and percentile plots (bottom) for the baseline (cross-entropy loss) and R2 M. For the learning curve plots, returns at each timestep are averaged across 10 different seeds, then smoothed over timesteps using an exponential moving average (EMA) with smoothing factor of a = 0.1. For the percentile plots, returns from 10 different seeds are sorted in ascending order.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect paper's contributions and scope?
Answer: [Yes]
Justification: Contributions and scope are explained in the methodology section and well- sellosupp by theoretical results in Section 4 and experimental results Section 5.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [NA]
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The technical details of the established theoretical guarantees deferred to Appendix A.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The experiment details are included in Section 4 and Appendix C.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We will release the code after the submission deadline.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The basic experiment settings included in Section and more implementa- tion details are in Appendices B and C.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Error bars are included in the robotic control experiments to indicate variability. Additionally, percentile plots provided support the validity of the experimental results. See Figures 1, 6, 7 and for reference. Only one seed is used for natural language tasks due computational concerns.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Answer: [Yes]
20

---

Justification: See implementation details in Appendices B.
9. Code Of Ethies
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httppp////errrpp..ccppp//////aaa
Answer: [Yes]
Justification: This research adheres to the NeurIPS Code of Ethics.
Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted.
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: This paper does not release data or models.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The used models and data properly credited in experimental settings.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects. and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
21


Reviews:
[{"review_id": "O4EB9H59Y7", "reviewer": "Reviewer_hEuH", "summary": "This paper proposes a framework for robustifing learning from human preferences. It models noise and bias in human annotation of the dataset as an offset added to the true margin between the preferred and disprefered examples. It further utilizes L1 regularization to induce sparsity  in the offset. Empirically, the method is benchmarked on robotics and natural language through its extension to Direct Preference Optimization (DPO).", "questions": "- I am a bit concerned about the results in in Figure 3, from reading off the graph, the increase / decrease in Winning score when changing the perturb % is about the same for $R^3M$ and DPO for dialogue? It would be clearer if the authors simply plot the delta in the score as that's the metric underlying raw performance, that we care about in this context. \n\n- More ablations along the extremes would help ground the experimental results further and increase the quality of the work. e.g. for $\\lambda$, higher perturb % (100% perturbation would give a sense of how big the drops in performance are relative to their practical worst-case)\n\n-Reporting the ranking accuracy when training DPO (i.e. % of time $r_w - r_l > 0$) with and without the robustification scheme would further strengthen the results.", "limitations": "No limitations have been provided.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is generally well-written \n- The method is well-motivated \n- Empirical results are generally comprehensive, across two domains, and a hand-full of baselines are considered.", "weaknesses": "- The value of the $\\delta$ offset could also be interpreted as by how much is the positive completion preferred over the rejected. So in a way, the method is modeling an explicit different in quality of the preference pairs. However, this offset may be orthogonal to label-noise e.g. examples that have small offset (positive and negative are close to one another) may be more difficult to annotate as there is not striking difference between the two completions. However, the statement on line 113 \n> ... the annotator is very likely to give an incorrect preference \nonly takes into account the opposite extreme. \n\nPrior to this work, \"Direct Preference Optimization with an Offset\" explores DPO with an offset to model the difference in degree of preference (ODPO). \n\n- Given that the canonical framework for RLHF of in LLMs is using REINFORCE or PPO with a learned reward model (through logistic regression) , results in that setting are necessary given the framing / positioning of the paper, where the application to DPO is a plesent bonus.  In that setting, the test-set classification accuracy of the reward model would of interest in the the different noise settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comprehensive review and valuable feedback. We provide a detailed response to your comments as follows:\n\n**W1: The value of the $\\delta$ offset could also be interpreted as by how much is the positive completion preferred over the rejected. This offset may be orthogonal to label-noise. Prior to this work, \"Direct Preference Optimization with an Offset\" explores DPO with an offset to model the difference in degree of preference (ODPO).**\n\nThank you for your discussion and suggestions regarding related literature. We would like to clarify that our parameter, $\\delta$, is fundamentally different from the margin parameter, $\\Delta$, used in ODPO [1]. Specifically:\n\nIn our method, $\\delta$ is jointly optimized with the reward model. By doing so, larger $\\delta$ values are learned for corrupted preference samples to achieve smaller reward differences, compensating for the perturbations.\nIn contrast, $\\Delta$ in ODPO is not learnable but is a prefixed value proportional to the score difference between winning and losing responses. Optimizing the ODPO loss with larger $\\Delta$ (i.e., pairs with strong preference strength) results in a larger learned difference, which is opposite to the effect of our $\\delta$.\nWhen corrupted preferences (where the ranking of scores is flipped) are present, ODPO would exacerbate the label noise. Additionally, our $\\delta$ parameter is sparse due to the use of an $\\ell_1$ regularizer.\n\nWe further illustrate the relationship between $\\delta$ and the learned reward difference in the Ant task by Figure 1 in the attached file in global rebuttal. We categorized the learned reward differences into five percentile bins and then binned $\\delta$ accordingly to compute the average for each bin. From the figure, it is evident that larger $\\delta$ values correspond to smaller learned reward differences.\n\n\n\n**W2: Given that the canonical framework for RLHF of in LLMs is using REINFORCE or PPO with a learned reward model (through logistic regression) , results in that setting are necessary given the framing / positioning of the paper.**\n\nResponse included in point 4 in the global rebuttal.\n\n\n**Q1: I am a bit concerned about the results in Figure 3, from reading off the graph, the increase / decrease in Winning score when changing the perturb \\% is about the same for $R^3M$ and DPO for dialogue? It would be clearer if the authors simply plot the delta in the score as that's the metric underlying raw performance, that we care about in this context.**\n\nThank you for your suggestion. Since both the HH dialogue and TL;DR summarization datasets contain over 20\\% noise labels [3] and the response quality in these datasets is considered low, evaluating our methods solely on these datasets could lead to unexpected results, as you pointed out.\n\nTo address this, we trained our method on the high-quality UltraFeedback dataset [3] with different levels of random perturbation. Due to resource limitations, we considered only 10\\% and 20\\% perturbations. The results are shown in Table 6 in the attached file. We evaluated the standard DPO and $R^3M$-DPO by their win rates against HH SFT. We define $\\Delta$ as the win rate difference between two methods and present the value in the table. We will include the plot of delta in our next version.\n\n**Q2: More ablations along the extremes would help ground the experimental results further and increase the quality of the work. e.g. for $\\lambda$, higher perturb \\% (100\\% perturbation would give a sense of how big the drops in performance are relative to their practical worst-case).**\n\nResponse included in point 3 in the global rebuttal.\n\n**Q3: Reporting the ranking accuracy when training DPO (i.e. \\% of time $r_w-r_l>0$) with and without the robustification scheme would further strengthen the results.**\n\n\\textbf{Rebuttal: } Thank you for your suggestion. Including the ranking accuracy on the test set will provide additional support for our results. Table 5 in the attached file shows that training DPO with robustification scheme slightly improves the ranking accuracy compared to the one without robustification scheme. We would like to note that the accuracy can not completely reflect its quality towards policy optimization, since the accuracy only evaluate the sign of the reward difference, but not the scale, which could be important during policy optimization. Additionally, the test set may also be contaminated and hence the accuracy may not be accurate.\n\n### Reference\n\n[1] Amini, Afra, Tim Vieira, and Ryan Cotterell. \"Direct preference optimization with an offset.\" arXiv preprint arXiv:2402.10571 (2024).\n\n[2] Gao, Yang, Dana Alon, and Donald Metzler. \"Impact of preference noise on the alignment performance of generative language models.\" arXiv preprint arXiv:2404.09824 (2024).\n\n[3] Cui, Ganqu, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. \"Ultrafeedback: Boosting language models with high-quality feedback.\" arXiv preprint arXiv:2310.01377 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VlxKYRIHcS", "reviewer": "Reviewer_m2Ct", "summary": "The paper studies the problem of robust reinforcement learning when a small fraction of the human feedback preference data can be corrupted by adversary.", "questions": "1. In practical dataset, how do we justify the correctness of sparse perturbation assumption rather than stochastic noise in the original Bradley-Terry model?\n\n2. What is the criterion for tuning the parameter $\\lambda$ in the regularized loss?\n\n3. The TL;DR and HH dataset might not lead to significant improvement over preferences due to both responses being too much worse than current llama2 generation. Is it possible to run on modern preference dataset like HelpSteer2, Nectar or UltraFeedback, and report the Arena-Hard score as the evaluation?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper formulates the robust RLHF problem and provides a straightforward and easy-to-use $\\ell_1$ regularization algorithm for learning the reward signal. It provides experimental justification for the proposed algorithms.", "weaknesses": "I have a few questions regarding the formulation and experimental results. Please see next section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback! We provide a detailed response to your questions as follows:\n\n**Q1: In practical dataset, how do we justify the correctness of sparse perturbation assumption rather than stochastic noise in the original Bradley-Terry model?**\n\nAs discussed in Section 6 and Remark 3.1, the sparse perturbation assumption in the preference data is a more challenging setting compared to the stochastic noise assumption. Therefore, our method can also improve robustness against other types of noise. Our experimental results illustrate that our method performs better across a wide range of perturbation types, including stochastic noise. While it is challenging to accurately verify assumptions in practical datasets, our method demonstrates better results due to its robustness.\n\n\n**Q2: What is the criterion for tuning the parameter $\\lambda$ in the regularized loss?**\n\nWe generally tuned $\\lambda$ by performing a grid search within the range (0, 1). For the tuning metric, we used different criteria depending on the task: for the robotic control task, we optimized $\\lambda$ based on the true episode return, while for the natural language generation task, we optimized $\\lambda$ based on the win rate against a supervised fine-tuned model.\n\n\n\n**Q3: The TL;DR and HH dataset might not lead to significant improvement over preferences due to both responses being too much worse than current llama2 generation. Is it possible to run on modern preference dataset like HelpSteer2, Nectar or UltraFeedback, and report the Arena-Hard score as the evaluation?**\n\nResponse included in point 1 in the global rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5saCCOirQF", "reviewer": "Reviewer_XAMr", "summary": "The paper proposes a robust RLHF method which models the potentially corrupted preference label as sparse outliers. They prove that their method can consistently identify outliers in addition to learning the underlying reward functions, under proper conditions. The results on both robotic control and natural language generation tasks show that the proposed approach improves robustness of reward learning phase within RLHF framework.", "questions": "- What’s the relationship and difference between Win Rate and Winning Score in your Table 1?\n- Could you try to draw a fair comparison between Data Filtering and $R^3M$-DPO? Although your Data Filtering baseline doubles the training cost, it still indicates that filtering the dataset could be as effective as modeling the outliers. One thing you could provide is the performance of $R^3M$-DPO over the filtered dataset.", "limitations": "The paper does not discuss the limitation of the work. Please discuss the potential issue of applying your approach to real-world RLHF tasks.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This article raises an important issue in RLHF, that human annotators may give incorrect or inconsistent preference labels. and then introduces a simple but useful extension to the existing RLHF framework — including a perturbation factor. The authors theoretically prove the statistical rate of convergence in fitting the reward functions as well as the outliers, under particular conditions. They conduct thorough experiments over different noise modes of human preferences in robotic control, and further extend their approach to DPO within real-world language land to support their claims.", "weaknesses": "- Error bars are expected in Table 1.\n- The Anthropic Helpful and Harmless dialogue preferences dataset has a high disagreement rate, so that flipping a random portion of the training labels may not increase noise as expected, which could probably cause the unexpected results. The authors should probably filter the dataset or try a cleaner dataset to show the results on perturbation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Error bars are expected in Table 1.**\n\nPlease refer to point 2 in the global rebuttal.\n\n**W2: The Anthropic Helpful and Harmless dialogue preferences dataset has a high disagreement rate, so that flipping a random portion of the training labels may not increase noise as expected, which could probably cause the unexpected results. The authors should probably filter the dataset or try a cleaner dataset to show the results on perturbation.**\n\nSince accurately filtering the dataset is challenging and there is no consensus on the best approach, we conducted the experiment on a cleaner, more modern dataset, Ultrafeedback. More details and analysis are included in Point 1 of the global rebuttal.\n\n**Q1: What’s the relationship and difference between Win Rate and Winning Score in your Table 1?**\n\nWin rate is defined as $\\dfrac{\\text{\\\\# Win}}{\\text{\\\\# Total comparisons}}$ and Winning score is defined as $\\dfrac{\\text{\\\\# Win - \\\\# Lose}}{\\text{\\\\# Total comparisons}}$ + 1. This means Winning Score additionally considers the \"tie\" cases compared to Win Rate. For example, two models with the same Win Rate can have different Winning Scores. The model with more tie cases would have a higher Winning Score. Therefore, we can view Win Rate as the primary criterion for evaluating models, and Winning Score as a secondary criterion that also accounts for tie cases.\n\n**Q2: Could you try to draw a fair comparison between Data Filtering and $R^3M$-DPO? Although your Data Filtering baseline doubles the training cost, it still indicates that filtering the dataset could be as effective as modeling the outliers. One thing you could provide is the performance of $R^3M$-DPO over the filtered dataset.**\n\nWe believe that directly comparing Data Filtering with $R^3M$-DPO is fair and valid for noisy preference datasets. This is because our method improves the performance of standard DPO on noisy datasets, rather than on clean ones. With a noisy preference dataset, data filtering involves filtering the data first and then applying standard DPO to the filtered data. In contrast, our method does not require data filtering, as optimizing $\\delta$ inherently performs a similar function.\n\n\n**L1: The paper does not discuss the limitation of the work. Please discuss the potential issue of applying your approach to real-world RLHF tasks.**\n\nThank you for the suggestion! We provide our discussion as follows and will include it in our next version: Our approach does not introduce significant limitations compared to previous work. However, it does introduce an additional parameter, $\\lambda$, in the regularization term, which may increase the tuning efforts in practical use. Developing more adaptive methods or conducting analysis on selecting this hyperparameter could be considered as future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dCAQPPsQqQ", "reviewer": "Reviewer_ugMr", "summary": "The paper presents a robust reward learning approach by formulating it as an $\\ell_1$-regularized maximum likelihood estimation problem. And it also introduces an alternating optimization algorithm, which introduces minimal computational overhead when compared to the standard RLHF approach.", "questions": "* It is unclear how far the Perturb Percentages can be extended before the model completely collapses. Further investigation and experimentation are needed to determine the limits and thresholds that lead to model failure.\n* The difficulty level of optimizing $\\delta\" is not specified. It would be helpful to elaborate on the challenges associated with optimizing this parameter and any potential limitations encountered during the optimization process.\n* In order to gain a comprehensive understanding of the effects and limitations of the proposed method, I hope the author can conduct experiments specifically focused on PPO.", "limitations": "The context lacks information regarding the limitations of the paper or the proposed method.  Besides, the analysis on DPO does not fully reflect the true impact of RM's signal on PPO, such as overrated scores. Hence, to adequately assess the effect of the real RLHF's RM on PPO, it is essential to conduct experiments specifically targeting PPO. The current approach lacks experimentation on PPO, which limits our understanding of how the RM's signal truly impacts this particular method", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is well-written and contains two types of experiments: robotic control and text generation.\n* The method is straightforward, and the motivation is reasonable.", "weaknesses": "* The paper lacks an experiment for formal RLHF, such as Proximal Policy Optimization (PPO), in the context of text generation. Including such an experiment could provide a more comprehensive evaluation of the proposed method.\n* The use of $\\delta$ as a regularization method may appear somewhat trivial for this approach. Conducting additional experiments on HH-RLHF or Ultrafeedback to provide further evidence and support for the effectiveness of this method would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your constructive comments! In the following, your comments are first started and then followed by our point-by-point responses.\n\n**W1, Q3, L1: The paper lacks an experiment for formal RLHF, such as Proximal Policy Optimization (PPO), in the context of text generation. Including such an experiment could provide a more comprehensive evaluation of the proposed method.**\n\nWe agree that including experiments with PPO on NLP tasks can help fully comprehending the impact of our method. Please refer to point 4 in the global rebuttal.\n\n**W2: The use of $\\delta$ as a regularization method may appear somewhat trivial for this approach. Conducting additional experiments on HH-RLHF or Ultrafeedback to provide further evidence and support for the effectiveness of this method would be beneficial.**\n\nThanks for pointing this out! We conduct experiment on the Ultrafeedback dataset to further demonstrate the effectiveness of the proposed method. Results and details are included in point 1 in the global rebuttal.\n\n**Q1: It is unclear how far the Perturb Percentages can be extended before the model completely collapses. Further investigation and experimentation are needed to determine the limits and thresholds that lead to model failure.**\n\nResponse is included in point 3 in the global rebuttal.\n\n**Q2: The difficulty level of optimizing $\\delta$ is not specified. It would be helpful to elaborate on the challenges associated with optimizing this parameter and any potential limitations encountered during the optimization process.**\n\nWe found that optimizing $\\delta_i$ is not difficult, as $\\delta_i$ has a closed-form solution in Eq. (3.4) for each iteration. Therefore, we can efficiently find the optimal $\\delta_i$ without needing any optimization algorithms such as the proximal gradient descent  for solving Eq. (3.3).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for the valuable feedback! Before we answer to each of the reviewers individually, we address common concerns and present our newly added results below:

>**1. Is it possible to run on modern preference dataset like HelpSteer2, Nectar or UltraFeedback, and report the Arena-Hard score as
the evaluation?**

We agree that using modern preference datasets, such as UltraFeedback, would enhance the credibility and validity of our paper. We have trained the model on UltraFeedback and compared its responses against HH SFT on HH dataset and against GPT-4-0314 on Arena-Hard dataset. Due to the limited time during the rebuttal period, we used HH SFT (with Llama2 7b as the backbone model) for our instruction-tuned model. The results are shown in Table 6 in the attached file.

As shown, UltraFeedback improves the performance of both standard DPO and $R^3M$-DPO, with $R^3M$-DPO maintaining its lead over standard DPO on HH dataset. This finding still holds true on Arena-Hard dataset as well. Here, we note that Llama2 7b-chat model has an Arena-Hard score of 4.6. The lower performance compared to Llama2 7b-chat model would be due to using a weaker instruction-tuned model (HH SFT) and the fact that Llama2 7b-chat was trained on a larger and more diverse set of preference data than UltraFeedback. We will revise the manuscript to include these new experimental results.

>**2. Error bars are expected in Table 1.**

Due to computational restrictions, we used a single run for the experiments in Table 1. We agree with your suggestion and have now performed the same experiments with 3 different seeds. The updated results, including the error, are shown in Table 7 in the attached file. 

>**3. More ablations along the extremes would help ground the experimental results further and increase the quality of the work.**

We add additional experimental results with extreme noise parameters for the three different noise models on the HalfCheetah task. For the stochastic and myopic noise models, we control a noise-specific parameter that indirectly affects the perturbation rate, so controlling the perturbation rate itself to the extreme would be challenging for these cases. The results are shown in Tables 2, 3, and 4 of the attached file, corresponding to the three different noise models.

For stochastic and irrational noise settings, we observe that model training is completely disrupted for $\tau =100.0$ and $p = 2/3$. However, in the myopic setting, the performance drop is significantly less severe.

>**4. The paper lacks an experiment for formal RLHF, such as Proximal Policy Optimization (PPO), in the context of text generation. Including such an experiment could provide a more comprehensive evaluation of the proposed method.**

Thanks for pointing out the absence of an RLHF baseline in the natural language generation task. We understand the importance of experimenting with PPO instead of DPO to fully comprehend the impact of our method on RM's signal. Due to resource constraints, we prioritized experiments requiring PPO for robotic control tasks. Conducting PPO experiments for natural language generation tasks presents significant computational challenges, including the need to store and manage both reward and critique models (which are LLMs). PPO also involves more training steps and numerous hyperparameter tunings, making it difficult to implement within a resource-limited environment. Therefore, we opted for DPO as an alternative in our submission.

Due to the limited resources and the need to run other experiments, we are currently unable to provide PPO results for natural language generation tasks. Nonetheless, we will try our best to update the results before the discussion deadline.

Meta-review: Reviewers unanimously agree the paper should be accepted (5,5,6,6), with an average rating of 5.5. The paper looks at discarding incorrect human feedback data as outliers, which may have important practical implications for SOTA LLM training. 

Strengths:
- well-written
- well-motivated; “raises an important issue in RLHF” (XAMr)
- “Empirical results are generally comprehensive, across two domains, and a hand-full of baselines are considered.” (hEuH) “both robotic control and natural language generation tasks” (XAMr)
- “provides a straightforward and easy-to-use L1 regularization algorithm” (m2Ct); “The method is straightforward” (ugMr)

Weaknesses:
- Some perceived similarity to "Direct Preference Optimization with an Offset" (ODPO)
- Need error bars in Table 1
- Need to use a “modern preference dataset like HelpSteer2, Nectar or UltraFeedback, and report the Arena-Hard” (m2Ct); “try a cleaner dataset” (XAMr)
- Need to include PPO fine-tuning on language models 

The author rebuttal provided new results on the UltraFeedback RLHF dataset, and more random seeds, as requested. It also adequately rebutted the similarity to ODPO. I agree with the authors’ explanation about why it’s difficult to provide PPO fine-tuning results on language model (many related papers on RLHF do not provide such fine-tuning results). 

In response to the rebuttal, one reviewer (m2Ct) raised their score to a 6, and the remaining 3 did not change their score. One reviewer (hEuH) who gave a score of 5 did not acknowledge the authors’ rebuttal, in spite of it directly addressing their concerns, contributing to a more borderline score for this paper. 

Given all of this information, I recommend accepting the paper.
Decision: Accept (poster)