Perplexity-aware Correction for Robust Alignment with Noisy Preferences
Authors: Keyi Kong
Xilie Xu
Di Wang
Jingfeng Zhang
Mohan Kankanhalli
Venue: NeurIPS 2024 poster (2024)
Keywords: Large language model
Label-noise learning
Primary area: safety_in_machine_learning

Abstract: Alignment techniques are critical in ensuring that large language models (LLMs) output helpful and harmless content by enforcing the LLM-generated content to align with human preferences. 
However, the existence of noisy preferences (NPs), where the responses are mistakenly labelled as chosen or rejected, could spoil the alignment, thus making the LLMs generate useless and even malicious content. 
Existing methods mitigate the issue of NPs from the loss perspective by adjusting the alignment loss based on a clean validation dataset.
Orthogonal to these loss-oriented methods, we propose perplexity-aware correction (PerpCorrect) from the data perspective for robust alignment which detects and corrects NPs based on the differences between the perplexity of the chosen and rejected responses (dubbed as PPLDiff). 
Intuitively, a higher PPLDiff indicates a higher probability of the NP because a rejected/chosen response which is mistakenly labelled as chosen/rejected is less preferable to be generated by an aligned LLM, thus having a higher/lower perplexity.
PerpCorrect works in three steps: 
(1) PerpCorrect aligns a surrogate LLM using the clean validation data to make the PPLDiff able to distinguish clean preferences (CPs) and NPs. 
(2) PerpCorrect further aligns the surrogate LLM by incorporating the reliable clean training data whose PPLDiff is extremely small and reliable noisy training data whose PPLDiff is extremely large after correction to boost the discriminatory power.
(3) Detecting and correcting NPs according to the PPLDiff obtained by the aligned surrogate LLM to obtain a denoised training dataset for robust alignment.
Comprehensive experiments validate that our proposed PerpCorrect can achieve state-of-the-art alignment performance under NPs.
Notably, PerpCorrect demonstrates practical utility by requiring only a modest amount of validation data and being compatible with various alignment techniques. 
Our code is available at [PerpCorrect](https://github.com/luxinyayaya/PerpCorrect).

Perplexity-aware Correction for Robust Alignment with Noisy Preferences
Keyi Kong
Xilie Xu Di Wang' Jingfeng Zhang*5t Mohan Kankanhalli
SShandong University National University of Singapore
3 SKin Abdullah University of Science and Technology 'The University of Auckland
'RKKEN Center for Advanced Intelligence Project (AIP)
luxinyayaya@mail sdu. edu. xuxilie@comp mus. edu. di www.wannnus..eeeemmmmm edu. sa, jingfeng- zhang@auckland nz, mohan@comp.. nus edu.
Abstract
Alignment techniques are critical in ensuring that large language models (LLMs) output helpful and harmless content by enforcing the LLM-generated content to align with human preferences. However, the existence of noisy preferences (NPs), where the responses are mistakenly labelled as chosen or rejected, could spoil the alignment, thus making the L.MS generate useless and even malicious content. Existing methods mitigate the issue of NPs from the loss perspective by adjusting the alignment loss based on a clean validation dataset. Orthogonal to these loss- oriented methods, we propose perplexity-awar correction (PerpCorrect) from the data perspective for robust alignment which detects and corrects NPs based the differences between the perplexity of the chosen and rejected responses (dubbed as PPLDiff). Intuitively, higher PPLDiff indicates a higher probability of the NP because rejected/chosen response which is mistakenly labelled chosen/rejected is less preferable to be generated by an aligned LLM,, thus having higher/lower perplexity. PerpCorrect works in three steps: (1) PerpCorrect aligns a surrogate LLM using the clean validation data to make the PPLDiff able to distinguish clean preferences (CPs) and NPs. (2) PerpCorrect further aligns the surrogate LLM by incorporating the reliable clean training data whose PPLDiff is extremely small and reliable noisy training data whose PPLDiff is extremely large after correction to boost the discriminatory power. (3) Detecting and correcting NPs according the PPLDiff obtained by the aligned surrogate LLM obtain a denoised training dataset for robust alignment. Comprehensive experiments validate that our proposed PerpCorrect can achieve state-of-the-at of-the-art alignment performance under NPs. Notably, PerpCorrect demonstrates practical utility by requiring only a modest amount of validation data and being compatible with various alignment techniques. Our code is available at PeerPorrect
Introduction
Alignment enables the safe utilization of the remarkable capabilities acquired by large language models (LLMs) through self-supervised learning on vast corpora 6 24 4. It refers to the process of
ensuring that the contents generated by LLMs are helpful, harmless, and aligned with human values and preferences [19] Reinforcement Learning from Human Feedback (RLHF) 9 has emerged
a primary technique for achieving alignment. Current technical routes
39 40 29 require
reward model to simulate human preference and use it to optimize the policy model outputs with
Equal contribution.
Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Performance of DPO series alignment methods
Performance of PPO seties alignment methods
96
Reward acetracy (%)
Reward aceuracy (%)
94
80
92
70
Vanila DPO
90
Vanilla PPO
rDPO
<PPO
60
:DPO
tPPO
PerpCorrectDPO
PerpCorrect-PPO
86
10
20
30
40
10
20
30
40
Proportion of noisy preferences (%)
Proportion of noisy preferences (%)
(a) DPO series methods.
(b) PPO series methods.
We evaluated various robust alignment methods under different proportions of noisy preferences using the Llama2-7B model, on the Golden HH dataset. The reward accuracy of both the vanilla DPO and PPO method significantly decreases the proportion of noisy preferences increases. Our method, perplexity-awee awaare correction (PerpCorrect), outperforms both the DPO and PPO series baselines across different proportions of noisy preferences.
Figure 1:
Proximal Policy Optimization (PPO) (27]. Current offline techniques such as Direct Preference
Optimisation (DPO) (26). Sequence Likelihood Calibration with Human Feedback (SLiC) [38] and Identity-Preferent Optimisation (IPO) (3. could directly align LLMs without intensively training a
reward model as employed in RLHE.
Recent studies [34 8 have shown there exist noisy preferences (NPs) that may lead to significant degradation in alignment performance. The issue of NPs, where the label of the actually cho- sen/rejected responses in training datasets is flipped as rejected/chosen, can arise from the biases of annotators 34 and malicious noise injection [5]. As shown in Figure [ when NPs are randomly injected into the training dataset, the conventional alignment method (e.g., DPO (26) and PPO (9) will have significantly degraded alignment performance measured by the reward accuracy. Such performance degradation could result in the generation of useless and even malicious content [34]. Therefore, necessitates developing robust alignment methods that can utilize datasets with NPs
effectively align the LLMs with human preferences.
Existing robust alignment methods are proposed from the loss perspective, which adjust the alignment loss using a clean validation dataset to mitigate the issue NPs. In particular, the conservative DPO (cDPO) 21 and robust-DPO (rDPO) [8] both estimate the proportion of NPs using the clean validation data via cross-validation and then adjust the original DPO loss based on the estimated proportion of NPs. However, Mitchell (21 and Chowdhury al. {8] overlooked the essential differences between noisy and clean preferences, which is critical for mitigating the issue of NPs.
To this end, we propose Perplexity-aware Correction (PerpCorrect) for robust alignment from the data perspective by leveraging the differences between noisy and clean preferences robust alignment. PerpCorrect detects and corrects NPs based on the difference between the perplexity of the chosen response and that of the rejected counterparts (dubbed as PPLDiff) obtained by an aligned surrogate LLM using the clean validation set. If an NP is detected, PerpCorrect will correct it by flipping the label of the rejected/chosen responses as chosen/rejected, Intuitively, rejected responses which are mistakenly labelled as chosen have higher perplexity since they are less consistent with human preferences and thus have a lower probability of being generated after alignment. Therefore, higher value of PPLDiff indicates a higher probability of the preferences being noisy. In this way, PerpCorrect leverages the differences between noisy and clean preferences (CPs) identified by PPLDiff detect NPs.
To make the PPL.iff able to distinguish CPs and NPs, PerpCorrect requires an aligned surrogate LLM for calculating PPLDiff. The density of PPLDiff obtained the noisy training dataset using

---

an unaligned surrogate LLM, which can be fitted as a normal distribution centered around zero (evidenced in Figure 2a). cannot discriminate CPs and NPs. Therefore, we align a surrogate LLM using the clean validation data. The density of PPL.Dff obtained by the aligned surrogate L.M in Figure can be fitted into two distinguishable normal distributions, thus being able to differentiate CPs and NPs.
However, there still exists a large overlap between two normal distributions after aligning only on the clean validation dataset, which could result in an unsatisfactory accuracy of NP detection. To this end, we iteratively align the model using more reliable clean training data with extremely low PPLDiff (located in the green area in Figure 2c) and reliable noisy training data with extremely large PPLDiff (located in the red area in Figure 2c) sampled from noisy training datasets. Finally, the two normal distributions are significantly separated as shown in Figure 2d which indicates that PPLDiff has an enhanced discriminatory power.
Benefiting from the strong discriminatory power of PPL.iff calculated by the aligned surrogate LLM, PerpCorrect outputs denoised training dataset for robust alignment by first detecting NPs based on a PPLDiff threshold and then correcting them. The data, whose PPLDiff is below a certain threshold (i.e., the black dotted line in Figure 2d selected as the x-coordinate of the two normal distributions' intersection, are identified as NPs and thus corrected by flipping the response's 's label. Notably, our proposed PerpCorrect is compatible with various alignment methods as well as robust aligmment methods [21] [ since the metric PPLDiff is agnostic to training algorithms and only requires an arguably small number of clean validation data (-50), thus making it practical.
Comprehensive empirical results, evaluated using the Llama2-7B 32 and phi-2 (0 models on the OpenAssistant Conversations (OASSTI) [17 and Golden HH 7 datasets, validate the effectiveness of our proposed PerpCorrect method in robustifying alignment with NPs. We empirically validate that PerpCorrect consistently obtains state-of-the-art performance among various proportions of NPs. Besides, we empirically demonstrate that PerpCorrect can effectively robustify various alignment techniques and robust alignment methods, validating its compatibility.
2 Literature Review and Preliminary
In this section, we introduce the related work about L.M alignment and provide preliminaries about the noisy preferences, perplexity, as well as various alignment methods.
2.1
LLM Alignment
In the domain of aligning LLMs with human preferences, pairwise preference methods are favored due to their lower cognitive burden on evaluators. Traditional online alignment approaches [32]
24 29 involve training reward models from these preferences to provide signals in reinforcement
learning. Recent offline alignment methods like Direct Preference Optimization (DPO) 26 Sequence Likelihood Calibration (SLiC) 3X and Identify Preference Optimization (IPO) streamlined this
process by directly using preference pairs to train LLMs, thus enhancing performance and reducing computational costs. Additionally, methods like RRHF (37] align LLMs using multiple ranked preferences, Jahneman-Tvvrreee labeled as good or bad, and Rejection Sampling Optimization (RSO) [18] address DPO's limitation in sampling preference pairs from the optimal policy through rejection sampling. However, NPs, arising from the biased human feedback, can determine the alignment performance [24] (34. Robust alignment methods like conservative DPO (cDPO) (21), robust DPO (rDPO) [ have been proposed to address these issues from the loss perspective. Our approach focuses on the data perspective to address these issues of NPs and is orthogonal to these robust alignment methods.
Optimization (KTO) 13 align L.MSs using a single preference
2.2 Preliminary
Noisy preferences (NPs). NPs refer to preference data in training datasets, whose label of the actually chosen/rejected responses flipped as rejected/chosen. Let = ((((1) y y/ (0) ) be the preference dataset consisting of N N preference data points. For each preference data point (x, you 9t) E D, x is the prompt input to LLMs, yo is the chosen response, and yr is the rejected response. We let D= (x((1)
-(3 _(t)
Yu D) ]=== be the noisy preference dataset (i.c., preference dataset
3

---

Threshold
Clean
Noiey
Density
Density
PPLDiff
PPLDiff
(a) Noisy and clean preferences cannot be distin- guished by PPL.ifff
(b) The large overlap between two distributions leads flawed NP detection.
Left bound
Right bound
Density
Density
PPLDif
PPLDif
(c) Aligning the surrogate I.I.M using extra reliahle training data.
(d) Separating and correcting noisy preferences based on the threshold.
We visualized the PPLDiff under the entire PerpCorrect process using Llama2-7B on Golden HH dataset with 20% noisy preferences. We the green dotted line represent the normal distribution formed by clean data, the red dotted line represents the normal distribution formed by noisy data, and the black dotted line represents the threshold.
Figure 2:
consisting noisy preferences) and denote preference data points that are not noisy clean preferences (CPs). Following Chowdhury et al. [8], we obtain the noisy preference dataset 2 using the standard random noise model 23 with the probability E (0, 50%) to change the data point into noisy preferences, i.e.
DD (ac(), 1005(())))))) (0) (z. ut
( =
(0 1
Perplexity (PPL). PPL 15] measures the probability that the LLM generates a sentence. A lower PPL of sentence indicates that the LLM has generated this sentence with high probability. PPL is defined as the average negative log-likelihood of sequence, i.c..
PPL(s; exp(
Σ log To (88(822))), 1=1
where is sequence composed of tokens and log ( (8:8(882)) denotes the log-likelihood of the i-th token given the preceding tokens Szz calculated by an LLM Wg.
Technical details of alignment methods. There usually three phases in RLHF pipeline 34 26
(1) supervised fine-tuning (SFT); (2) reward modeling; (3) reinforcement learning (RL. optimization.
In the SFT phase, LLM is fine-tuned via supervised learning on high-quality task-related data.

---

We denote the LLM after the SFT phase as TSFT. In the reward modeling phase, the reward model introduced simulate human preferences. Given a preference dataset, reward model rw (2, y) parameterized by w, which takes prompt and response y input and outputs real number representing the reward score, can be optimized via minimizing the following loss function:
Cn(ww: D) = - (2. [log (1, the) (1, 3n))]
where a is the logistic function. the RL optimization phase, the objective function is as follows: max D. (T (r, y) 3 (log To (y/az) log [=(()/=))]
(4)
where (z)1)) represents the probability that the LLM parameterized by 0 > 0 generates the response y given the prompt v; Tref is a reference I.LM to maintain the generation ability of the aligned model, and / is hyper-parameter ensure the similarity between To(y .x) and Tref (r). We take ØSFT
as the reference LLM zef following Ouyang et al. (24).
Recently, offline alignment methods directly leverages preferences in preference datasets, bypassing the need learn a reward model in RLHE. The LLM parameters are optimized by minimizing the following loss funetion:
[TTT Tyat) = (x wDD (G( ywes M1; 0)]
(5)
where the function G changes with the alignment method. To be specific, DPO SLiC (38 uses a hinge loss, and IPO [3] uses square loss:
226 uses BCE loss,
Io (B x)
T π)) (ir x)) log To (²)
(D, Mur yr: 0) log
[ [Blog
3log
(6)
(Be x))
Gsuc thes yı; 0) max {0,1
Blog
(o
[x)
(7)
Teet (No
xx)
Terl(l x))
Te (Uw x))
To
xx)
GI, you yt; 0) 1og
3 log
(8)
Tref (Nhr
x7)
Tea 1
x7)
To mitigate the issue of NPs, cDPO (1] and rDPO adjust the DPO loss based on the estimated
proportion of NPs e' using a clean validation dataset Dval = ((xo(() you Vi M
(2) No.i consisting of
Nyal EN clean preference data points, i.e.
GADPO(I, Dur, Tt; 0) = (1 IDPO( Iw. Ti; 0) + s GDpo (a,, D1, Fw 0),
(9)
(I, Iw. T1; 0) (1 e' e')GrPo (æ. Tw Dt; 0) - E' Goroo in, Dwi 0)
(10)
2e'
Perplexity-aware Correction for Robust Alignment
This section introduces Perplexity-awand Correction (PerpCorrect) for robust alignment with NPs. In Section 3.] we introduce a novel metric called PPLDiff and then illustrate the pipeline of PerpCorrect to detect and correct NPs based on PPLDiff. In Section 3.2 we demonstrate how to adapt our proposed PerpCorrect with various alignment methods to achieve robust alignment.
3.1 Perplexity-aware Correction (PerpCorrect)
In this subsection, we introduce PerpCorrect which employs a novel metric called PPLDiff as the foundation for detecting and correcting NPs. The algorithm of PerpCorrect is described in Algorithm
PPLDiff. PPLDiff measures the difference between the PPL of chosen response and that of the rejected response. Given a preference data point (x., Twe Dt) E D sampled from the noisy training dataset DD and an LLM PPL.Dff is defined as follows:
(11) where [x; y] indicates the concatenation of the prompt r and the response Intuitively, if data point
PPLDiff(r. Th. Tr: 8) log PPL([x; g..l: 0) log PPL([x: 8)]:
a clean preference, the PPL([a; D.:]; 0) will be lower than PPL((a; ūr]; 0) because the sequence
[x: Dw] is more aligned with human values and thus has a higher probability of being generated by aligned LLMss. a result, PPL.Dfff will be lower compared to NPs, which PPI.([2; Go] ] 0) is
higher than PPL( vi); 0). This difference allows to distinguish CPs and NPs based on PPLDiff.
5

---

Aligning a surrogate LLM only using clean validation data. Here, we utilize clean validation dataset Dval to obtain an aligned surrogate LLM to make PPLDiff able to distinguish CPs and NPs. We empirically find that the PPL.Dff values of CPs and NPs calculated by an unaligned LLM in the noisy training dataset were initially indistinguishable as shown Figure 2al making impossible to differentiate the NPs from CPs. This is because an unaligned LLM lacks the necessary preferences to distinguish NPs and CPs.
Therefore, we introduce surrogate LLM тe. parameterized by 0' to replace the unaligned LLM and use it for calculating PPLDiff. We optimize the surrogate LLM me using the clean validation dataset Vaal as follows:
max B (x).eeni))Dou (Gro Uw. Ve; 0-)
(12)
After aligning the surrogate LLM, the DPL values of NPs calculated by the surrogate LLM TTy are significantly increased and those of CPs are significantly decreased, forming two distinct distributions as shown in Figure 2 This is because the aligned surrogate I.I.M is trained to generate responses that align with human preferences, enhancing its ability to distinguish between NPs and CPs based on PPL.iff.
To separate CPs and NPs in the noisy training dataset without knowing the oracle preferences, we leverage the Levenberg -arqquard (LM) algorithm to find two normal distributions that fit the density of PPLDiff calculated by the aligned surrogate LLM. Specifically, the LM algorithm returns the constants E, pi, that satisfies the following condition:
Mxle,, f, 5) = (1 - --) (2) [7, σ²) + fnoisy (z) - [,, )²),
(13)
1
cxp( - 12)
where f(zlu,, 0²) = V202T
(14)
20
Note that r is the PPLDiff value and h(z|E, T) is the superposition of these two normal distribution. We denote Selean (x) Jr, o²) as the normal distribution fitting the PPLDiff of CPs and fnoisy (1rl - Jr, o²) as the normal distribution fitting the PPLDiff of NPs since the PPLDiff of NPs is intuitively higher than that of CPs. this way, we can obtain two distinguishable normal distributions to separate NPs and CPs as shown in the green and red dotted lines of Figure 2b without knowing the oracle preferences.
Further aligning the surrogate LLM using extra reliable training data from noisy training datasets, After aligning only using the clean validation datasets, the discriminatory power of the PPI.Dfff is still far from satisfactory because of the large overlap between the two normal distributions. Therefore, we align the surrogate L.MM with more reliable training data to make the PPL.iff of CPs and that of NPs more separable. We iteratively align the surrogate LLM zg using more reliably clean training data whose PPLDiff is extremely small and reliably noisy training data whose PPLDiff is extremely large after correction by flipping the label of the response.
Specifically, at epoch E N, we select (1 1)- a |DD of the training data along with the clean validation data for further alignment where a (0, 1) is the selection ratio and [D] N is the number of data points in noisy training dataset. As shown in Lines 33
45 of Algorithm 2 the selected
reliable training dataset D/2 consists of (t 1) a (1 - ž) |D] reliably clean training data whose (1-E) percent and (t - 1) ...E- |DD| reliably noisy training
PPL values are smallest (t 1)- data after correction. Note that the reliably clean training data are the data points whose PPLDiff values are smallest (1 - 1)
(1 2) percent (located in the green area of Figure 2c and the
reliably noisy training data whose PPLDiff values are largest (1 - 1) a percent (located in the red area of Figure 2ch among all the training data points.
Detecting and correcting NPs based on PPLDiff to output denoised training dataset. Based on the PPLDiff calculated by the aligned surrogate LLM, PerpCorrect detects and corrects NPs whose PPLDiff value is lower than a certain threshold. We take the x-coordinate of the intersection of the two normal distributions as the threshold (the black dotted line in Figure 2d). As shown in
Lines 31 data points whose PPLDiff values are larger than this threshold are identified as CPs (the green area in Figure 2d), and other data points are identified as NPs requiring correction (the red area in Figure 2dh. In this way, we can obtain denoised training dataset for robust alignment.

---

Algorithm Robust Alignment via Perplexity-aware Correction (PerpCorrect)
1: Input: Noisy training dataset D, clean validation dataset Vval, and pre-trained L.MM we parame-
terized by
2: Output: Robust alignment model ie
3: // Stage Supervised fine-tuning (SFT)
4: < Supervised fine-tuned LLM we. (Details in Appendix C.3
5: // Stage II: Perplexity-aware correction using the surrogate LLM
6: D Ddenoised. denoised f Perplexity- -ware Correction (To, D, Dval) (Details in Algorithm
7: // Stage III: Alignment with denoised dataset
8: < Aligned LLM using D Deenoised and denoized (Details in Appendix C33
Further, we select an optimal denoised training dataset to further enhance the performance of robust alignment according to the intersection area of the two normal distributions. We denote the intersection area of two normal distributions as the estimated NP proportion of the denoised training dataset, i.e.,
{ inf
min ( Jetean (3/7i,
(2) jr, (2,,
(15)
where e'pc calculates the ratio of noisy data points which are not detected by PerpCorrect (i.e., the green area enclosed by the black and red lines in Figure 2dh and the clean data points which are mistakenly detected by PerpCorrect (i.e., the red area enclosed by the black and green lines in Figure 2d). In this way, EPC can efficiently calculate the NP proportion of the denoised training dataset. We take the denoised training dataset with the smallest PC among multiple iterations the optimal one for robust alignment to boost alignment performance.
3.2 Robust Alignment
Here, we introduce how to adapt PerpCorrect to robustify various alignment methods and demonstrate the algorithm of robust alignment via PerpCorrect in Algorithm
general, the pipeline of the robust alignment based on PerpCorrect contains three stages: SFr,,PerpCrrect and alignment. We will first conduct SFT, following Christiano et al. 99, to boost the performance of pre-trained LLM by boosting its skills for specific tasks. Next, we will conduct PerpCorrect to detect and correct NPs and output an optimal denoised training dataset Dienoised the smallest E'c in Eq. 15 Finally, we obtain aligned LLM from the SFT model using the denoised training dataset Decoised via alignment (i.e., Line g in Algorithm[I] [.]
Because our proposed PerpCorrect is agnostic alignment methods and model structures, PerpCor- rect is applicable to robustify both online alignment methods such as RLHF (PPO) g and offline alignment methods including DPO (26). SLiC [38). and IPO (3) Besides, is compatible with existing loss-oriented robust alignment methods, such as cDPO (2I] and rDPO (8), based the estimated proportion of NPs. Note that cDPO and rDPO require conducting compu- tationally expensive cross- cross-valide validdaiio to tune the estimated proportion NPs. We can efficiently estimate the proportion of NPs by utilizing the fitted normal distributions during PerpCorrect, i.e., e'pc in Eq. 15 Therefore, we can combine PerpCorrect with a wide range of existing alignment methods achieve robust alignment with NPs.
proposed PerpCorrect
4 Experiments
In this section, we demonstrate that our proposed PerpCorrect achieves state-of-the-ae alignment performance under different proportion of NPs and have good compatibility with other alignment methods. In Section 4.1 PerpCorrect combined with DPO (26] achieves sfate-of-the-an alignment performance than existing baselines (Section 1. including DPO 226], cDPO (21), and rDPO (8] In Section 4.2 we further analyze the impact of the number of validation data and verified the compatibility of PerpCorrect with online and offline alignment methods and robust alignment methods.
The training details and compute resources reported in Appendix C.I
7

---

Table 1: Average reward accuracy of DPO ries alignment methods using Llama2-7B on the Golden HH dataset. The standard deviation of reward accuracy is reported in Table
Table 2: Average reward aceuracy of PPO se-
ries alignment methods using Llama2-7B on the Golden HH dataset. The standard deviation of reward accuracy is reported in Table
Proportion of noisy preferences (%)
Proportion of noisy preferences (%)
Method
Method
10
20
30
40
10
20
30
40
Vanilla DPO
92.53%
82.62%
68.50%
53.15%
Vanilla
96.6%%
92.71%
90.21%
86.29%
cDPO
96.04%
90.85%
83.23%
65. 65.60%
cPPO
96. 18%
93.63%
90.62%
88.02%
:DPO
96.65%
95.22%
93.90%
90.45%
:PPO
95.92%
93.73%
92.05%
90.62%
PepCoonect-DPe
97.51%
96.24%
95.53%
94.92%
PerpCorrect-Pre
96.38%
94.04%
93.99%
93.17%
Table Average reward accuracy of DPO series alignment methods using phi-2 on the Golden HH dataset. The standard deviation of reward
Table 4: Average reward accuracy of DPO series alignment methods using phi-2 on the OASSTI dataset. The standard deviation of reward accu- racy is reported in Table I1
accuracy is reported in Table TO
Method
Proportion noisy preférences (%)
Method
Proportion noisy preferences (%)
10
20
30
40
10
20
30
40
Vanilla
93.19%.
85.57%
73.07%
54.98%
Vanilla DPO
66.94%
62.61%
58.44%
52.42%
cDPO
97.21%
92.63%
81.05%
66.72%
cDPO
67.30%
61.44%
54.87%
49.21
:DPO
96.49%
95.73%
93.34%
84.55%
rDPO
63.95%
59.47%
56.45%
45.20%
PerpCorret-DOO
98.17%
97.05%
97.66
96.39%
PerpCoreet-DPO
71.34%
69.04%
68.27%
68.49%
Datasets. We utilize two preference datasets, namely Assistatt Conversations (OASST1) 17
and Golden HH (7]. The processed OASSTI dataset comprises 17,939 training samples and 951
testing samples and the processed Golden HH dataset consists of 12,066 training samples and 634
testing samples. The description and processing details of these datasets are provided in Appendix C.2
Models. Our evaluation leverages two distinct series of operesourred LLMs with different parameter sizes: Llama2-7B [32 and phi-2 [20]. We acquire the checkpoints from their official repositories on Hugging Face. The LLMs used for PerpCorrect and those for robust alignment share the same model structure and initialization.
Baselines. We adopt vanilla DPO 226 and two robust aligmment methods, cDPO 21 and rDPO 88) as baselines. For their detailed implementation, we utilize and adapt the transformers and TRL libraries provided by the Hugging Face community.
Metrics. In accordance with Chowdhury et al. we employ the winning rate of policy generations against the selected preferences on the test dataset our primary metric. This metric applies to vanilla DPO 26) cDPO [1], rDPO [8], as well as other offline alignment methods including SLiC (38] and IPO 3. Additionally, we utilize the winning rate of the reward model score for the chosen preferences on the test dataset as our metric for vanilla PPO 24], cPPO 21 34 two metrics are collectively called reward accuracy.
and rPPO 88. These
4.1 PerpCorrect Achieves the State-of-the-Art Robust Alignment Performance
The empirical results demonstrate that our method, PerpCorrect, achieves state-of-the-arr robust alignment performance, surpassing existing basclines such as vanilla DPO (26), cDPO [21]. and rDPO (8). This is evident across various proportions of noisy preferences using different datasets and LLMs.
Comparison using different LLMs. Tables and 3 demonstrate the average reward accuracy of the DPO series alignment methods on the Golden HH (7) dataset using Llama2-7B 32 and phi-2 (20]. At proportion of the NPs & 40%. PerpCorrect increases the reward accuracy by 41.77% (from 53, 15% to 94.92%) using Llama2-7B and by 41.41% (from 54.98% to 96.39%) using
phi-2, The empirical result validates that our proposed PerpCorrect can be used different sizes of LLMs and achieve better alignment performance than baselines.
Comparison on different datasets. Tables 3 and demonstrate the average reward accuracy the DPO series alignment methods the Golden HH and OASSTI [17] datasets using phi-2 [20).

---

Table 5: Impact the number of clean validation data evaluated on the Golden HH dataset using
Llama2-7B with proportion of NPs 40%.
Number Reward accuracy
10
30
40
50
100
200
81.40%
88.26 26%
94.21%
94.21% 21%
95.43%
95.43%
96.04%
Table 6: Average reward accuracy and improvements of the offline and robust alignment methods, as well as those combined with PerpCorrect, using Llama2-BB on the Golden HH dataset. The standard deviation of reward accuracy and improvements is reported in Table 12
Proportion of noisy preferences (%)
Method DPO 92.53%
10
20
30
40
82.62%
68.50%
53.15%
PerpCorrect-DPe
97.51%
96.24%
95.53%
94.92%
A
+4.98%
+13.62%
+27.03%
+41.77%
SLiC
96.70%
87.75%
76.17%
58.59%
PerpCorrect-SLiC
96.95%
95.02%
95.38%
94.61%
A
+0.25%
+7.27%
+19.21%
+36.02%
IPO
98.07%
92.73%
79.17%
61.64%
PerpCorrect-IPO
98.73%
97.66%
97.82%
97.56%
A
+0.66%
+4.93%
+18.65%
+35.92%
cDPO
96.04%
90.85%
83.23%
65.60%
PerpCorrect-DDPO
98.12%
97.31%
94.97%
88.36%
A
+2.08%
+6.46%
+11.74%
+22.76%
rDРО 96.65%
95.22%
93.90%
90.45%
PerpCorret-rDDOO A
95.99%
95.02%
94.77%
95.73%
-0.66%
-0.20%
+0.87%
+5.28%
The empirical results reveal significant discrepancy in average reward accuracy between the more complex OASSTI dataset and the Golden HH dataset. The performance of other robust alignment methods is found to be unsatisfactory on the OASSTI dataset, often not surpassing the vanilla DPO. In contrast, our method PerpCorrect consistently maintains strong alignment performance across varying proportions of noisy preferences. In general, our method PerpCorrect can achieve better alignment performance than baselines across different datasets.
4.2 Ablation Study
the impact of the number of clean validation data points. We conducted experiments the Golden HH dataset using Llama2- 7B with proportion of NPs validation data points increases, the performance of method, PerpCorrect, also improves. However, when the number is too large, the improvement in performance is not obvious, and the cost of manual annotation significantly increases.
Impact of the number of clean validation data.
40%. The empirical results indicate that as the number of clean
Compatibility with online alignment method RLHF (PPO). We adopt vanilla PPO (24). cPPO 21 34 and rPPO [ baselines. Table shows the alignment performance of PPO series alignment methods on the Golden HH [7] dataset using Llama2-78 7B. Although vanilla PPO has good performance when the proportion of NPs is low, still declines significantly when the proportion is high. PerpCorrect maintains desirable alignment performances when the proportion of NPs high. Our empirical results show that PerpCorrect has desirable compatibility with online alignment method RLHF (PPO).
Compatibility with various offline alignmet methods. Table 6 presents the average reward accuracy and improvements of original offline alignment methods compared to those combined with PerpCorrect. Our experiments, conducted on the Golden HH dataset using Llama2-7B, reveal that the reward accuracy of SLiC (38 and IPO 3 both significantly decrease as the proportion of NPs increases, similar to vanilla DPO 26 However, our method PerpCorrect enhances their alignment performance across different proportions of NPs. Notably, IPO combined with PerpCorrect achieves

---

the best alignment performance. We conjecture the main reason is that the proportion of NPs in the denoised dataset very low and IPO performs better than other methods under a low proportion of NPs. These empirical results demonstrate that our method has good compatibility with various offline alignment methods.
Compatibility with robust alignment methods. 6 the average reward accuracy and improvements of robust alignment methods compared those combined with PerpCorrect. Our method, PerpCorrect, can significantly enhance the performance of cDPO [21]. and provide modest improvement for rDPO [ under almost all proportion of NPs. The empirical results show that our method has good compatibility with robust alignment methods.
5 Conclusions
This paper proposes method called perplexity-aware correction (PerpCorrect), as an effective approach for robust alignment with noisy preferences (NPs), PerpCorrect utilizes a surrogate LLM to calculate novel metric, PPLDiff, and further detects and corrects NPs from clean preferences (CPs) based on it. PerpCorrect consists three steps: (1) PerpCorrect aligns a surrogate LLM using the clean validation dataset, enabling PPLDiff to distinguish between CPs and NPs. (2) PerpCorrect enhances the discrimination power of PPL by aligning the surrogate LLM with more reliable training data. (3) PerpCorrect detects and corrects NPs from CPs based on calculated threshold and obtains a denoised training dataset. The paper further proposes a robust alignment pipeline, consisting of three stages SFT, PerpCorrect, and alignment, achieve robust alignment with NPs. The experimental results validate that PerpCorrect achieves state-of-the-ee art alignment performance and has good compatibility with other online, offline, and robust alignment methods. Therefore, PerpCorrect can be an effective method to mitigate the impact of NPs and can be used for robust alignment. Future research directions include: (1) Improving the time efficiency of PerpCorrect and (2) Reducing the amount of clean validation data required to achieve the same alignment performance.
Limitations
We discuss some limitations of this work to stimulate further research in this direction. Our limitations mainly stem from two aspects: time efficiency issues caused by multiple calculations of PPLDiff and repeated training of a surrogate LLM, and the need for a validation dataset.
Time efficiency. Iteratively calculating the PPLDiff value for each data point and aligning surrogate LLM is time-consuning. Selecting reliably training data and denoising the training dataset requires that the PPLDiff value be calculated for each data point during each epoch, which may cause unnecessary calculations for CPs and NPs that can already be clearly distinguished. Besides, aligning surrogate LLM with same size the LLM for alignment multiple times time-consuming. The detailed discussion is in the Appendix B
Validation dataset. PerpCorrect requires validation dataset for aligning a surrogate LLM. How- ever, manually annotating validation dataset is complex and labor-intensive in practice. As shown in T there a significant disparity alignment performance when comparing the use of 10 clean samples to 50 clean samples. Exploring how to use fewer clean samples or even no clean samples to achieve the same or better performance is problem worth further investigation.
Acknowledgements
This research supported by the National Research Foundation, Singapore under its Strategic Capability Research Centres Funding Initiative, the funding BAS/1/1689-01-00 URF/1/4663-01-00 REI/1/5232-01-01, REI/1/5332-01-001 and URF/1/5508-01-01 from KAUST, and funding from KAUST- - Center of Excellence for Generative AI, under award number 5940. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
10

---

References
[1] 01. AI, i, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang. Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by 0l.ai, 2024. URL
https://arxiv. org/ http//ab/2440..000 04652
[2] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra
Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malar- tic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series of open language models, 2023. URL httpp://aazae org/ (rrg/abs/23111 16867
[3] Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm understand learning from human preferences, 2023.
[4] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022. URL https: /arxiv.xvvv/// org/ abs////////// /tbb//224444 05862
[5] Tim Baumgärtner, Yang Gao, Dana Alon, and Donald Metzler. Best-of-venom: Attacking rlhf by injecting poisoned preference data, 2024.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[7] Tianchi Cai, Xierui Song, Jiyan Jiang, Fei Teng. Jinjie Gu, and Guannan Zhang. Ulma: Unified language model alignment with human demonstration and point-wise preference, 2024.
[8] Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust DPO: Aligning language models with noisy feedback. In Forty-first International Conference on Machine Learning, 2024. URL https:///ppperevieet wwt./ooooomdddddppooooooo
[9] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[10] Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations, 2024. URL https::/
jet//forma/ddTTyyFFOOKX
[11] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoye. Qlora: Efficient finetuning of quantized 1lms. Advances in Neural Information Processing Systems, 36, 2024.
[12] Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings the 39th International Conference on Ma- chine Learning, volume 162 of Proceedings of Machine Learning Research, pages 9988-6008 www.mmmerss.....mmmmmmmmmmmmm...
PMLR, 17-23 Jul 2022. URL httttt:::/ppoceeeiiiiii mlr press / /ttp///aaaaaaaaa222a.
html
[13] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela, Kto: Model alignment prospect theoretic optimization, 2024.

---

[14] Team GLM, Aohan Zeng, Bin Xu, Bowen Wang. Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang. Jiale Cheng. Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang. Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang. Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang. Chatglm: A family of large language models from glm- 130b glm-4 all tools, 2024.
[15] Fred Jelinek, Robert L. Mercer, Lalit Bahl, and James K Baker. Perplexity a measure the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1): S63-S63. S63, 1977.
[16] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL org/ /bs/ 06825
[17] Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen, Oliver Stanley, Richard Nagyfi, et al. Openassistant cunversations-demorrrtaii large language model alignment. Advances in Neural Information Processing Systems, 36, 2024.
[18] Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. In The Twelfth International Conference on Learning Representations, 2024. URL BRLhhttps:/// htttps://openrviiiee
net/forum?id-xbii QDwwwr
[19] Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. Trustworthy llms: a survey and guideline for evaluating large language models' alignment, 2024.
[20] Microsoft.
The surprising power of small language mod- URL https: //ww:///wwwww http://wwwccooffftto/////// com/ en us/research/blog/ phi-2 2-e surprising-pooeeee powee-owerrrooo swwsssaalllaaaaaaa -aagammaggggggemmooooooooo models
Phi-2:
els. 2023.
[21] Eric Mitchell. A note on dpo with noisy preferences and relationship to ipo, 2023. URL
https://erricmitcchelll. aww..po.oppp.o pddf
[22]
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John with human feedback, 2022. URL
Schulman. Webgpt: Browscr-assisted
https://arxxt http:///axxi..../abbbbbbae wwtp:oor//ab///i/e/ 2112. 09332
[23] Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances neural information processing systems, 26, 2013.
[24] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[25] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training, 2018.
[26) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Thirth-seventh Conference on Neural Information Processing Systems, 2023. URL.
12

---

[27] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
[28] Damien Sileo. tasksource: Structured dataset preprocessing annotations for frictionless extreme
multi-task learning and evaluation. arXiv preprint arXiv:2301.05945, 2023. URL https
//arxiv.org/ wtw.oo//g/...........oo
[29] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3000 2020.
[30] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-follooing Ilama model. https://8 thub. www/tatsu- taatttatsaaaaa 1ab/stann/tttttteee ord alpaca 2023.
[31]
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev. Matt Hoffman. Shantanu Thakoor, lean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchi- son, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov. David Weinberger, Dimple Vijaykumar, Dominika Rogozińska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Pluciissk, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh I. Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Mau Davidow, Matt Miller. Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan. Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauban, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona COmanesou, Ramona Merhej. Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Arool, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open language models practical size, 2024. URL https://arras http///xxivv.rrrr abs 2008./24000000
[32] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nic, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta,
13

---

Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Ilyann Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnie, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[33] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale. Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams. Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarow, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL lattps:///axxtt http//orx.ivo//////////// abs/ 2307 09288
[34] Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, Songyang Gao, Nuo Xu, Yuhao Zhou, Xiaoran Fan, Zhiheng Xi, Jun Zhao, Xiao Wang. Tao Ji, Hang Yan, Lixing Shen, Zhan Chen, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang. Zuxuan Wu, and Yu-Gang Jiang. Secrets of rlhf in large language models part ii: Reward modeling, 2024.
[35]
An Yang, Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge. Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL actps://arxiv. org/ abs/ 2000/ 10671
[36] Jingwei Yi, Rui Ye, Qisi Chen, Bin Benjamin Zhu, Siheng Chen, Defu Lian, Guangzhong Sun, Xing Xie, and Fangzhao Wu. Open-source can be dangerous: On the vulnerability of value alignment in open-source LLMs, 2024. URL NIouOOCDex
[37] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprins arXiv:230..05302, 2023.
[38] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback, 2023.
[39] Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, Limao Xiong. Lu Chen, Zhiheng Xi. Nuo Xu, Wenbin Lai, Minghao Zhu, Cheng Chang, Zhangyue Yin, Rongxiang Weng, Wensen Cheng, Haoran Huang, Tianxiang Sun, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, and Xuanjing Huang. Secrets of rlhf in large language models part i: Ppo. 2023.
[40] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 190.00999, 2019.
14

---

A Broader Impacts
Our proposed PerpCorrect and robust alignment pipeline offers a solution for achieving state-of-the- art performance in robust alignment under noisy preferences. PerpCorrect designed to effectively reduce malicious noise in the dataset and mitigate biases introduced by human annotators, ensuring that the trained language model (LLM) accurately aligned with true human preferences.
Moreover, we recognize potential risk: if malicious users exploit our method for reverse training, they might compromise the security mechanisms of existing open-source LLMs. Existing research has demonstrated the possibility of reverse training [36].
B Time Efficiency Analysis
The additional computational overhead is primarily attributed to PerpCorrect (Section 3.1. Table 7 presents both theoretical and empirical runtime comparisons, where X represents the theoretical time required for Alignment (Section 3.2) or other baselines.
In theory, during the PerpCorrect process, we need calculate PPLDiff and train the surrogate model in each epoch. The computation time introduced by PerpCorrect is approximately 3 Alignment or other baselines.
I that of the
The calculation of PPLDiff in each epoch requires only oo the time needed for robust alignment. The primary computational load in robust alignment arises from the complexity of forwarding and back paok-propag,tton, while the complexities of gradient updates and parameter updates are relatively low. Additionally, back-propagation takes twice long as forwarding. In addition, the calculation of PPLDiff only requires forwarding.
For surrogate model training, PerpCorrect utilized data points that represented i x a of the total dataset during epoch t. Since both i and a are small, the time required for surrogate model training can be approximately ignored.
In practice, Our entire robust alignment pipeline (~224 hours) takes only twice as long as the baseline (~112 hours). We set T = 5 and a = 2. and used the AdamW optimizer. The practical efficiency of the PerpCorrect is due to the use of fp32 precision by the AdamW optimizer, which increases the GPU's calculation time during the robust alignment process.
Table 7: Comparison of theoretical and practical running times for PerpCorrect and baselines.
Stage
Thcoretical Running Time Practical Running Time
PerpCorrect
x X
12 hours
Alignment (or baselines)
X
12 hours
Total
(1 + T X
24 hours
C Implementation details
C.1 Training details and compute resources.
We utilized the Qlora method [1] for fine-tuning the LLMs, executed on RTX 4090 GPUs with 24 GB of memory. Hyperparameters were as follows:
32, Iora_ dropout 0.1,
and lora alpha 16. For SFT, we use the alpaca dataset [30 and set learning_rate rate 2c 4 and batch size 20. For our PerpCorrect stage II, we set 8 0.1. learning_rate rate batch size = 4, T = 5, and = 0.02. For our PerpCorrect stage III and all other alignment methods, = 3e - and batch_size 20. Other details not mentioned, we follow the default setting in TRL library. Each experiment, involving specific method and proportion of NPs, could be completed using a single RTX 4090 GPU within 24 hours on the Golden HH dataset and within 72 hours the OASSTI dataset.
we set 3 = 0. learning_ren
15

---

C.2 Description and Processing Details of the Datasets
OpenAssistant Conversations Dataset (OASST1). The original OASSTI dataset is an assistant-style conversation corpus generated and annotated by humans. It consists of over 10,000 fully annotated conversations in 35 different languages. Sileo [28] converted these conversations into a preference dataset comprising 17, 77966 training samples and 952 testing samples. After filtering out conversations with one or fewer letters, we obtained preference dataset with 17,939 training samples and 951 testing samples.
Golden HH. The Golden HH dataset 7 is variant of the Anthropic Helpful and Harmless (HH) dataset 4 It originally contains 42,537 training samples and 2,312 testing samples as part of preference dataset. Each sample has two keys: one representing the prompt x and the chosen response Yor, and the other representing the prompt x and the rejected response yr- We first converted the dataset into a triple form: prompt z, chosen response Yu. and rejected response yt, retaining only one-turn conversation data. After filtering out samples with one or fewer letters, obtained preference dataset with 12,066 training samples and 654 testing samples.
C.3 Detailed Robust Alignment via Perplexity-aware Correction
Supervised Fine-Tuniing (SFT). The objective of Supervised Fine-Tuning (SFT) is to enhance the performance of a pre-trained large language model (LLM) by refining its abilities for specific tasks. As demonstrated by prior work 99 25 24 this can be achieved by utilizing supervised fine-tuning with a specialized dataset tailored to the target task. The SFT dataset is annotated with labels, providing examples that directly relevant the task. Specifically, for each data point (x, y) in the SFT dataset, æ represents the prompt given to the LLM, and represents the expected response that the model should generate based on the prompt The process involves fine-tuning the LLM by maximizing the log-like ligeiikl the correct responses y given the prompts æ. Through this method, the model learns produce more accurate and task outputs, thereby significantly improving its performance on the given task.
Perplexity-aware Correction (PerpCorrect). We demonstrate the entire PerpCorrect algorithm in
Algorithm 2
Alignment. We can achieve alignment using the denoised training dataset D denoised with an estimated proportion of NPs denoised. For offline alignment methods such as DPO, SLiC, and IPO, we can directly optimize the LLM using the denoised training dataset DD, Denoised based on the loss functions defined Eqs. 6 For loss-based robust alignment methods, including cDPO and rDPO, we set e' 5 demoisse and then optimize the LLM using the denoised training dataset Deeenissd according to the loss functions mentioned in Eqs. and T0 For the online alignment method RL.FF (PPO), we first train a reward model using the denoised training dataset DD Denoised based on the loss function described in Eq. 3 Subsequently, we further optimize the LLM using PPO according to the objective function detailed in Eq.
D Extended Experimental Results
D.1 Experiment Statistical Significance
Tables 8
demonstrates the standard deviation of the reward accuracy reported in Tables 4 and
D.2 Average PPLDiff Values of Data from Different Datasets Calculated by Unaligned LLMs
We randomly selected 10,000 data points from each dataset and calculated PPLDiff using different LLMss. The datasets and LLMSs are downloaded from the Huggingface website. The average PPLDiff values are reported in the Table 13
16

---

Table 8: Standard deviation of reward accuracy for DPO series alignment methods using Llama2- 7B on the Golden HH dataset. The average ward accuracy is reported Table T
Table 9: Standard deviation of reward accuracy for PPO series alignment methods using Llama2- 7B on the Golden HH dataset. The average re- ward accuracy is reported in T
Proportion of noisy preferences (%)
Proportion of noisy preferences (%)
Method
Method
10
20
30
40
10
20
30
40
vanilla DPO
0.81%
0.40%
2.52%
2.60%
vanilla PPO
0.15%
1.30%
4.05%
0.77%
1.15%
0.81%
1.64%
cPPO
0.15%
1.53%
4.61%
5.89%
:DPO
0.26%
1.53%
0.95%
1.92%
rPPO
0.62%
1.38%
1.55%
5.29%
PerCorrect-DPO
0.63%
0.87%
1.73%
0.63%
PerpCorrect-PPO
0.35%
1.15%
1.34%
1.57%
Table 10: Standard deviation of reward accuracy for DPO series alignment methods using phi-2 on the Golden HH dataset. The average reward accuracy is reported in Table 3
Table 11: Standard deviation of reward accuracy for DPO series alignment methods using phi-2 the OASST1 dataset. The average reward accuracy is reported in Table
Proportion of noisy preferences (%)
Proportion noisy preferences
Method
Method
10
20
30
40
10
20
30
40
vamilla DPO
0.92%
0.633%
1.28%
0.98%
vanilla PPO
0.70%
1.30%
2.46%
1.64%
cDPO
0.35%
0.54%
0.49%
1.84%
cPPO
0.46%
0.87%
0.56%
0.91%
rDPO
0.30%
0.15%
1.67%
7.82%
rPPO
0.96%
2.29%
1.24%
PerpCoreet-DPO
0.81%
1.27%
0.63%
1.07%
PerpCorrect-POO
0.35%
1.89%
1.35%
1.06%
Table 12: Standard deviation of reward accuracy and improvements of the offline and robust alignment methods, as well as those combined with PerpCorrect, using Llama2-7B on the Golden HH dataset. The average reward accuracy is reported in Table 6
Proportion of noisy preferences (%)
Method DPO 0.81%
10
20
30
40
0.40%
1.28%
0.98%
PerpCorrect-DPO
0.63%
0.87%
1.73%
0.63%
A
0.98%
0.89%
0.97%
1.98%
SLiC
1.91%
1.33%
6.77%
8.02%
PerpCorrect-SI
1.76%
1.45%
1.16%
1.50%
A
0.63%
0.38%
5.62%
6.52%
IPO
0.84%
0.63%
3.48%
0.32%
PerpCorrect-IPO
0.32%
1.68%
0.23%
1.25%
0.98%
2.29%
3.26%
1.10%
cDPO
1.15%
0.81%
1.76%
1.64%
PerpCorrect-DDPO
0.69%
0.75%
0.81%
0.72%
A
0.47%
1.31%
1.85%
2.36%
rDРO
0.26%
1.53%
0.95%
1.92%
0.63%
0.23%
1.84%
1.19%
A
0.84%
1.48%
2.19%
2.31%
17

---

Table 13: Average PPLDiff values of randomly selected data points across datasets calculated by different LLMs. "Avg." refers to the average PPLDiff value over all the datasets.
Dataset
Model
Avg.
HH-RLHF
SafeRLHF [10
SHP [2
WebGPT 22
Qwen2-1.5B
35
-0.140
-0.002
0.040
-0.018
-0.030
Qwen2-1.5B-Instreee
35
-0.149
-0.009
0.046
-0.021
-0.033
Yi-1.5-6B [
-0.158
-0.103
0.105
-0,036
-0.088
Yi-1.5-6B- Chat
-0.159
-0.054
0,069
-0,040
-0.46
gemma-2-2b [1]
-0.140
+0.051
0.001
-0.024
00.53
gemma-2-2b-it
-0.163
+0.053
0.063
+0.028
0..45
falcon-7b 2
-0.113
+0.001
0.037
+0.016
-0.23
falcon-7b-instrue
2
-0.121
0.021
0.039
-0.017
-0.019
Mistral-7B-v0.3 [16
-0.133
-0.45
0,048
-0.026
-0,039
Mistral-7B--nnstruct--..
16
-0.201
-0.058
0.065
-0.038
-0,058
glm-4-9b
14
-0.134
-0.019
0.045
-0.027
-0,034
31m-4-9b-chat
1m
[4
-0.135
-0.019
0.049
+0.028
-0.033
Llama-2-77---
33
-0.133
-0.552
0,051
-0.028
-0,040
Llama-2-766
2-m-ch---
33
-0.139
-0.011
0.063
-0.032
-0.037
18

---

Algorithm 2 Perplexity- aware Correction (PerpCorrect)
1: Input: Noisy training dataset D, clean validation dataset Val. LLM parameterized by 0
2: Output: Denoised training dataset D decuiised and estimated proportion of NPs denoised
3: Tor f- Te, D/2 +- 0, E denoised f- 1, D denoised + D,
4: for epoch f 0, .... 7 do
5: // Aligning the surrogate LLM
6: Tee < Alignment (me Di U vaa)
7: // Calculatiing the PPLDiff values for each data point
8:
Ω
2+0 0
9:
for (ī, Nw: 11) D do
log PPL( + Twi 0') log PPL(x + 0i; 0")
+ ( Yur Bt. =))
end for
// Fitting PPLDiff density of noisy training dataset
Fitted parameters using Wyvenberg-Marquare algorithm with 15: // Estimating NPs proportion of the denoised training dataset
14: E, a. 5 <
16:
E'PC + Estimated proportion of NPs using the Eq 15 based on E, (3, ō
17:
// Keeping denoised training dataset with the smallest denoised
18:
if e'pc < <Edenoised then
19:
dennised f SPC
20:
// Calculating the Threshold 7
21:
x-coordinate of the intersection of the two normal distributions(E, ō)
22:
I/ Distinguishing CPs and NPs based on the threshold and correcting NPs
23:
DCP: + 0, DNPs f- (
24:
for (T, Dwr Wi, z) E f do
25:
if > then
26:
Dcr. Ders U {(ē Iw: Dn)
27:
else
28:
DNP: DNPs ((ū, ūt Tw))
29:
end if
end for
D Donised + Deps D
32:
end if
33:
Dclean + 0. Dooisy +- 0
34:
I/ Calculating the left bound n. and the right bound Tr
35:
ni T 1)- a (1 - E) -th smallest PPLDiff value in Ω
36:
T, + (t- a E- 1D largest PPLDiff value in
37:
// Finding extra reliable training data
38:
for (ā, Tno in, z) E Ω do
39:
if 2 < then
40:
Dclean + Dclean U (θt Tw, Ot)
41:
end if
42:
if = Tr then
43:
DNois +- DNoiey U ((T, Dr. Tw))
44:
end if
45:
end for
46:
D'11 Delean U DNoisy
end for
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Our introduction covers our contributions, main methods and experimental results.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: We discussed the efficiency issues and data volume requirements of our method PerpCorrect in the Conclusions section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
20

---

Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We show all the experiment detail in the Experiments section and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide open access to our code using Github.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //iiss.cc/pill wttpcccpppppppppeeeeeeeeeeeeeeee httpa//gguddeee/eeeeabiiissssss /Cc////oddiiisssseeeeeee ssiiiiiiccyyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We show the entire experimental details the Experiments section and Appendix and provide open access to the code using Github.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report the standard deviation in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed sufficient information on the computer resources in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httppp////earrrp..ccc/o//l///aaa
Answer: [Yes]
Justification:
We follow the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential impacts in Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our robust alignment method does not have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The models and datasets the used are open-sourced, and follow their license and terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our training code are open-source on GitHub.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "ZNH0RgDppD", "reviewer": "Reviewer_JSSg", "summary": "The paper introduces Perplexity-aware Correction (PerpCorrect), a method for robust alignment of large language models (LLMs) with noisy preferences (NPs). PerpCorrect detects and corrects NPs by analyzing the perplexity difference (PPLDiff) between chosen and rejected responses. The approach involves aligning a surrogate LLM with clean validation data, iteratively refining with reliable training data, and ultimately producing a denoised training dataset. Experiments demonstrate that PerpCorrect significantly improves alignment performance and is compatible with various alignment techniques.", "questions": "See Weaknesses.", "limitations": "Limitations have been discussed in Conclusion section.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The problem of aligning LLMs with human preferences while effectively handling noisy data is a significant challenge in the field of AI, and this paper provides a valuable contribution towards solving it.\n\n2. This paper introduces PPLDiff, a novel metric for distinguishing between clean and noisy preferences, enhancing the precision of corrections. The proposed method effectively handles noisy preferences (NPs), improving the reliability of LLM alignment.\n\n3. PerpCorrect obtains significant improvements compared to previous methods, requiring only a modest amount of clean validation data, making it practical for real-world applications.", "weaknesses": "1. Implementing PerpCorrect involves additional steps of calculating and analyzing perplexity differences (PPLDiff), which can add complexity to the alignment process. Can the authors discuss the complexity and additional computation time of the method?\n\n2. The method requires a clean validation dataset to align the surrogate LLM initially. The need for manually annotated validation data can be labor-intensive and may not always be feasible in large-scale applications.\n\n3. The method assumes that the perplexity of clean and noisy preferences will differ consistently, which may not always hold true across all datasets and model configurations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your comments! Please find our replies below.\n\n``[Reply to W1] Please refer to our general author rebuttal [R1].`` \n\n``[Reply to W2] Please refer to our general author rebuttal [R2].``\n\n``[Reply to W3] We provide a discussion of the assumption as follows.``\n\n**Technically**,  this assumption is inspired by the loss function of DPO (As shown in Eqs. 5-6). According to Rafailov et al. [1], the gradient can be written as: \n$$\n\\nabla\\_\\theta\\mathcal{L}\\_{\\mathrm{DPO}}(\\pi\\_\\theta;\\pi\\_{\\mathrm{ref}})=-\\beta\\mathbb{E}\\_{(x,y\\_w,y\\_l)\\sim\\mathcal{D}}\\bigg[\\sigma(\\beta\\log\\frac{\\pi\\_\\theta(y\\_l|x)}{\\pi_\\text{ref}(y\\_l|x)}-\\beta\\log\\frac{\\pi\\_\\theta(y\\_w|x)}{\\pi\\_\\text{ref}(y\\_w|x)})\\bigg[\\nabla_\\theta\\log\\pi(y\\_w| x)-\\nabla_\\theta\\log\\pi(y\\_l| x)\\bigg]\\bigg],\n$$\nwhere $\\nabla_\\theta\\log\\pi(y_w| x)$ increases likelihood of $y_w$ and $\\nabla_\\theta\\log\\pi(y_l| x)$ decreases likelihood of $y_l$. Furthermore, for CPs where $(x,\\tilde{y_w},\\tilde{y_l}) = (x,y_w,y_l)$, the gradient directs the decrease of $\\mathrm{PPL}([x;\\tilde{y}\\_{w}];\\theta)$ and the increase of $\\mathrm{PPL}([x;\\tilde{y}\\_{l}];\\theta)$. This ultimately leads to the reduction of $\\mathrm{PPLDiff}(x,\\tilde{y}\\_w,\\tilde{y}\\_l;\\theta)$. Conversely, for NPs, the gradient leads to an opposite effect. As a result, the PPLDiff of CPs and NPs will consistently differ.\n\n**Empirically**, the results of our method, PerpCorrect, using different LLMs (including phi-2 and Llama-2) on various datasets (such as the Golden HH and OASST1 datasets) have verified our assumption.\n\n[1] Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeruIPS 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eBAG6XTNkn", "reviewer": "Reviewer_JfpF", "summary": "The paper proposed a novel method to mitigate the preference noise in alignment. The authors first provide insights into how the PPLDiff can recognize the noisy preferences and then use the PPLDiff to select and correct noise preferences. Extensive experiments demonstrate that the method can significantly improve the robustness under high noise ratios.", "questions": "* In Line 50, \"However, Mitchell [15] and Chowdhury et al. [6] overlooked the essential, differences between noisy and clean preferences, which is critical for mitigating the issue of NPs.\" What is the overlooked difference here? The logic is not clear.", "limitations": "No obvious limitations.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper proposed a novel detect and correct paradigm for noise-robust alignment. \n* Novel insights were revealed for the failure with the noise preferences.", "weaknesses": "* Some writing is not clear.\n   - In Line 50, \"However, Mitchell [15] and Chowdhury et al. [6] overlooked the essential, differences between noisy and clean preferences, which is critical for mitigating the issue of NPs.\" What is the overlooked difference here? The logic is not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your comments! Please find our replies below.\n\n``[Reply to Q1] The overlooked difference is that NPs have incorrect labels, which can be identified using PPLDiff.``\n\nBoth cDPO (ref to Eq. 9) and rDPO (ref to Eq. 10) use a universal loss to treat CPs and NPs. They overlooked the difference between CPs and NPs, that is, NPs have incorrect labels, while CPs have correct labels. This results in the underutilization of CPs and the negative impact of NPs.\n\nIn contrast, our approach, PerpCorrect, addresses this issue more effectively by using PPLDiff to identify and correct NPs from a data perspective rather than a loss perspective.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9NoDytKY4Z", "reviewer": "Reviewer_LN38", "summary": "This paper presents a novel method called PerpCorrect for robust alignment in large language models, particularly in the presence of noisy preferences in training data. PerpCorrect addresses noisy preferences by evaluating the perplexity difference (PPLDiff) between selected and rejected responses. By leveraging PPLDiff to identify and rectify these noisy preferences, PerpCorrect generates a denoised training dataset. Experimental results demonstrate that PerpCorrect achieves state-of-the-art alignment performance with only a small amount of validation data.", "questions": "1. Will the PPLDiff calculate on the new dataset, using models similar to those used with other data (e.g., Llama-2-7b-chat), result in a normal distribution with a mean of 0?\n\n2. What are the reasons that make PrepCorrect-IPO perform better than PrepCorrect-cDPO and PrepCorrect-rDPO? Please analyze this phenomenon.", "limitations": "As discussed in the paper.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well-written, offering a comprehensive explanation of the proposed method, including detailed descriptions, mathematical formulas, and pseudocode. \n2. The proposed method can be integrated with both existing online and offline alignment methods, enhancing their robustness without requiring significant modifications to their core methodologies.\n3. PerpCorrect demonstrates strong performance with only a modest number of clean validation data points, which is beneficial in scenarios where such data is scarce or expensive to obtain.\n4. The proposed method has been empirically tested on several datasets and models, consistently showing superior performance over traditional methods in the presence of noisy data.", "weaknesses": "1. Tables 3 and 4 do not include the standard deviations for reward accuracy.\n2. Despite requiring fewer validation data points, its performance and accuracy still heavily depend on the availability of a high-quality clean validation dataset to initially train the surrogate LLM. Therefore, it is necessary to discuss potential solutions to this limitation.\n3. The process of calculating perplexity differences (PPLDiff) for each data point and aligning a surrogate LLM multiple times could be computationally intensive and time-consuming. Therefore, this additional cost needs to be discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your comments! Please find our replies below.\n\n``[Reply to W1] Please refer to our general author rebuttal [R3].``\n\n``[Reply to W2] Please refer to our general author rebuttal [R2].``\n\n``[Reply to W3] Please refer to our general author rebuttal [R1].`` \n\n``[Reply to Q1] We validated this assumption by conducting experiments on other datasets using various models. ``  \n\nWe randomly selected 10,000 data points from each dataset and calculated PPLDiff using different models.  The datasets and LLMs are downloaded from the Huggingface website.\n\nThe following table report the mean of PPLDiff values.\n\n|          Model           | HH-RLHF [1] | SafeRLHF [2] | SHP [3] | WebGPT [4] |  Avg.  |\n| :----------------------: | :---------: | :----------: | :-----: | :--------: | :----: |\n|        Qwen2-1.5B        |   -0.140    |    -0.002    |  0.040  |   -0.018   | -0.030 |\n|   Qwen2-1.5B-Instruct    |   -0.149    |    -0.009    |  0.046  |   -0.021   | -0.033 |\n|        Yi-1.5-6B         |   -0.158    |    -0.103    |  0.105  |   -0.036   | -0.048 |\n|      Yi-1.5-6B-Chat      |   -0.159    |    -0.054    |  0.069  |   -0.040   | -0.046 |\n|        gemma-2-2b        |   -0.140    |    -0.051    |  0.001  |   -0.024   | -0.053 |\n|      gemma-2-2b-it       |   -0.163    |    -0.053    |  0.063  |   -0.028   | -0.045 |\n|        falcon-7b         |   -0.113    |    -0.001    |  0.037  |   -0.016   | -0.023 |\n|    falcon-7b-instruct    |   -0.121    |    0.021     |  0.039  |   -0.017   | -0.019 |\n|     Mistral-7B-v0.3      |   -0.133    |    -0.045    |  0.048  |   -0.026   | -0.039 |\n| Mistral-7B-Instruct-v0.3 |   -0.201    |    -0.058    |  0.065  |   -0.038   | -0.058 |\n|         glm-4-9b         |   -0.134    |    -0.019    |  0.045  |   -0.027   | -0.034 |\n|     glm-4-9b-chat-1m     |   -0.135    |    -0.019    |  0.049  |   -0.028   | -0.033 |\n|      Llama-2-7b-hf       |   -0.133    |    -0.052    |  0.051  |   -0.028   | -0.040 |\n|    Llama-2-7b-chat-hf    |   -0.139    |    -0.041    |  0.063  |   -0.032   | -0.037 |\n\nThe following table report the standard deviation of PPLDiff values.\n\n|          Model           | HH-RLHF [1] | SafeRLHF [2] | SHP [3] | WebGPT [4] | Avg.  |\n| :----------------------: | :---------: | :----------: | :-----: | :--------: | :---: |\n|        Qwen2-1.5B        |    0.764    |    0.475     |  0.367  |   0.361    | 0.492 |\n|   Qwen2-1.5B-Instruct    |    0.798    |    0.491     |  0.385  |   0.369    | 0.511 |\n|        Yi-1.5-6B         |    1.119    |    0.614     |  0.564  |   0.500    | 0.699 |\n|      Yi-1.5-6B-Chat      |    0.855    |    0.493     |  0.417  |   0.373    | 0.534 |\n|        gemma-2-2b        |    0.885    |    0.517     |  0.702  |   0.373    | 0.619 |\n|      gemma-2-2b-it       |    0.975    |    0.535     |  0.467  |   0.378    | 0.588 |\n|        falcon-7b         |    0.631    |    0.400     |  0.324  |   0.325    | 0.420 |\n|    falcon-7b-instruct    |    0.648    |    0.436     |  0.342  |   0.344    | 0.442 |\n|     Mistral-7B-v0.3      |    0.719    |    0.411     |  0.332  |   0.313    | 0.444 |\n| Mistral-7B-Instruct-v0.3 |    0.981    |    0.482     |  0.385  |   0.343    | 0.548 |\n|         glm-4-9b         |    0.709    |    0.454     |  0.357  |   0.350    | 0.467 |\n|     glm-4-9b-chat-1m     |    0.719    |    0.456     |  0.367  |   0.356    | 0.474 |\n|      Llama-2-7b-hf       |    0.670    |    0.390     |  0.322  |   0.309    | 0.423 |\n|    Llama-2-7b-chat-hf    |    0.709    |    0.411     |  0.360  |   0.339    | 0.455 |\n\n``[Reply to Q2] We conjecture the main reason is that IPO performs better than cDPO and rDPO under a low proportion of NPs. `` \n\nThe proportion of NPs in the dataset corrected by our method is very low (**~10%**). We provide the proportion of NPs before and after using our method on the Golden HH dataset with the Llama-2 model in the table below.\n\n| Before        | After  | $\\Delta$ |\n| ------------- | ------ | -------- |\n| 9.86% (~10%)  | 6.05%  | 3.81%    |\n| 19.93% (~20%) | 9.76%  | 10.17%   |\n| 29.94% (~30%) | 14.14% | 15.80%   |\n| 40.06% (~40%) | 13.98% | 26.08%   |\n\nAccording to Table 6, the performance of IPO under low NPs (10%) is the main reason for the good performance of PerpCorrect-IPO. PerpCorrect-cDPO and PerpCorrect-rDPO are not as effective as PerpCorrect-IPO because both cDPO and rDPO are robust alignment methods that cannot fully utilize CPs.\n\n[1] Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. ArXiv 2022\n\n[2] Safe RLHF: Safe Reinforcement Learning from Human Feedback. ICLR 2024\n\n[3] Understanding Dataset Difficulty with V-Usable Information. ICML 2022\n\n[4] WebGPT: Browser-assisted question-answering with human feedback. ArXiv 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VOCtFT3bb7", "reviewer": "Reviewer_J5j5", "summary": "The paper proposes Perplexity-Aware Correction (PerpCorrect) to improve the alignment of large language models (LLMs) by detecting and correcting noisy preferences (NPs). PerpCorrect identifies NPs using the differences in perplexity (PPLDiff) between chosen and rejected responses. The method involves aligning a surrogate LLM with clean validation data, refining it with reliably clean and noisy data, and correcting NPs based on PPLDiff. Experiments demonstrate that PerpCorrect enhances alignment performance, is practical with a small amount of validation data, and is compatible with various alignment techniques.", "questions": "I have two minor questions about the experiments:\n\n1. As shown in Tables 1 and 3, the reward accuracy of other DPO family alignment methods evaluated with Llama2-7B is better than that evaluated with phi-2 (2B). However, PrepCorrect performs better when evaluated on phi-2 (2B). Could you give some explanation for this phenomenon?\n\n2. As shown in Table 6, PerpCorrect shows significant improvement when combined with other methods, such as SLiC, IPO, and cDPO, but less improvement when combined with rDPO.  Could you give some analyses of this phenomenon?", "limitations": "The paper explains its limitations in the appendix, such as time efficiency and the requirement for a validation dataset.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper innovatively addresses the impact of noisy preferences on alignment from the perspective of correcting noisy data. \n\n2. The experiments are comprehensive, which cover existing online, offline, and robust alignment methods, such as PPO, DPO, cDPO, and rDPO. The results show that PerpCorrect achieves the state-of-the-art robust alignment preference evaluated with different LLMs and datasets. Besides, PerpCorrect can be further combined with the other loss-based robust alignment methods.", "weaknesses": "1. The standard deviation of reward accuracy is not reported in Tables 3, 4, and 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your comments! Please find our replies below.\n\n``[Reply to W1] Please refer to our general author rebuttal [R3].  `` \t\n\n``[Reply to Q1] We provide a discussion about this phenomenon as follows. ``  \n\nWe found that the two models have similar proportion of NPs in the denoised dataset using PerpCorrent. The table below reports the proportion of NPs before and after PerpCorrect with phi-2 and Llama2, respectively. These empirical results suggest that PerpCorrect (Stage II) is not the main reason why PerpCorrect-DPO performs better when evaluated on phi-2 (2B).\n\n| Before        | After (Llama2) | After (phi-2) |\n| ------------- | -------------- | ------------- |\n| 9.86% (~10%)  | 6.05%          | 6.41%         |\n| 19.93% (~20%) | 9.76%          | 7.33%         |\n| 29.94% (~30%) | 14.14%         | 14.5%         |\n| 40.06% (~40%) | 13.98%         | 16.06%        |\n\nAs shown in Table 3, the reward accuracy of vanilla DPO on the phi-2 is better than that on the Llama2-7B. We conjecture that the phi-2 model has better robustness and that is why PerpCorrect performs better when evaluated on phi-2.\n\n``[Reply to Q2] The superior performance of rDPO and the gap between the estimated and actural proportion of NPs are the two main reasons.``\n\n**The superior performance of rDPO:** Under different proportions of NPs (ranging from 10% to 40%), rDPO demonstrated superior performance, achieving greater than 90% reward accuracy. This performance surpasses that of other methods such as cDPO, SliC, and IPO, leaving less room for improvement.\n\n**The gap between the estimated and actural proportion of NPs:** One reason why PerpCorrect-rDPO is not as effective as rDPO for certain proportions of NPs (10% and 20%) is that there is a gap between the estimated and actural proportion of NPs in the denoised dataset based on Eq. 15. We present the estimated and actual proportions of NPs in the denoised dataset in the table below.\n\n| The Estimated Proportion of NPs | The Actual Proportion of NPs |\n| ------------------------------- | ---------------------------- |\n| 4.36%                           | 6.05%                        |\n| 8.34%                           | 9.76%                        |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their insightful comments and suggestions. Please find our replies below.

``[R1] We discuss the complexity and additional computation time of method as follows.``

The additional computation time is primarily due to PerpCorrect. Both the theoretical and practical times are reported in the table below (using XXX to represent the theoretical time required for robust alignment) and are followed by a detailed analysis.

| Stage                                             | Theoretical Time       | Practical Time |
| ------------------------------------------------- | ---------------------- | -------------- |
| PerpCorrect (Stage II)                            | $\frac{T}{3} \times X$ | ~12 hours      |
| Robust alignment (Stage III and baseline methods) | $X$                    | ~12 hours      |
| Total                                             | $(1+\frac T3) X$       | ~24 hours      |

**In theory**, during the PerpCorrect process, we need to calculate PPLDiff and train the surrogate model in each epoch. The computation time introduced by PerpCorrect (Stage II) is approximately $\frac{T}{3}$ that of the robust alignment (Stage III and baseline methods).

- The calculation of PPLDiff in each epoch requires only $\frac{1}{3}$ of the time needed for robust alignment (Stage III). The primary computational load in robust alignment arises from the complexity of forwarding and back-propagation, while the complexities of gradient updates and parameter updates are relatively low. Additionally, back-propagation takes twice as long as forwarding. In addition, the calculation of PPLDiff only requires forwarding.
- For surrogate model training, PerpCorrect utilized data points that represented $t\times \alpha$ of the total dataset during epoch $t$. Since both $\alpha$ and $t$ are small, the time required for surrogate model training can be approximately ignored.

**In practice**, Our entire robust alignment pipeline (**~24h**) takes only **twice** as long as the baseline (**~12h**). We set $T=5$ and $\alpha=2\%$, and used the AdamW optimizer. The practical efficiency of the PerpCorrect is due to the use of fp32 precision by the AdamW optimizer, which increases the GPU's calculation time during the robust alignment process.

``[R2] We provide a discussion about the need for clean datasets and potential solutions as follows. ``

We only need 50 clean data points, which constitutes **less than 0.5%** of the entire dataset. In practice, such small-scale annotation is feasible. Reviewing and labeling data are essential steps to enhance data quality. For instance, Scale AI utilizes a large workforce to review and ensure the high quality of their data [1].

To address the challenge of obtaining sufficient clean datasets, one potential approach is to employ the LLM-as-a-judge method, which allows models to self-annotate and thereby reduce reliance on manually curated clean data. This method has been discussed in recent works [2] [3] [4], and could serve as a promising solution to this limitation. 

``[R3] We repeated the experiments three times with the same settings for Tables 3, 4, and 6, and the average reward accuracy and standard deviation are reported in the PDF file.``

[1] Scale AI, https://scale.com/docs

[2] Self-Rewarding Language Models, ICML 2024

[3] Constitutional AI: Harmlessness from AI Feedback, Arxiv 2024

[4] Trustllm: Trustworthiness in large language models, ICML 2024

Meta-review: All four reviewers provided positive recommendations for this work and recognized that the problem of noisy preference is significant, the proposed method is novel, the experiments are comprehensive. The reviewers also pointed out that it can be better if the authors can discuss the limitation of requiring high-quality clean validation dataset, and report the standard deviation in the final version. As a result, I recommend this paper for acceptance.
Decision: Accept (poster)