Perplexity-aware Correction for Robust Alignment with Noisy Preferences

Authors: keyi  kong
xilie  xu
di  wang
jingfeng  zhang
mohan  kankanhalli
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 20:05:18

Abstract: Alignment techniques are critical in ensuring that large language models (LLMs) output helpful and harmless content by enforcing the LLM-generated content to align with human preferences. 
However, the existence of noisy preferences (NPs), where the responses are mistakenly labelled as chosen or rejected, could spoil the alignment, thus making the LLMs generate useless and even malicious content. 
Existing methods mitigate the issue of NPs from the loss perspective by adjusting the alignment loss based on a clean validation dataset.
Orthogonal to these loss-oriented methods, we propose perplexity-aware correction (PerpCorrect) from the data perspective for robust alignment which detects and corrects NPs based on the differences between the perplexity of the chosen and rejected responses (dubbed as PPLDiff). 
Intuitively, a higher PPLDiff indicates a higher probability of the NP because a rejected/chosen response which is mistakenly labelled as chosen/rejected is less preferable to be generated by an aligned LLM, thus having a higher/lower perplexity.
PerpCorrect works in three steps: 
(1) PerpCorrect aligns a surrogate LLM using the clean validation data to make the PPLDiff able to distinguish clean preferences (CPs) and NPs. 
(2) PerpCorrect further aligns the surrogate LLM by incorporating the reliable clean training data whose PPLDiff is extremely small and reliable noisy training data whose PPLDiff is extremely large after correction to boost the discriminatory power.
(3) Detecting and correcting NPs according to the PPLDiff obtained by the aligned surrogate LLM to obtain a denoised training dataset for robust alignment.
Comprehensive experiments validate that our proposed PerpCorrect can achieve state-of-the-art alignment performance under NPs.
Notably, PerpCorrect demonstrates practical utility by requiring only a modest amount of validation data and being compatible with various alignment techniques. 
Our code is available at [PerpCorrect](https://github.com/luxinyayaya/PerpCorrect).

Main paper: 
 introduction: 
 Alignment enables the safe utilization of the remarkable capabilities acquired by large language models (LLMs) through self-supervised learning on vast corpora [6, 24, 4] . It refers to the process of ensuring that the contents generated by LLMs are helpful, harmless, and aligned with human values and preferences [19] . Reinforcement Learning from Human Feedback (RLHF) [9] has emerged as a primary technique for achieving alignment. Current technical routes [39, 40, 29] Figure 1 : We evaluated various robust alignment methods under different proportions of noisy preferences using the Llama2-7B model, on the Golden HH dataset. The reward accuracy of both the vanilla DPO and PPO method significantly decreases as the proportion of noisy preferences increases. Our method, perplexity-aware correction (PerpCorrect), outperforms both the DPO and PPO series baselines across different proportions of noisy preferences.Proximal Policy Optimization (PPO) [27] . Current offline techniques such as Direct Preference Optimisation (DPO) [26] , Sequence Likelihood Calibration with Human Feedback (SLiC) [38] and Identity-Preference Optimisation (IPO) [3] , could directly align LLMs without intensively training a reward model as employed in RLHF.Recent studies [34, 8] have shown there exist noisy preferences (NPs) that may lead to significant degradation in alignment performance. The issue of NPs, where the label of the actually chosen/rejected responses in training datasets is flipped as rejected/chosen, can arise from the biases of annotators [34] and ẗhe malicious noise injection [5] . As shown in Figure 1 , when NPs are randomly injected into the training dataset, the conventional alignment method (e.g., DPO [26] and PPO [9] ) will have significantly degraded alignment performance measured by the reward accuracy. Such performance degradation could result in the generation of useless and even malicious content [34] . Therefore, it necessitates developing robust alignment methods that can utilize datasets with NPs to effectively align the LLMs with human preferences.Existing robust alignment methods are proposed from the loss perspective, which adjust the alignment loss using a clean validation dataset to mitigate the issue of NPs. In particular, the conservative DPO (cDPO) [21] and robust-DPO (rDPO) [8] both estimate the proportion of NPs using the clean validation data via cross-validation and then adjust the original DPO loss based on the estimated proportion of NPs. However, Mitchell [21] and Chowdhury et al. [8] overlooked the essential differences between noisy and clean preferences, which is critical for mitigating the issue of NPs.To this end, we propose Perplexity-aware Correction (PerpCorrect) for robust alignment from the data perspective by leveraging the differences between noisy and clean preferences for robust alignment. PerpCorrect detects and corrects NPs based on the difference between the perplexity of the chosen response and that of the rejected counterparts (dubbed as PPLDiff) obtained by an aligned surrogate LLM using the clean validation set. If an NP is detected, PerpCorrect will correct it by flipping the label of the rejected/chosen responses as chosen/rejected. Intuitively, rejected responses which are mistakenly labelled as chosen have a higher perplexity since they are less consistent with human preferences and thus have a lower probability of being generated after alignment. Therefore, a higher value of PPLDiff indicates a higher probability of the preferences being noisy. In this way, PerpCorrect leverages the differences between noisy and clean preferences (CPs) identified by PPLDiff to detect NPs.To make the PPLDiff able to distinguish CPs and NPs, PerpCorrect requires an aligned surrogate LLM for calculating PPLDiff. The density of PPLDiff obtained on the noisy training dataset using an unaligned surrogate LLM, which can be fitted as a normal distribution centered around zero (evidenced in Figure 2a ), cannot discriminate CPs and NPs. Therefore, we align a surrogate LLM using the clean validation data. The density of PPLDiff obtained by the aligned surrogate LLM in Figure 2b can be fitted into two distinguishable normal distributions, thus being able to differentiate CPs and NPs.However, there still exists a large overlap between two normal distributions after aligning only on the clean validation dataset, which could result in an unsatisfactory accuracy of NP detection. To this end, we iteratively align the model using more reliable clean training data with extremely low PPLDiff (located in the green area in Figure 2c ) and reliable noisy training data with extremely large PPLDiff (located in the red area in Figure 2c ) sampled from noisy training datasets. Finally, the two normal distributions are significantly separated as shown in Figure 2d , which indicates that PPLDiff has an enhanced discriminatory power.Benefiting from the strong discriminatory power of PPLDiff calculated by the aligned surrogate LLM, PerpCorrect outputs a denoised training dataset for robust alignment by first detecting NPs based on a PPLDiff threshold and then correcting them. The data, whose PPLDiff is below a certain threshold (i.e., the black dotted line in Figure 2d ) selected as the x-coordinate of the two normal distributions' intersection, are identified as NPs and thus corrected by flipping the response's label. Notably, our proposed PerpCorrect is compatible with various alignment methods as well as robust alignment methods [21, 8] since the metric PPLDiff is agnostic to training algorithms and only requires an arguably small number of clean validation data (~50), thus making it practical.Comprehensive empirical results, evaluated using the Llama2-7B [32] and phi-2 [20] models on the OpenAssistant Conversations (OASST1) [17] and Golden HH [7] datasets, validate the effectiveness of our proposed PerpCorrect method in robustifying alignment with NPs. We empirically validate that PerpCorrect consistently obtains state-of-the-art performance among various proportions of NPs. Besides, we empirically demonstrate that PerpCorrect can effectively robustify various alignment techniques and robust alignment methods, validating its compatibility.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Our introduction covers our contributions, main methods and experimental results.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
literature review and preliminary: 
 In this section, we introduce the related work about LLM alignment and provide preliminaries about the noisy preferences, perplexity, as well as various alignment methods.In the domain of aligning LLMs with human preferences, pairwise preference methods are favored due to their lower cognitive burden on evaluators. Traditional online alignment approaches [32, 24, 29] involve training reward models from these preferences to provide signals in reinforcement learning. Recent offline alignment methods like Direct Preference Optimization (DPO) [26] , Sequence Likelihood Calibration (SLiC) [38] , and Identify Preference Optimization (IPO) [3] streamlined this process by directly using preference pairs to train LLMs, thus enhancing performance and reducing computational costs. Additionally, methods like RRHF [37] align LLMs using multiple ranked preferences, Kahneman-Tversky Optimization (KTO) [13] align LLMs using a single preference labeled as good or bad, and Rejection Sampling Optimization (RSO) [18] address DPO's limitation in sampling preference pairs from the optimal policy through rejection sampling. However, NPs, arising from the biased human feedback, can determine the alignment performance [24, 34] . Robust alignment methods like conservative DPO (cDPO) [21] , robust DPO (rDPO) [8] have been proposed to address these issues from the loss perspective. Our approach focuses on the data perspective to address these issues of NPs and is orthogonal to these robust alignment methods.Noisy preferences (NPs). NPs refer to preference data in training datasets, whose label of the actually chosen/rejected responses is flipped as rejected/chosen. Let D = {(x (i) , y(i) w , y (i) l )} Ni=1 be the preference dataset consisting of N ∈ N preference data points. For each preference data point (x, y w , y l ) ∈ D, x is the prompt input to LLMs, y w is the chosen response, and y l is the rejected response. We let D = {(x (i) , ỹ(i) w , ỹ(i) l )} N i=1 be the noisy preference dataset (i.e., preference dataset consisting noisy preferences) and denote preference data points that are not noisy as clean preferences (CPs). Following Chowdhury et al. [8] , we obtain the noisy preference dataset D using the standard random noise model [23] with the probability ε ∈ (0, 50%) to change the data point into noisy preferences, i.e.EQUATIONPerplexity (PPL). PPL [15] measures the probability that the LLM generates a sentence. A lower PPL of a sentence indicates that the LLM has generated this sentence with a high probability. PPL is defined as the average negative log-likelihood of a sequence, i.e.,EQUATIONwhere s is a sequence composed of t tokens and log π θ (s i |s <i ) denotes the log-likelihood of the i-th token given the preceding tokens s <i calculated by an LLM π θ .Technical details of alignment methods. There are usually three phases in RLHF pipeline [34, 26] :(1) supervised fine-tuning (SFT); (2) reward modeling; (3) reinforcement learning (RL) optimization.In the SFT phase, an LLM is fine-tuned via supervised learning on high-quality task-related data.We denote the LLM after the SFT phase as π SFT . In the reward modeling phase, the reward model is introduced to simulate human preferences. Given a preference dataset, a reward model r ω (x, y) parameterized by ω, which takes prompt x and response y as input and outputs a real number representing the reward score, can be optimized via minimizing the following loss function: L R (r ω , D) = -E (x,yw,y l )∼D [log σ(r ω (x, y w )r ω (x, y l ))] ,(3) where σ is the logistic function. In the RL optimization phase, the objective function is as follows:EQUATIONwhere π θ (y|x) represents the probability that the LLM parameterized by θ > 0 generates the response y given the prompt x, π ref is a reference LLM to maintain the generation ability of the aligned model, and β is a hyper-parameter to ensure the similarity between π θ (y | x) and π ref (y | x). We take π SFT as the reference LLM π ref following Ouyang et al. [24] .Recently, offline alignment methods directly leverages preferences in preference datasets, bypassing the need to learn a reward model in RLHF. The LLM parameters are optimized by minimizing the following loss function:EQUATIONwhere the function G changes with the alignment method. To be specific, DPO [26] uses a BCE loss, SLiC [38] uses a hinge loss, and IPO [3] uses a square loss:EQUATIONEQUATIONEQUATIONTo mitigate the issue of NPs, cDPO [21] and rDPO [8] adjust the DPO loss based on the estimated proportion of NPs ε ′ using a clean validation dataset D val = {(x (i) , yl )} N val i=1 consisting of N val ∈ N clean preference data points, i.e.G cDPO (x, ỹw , ỹl ;EQUATIONEQUATION)3 Perplexity-aware Correction for Robust AlignmentThis section introduces Perplexity-aware Correction (PerpCorrect) for robust alignment with NPs. In Section 3.1, we introduce a novel metric called PPLDiff and then illustrate the pipeline of PerpCorrect to detect and correct NPs based on PPLDiff. In Section 3.2, we demonstrate how to adapt our proposed PerpCorrect with various alignment methods to achieve robust alignment.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discussed the efficiency issues and data volume requirements of our method PerpCorrect in the Conclusions section.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
perplexity-aware correction (perpcorrect): 
 In this subsection, we introduce PerpCorrect which employs a novel metric called PPLDiff as the foundation for detecting and correcting NPs. The algorithm of PerpCorrect is described in Algorithm 2.PPLDiff. PPLDiff measures the difference between the PPL of chosen response and that of the rejected response. Given a preference data point (x, ỹw , ỹl ) ∈ D sampled from the noisy training dataset D and an LLM π θ , PPLDiff is defined as follows:PPLDiff(x, ỹw , ỹl ; θ) = log PPL([x; ỹw ]; θ)log PPL([x; ỹl ]; θ), (11) where [x; y] indicates the concatenation of the prompt x and the response y. Intuitively, if a data point is a clean preference, the PPL([x; ỹw ]; θ) will be lower than PPL([x; ỹl ]; θ) because the sequence [x; ỹw ] is more aligned with human values and thus has a higher probability of being generated by aligned LLMs. As a result, it PPLDiff will be lower compared to NPs, which PPL([x; ỹw ]; θ) is higher than PPL([x; ỹl ]; θ). This difference allows us to distinguish CPs and NPs based on PPLDiff.Aligning a surrogate LLM only using clean validation data. Here, we utilize a clean validation dataset D val to obtain an aligned surrogate LLM to make PPLDiff able to distinguish CPs and NPs. We empirically find that the PPLDiff values of CPs and NPs calculated by an unaligned LLM in the noisy training dataset were initially indistinguishable as shown in Figure 2a , making it impossible to differentiate the NPs from CPs. This is because an unaligned LLM lacks the necessary preferences to distinguish NPs and CPs.Therefore, we introduce a surrogate LLM π θ ′ parameterized by θ ′ to replace the unaligned LLM and use it for calculating PPLDiff. We optimize the surrogate LLM π θ ′ using the clean validation dataset D val as follows:EQUATIONAfter aligning the surrogate LLM, the PPLDiff values of NPs calculated by the surrogate LLM π θ ′ are significantly increased and those of CPs are significantly decreased, forming two distinct distributions as shown in Figure 2b . This is because the aligned surrogate LLM is trained to generate responses that align with human preferences, enhancing its ability to distinguish between NPs and CPs based on PPLDiff.To separate CPs and NPs in the noisy training dataset without knowing the oracle preferences, we leverage the Levenberg-Marquardt (LM) algorithm to find two normal distributions that fit the density of PPLDiff calculated by the aligned surrogate LLM. Specifically, the LM algorithm returns the constants ε, μ, σ that satisfies the following condition:EQUATIONwhereEQUATIONNote that x is the PPLDiff value and h(x|ε, μ, σ) is the superposition of these two normal distribution. We denote f clean (x|μ, σ2 ) as the normal distribution fitting the PPLDiff of CPs and f noisy (x|μ, σ2 ) as the normal distribution fitting the PPLDiff of NPs since the PPLDiff of NPs is intuitively higher than that of CPs. In this way, we can obtain two distinguishable normal distributions to separate NPs and CPs as shown in the green and red dotted lines of Figure 2b without knowing the oracle preferences.Further aligning the surrogate LLM using extra reliable training data from noisy training datasets. After aligning only using the clean validation datasets, the discriminatory power of the PPLDiff is still far from satisfactory because of the large overlap between the two normal distributions. Therefore, we align the surrogate LLM with more reliable training data to make the PPLDiff of CPs and that of NPs more separable. We iteratively align the surrogate LLM π θ ′ using more reliably clean training data whose PPLDiff is extremely small and reliably noisy training data whose PPLDiff is extremely large after correction by flipping the label of the response. Detecting and correcting NPs based on PPLDiff to output a denoised training dataset. Based on the PPLDiff calculated by the aligned surrogate LLM, PerpCorrect detects and corrects NPs whose PPLDiff value is lower than a certain threshold. We take the x-coordinate of the intersection of the two normal distributions as the threshold (the black dotted line in Figure 2d ). As shown in Lines 23-31, data points whose PPLDiff values are larger than this threshold are identified as CPs (the green area in Figure 2d ), and other data points are identified as NPs requiring correction (the red area in Figure 2d ). In this way, we can obtain a denoised training dataset for robust alignment. Further, we select an optimal denoised training dataset to further enhance the performance of robust alignment according to the intersection area of the two normal distributions. We denote the intersection area of two normal distributions as the estimated NP proportion of the denoised training dataset, i.e.,D ′ t consists of (t -1) • α • (1 -ε) • | D| reliably clean training data whose PPLDiff values are smallest (t -1) • α • (1 -ε) percent and (t -1) • α • ε • | D|EQUATIONwhere ε ′ P C calculates the ratio of noisy data points which are not detected by PerpCorrect (i.e., the green area enclosed by the black and red lines in Figure 2d ) and the clean data points which are mistakenly detected by PerpCorrect (i.e., the red area enclosed by the black and green lines in Figure 2d ). In this way, ε ′ P C can efficiently calculate the NP proportion of the denoised training dataset. We take the denoised training dataset with the smallest ε ′ P C among multiple iterations as the optimal one for robust alignment to boost alignment performance.Here, we introduce how to adapt PerpCorrect to robustify various alignment methods and demonstrate the algorithm of robust alignment via PerpCorrect in Algorithm 1. In general, the pipeline of the robust alignment based on PerpCorrect contains three stages: SFT, PerpCorrect, and alignment. We will first conduct SFT, following Christiano et al. [9] , to boost the performance of a pre-trained LLM by boosting its skills for specific tasks. Next, we will conduct PerpCorrect to detect and correct NPs and output an optimal denoised training dataset Ddenoised the smallest ε ′ P C in Eq. 15. Finally, we can obtain an aligned LLM from the SFT model using the denoised training dataset Ddenoised via alignment (i.e., Line 8 in Algorithm 1).Because our proposed PerpCorrect is agnostic to alignment methods and model structures, PerpCorrect is applicable to robustify both online alignment methods such as RLHF (PPO) [9] and offline alignment methods including DPO [26] , SLiC [38] , and IPO [3] . Besides, our proposed PerpCorrect is compatible with existing loss-oriented robust alignment methods, such as cDPO [21] and rDPO [8] , based on the estimated proportion of NPs. Note that cDPO and rDPO require conducting computationally expensive cross-validation to tune the estimated proportion of NPs. We can efficiently estimate the proportion of NPs by utilizing the fitted normal distributions during PerpCorrect, i.e., ε ′ P C in Eq. 15. Therefore, we can combine PerpCorrect with a wide range of existing alignment methods to achieve robust alignment with NPs.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments: 
 In this section, we demonstrate that our proposed PerpCorrect achieves state-of-the-art alignment performance under different proportion of NPs and have good compatibility with other alignment methods. In Section 4.1, PerpCorrect combined with DPO [26] achieves state-of-the-art alignment performance than existing baselines (Section 4.1), including DPO [26] , cDPO [21] , and rDPO [8] . In Section 4.2, we further analyze the impact of the number of validation data and verified the compatibility of PerpCorrect with online and offline alignment methods and robust alignment methods. The training details and compute resources are reported in Appendix C.1. Datasets. We utilize two preference datasets, namely OpenAssistant Conversations (OASST1) [17] and Golden HH [7] . The processed OASST1 dataset comprises 17,939 training samples and 951 testing samples and the processed Golden HH dataset consists of 12,066 training samples and 654 testing samples. The description and processing details of these datasets are provided in Appendix C.2. Models. Our evaluation leverages two distinct series of open-sourced LLMs with different parameter sizes: Llama2-7B [32] and phi-2 [20] . We acquire the checkpoints from their official repositories on Hugging Face. The LLMs used for PerpCorrect and those for robust alignment share the same model structure and initialization.Baselines. We adopt vanilla DPO [26] and two robust alignment methods, cDPO [21] and rDPO [8] , as baselines. For their detailed implementation, we utilize and adapt the transformers and TRL libraries provided by the Hugging Face community.Metrics. In accordance with Chowdhury et al. [8] , we employ the winning rate of policy generations against the selected preferences on the test dataset as our primary metric. This metric applies to vanilla DPO [26] , cDPO [21] , rDPO [8] , as well as other offline alignment methods including SLiC [38] and IPO [3] . Additionally, we utilize the winning rate of the reward model score for the chosen preferences on the test dataset as our metric for vanilla PPO [24] , cPPO [21, 34] , and rPPO [8] . These two metrics are collectively called reward accuracy.The empirical results demonstrate that our method, PerpCorrect, achieves state-of-the-art robust alignment performance, surpassing existing baselines such as vanilla DPO [26] , cDPO [21] , and rDPO [8] . This is evident across various proportions of noisy preferences ε using different datasets and LLMs.Comparison using different LLMs. Tables 1 and 3 demonstrate the average reward accuracy of the DPO series alignment methods on the Golden HH [7] dataset using Llama2-7B [32] and phi-2 [20] . At a proportion of the NPs ε = 40%, PerpCorrect increases the reward accuracy by 41.77% (from 53.15% to 94.92%) using Llama2-7B and by 41.41% (from 54.98% to 96.39%) using phi-2. The empirical result validates that our proposed PerpCorrect can be used on different sizes of LLMs and achieve better alignment performance than baselines.Comparison on different datasets. Tables 3 and 4 demonstrate the average reward accuracy of the DPO series alignment methods on the Golden HH [7] and OASST1 [17] datasets using phi-2 [20] . The empirical results reveal a significant discrepancy in average reward accuracy between the more complex OASST1 dataset and the Golden HH dataset. The performance of other robust alignment methods is found to be unsatisfactory on the OASST1 dataset, often not surpassing the vanilla DPO.In contrast, our method PerpCorrect consistently maintains strong alignment performance across varying proportions of noisy preferences. In general, our method PerpCorrect can achieve better alignment performance than baselines across different datasets.Impact of the number of clean validation data. Table 5 illustrates the impact of the number of clean validation data points. We conducted experiments on the Golden HH dataset using Llama2-7B with a proportion of NPs ε = 40%. The empirical results indicate that as the number of clean validation data points increases, the performance of our method, PerpCorrect, also improves. However, when the number is too large, the improvement in performance is not obvious, and the cost of manual annotation significantly increases.Compatibility with online alignment method RLHF (PPO). We adopt vanilla PPO [24] , cPPO [21, 34] , and rPPO [8] as baselines. Table 2 shows the alignment performance of PPO series alignment methods on the Golden HH [7] dataset using Llama2-7B. Although vanilla PPO has good performance when the proportion of NPs is low, it still declines significantly when the proportion is high. PerpCorrect maintains desirable alignment performances when the proportion of NPs is high. Our empirical results show that PerpCorrect has desirable compatibility with online alignment method RLHF (PPO).Compatibility with various offline alignment methods. Table 6 presents the average reward accuracy and improvements of original offline alignment methods compared to those combined with PerpCorrect. Our experiments, conducted on the Golden HH dataset using Llama2-7B, reveal that the reward accuracy of SLiC [38] and IPO [3] both significantly decrease as the proportion of NPs increases, similar to vanilla DPO [26] . However, our method PerpCorrect enhances their alignment performance across different proportions of NPs. Notably, IPO combined with PerpCorrect achieves the best alignment performance. We conjecture the main reason is that the proportion of NPs in the denoised dataset is very low and IPO performs better than other methods under a low proportion of NPs. These empirical results demonstrate that our method has good compatibility with various offline alignment methods.Compatibility with robust alignment methods. Table 6 shows the average reward accuracy and improvements of robust alignment methods compared to those combined with PerpCorrect. Our method, PerpCorrect, can significantly enhance the performance of cDPO [21] , and provide a modest improvement for rDPO [8] under almost all proportion of NPs. The empirical results show that our method has good compatibility with robust alignment methods. 
conclusions: 
 This paper proposes a method called perplexity-aware correction (PerpCorrect), as an effective approach for robust alignment with noisy preferences (NPs Validation dataset. PerpCorrect requires a validation dataset for aligning a surrogate LLM. However, manually annotating a validation dataset is complex and labor-intensive in practice. As shown in Table 5 , there is a significant disparity in alignment performance when comparing the use of 10 clean samples to 50 clean samples. Exploring how to use fewer clean samples or even no clean samples to achieve the same or better performance is a problem worth further investigation. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We provide detailed sufficient information on the computer resources in the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We discuss the potential impacts in Appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our robust alignment method does not have a high risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The models and datasets the we used are open-sourced, and we follow their license and terms of use.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Our proposed PerpCorrect and robust alignment pipeline offers a solution for achieving state-of-theart performance in robust alignment under noisy preferences. PerpCorrect is designed to effectively reduce malicious noise in the dataset and mitigate biases introduced by human annotators, ensuring that the trained language model (LLM) is accurately aligned with true human preferences.Moreover, we recognize a potential risk: if malicious users exploit our method for reverse training, they might compromise the security mechanisms of existing open-source LLMs. Existing research has demonstrated the possibility of reverse training [36] .The additional computational overhead is primarily attributed to PerpCorrect (Section 3.1). The calculation of PPLDiff in each epoch requires only 1 3 of the time needed for robust alignment. The primary computational load in robust alignment arises from the complexity of forwarding and back-propagation, while the complexities of gradient updates and parameter updates are relatively low. Additionally, back-propagation takes twice as long as forwarding. In addition, the calculation of PPLDiff only requires forwarding.For surrogate model training, PerpCorrect utilized data points that represented t × α of the total dataset during epoch t. Since both t and α are small, the time required for surrogate model training can be approximately ignored.In practice, Our entire robust alignment pipeline (∼24 hours) takes only twice as long as the baseline (∼12 hours). We set T = 5 and α = 2, and used the AdamW optimizer. The practical efficiency of the PerpCorrect is due to the use of fp32 precision by the AdamW optimizer, which increases the GPU's calculation time during the robust alignment process. We utilized the Qlora method [11] for fine-tuning the LLMs, executed on RTX 4090 GPUs with 24 GB of memory. Hyperparameters were set as follows: lora_rank = 32, lora_dropout = 0.1, and lora_alpha = 16. For SFT, we use the alpaca dataset [30] and set learning_rate = 2e -4 and batch_size = 20. For our PerpCorrect stage II, we set β = 0.1, learning_rate = 1e -3, batch_size = 4, T = 5, and α = 0.02. For our PerpCorrect stage III and all other alignment methods, we set β = 0.1, learning_rate = 3e -4, and batch_size = 20. Other details not mentioned, we follow the default setting in TRL library. Each experiment, involving a specific method and proportion of NPs, could be completed using a single RTX 4090 GPU within 24 hours on the Golden HH dataset and within 72 hours on the OASST1 dataset.OpenAssistant Conversations Dataset (OASST1). The original OASST1 dataset [17] is an assistant-style conversation corpus generated and annotated by humans. It consists of over 10,000 fully annotated conversations in 35 different languages. Sileo [28] converted these conversations into a preference dataset comprising 17,966 training samples and 952 testing samples. After filtering out conversations with one or fewer letters, we obtained a preference dataset with 17,939 training samples and 951 testing samples.Golden HH. The Golden HH dataset [7] is a variant of the Anthropic Helpful and Harmless (HH) dataset [4] . It originally contains 42,537 training samples and 2,312 testing samples as part of a preference dataset. Each sample has two keys: one representing the prompt x and the chosen response y w , and the other representing the prompt x and the rejected response y l . We first converted the dataset into a triple form: prompt x, chosen response y w , and rejected response y l , retaining only one-turn conversation data. After filtering out samples with one or fewer letters, we obtained a preference dataset with 12,066 training samples and 654 testing samples.Supervised Fine-Tuning (SFT). The objective of Supervised Fine-Tuning (SFT) is to enhance the performance of a pre-trained large language model (LLM) by refining its abilities for specific tasks. As demonstrated by prior work [9, 25, 24] , this can be achieved by utilizing supervised fine-tuning with a specialized dataset tailored to the target task. The SFT dataset is annotated with labels, providing examples that are directly relevant to the task. Specifically, for each data point (x, y) in the SFT dataset, x represents the prompt given to the LLM, and y represents the expected response that the model should generate based on the prompt x. The process involves fine-tuning the LLM by maximizing the log-likelihood of the correct responses y given the prompts x. Through this method, the model learns to produce more accurate and task-specific outputs, thereby significantly improving its performance on the given task. Perplexity-aware Correction (PerpCorrect). We demonstrate the entire PerpCorrect algorithm in Algorithm 2.Alignment. We can achieve alignment using the denoised training dataset Ddenoised with an estimated proportion of NPs ε ′ denoised . For offline alignment methods such as DPO, SLiC, and IPO, we can directly optimize the LLM using the denoised training dataset Ddenoised based on the loss functions defined in Eqs. 6-8. For loss-based robust alignment methods, including cDPO and rDPO, we set ε ′ = ε ′ denoised and then optimize the LLM using the denoised training dataset Ddenoised according to the loss functions mentioned in Eqs. 9 and 10. For the online alignment method RLHF (PPO), we first train a reward model using the denoised training dataset Ddenoised based on the loss function described in Eq. 3. Subsequently, we further optimize the LLM using PPO according to the objective function detailed in Eq. 4.Tables 8-12 demonstrates the standard deviation of the reward accuracy reported in Tables 1-4 and 6.We randomly selected 10,000 data points from each dataset and calculated PPLDiff using different LLMs. The datasets and LLMs are downloaded from the Huggingface website. The average PPLDiff values are reported in the Table 13 . 12 : Standard deviation of reward accuracy and improvements of the offline and robust alignment methods, as well as those combined with PerpCorrect, using Llama2-7B on the Golden HH dataset. The average reward accuracy is reported in Table 6 .Proportion of noisy preferences (%) 10 20 30 40 DPO 0.81% 0.40% 1.28% 0.98% PerpCorrect-DPO 0.63% 0.87% 1.73% 0.63% ∆ 0.98% 0.89% 0.97% 1.98% SLiC 1.91% 1.33% 6.77% 8.02% PerpCorrect-SLiC 1.76% 1.45% 1.16% 1.50% ∆ 0.63% 0.38% 5.62% 6.52% IPO 0.84% 0.63% 3.48% 0.32% PerpCorrect-IPO 0.32% 1.68% 0.23% 1.25% ∆ 0.98% 2.29% 3.26% 1.10% cDPO 1.15% 0.81% 1.76% 1.64% PerpCorrect-cDPO 0.69% 0.75% 0.81% 0.72% ∆ 0.47% 1.31% 1.85% 2.36% rDPO 0.26% 1.53% 0.95% 1.92% PerpCorrect-rDPO 0.63% 0.23% 1.84% 1.19% ∆ 0.84% 1.48% 2.19% 2.31% Avg. HH-RLHF [4] SafeRLHF [10] SHP [12] WebGPT [22] Qwen2-1.5B [35] -0.140 -0.002 0.040 -0.018 -0.030 Qwen2-1.5B-Instruct [35] - -0.140 -0.051 0.001 -0.053 gemma-2-2b-it [31] -0.163 -0.053 0.063 -0.028 -0.045 falcon-7b [2] -0.113 -0.001 0.037 -0.016 -0.023 falcon-7b-instruct [2] -0.121 0.021 0.039 -0.017 -0.019 Mistral-7B-v0.3 [16] -0.133 -0.045 0.048 -0.026 -0.039 Mistral-7B-Instruct-v0.3 [16] -0.201 -0.058 0.065 -0.038 -0.058 glm-4-9b [14] -0.134 -0.019 0.045 -0.027 -0.034 glm-4-9b-chat-1m [14] -0.135 -0.019 0.049 -0.028 -0.033 Llama-2-7b-hf [33] -0.133 -0.052 0.051 -0.028 -0.040 Llama-2-7b-chat-hf [33] -0.139 -0.041 0.063 -0.032 -0.037 π θ ′ ← Alignment (π θ ′ , D ′ t ∪ D val )NeurIPS Paper ChecklistJustification: We follow the Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: All four reviewers provided positive recommendations for this work and recognized that the problem of noisy preference is significant, the proposed method is novel, the experiments are comprehensive. The reviewers also pointed out that it can be better if the authors can discuss the limitation of requiring high-quality clean validation dataset, and report the standard deviation in the final version. As a result, I recommend this paper for acceptance.

Reviews: review: paper_summary: paper_summary: The paper introduces Perplexity-aware Correction (PerpCorrect), a method for robust alignment of large language models (LLMs) with noisy preferences (NPs). PerpCorrect detects and corrects NPs by analyzing the perplexity difference (PPLDiff) between chosen and rejected responses. The approach involves aligning a surrogate LLM with clean validation data, iteratively refining with reliable training data, and ultimately producing a denoised training dataset. Experiments demonstrate that PerpCorrect significantly improves alignment performance and is compatible with various alignment techniques.
strength_weakness: strength_weakness: 1. Implementing PerpCorrect involves additional steps of calculating and analyzing perplexity differences (PPLDiff), which can add complexity to the alignment process. Can the authors discuss the complexity and additional computation time of the method?

2. The method requires a clean validation dataset to align the surrogate LLM initially. The need for manually annotated validation data can be labor-intensive and may not always be feasible in large-scale applications.

3. The method assumes that the perplexity of clean and noisy preferences will differ consistently, which may not always hold true across all datasets and model configurations.
questions: questions: See Weaknesses.
limitations: limitations: Limitations have been discussed in Conclusion section.
paper_summary: paper_summary: The paper proposed a novel method to mitigate the preference noise in alignment. The authors first provide insights into how the PPLDiff can recognize the noisy preferences and then use the PPLDiff to select and correct noise preferences. Extensive experiments demonstrate that the method can significantly improve the robustness under high noise ratios.
strength_weakness: strength_weakness: * Some writing is not clear.
   - In Line 50, "However, Mitchell [15] and Chowdhury et al. [6] overlooked the essential, differences between noisy and clean preferences, which is critical for mitigating the issue of NPs." What is the overlooked difference here? The logic is not clear.
questions: questions: * In Line 50, "However, Mitchell [15] and Chowdhury et al. [6] overlooked the essential, differences between noisy and clean preferences, which is critical for mitigating the issue of NPs." What is the overlooked difference here? The logic is not clear.
limitations: limitations: No obvious limitations.
paper_summary: paper_summary: This paper presents a novel method called PerpCorrect for robust alignment in large language models, particularly in the presence of noisy preferences in training data. PerpCorrect addresses noisy preferences by evaluating the perplexity difference (PPLDiff) between selected and rejected responses. By leveraging PPLDiff to identify and rectify these noisy preferences, PerpCorrect generates a denoised training dataset. Experimental results demonstrate that PerpCorrect achieves state-of-the-art alignment performance with only a small amount of validation data.
strength_weakness: strength_weakness: 1. Tables 3 and 4 do not include the standard deviations for reward accuracy.
2. Despite requiring fewer validation data points, its performance and accuracy still heavily depend on the availability of a high-quality clean validation dataset to initially train the surrogate LLM. Therefore, it is necessary to discuss potential solutions to this limitation.
3. The process of calculating perplexity differences (PPLDiff) for each data point and aligning a surrogate LLM multiple times could be computationally intensive and time-consuming. Therefore, this additional cost needs to be discussed.
questions: questions: 1. Will the PPLDiff calculate on the new dataset, using models similar to those used with other data (e.g., Llama-2-7b-chat), result in a normal distribution with a mean of 0?

2. What are the reasons that make PrepCorrect-IPO perform better than PrepCorrect-cDPO and PrepCorrect-rDPO? Please analyze this phenomenon.
limitations: limitations: As discussed in the paper.
paper_summary: paper_summary: The paper proposes Perplexity-Aware Correction (PerpCorrect) to improve the alignment of large language models (LLMs) by detecting and correcting noisy preferences (NPs). PerpCorrect identifies NPs using the differences in perplexity (PPLDiff) between chosen and rejected responses. The method involves aligning a surrogate LLM with clean validation data, refining it with reliably clean and noisy data, and correcting NPs based on PPLDiff. Experiments demonstrate that PerpCorrect enhances alignment performance, is practical with a small amount of validation data, and is compatible with various alignment techniques.
strength_weakness: strength_weakness: 1. The standard deviation of reward accuracy is not reported in Tables 3, 4, and 6.
questions: questions: I have two minor questions about the experiments:

1. As shown in Tables 1 and 3, the reward accuracy of other DPO family alignment methods evaluated with Llama2-7B is better than that evaluated with phi-2 (2B). However, PrepCorrect performs better when evaluated on phi-2 (2B). Could you give some explanation for this phenomenon?

2. As shown in Table 6, PerpCorrect shows significant improvement when combined with other methods, such as SLiC, IPO, and cDPO, but less improvement when combined with rDPO.  Could you give some analyses of this phenomenon?
limitations: limitations: The paper explains its limitations in the appendix, such as time efficiency and the requirement for a validation dataset.

Comments: comment: I appreciate the authors' response, which has solved my concerns.
Thank you for your reply, my concerns have been resolved.
Thank you for the response which addressed my concerns.