Title: Perplexity-aware Correction for Robust Alignment with Noisy Preferences
Venue: NeurIPS 2024
Accepted: yes
Keywords: Large language model

Label-noise learning
Abstract: Alignment techniques are critical in ensuring that large language models (LLMs) output helpful and harmless content by enforcing the LLM-generated content to align with human preferences. 
However, the existence of noisy preferences (NPs), where the responses are mistakenly labelled as chosen or rejected, could spoil the alignment, thus making the LLMs generate useless and even malicious content. 
Existing methods mitigate the issue of NPs from the loss perspective by adjusting the alignment loss based on a clean validation dataset.
Orthogonal to these loss-oriented methods, we propose perplexity-aware correction (PerpCorrect) from the data perspective for robust alignment which detects and corrects NPs based on the differences between the perplexity of the chosen and rejected responses (dubbed as PPLDiff). 
Intuitively, a higher PPLDiff indicates a higher probability of the NP because a rejected/chosen response which is mistakenly labelled as chosen/rejected is less preferable to be generated by an aligned LLM, thus having a higher/lower perplexity.
PerpCorrect works in three steps: 
(1) PerpCorrect aligns a surrogate LLM using the clean validation data to make the PPLDiff able to distinguish clean preferences (CPs) and NPs. 
(2) PerpCorrect further aligns the surrogate LLM by incorporating the reliable clean training data whose PPLDiff is extremely small and reliable noisy training data whose PPLDiff is extremely large after correction to boost the discriminatory power.
(3) Detecting and correcting NPs according to the PPLDiff obtained by the aligned surrogate LLM to obtain a denoised training dataset for robust alignment.
Comprehensive experiments validate that our proposed PerpCorrect can achieve state-of-the-art alignment performance under NPs.
Notably, PerpCorrect demonstrates practical utility by requiring only a modest amount of validation data and being compatible with various alignment techniques. 
Our code is available at [PerpCorrect](https://github.com/luxinyayaya/PerpCorrect).

Perplexity-aware Correction for Robust Alignment with Noisy Preferences

Abstract

Alignment techniques are critical in ensuring that large language models (LLMs) output helpful and harmless content by enforcing the LLM-generated content to align with human preferences. However, the existence of noisy preferences (NPs), where the responses are mistakenly labelled as chosen or rejected, could spoil the alignment, thus making the LLMs generate useless and even malicious content. Existing methods mitigate the issue of NPs from the loss perspective by adjusting the alignment loss based on a clean validation dataset. Orthogonal to these lossoriented methods, we propose perplexity-aware correction (PerpCorrect) from the data perspective for robust alignment which detects and corrects NPs based on the differences between the perplexity of the chosen and rejected responses (dubbed as PPLDiff). Intuitively, a higher PPLDiff indicates a higher probability of the NP because a rejected/chosen response which is mistakenly labelled as chosen/rejected is less preferable to be generated by an aligned LLM, thus having a higher/lower perplexity. PerpCorrect works in three steps: (1) PerpCorrect aligns a surrogate LLM using the clean validation data to make the PPLDiff able to distinguish clean preferences (CPs) and NPs. (2) PerpCorrect further aligns the surrogate LLM by incorporating the reliable clean training data whose PPLDiff is extremely small and reliable noisy training data whose PPLDiff is extremely large after correction to boost the discriminatory power. (3) Detecting and correcting NPs according to the PPLDiff obtained by the aligned surrogate LLM to obtain a denoised training dataset for robust alignment. Comprehensive experiments validate that our proposed PerpCorrect can achieve state-of-the-art alignment performance under NPs. Notably, PerpCorrect demonstrates practical utility by requiring only a modest amount of validation data and being compatible with various alignment techniques. Our code is available at PerpCorrect.

1 Introduction

Alignment enables the safe utilization of the remarkable capabilities acquired by large language models (LLMs) through self-supervised learning on vast corpora [6, 24, 4]. It refers to the process of ensuring that the contents generated by LLMs are helpful, harmless, and aligned with human values and preferences [19]. Reinforcement Learning from Human Feedback (RLHF) [9] has emerged as a primary technique for achieving alignment. Current technical routes [39, 40, 29] require a reward model to simulate human preference and use it to optimize the policy model outputs with

10

20

30

40

Proportion of noisy preferences (%)

60

70

80

90

Reward accuracy (%)

Performance of DPO series alignment methods

Vanilla DPO

cDPO

rDPO

PerpCorrect-DPO

(a) DPO series methods.

10

20

30

40

Proportion of noisy preferences (%)

86

88

90

92

94

96

Reward accuracy (%)

Performance of PPO series alignment methods

Vanilla PPO

cPPO

rPPO

PerpCorrect-PPO

(b) PPO series methods.

Proximal Policy Optimization (PPO) [27]. Current offline techniques such as Direct Preference Optimisation (DPO) [26], Sequence Likelihood Calibration with Human Feedback (SLiC) [38] and Identity-Preference Optimisation (IPO) [3], could directly align LLMs without intensively training a reward model as employed in RLHF.

Recent studies [34, 8] have shown there exist noisy preferences (NPs) that may lead to significant degradation in alignment performance. The issue of NPs, where the label of the actually chosen/rejected responses in training datasets is flipped as rejected/chosen, can arise from the biases of annotators [34] and malicious noise injection [5]. As shown in Figure 1, when NPs are randomly injected into the training dataset, the conventional alignment method (e.g., DPO [26] and PPO [9]) will have significantly degraded alignment performance measured by the reward accuracy. Such performance degradation could result in the generation of useless and even malicious content [34]. Therefore, it necessitates developing robust alignment methods that can utilize datasets with NPs to effectively align the LLMs with human preferences.

Existing robust alignment methods are proposed from the loss perspective, which adjust the alignment loss using a clean validation dataset to mitigate the issue of NPs. In particular, the conservative DPO (cDPO) [21] and robust-DPO (rDPO) [8] both estimate the proportion of NPs using the clean validation data via cross-validation and then adjust the original DPO loss based on the estimated proportion of NPs. However, Mitchell [21] and Chowdhury et al. [8] overlooked the essential differences between noisy and clean preferences, which is critical for mitigating the issue of NPs.

To this end, we propose Perp lexity-aware Correct ion (PerpCorrect) for robust alignment from the data perspective by leveraging the differences between noisy and clean preferences for robust alignment. PerpCorrect detects and corrects NPs based on the difference between the perplexity of the chosen response and that of the rejected counterparts (dubbed as PPLDiff) obtained by an aligned surrogate LLM using the clean validation set. If an NP is detected, PerpCorrect will correct it by flipping the label of the rejected/chosen responses as chosen/rejected. Intuitively, rejected responses which are mistakenly labelled as chosen have a higher perplexity since they are less consistent with human preferences and thus have a lower probability of being generated after alignment. Therefore, a higher value of PPLDiff indicates a higher probability of the preferences being noisy. In this way, PerpCorrect leverages the differences between noisy and clean preferences (CPs) identified by PPLDiff to detect NPs.

To make the PPLDiff able to distinguish CPs and NPs, PerpCorrect requires an aligned surrogate LLM for calculating PPLDiff. The density of PPLDiff obtained on the noisy training dataset using

an unaligned surrogate LLM, which can be fitted as a normal distribution centered around zero (evidenced in Figure 2a), cannot discriminate CPs and NPs. Therefore, we align a surrogate LLM using the clean validation data. The density of PPLDiff obtained by the aligned surrogate LLM in Figure 2b can be fitted into two distinguishable normal distributions, thus being able to differentiate CPs and NPs.

However, there still exists a large overlap between two normal distributions after aligning only on the clean validation dataset, which could result in an unsatisfactory accuracy of NP detection. To this end, we iteratively align the model using more reliable clean training data with extremely low PPLDiff (located in the green area in Figure 2c) and reliable noisy training data with extremely large PPLDiff (located in the red area in Figure 2c) sampled from noisy training datasets. Finally, the two normal distributions are significantly separated as shown in Figure 2d, which indicates that PPLDiff has an enhanced discriminatory power.

Benefiting from the strong discriminatory power of PPLDiff calculated by the aligned surrogate LLM, PerpCorrect outputs a denoised training dataset for robust alignment by first detecting NPs based on a PPLDiff threshold and then correcting them. The data, whose PPLDiff is below a certain threshold (i.e., the black dotted line in Figure 2d) selected as the x-coordinate of the two normal distributions' intersection, are identified as NPs and thus corrected by flipping the response's label. Notably, our proposed PerpCorrect is compatible with various alignment methods as well as robust alignment methods [21, 8] since the metric PPLDiff is agnostic to training algorithms and only requires an arguably small number of clean validation data (~50), thus making it practical.

Comprehensive empirical results, evaluated using the Llama2-7B [32] and phi-2 [20] models on the OpenAssistant Conversations (OASST1) [17] and Golden HH [7] datasets, validate the effectiveness of our proposed PerpCorrect method in robustifying alignment with NPs. We empirically validate that PerpCorrect consistently obtains state-of-the-art performance among various proportions of NPs. Besides, we empirically demonstrate that PerpCorrect can effectively robustify various alignment techniques and robust alignment methods, validating its compatibility.

2 Literature Review and Preliminary

In this section, we introduce the related work about LLM alignment and provide preliminaries about the noisy preferences, perplexity, as well as various alignment methods.

2.1 LLMAlignment

In the domain of aligning LLMs with human preferences, pairwise preference methods are favored due to their lower cognitive burden on evaluators. Traditional online alignment approaches [32, 24, 29] involve training reward models from these preferences to provide signals in reinforcement learning. Recent offline alignment methods like Direct Preference Optimization (DPO) [26], Sequence Likelihood Calibration (SLiC) [38], and Identify Preference Optimization (IPO) [3] streamlined this process by directly using preference pairs to train LLMs, thus enhancing performance and reducing computational costs. Additionally, methods like RRHF [37] align LLMs using multiple ranked preferences, Kahneman-Tversky Optimization (KTO) [13] align LLMs using a single preference labeled as good or bad, and Rejection Sampling Optimization (RSO) [18] address DPO's limitation in sampling preference pairs from the optimal policy through rejection sampling. However, NPs, arising from the biased human feedback, can determine the alignment performance [24, 34]. Robust alignment methods like conservative DPO (cDPO) [21], robust DPO (rDPO) [8] have been proposed to address these issues from the loss perspective. Our approach focuses on the data perspective to address these issues of NPs and is orthogonal to these robust alignment methods.

2.2 Preliminary

Noisy preferences (NPs). NPs refer to preference data in training datasets, whose label of the actually chosen/rejected responses is flipped as rejected/chosen. Let D = { ( x ( i ) , y ( i ) w , y ( i ) l ) } N i =1 be the preference dataset consisting of N ∈ N preference data points. For each preference data point ( x, y w , y l ) ∈ D , x is the prompt input to LLMs, y w is the chosen response, and y l is the rejected response. We let ˜ D = { ( x ( i ) , ˜ y ( i ) w , ˜ y ( i ) l ) } N i =1 be the noisy preference dataset (i.e., preference dataset

-

5

0

5

Density

PPLDiff

-

5

0

5

PPLDiff

Density

Left bound

Right bound

-

5

0

5

Density

Threshold

Clean

Noisy

PPLDiff

-

5

0

5

PPLDiff

Density

consisting noisy preferences) and denote preference data points that are not noisy as clean preferences (CPs). Following Chowdhury et al. [8], we obtain the noisy preference dataset ˜ D using the standard random noise model [23] with the probability ε ∈ (0 , 50%) to change the data point into noisy preferences, i.e.

Perplexity (PPL). PPL [15] measures the probability that the LLM generates a sentence. A lower PPL of a sentence indicates that the LLM has generated this sentence with a high probability. PPL is defined as the average negative log-likelihood of a sequence, i.e.,

where s is a sequence composed of t tokens and log π θ ( s i | s <i ) denotes the log-likelihood of the i -th token given the preceding tokens s <i calculated by an LLM π θ .

Technical details of alignment methods. There are usually three phases in RLHF pipeline [34, 26]: (1) supervised fine-tuning (SFT); (2) reward modeling; (3) reinforcement learning (RL) optimization. In the SFT phase, an LLM is fine-tuned via supervised learning on high-quality task-related data.

We denote the LLM after the SFT phase as π SFT . In the reward modeling phase, the reward model is introduced to simulate human preferences. Given a preference dataset, a reward model r ω ( x, y ) parameterized by ω , which takes prompt x and response y as input and outputs a real number representing the reward score, can be optimized via minimizing the following loss function:

where σ is the logistic function. In the RL optimization phase, the objective function is as follows:

where π θ ( y | x ) represents the probability that the LLM parameterized by θ > 0 generates the response y given the prompt x , π ref is a reference LLM to maintain the generation ability of the aligned model, and β is a hyper-parameter to ensure the similarity between π θ ( y | x ) and π ref ( y | x ) . We take π SFT as the reference LLM π ref following Ouyang et al. [24].

Recently, offline alignment methods directly leverages preferences in preference datasets, bypassing the need to learn a reward model in RLHF. The LLM parameters are optimized by minimizing the following loss function:

where the function G changes with the alignment method. To be specific, DPO [26] uses a BCE loss, SLiC [38] uses a hinge loss, and IPO [3] uses a square loss:

To mitigate the issue of NPs, cDPO [21] and rDPO [8] adjust the DPO loss based on the estimated proportion of NPs ε ′ using a clean validation dataset D val = { ( x ( i ) , y ( i ) w , y ( i ) l ) } N val i =1 consisting of N val ∈ N clean preference data points, i.e.

3 Perplexity-aware Correction for Robust Alignment

This section introduces Perp lexity-aware Correct ion (PerpCorrect) for robust alignment with NPs. In Section 3.1, we introduce a novel metric called PPLDiff and then illustrate the pipeline of PerpCorrect to detect and correct NPs based on PPLDiff. In Section 3.2, we demonstrate how to adapt our proposed PerpCorrect with various alignment methods to achieve robust alignment.

3.1 Perplexity-aware Correction (PerpCorrect)

In this subsection, we introduce PerpCorrect which employs a novel metric called PPLDiff as the foundation for detecting and correcting NPs. The algorithm of PerpCorrect is described in Algorithm 2.

PPLDiff. PPLDiff measures the difference between the PPL of chosen response and that of the rejected response. Given a preference data point ( x, ˜ y w , ˜ y l ) ∈ ˜ D sampled from the noisy training dataset ˜ D and an LLM π θ , PPLDiff is defined as follows:

where [ x ; y ] indicates the concatenation of the prompt x and the response y . Intuitively, if a data point is a clean preference, the PPL([ x ; ˜ y w ]; θ ) will be lower than PPL([ x ; ˜ y l ]; θ ) because the sequence [ x ; ˜ y w ] is more aligned with human values and thus has a higher probability of being generated by aligned LLMs. As a result, it PPLDiff will be lower compared to NPs, which PPL([ x ; ˜ y w ]; θ ) is higher than PPL([ x ; ˜ y l ]; θ ) . This difference allows us to distinguish CPs and NPs based on PPLDiff.

Aligning a surrogate LLM only using clean validation data. Here, we utilize a clean validation dataset D val to obtain an aligned surrogate LLM to make PPLDiff able to distinguish CPs and NPs. We empirically find that the PPLDiff values of CPs and NPs calculated by an unaligned LLM in the noisy training dataset were initially indistinguishable as shown in Figure 2a, making it impossible to differentiate the NPs from CPs. This is because an unaligned LLM lacks the necessary preferences to distinguish NPs and CPs.

Therefore, we introduce a surrogate LLM π θ ′ parameterized by θ ′ to replace the unaligned LLM and use it for calculating PPLDiff. We optimize the surrogate LLM π θ ′ using the clean validation dataset D val as follows:

After aligning the surrogate LLM, the PPLDiff values of NPs calculated by the surrogate LLM π θ ′ are significantly increased and those of CPs are significantly decreased, forming two distinct distributions as shown in Figure 2b. This is because the aligned surrogate LLM is trained to generate responses that align with human preferences, enhancing its ability to distinguish between NPs and CPs based on PPLDiff.

To separate CPs and NPs in the noisy training dataset without knowing the oracle preferences, we leverage the Levenberg-Marquardt (LM) algorithm to find two normal distributions that fit the density of PPLDiff calculated by the aligned surrogate LLM. Specifically, the LM algorithm returns the constants ¯ ε, ¯ µ, ¯ σ that satisfies the following condition:

Note that x is the PPLDiff value and h ( x | ¯ ε, ¯ µ, ¯ σ ) is the superposition of these two normal distribution. We denote f clean ( x | ¯ µ, ¯ σ 2 ) as the normal distribution fitting the PPLDiff of CPs and f noisy ( x | -¯ µ, ¯ σ 2 ) as the normal distribution fitting the PPLDiff of NPs since the PPLDiff of NPs is intuitively higher than that of CPs. In this way, we can obtain two distinguishable normal distributions to separate NPs and CPs as shown in the green and red dotted lines of Figure 2b without knowing the oracle preferences.

Further aligning the surrogate LLM using extra reliable training data from noisy training datasets. After aligning only using the clean validation datasets, the discriminatory power of the PPLDiff is still far from satisfactory because of the large overlap between the two normal distributions. Therefore, we align the surrogate LLM with more reliable training data to make the PPLDiff of CPs and that of NPs more separable. We iteratively align the surrogate LLM π θ ′ using more reliably clean training data whose PPLDiff is extremely small and reliably noisy training data whose PPLDiff is extremely large after correction by flipping the label of the response.

Specifically, at epoch t ∈ N , we select ( t -1) · α · | ˜ D| of the training data along with the clean validation data for further alignment where α ∈ (0 , 1) is the selection ratio and | ˜ D| = N is the number of data points in noisy training dataset. As shown in Lines 33-45 of Algorithm 2, the selected reliable training dataset D ′ t consists of ( t -1) · α · (1 -¯ ε ) · | ˜ D| reliably clean training data whose PPLDiff values are smallest ( t -1) · α · (1 -¯ ε ) percent and ( t -1) · α · ¯ ε · | ˜ D| reliably noisy training data after correction. Note that the reliably clean training data are the data points whose PPLDiff values are smallest ( t -1) · α · (1 -¯ ε ) percent (located in the green area of Figure 2c), and the reliably noisy training data whose PPLDiff values are largest ( t -1) · α · ¯ ε percent (located in the red area of Figure 2c) among all the training data points.

Detecting and correcting NPs based on PPLDiff to output a denoised training dataset. Based on the PPLDiff calculated by the aligned surrogate LLM, PerpCorrect detects and corrects NPs whose PPLDiff value is lower than a certain threshold. We take the x-coordinate of the intersection of the two normal distributions as the threshold (the black dotted line in Figure 2d). As shown in Lines 23-31, data points whose PPLDiff values are larger than this threshold are identified as CPs (the green area in Figure 2d), and other data points are identified as NPs requiring correction (the red area in Figure 2d). In this way, we can obtain a denoised training dataset for robust alignment.

Algorithm 1 Robust Alignment via Perplexity-aware Correction (PerpCorrect)

1: Input: Noisy training dataset ˜ D , clean validation dataset D val , and pre-trained LLM π θ parameterized by θ

2: Output: Robust alignment model π θ

3: // Stage I: Supervised fine-tuning (SFT)

4: π θ ← Supervised fine-tuned LLM π θ . (Details in Appendix C.3)

5: // Stage II: Perplexity-aware correction using the surrogate LLM

6: ˜ D denoised , ε ′ denoised ← Perplexity-aware Correction ( π θ , ˜ D , D val ) (Details in Algorithm 2)

7: // Stage III: Alignment with denoised dataset

8: π θ ← Aligned LLM π θ using ˜ D denoised and ε ′ denoised (Details in Appendix C.3)

Further, we select an optimal denoised training dataset to further enhance the performance of robust alignment according to the intersection area of the two normal distributions. We denote the intersection area of two normal distributions as the estimated NP proportion of the denoised training dataset, i.e.,

where ε ′ PC calculates the ratio of noisy data points which are not detected by PerpCorrect (i.e., the green area enclosed by the black and red lines in Figure 2d) and the clean data points which are mistakenly detected by PerpCorrect (i.e., the red area enclosed by the black and green lines in Figure 2d). In this way, ε ′ PC can efficiently calculate the NP proportion of the denoised training dataset. We take the denoised training dataset with the smallest ε ′ PC among multiple iterations as the optimal one for robust alignment to boost alignment performance.

3.2 Robust Alignment

Here, we introduce how to adapt PerpCorrect to robustify various alignment methods and demonstrate the algorithm of robust alignment via PerpCorrect in Algorithm 1. In general, the pipeline of the robust alignment based on PerpCorrect contains three stages: SFT, PerpCorrect, and alignment. We will first conduct SFT, following Christiano et al. [9], to boost the performance of a pre-trained LLM by boosting its skills for specific tasks. Next, we will conduct PerpCorrect to detect and correct NPs and output an optimal denoised training dataset ˜ D denoised the smallest ε ′ PC in Eq. 15. Finally, we can obtain an aligned LLM from the SFT model using the denoised training dataset ˜ D denoised via alignment (i.e., Line 8 in Algorithm 1).

Because our proposed PerpCorrect is agnostic to alignment methods and model structures, PerpCorrect is applicable to robustify both online alignment methods such as RLHF (PPO) [9] and offline alignment methods including DPO [26], SLiC [38], and IPO [3]. Besides, our proposed PerpCorrect is compatible with existing loss-oriented robust alignment methods, such as cDPO [21] and rDPO [8], based on the estimated proportion of NPs. Note that cDPO and rDPO require conducting computationally expensive cross-validation to tune the estimated proportion of NPs. We can efficiently estimate the proportion of NPs by utilizing the fitted normal distributions during PerpCorrect, i.e., ε ′ PC in Eq. 15. Therefore, we can combine PerpCorrect with a wide range of existing alignment methods to achieve robust alignment with NPs.

4 Experiments

In this section, we demonstrate that our proposed PerpCorrect achieves state-of-the-art alignment performance under different proportion of NPs and have good compatibility with other alignment methods. In Section 4.1, PerpCorrect combined with DPO [26] achieves state-of-the-art alignment performance than existing baselines (Section 4.1), including DPO [26], cDPO [21], and rDPO [8]. In Section 4.2, we further analyze the impact of the number of validation data and verified the compatibility of PerpCorrect with online and offline alignment methods and robust alignment methods. The training details and compute resources are reported in Appendix C.1.

Table 2: Average reward accuracy of PPO series alignment methods using Llama2-7B on the Golden HH dataset. The standard deviation of reward accuracy is reported in Table 9.

Datasets. We utilize two preference datasets, namely OpenAssistant Conversations (OASST1) [17] and Golden HH [7]. The processed OASST1 dataset comprises 17,939 training samples and 951 testing samples and the processed Golden HH dataset consists of 12,066 training samples and 654 testing samples. The description and processing details of these datasets are provided in Appendix C.2.

Models. Our evaluation leverages two distinct series of open-sourced LLMs with different parameter sizes: Llama2-7B [32] and phi-2 [20]. We acquire the checkpoints from their official repositories on Hugging Face. The LLMs used for PerpCorrect and those for robust alignment share the same model structure and initialization.

Baselines. We adopt vanilla DPO [26] and two robust alignment methods, cDPO [21] and rDPO [8], as baselines. For their detailed implementation, we utilize and adapt the transformers and TRL libraries provided by the Hugging Face community.

Metrics. In accordance with Chowdhury et al. [8], we employ the winning rate of policy generations against the selected preferences on the test dataset as our primary metric. This metric applies to vanilla DPO [26], cDPO [21], rDPO [8], as well as other offline alignment methods including SLiC [38] and IPO [3]. Additionally, we utilize the winning rate of the reward model score for the chosen preferences on the test dataset as our metric for vanilla PPO [24], cPPO [21, 34], and rPPO [8]. These two metrics are collectively called reward accuracy.

4.1 PerpCorrect Achieves the State-of-the-Art Robust Alignment Performance

The empirical results demonstrate that our method, PerpCorrect, achieves state-of-the-art robust alignment performance, surpassing existing baselines such as vanilla DPO [26], cDPO [21], and rDPO [8]. This is evident across various proportions of noisy preferences ε using different datasets and LLMs.

Comparison using different LLMs. Tables 1 and 3 demonstrate the average reward accuracy of the DPO series alignment methods on the Golden HH [7] dataset using Llama2-7B [32] and phi-2 [20]. At a proportion of the NPs ε = 40% , PerpCorrect increases the reward accuracy by 41.77% (from 53.15% to 94.92%) using Llama2-7B and by 41.41% (from 54.98% to 96.39%) using phi-2. The empirical result validates that our proposed PerpCorrect can be used on different sizes of LLMs and achieve better alignment performance than baselines.

Comparison on different datasets. Tables 3 and 4 demonstrate the average reward accuracy of the DPO series alignment methods on the Golden HH [7] and OASST1 [17] datasets using phi-2 [20].

The empirical results reveal a significant discrepancy in average reward accuracy between the more complex OASST1 dataset and the Golden HH dataset. The performance of other robust alignment methods is found to be unsatisfactory on the OASST1 dataset, often not surpassing the vanilla DPO. In contrast, our method PerpCorrect consistently maintains strong alignment performance across varying proportions of noisy preferences. In general, our method PerpCorrect can achieve better alignment performance than baselines across different datasets.

4.2 Ablation Study

Impact of the number of clean validation data. Table 5 illustrates the impact of the number of clean validation data points. We conducted experiments on the Golden HH dataset using Llama27B with a proportion of NPs ε = 40% . The empirical results indicate that as the number of clean validation data points increases, the performance of our method, PerpCorrect, also improves. However, when the number is too large, the improvement in performance is not obvious, and the cost of manual annotation significantly increases.

Compatibility with online alignment method RLHF (PPO). We adopt vanilla PPO [24], cPPO [21, 34], and rPPO [8] as baselines. Table 2 shows the alignment performance of PPO series alignment methods on the Golden HH [7] dataset using Llama2-7B. Although vanilla PPO has good performance when the proportion of NPs is low, it still declines significantly when the proportion is high. PerpCorrect maintains desirable alignment performances when the proportion of NPs is high. Our empirical results show that PerpCorrect has desirable compatibility with online alignment method RLHF (PPO).

Compatibility with various offline alignmet methods. Table 6 presents the average reward accuracy and improvements of original offline alignment methods compared to those combined with PerpCorrect. Our experiments, conducted on the Golden HH dataset using Llama2-7B, reveal that the reward accuracy of SLiC [38] and IPO [3] both significantly decrease as the proportion of NPs increases, similar to vanilla DPO [26]. However, our method PerpCorrect enhances their alignment performance across different proportions of NPs. Notably, IPO combined with PerpCorrect achieves

the best alignment performance. We conjecture the main reason is that the proportion of NPs in the denoised dataset is very low and IPO performs better than other methods under a low proportion of NPs. These empirical results demonstrate that our method has good compatibility with various offline alignment methods.

Compatibility with robust alignment methods. Table 6 shows the average reward accuracy and improvements of robust alignment methods compared to those combined with PerpCorrect. Our method, PerpCorrect, can significantly enhance the performance of cDPO [21], and provide a modest improvement for rDPO [8] under almost all proportion of NPs. The empirical results show that our method has good compatibility with robust alignment methods.

5 Conclusions

This paper proposes a method called perplexity-aware correction (PerpCorrect), as an effective approach for robust alignment with noisy preferences (NPs). PerpCorrect utilizes a surrogate LLM to calculate a novel metric, PPLDiff, and further detects and corrects NPs from clean preferences (CPs) based on it. PerpCorrect consists of three steps: (1) PerpCorrect aligns a surrogate LLM using the clean validation dataset, enabling PPLDiff to distinguish between CPs and NPs. (2) PerpCorrect enhances the discrimination power of PPLDiff by aligning the surrogate LLM with more reliable training data. (3) PerpCorrect detects and corrects NPs from CPs based on a calculated threshold and obtains a denoised training dataset. The paper further proposes a robust alignment pipeline, consisting of three stages SFT, PerpCorrect, and alignment, to achieve robust alignment with NPs. The experimental results validate that PerpCorrect achieves state-of-the-art alignment performance and has good compatibility with other online, offline, and robust alignment methods. Therefore, PerpCorrect can be an effective method to mitigate the impact of NPs and can be used for robust alignment. Future research directions include: (1) Improving the time efficiency of PerpCorrect and (2) Reducing the amount of clean validation data required to achieve the same alignment performance.

Limitations

We discuss some limitations of this work to stimulate further research in this direction. Our limitations mainly stem from two aspects: time efficiency issues caused by multiple calculations of PPLDiff and repeated training of a surrogate LLM, and the need for a validation dataset.

Time efficiency. Iteratively calculating the PPLDiff value for each data point and aligning a surrogate LLM is time-consuming. Selecting reliably training data and denoising the training dataset requires that the PPLDiff value be calculated for each data point during each epoch, which may cause unnecessary calculations for CPs and NPs that can already be clearly distinguished. Besides, aligning a surrogate LLM with same size as the LLM for alignment multiple times is time-consuming. The detailed discussion is in the Appendix B.

Validation dataset. PerpCorrect requires a validation dataset for aligning a surrogate LLM. However, manually annotating a validation dataset is complex and labor-intensive in practice. As shown in Table 5, there is a significant disparity in alignment performance when comparing the use of 10 clean samples to 50 clean samples. Exploring how to use fewer clean samples or even no clean samples to achieve the same or better performance is a problem worth further investigation.

References

01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by 01.ai, 2024. URL https://arxiv.org/abs/2403.04652 .

Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series of open language models, 2023. URL https://arxiv.org/abs/2311.16867 .

Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences, 2023.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022. URL https://arxiv.org/abs/2204.05862 .

Tim Baumgärtner, Yang Gao, Dana Alon, and Donald Metzler. Best-of-venom: Attacking rlhf by injecting poisoned preference data, 2024.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Tianchi Cai, Xierui Song, Jiyan Jiang, Fei Teng, Jinjie Gu, and Guannan Zhang. Ulma: Unified language model alignment with human demonstration and point-wise preference, 2024.

Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust DPO: Aligning language models with noisy feedback. In Forty-first International Conference on Machine Learning , 2024. URL https://openreview.net/forum?id=yhpDKSw7yA .

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.

Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview. net/forum?id=TyFrPOKYXw .

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems , 36, 2024.

Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V -usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 5988-6008. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/ethayarajh22a. html .

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization, 2024.

Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang. Chatglm: A family of large language models from glm-130b to glm-4 all tools, 2024.

Fred Jelinek, Robert L Mercer, Lalit R Bahl, and James K Baker. Perplexity-a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America , 62(S1): S63-S63, 1977.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https://arxiv.org/abs/2310.06825 .

Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, et al. Openassistant conversations-democratizing large language model alignment. Advances in Neural Information Processing Systems , 36, 2024.

Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview. net/forum?id=xbjSwwrQOe .

Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. Trustworthy llms: a survey and guideline for evaluating large language models' alignment, 2024.

Microsoft. Phi-2: The surprising power of small language models, 2023. URL https://www.microsoft.com/en-us/research/blog/ phi-2-the-surprising-power-of-small-language-models .

Eric Mitchell. A note on dpo with noisy preferences and relationship to ipo, 2023. URL https://ericmitchell.ai/cdpo.pdf .

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answering with human feedback, 2022. URL https://arxiv.org/abs/2112.09332 .

Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances in neural information processing systems , 26, 2013.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730-27744, 2022.

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/forum?id=HPuSIXJaa9 .

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.

Damien Sileo. tasksource: Structured dataset preprocessing annotations for frictionless extreme multi-task learning and evaluation. arXiv preprint arXiv:2301.05948 , 2023. URL https: //arxiv.org/abs/2301.05948 .

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems , 33:3008-3021, 2020.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca , 2023.

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozi´ nska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimczak-Pluci´ nska, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open language models at a practical size, 2024. URL https://arxiv.org/abs/2408.00118 .

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta,

Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https://arxiv.org/abs/2307.09288 .

Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, Songyang Gao, Nuo Xu, Yuhao Zhou, Xiaoran Fan, Zhiheng Xi, Jun Zhao, Xiao Wang, Tao Ji, Hang Yan, Lixing Shen, Zhan Chen, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. Secrets of rlhf in large language models part ii: Reward modeling, 2024.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL https://arxiv.org/abs/2407.10671 .

Jingwei Yi, Rui Ye, Qisi Chen, Bin Benjamin Zhu, Siheng Chen, Defu Lian, Guangzhong Sun, Xing Xie, and Fangzhao Wu. Open-source can be dangerous: On the vulnerability of value alignment in open-source LLMs, 2024. URL https://openreview.net/forum?id= NIouO0C0ex .

Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302 , 2023.

Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback, 2023.

Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, Limao Xiong, Lu Chen, Zhiheng Xi, Nuo Xu, Wenbin Lai, Minghao Zhu, Cheng Chang, Zhangyue Yin, Rongxiang Weng, Wensen Cheng, Haoran Huang, Tianxiang Sun, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, and Xuanjing Huang. Secrets of rlhf in large language models part i: Ppo, 2023.

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 , 2019.

A Broader Impacts

Our proposed PerpCorrect and robust alignment pipeline offers a solution for achieving state-of-theart performance in robust alignment under noisy preferences. PerpCorrect is designed to effectively reduce malicious noise in the dataset and mitigate biases introduced by human annotators, ensuring that the trained language model (LLM) is accurately aligned with true human preferences.

Moreover, we recognize a potential risk: if malicious users exploit our method for reverse training, they might compromise the security mechanisms of existing open-source LLMs. Existing research has demonstrated the possibility of reverse training [36].

B Time Efficiency Analysis

The additional computational overhead is primarily attributed to PerpCorrect (Section 3.1). Table 7 presents both theoretical and empirical runtime comparisons, where X represents the theoretical time required for Alignment (Section 3.2) or other baselines.

In theory, during the PerpCorrect process, we need to calculate PPLDiff and train the surrogate model in each epoch. The computation time introduced by PerpCorrect is approximately T 3 that of the Alignment or other baselines.

The calculation of PPLDiff in each epoch requires only 1 3 of the time needed for robust alignment. The primary computational load in robust alignment arises from the complexity of forwarding and back-propagation, while the complexities of gradient updates and parameter updates are relatively low. Additionally, back-propagation takes twice as long as forwarding. In addition, the calculation of PPLDiff only requires forwarding.

For surrogate model training, PerpCorrect utilized data points that represented t × α of the total dataset during epoch t . Since both t and α are small, the time required for surrogate model training can be approximately ignored.

In practice, Our entire robust alignment pipeline ( ∼ 24 hours) takes only twice as long as the baseline ( ∼ 12 hours). We set T = 5 and α = 2 , and used the AdamW optimizer. The practical efficiency of the PerpCorrect is due to the use of fp32 precision by the AdamW optimizer, which increases the GPU's calculation time during the robust alignment process.

C Implementation details

C.1 Training details and compute resources.

We utilized the Qlora method [11] for fine-tuning the LLMs, executed on RTX 4090 GPUs with 24 GB of memory. Hyperparameters were set as follows: lora_rank = 32 , lora_dropout = 0 . 1 , and lora_alpha = 16 . For SFT, we use the alpaca dataset [30] and set learning_rate = 2 e -4 and batch_size = 20 . For our PerpCorrect stage II, we set β = 0 . 1 , learning_rate = 1 e -3 , batch_size = 4 , T = 5 , and α = 0 . 02 . For our PerpCorrect stage III and all other alignment methods, we set β = 0 . 1 , learning_rate = 3 e -4 , and batch_size = 20 . Other details not mentioned, we follow the default setting in TRL library. Each experiment, involving a specific method and proportion of NPs, could be completed using a single RTX 4090 GPU within 24 hours on the Golden HH dataset and within 72 hours on the OASST1 dataset.

∼

C.2 Description and Processing Details of the Datasets

OpenAssistant Conversations Dataset (OASST1). The original OASST1 dataset [17] is an assistant-style conversation corpus generated and annotated by humans. It consists of over 10,000 fully annotated conversations in 35 different languages. Sileo [28] converted these conversations into a preference dataset comprising 17,966 training samples and 952 testing samples. After filtering out conversations with one or fewer letters, we obtained a preference dataset with 17,939 training samples and 951 testing samples.

Golden HH. The Golden HH dataset [7] is a variant of the Anthropic Helpful and Harmless (HH) dataset [4]. It originally contains 42,537 training samples and 2,312 testing samples as part of a preference dataset. Each sample has two keys: one representing the prompt x and the chosen response y w , and the other representing the prompt x and the rejected response y l . We first converted the dataset into a triple form: prompt x , chosen response y w , and rejected response y l , retaining only one-turn conversation data. After filtering out samples with one or fewer letters, we obtained a preference dataset with 12,066 training samples and 654 testing samples.

C.3 Detailed Robust Alignment via Perplexity-aware Correction

Supervised Fine-Tuning (SFT). The objective of Supervised Fine-Tuning (SFT) is to enhance the performance of a pre-trained large language model (LLM) by refining its abilities for specific tasks. As demonstrated by prior work [9, 25, 24], this can be achieved by utilizing supervised fine-tuning with a specialized dataset tailored to the target task. The SFT dataset is annotated with labels, providing examples that are directly relevant to the task. Specifically, for each data point ( x, y ) in the SFT dataset, x represents the prompt given to the LLM, and y represents the expected response that the model should generate based on the prompt x . The process involves fine-tuning the LLM by maximizing the log-likelihood of the correct responses y given the prompts x . Through this method, the model learns to produce more accurate and task-specific outputs, thereby significantly improving its performance on the given task.

Perplexity-aware Correction (PerpCorrect). We demonstrate the entire PerpCorrect algorithm in Algorithm 2.

Alignment. We can achieve alignment using the denoised training dataset ˜ D denoised with an estimated proportion of NPs ε ′ denoised . For offline alignment methods such as DPO, SLiC, and IPO, we can directly optimize the LLM using the denoised training dataset ˜ D denoised based on the loss functions defined in Eqs. 6-8. For loss-based robust alignment methods, including cDPO and rDPO, we set ε ′ = ε ′ denoised and then optimize the LLM using the denoised training dataset ˜ D denoised according to the loss functions mentioned in Eqs. 9 and 10. For the online alignment method RLHF (PPO), we first train a reward model using the denoised training dataset ˜ D denoised based on the loss function described in Eq. 3. Subsequently, we further optimize the LLM using PPO according to the objective function detailed in Eq. 4.

D Extended Experimental Results

D.1 Experiment Statistical Significance

Tables 8-12 demonstrates the standard deviation of the reward accuracy reported in Tables 1-4 and 6.

D.2 Average PPLDiff Values of Data from Different Datasets Calculated by Unaligned LLMs

We randomly selected 10,000 data points from each dataset and calculated PPLDiff using different LLMs. The datasets and LLMs are downloaded from the Huggingface website. The average PPLDiff values are reported in the Table 13.

Algorithm 2 Perplexity-aware Correction (PerpCorrect)

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our introduction covers our contributions, main methods and experimental results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discussed the efficiency issues and data volume requirements of our method PerpCorrect in the Conclusions section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We show all the experiment detail in the Experiments section and Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide open access to our code using Github.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We show the entire experimental details in the Experiments section and Appendix and provide open access to the code using Github.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the standard deviation in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide detailed sufficient information on the computer resources in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We follow the Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the potential impacts in Appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our robust alignment method does not have a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The models and datasets the we used are open-sourced, and we follow their license and terms of use.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Our training code are open-source on GitHub.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Perplexity-Aware Correction (PerpCorrect) to improve the alignment of large language models (LLMs) by detecting and correcting noisy preferences (NPs). PerpCorrect identifies NPs using the differences in perplexity (PPLDiff) between chosen and rejected responses. The method involves aligning a surrogate LLM with clean validation data, refining it with reliably clean and noisy data, and correcting NPs based on PPLDiff. Experiments demonstrate that PerpCorrect enhances alignment performance, is practical with a small amount of validation data, and is compatible with various alignment techniques.\n\nSTRENGTHS:\n1. This paper innovatively addresses the impact of noisy preferences on alignment from the perspective of correcting noisy data. \n\n2. The experiments are comprehensive, which cover existing online, offline, and robust alignment methods, such as PPO, DPO, cDPO, and rDPO. The results show that PerpCorrect achieves the state-of-the-art robust alignment preference evaluated with different LLMs and datasets. Besides, PerpCorrect can be further combined with the other loss-based robust alignment methods.\n\nWEAKNESSES:\n1. The standard deviation of reward accuracy is not reported in Tables 3, 4, and 6.\n\nQUESTIONS:\nI have two minor questions about the experiments:\n\n1. As shown in Tables 1 and 3, the reward accuracy of other DPO family alignment methods evaluated with Llama2-7B is better than that evaluated with phi-2 (2B). However, PrepCorrect performs better when evaluated on phi-2 (2B). Could you give some explanation for this phenomenon?\n\n2. As shown in Table 6, PerpCorrect shows significant improvement when combined with other methods, such as SLiC, IPO, and cDPO, but less improvement when combined with rDPO.  Could you give some analyses of this phenomenon?\n\nLIMITATIONS:\nThe paper explains its limitations in the appendix, such as time efficiency and the requirement for a validation dataset.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel method called PerpCorrect for robust alignment in large language models, particularly in the presence of noisy preferences in training data. PerpCorrect addresses noisy preferences by evaluating the perplexity difference (PPLDiff) between selected and rejected responses. By leveraging PPLDiff to identify and rectify these noisy preferences, PerpCorrect generates a denoised training dataset. Experimental results demonstrate that PerpCorrect achieves state-of-the-art alignment performance with only a small amount of validation data.\n\nSTRENGTHS:\n1. The paper is well-written, offering a comprehensive explanation of the proposed method, including detailed descriptions, mathematical formulas, and pseudocode. \n2. The proposed method can be integrated with both existing online and offline alignment methods, enhancing their robustness without requiring significant modifications to their core methodologies.\n3. PerpCorrect demonstrates strong performance with only a modest number of clean validation data points, which is beneficial in scenarios where such data is scarce or expensive to obtain.\n4. The proposed method has been empirically tested on several datasets and models, consistently showing superior performance over traditional methods in the presence of noisy data.\n\nWEAKNESSES:\n1. Tables 3 and 4 do not include the standard deviations for reward accuracy.\n2. Despite requiring fewer validation data points, its performance and accuracy still heavily depend on the availability of a high-quality clean validation dataset to initially train the surrogate LLM. Therefore, it is necessary to discuss potential solutions to this limitation.\n3. The process of calculating perplexity differences (PPLDiff) for each data point and aligning a surrogate LLM multiple times could be computationally intensive and time-consuming. Therefore, this additional cost needs to be discussed.\n\nQUESTIONS:\n1. Will the PPLDiff calculate on the new dataset, using models similar to those used with other data (e.g., Llama-2-7b-chat), result in a normal distribution with a mean of 0?\n\n2. What are the reasons that make PrepCorrect-IPO perform better than PrepCorrect-cDPO and PrepCorrect-rDPO? Please analyze this phenomenon.\n\nLIMITATIONS:\nAs discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

comments: SUMMARY:
The paper proposed a novel method to mitigate the preference noise in alignment. The authors first provide insights into how the PPLDiff can recognize the noisy preferences and then use the PPLDiff to select and correct noise preferences. Extensive experiments demonstrate that the method can significantly improve the robustness under high noise ratios.

STRENGTHS:
* The paper proposed a novel detect and correct paradigm for noise-robust alignment. 
* Novel insights were revealed for the failure with the noise preferences.

WEAKNESSES:
* Some writing is not clear.
   - In Line 50, "However, Mitchell [15] and Chowdhury et al. [6] overlooked the essential, differences between noisy and clean preferences, which is critical for mitigating the issue of NPs." What is the overlooked difference here? The logic is not clear.

QUESTIONS:
* In Line 50, "However, Mitchell [15] and Chowdhury et al. [6] overlooked the essential, differences between noisy and clean preferences, which is critical for mitigating the issue of NPs." What is the overlooked difference here? The logic is not clear.

LIMITATIONS:
No obvious limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Perplexity-aware Correction (PerpCorrect), a method for robust alignment of large language models (LLMs) with noisy preferences (NPs). PerpCorrect detects and corrects NPs by analyzing the perplexity difference (PPLDiff) between chosen and rejected responses. The approach involves aligning a surrogate LLM with clean validation data, iteratively refining with reliable training data, and ultimately producing a denoised training dataset. Experiments demonstrate that PerpCorrect significantly improves alignment performance and is compatible with various alignment techniques.\n\nSTRENGTHS:\n1. The problem of aligning LLMs with human preferences while effectively handling noisy data is a significant challenge in the field of AI, and this paper provides a valuable contribution towards solving it.\n\n2. This paper introduces PPLDiff, a novel metric for distinguishing between clean and noisy preferences, enhancing the precision of corrections. The proposed method effectively handles noisy preferences (NPs), improving the reliability of LLM alignment.\n\n3. PerpCorrect obtains significant improvements compared to previous methods, requiring only a modest amount of clean validation data, making it practical for real-world applications.\n\nWEAKNESSES:\n1. Implementing PerpCorrect involves additional steps of calculating and analyzing perplexity differences (PPLDiff), which can add complexity to the alignment process. Can the authors discuss the complexity and additional computation time of the method?\n\n2. The method requires a clean validation dataset to align the surrogate LLM initially. The need for manually annotated validation data can be labor-intensive and may not always be feasible in large-scale applications.\n\n3. The method assumes that the perplexity of clean and noisy preferences will differ consistently, which may not always hold true across all datasets and model configurations.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nLimitations have been discussed in Conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-14"}