Impact of Preference Noise on the Alignment Performance of Generative Language Models
Authors: Yang Gao
Dana Alon
Donald Metzler
Venue: COLM (2024)
Keywords: LM Alignment, Noisy Data, Preference Learning
TL;DR: We studied how noise in preference pairs affect the alignment performance, and how to mitigate its negative impact.

Abstract: A key requirement in developing Generative Language Models (GLMs) is to have their values aligned with human’s values. Preference-based alignment is a widely used paradigm for this purpose, in which preferences over generation pairs are first elicited from human annotators or AI systems, and then fed into some alignment techniques, e.g., Direct Preference Optimization. However, a substantial percent (up to 42%) of the preference pairs used in GLM alignment are noisy, and it remains unclear how the noise affect the alignment performance and how to mitigate their negative impact. In this paper, we propose a framework to inject desirable amounts and types of noise to the preferences, and systematically study the impact of preference noise on the alignment performance in two tasks (summarization and dialogue generation). We find that the alignment performance can be highly sensitive to the noise rates in the preference data: e.g., a 10 percentage points (pp) increase of the noise rate can lead to 30 pp drop in the alignment performance (in win rate). To mitigate the impact of noise, confidence-based data filtering shows significant benefit when certain types of noise are present.  We hope our work can help the community better understand and mitigate the impact of preference noise in GLM alignment.

Published as a conference paper at COLM 2024
Impact of Preference Noise on the Alignment Performance of Generative Language Models
Yang Gao, Dana Alon & Donald Metzler
Google DeepMind (gaostayyang, danama, metzler Ggoogle. com
manama,
Abstract
A key requirement in developing Generative Language Models (GLMs) is have their values aligned with human values. Preference-bused aligmment is a widely used paradigm for this purpose, in which preferences over generation pairs are first elicited from human annotators or AT systems, and then fed into some alignment techniques, e.g., Direct Preference Optimization. However, substantial percent (20 40%) of the preference pairs used in GLM alignment are noisy, and it remains unclear how the noise affect the alignment performance and how to mitigate their negative impact. In this paper, we propose framework to inject desirable amounts and types of noise to the preferences, and systematically study the impact of preference noise on the alignment performance in two tasks (summarization and dialogue generation). We find that the alignment performance can be highly sensitive to the noise rates in the preference data: e.g., 10 percentage points (pp) increase of the noise rate can lead to 30 pp drop in the alignment performance (in win rate). To mitigate the impact of noise, confidence- data filfering shows significant benefit when certain types of noise are present. We hope our work can help the community better understand and mitigate the impact of preference noise in GLM alignment.
1 Introduction
As the capabilities of Generative Language Models (GLMs) keep improving through pre- training at large scale, methods for aligning GLMs with human preferences, i.e., steering GLMs follow user instructions effectively and safely, have attracted increasing attention @ al. 2023). A widely used paradigm to align GLMs with human values is to first collect binary preferences on generation pairs, and then use techniques like Proximal Policy Optimization (a Reinforcement Learning algorithm, (Schulman et al.) 2017)), Direct Preference Optimization (DPO, (Rafailov et al. 2023)), or Sequence Likelihood Calibration (SLiC, (Zhao et al., 2023)) to align the GLMs with the collected preferences. The binary preferences human annotators, trained Reward Models (RMs, (Ouyang et al., 2022)), or Constitutional AT agents (Bai et al., 2022b Lee et al.) 2024). Preference-based GLM alignment has proven to be highly effective in improving the safety and usability of GLMs, and hence has been used
be provided by
to develop both open-source (Touvron et al., 2023
Gemma 2024) and proprietary (Google)
2023 Achiam et al. 2023) GLMs.
However, the binary preferences used in GLM alignment are often noisy, i.e, containing preferences that disagree with the ground truth (e.g,, preferences provided by domain experts). Zheng et al. (2023) report that 19-37% preferences provided by crowd workers are noisy. Similar noise rates are also observed in preferences provided by RMs and Constitutional AI. Table summarizes the noise rates of preferences used in recent GLM alignment works. It is generally believed that the lower the noise rates in the preferences, the better the final alignment performance (Lee et al.) 2024), but it remains unclear what is the quantitative relation between noise rates and alggmment performance, and, furthermore, how to mitigate the negative impact of preference noise on alignment performance. In this paper, we answer these questions with systematic empirical studies.
1

---

Published as a conference paper at COLM 2024
Oracle
Task
Noise
Reference
Human
MTBench
19-37
Zheng al.
2023
MTBench
15-34
Zheng et al
2023
TL;DR
22
Lee al.
2024
Constitutional AI
CBArena
22-36
Zheng al. 2023
AntHH
27.9-30,9
Lee 2024]
MetaHS
41.4-419
louvron et al.
2023
TL;DR
21.3-27
Zhao et al. 2023
Munos et al. 2023
SHP
26.3
Cw et al 2025
Reward Models
MetaHS
35.5-36.8
louvron et 2023
WebGPT 34.8
(Cu et 2023
Table 1: Preference noise are observed in a wide range of tasks, including video game (Atari), QA (MTBench, StanfordHumanPrrfeeee Summarization (TL;DR), and Dialogue (WebGPT, ChatBotArena, AntropicHeipfulHarmees, MetaHelpfulSafety).
Generation
Preference
Alignment Techniques
Aligned GLM
Provide Preforences
Evaluate
Inject Simulated Noïse
Noisy Preference Cracle
Gold Preference Oracle
Figure 1: Our framework for evaluating the impact of preference noise on GLM alignment.
There exist frameworks for studying the influence of preference noise on alignment perfor- mance in Robotics, e.g,, the B-Pref framework by Lee et al. (2021a). They assume a gold reward model is available, and design different strategies corrupt the gold reward model to provide (simulated) noisy preferences (see for more details). Although they have been successfully used benchmark and compare different preference-basee
RL algorithms in Robotics tasks, they are not applicable for GLM aligmment, mainly for two reasons: (i) some of their noise simulation strategies are unsuitable for NLP tasks; and (ii) the alignment techniques used in Robotics (e.g., PREBBLE (Lee et al., 2021b)) are different from those used in GLM (e.g., DPO and SLiC). To alleviate these problems, we propose new framework whose noise-simulation strategies and alignment techniques are tailored for GLM alignment. Figure Tigureeilluae iillustrates the framework.
With the proposed framework, perform controlled experiments to study the impact of preference noise on alignment performance on two tasks, summarization (Stiennon et al., 2020) and dialogue generation (Bai al. (022a). We find that even with high (45%) noise rates, GLM alignment is still beneficial (i.e., yielding 50%+ win rate). However, alignment performance also highly sensitive to noise rates: a percentage points (pp) increase of noise rates can lead to 30 pp drop in the alignment performance (in terms of win rate). We also explore different strategies to mitigate the negative impact of preference noise on align- ment performance. We find that some widely used regularization methods fail to mitigate data selection can effectively improve performance
the negative impact, but confidence-bue in realistic settings. We hope findings help GLM developers better understand the impact of preference noise on alignment performance, and that our framework can facilitate the exploration of more effective and noise-robust alignment methods.
2

---

Published as a conference paper at COLM 2024
2 Related Works
Preference- GLM alignment. Pairwise preferences are widely used in Al alignment, because controlled experiments have suggested that asking for preferences places lower cognitive burden on the human subjects than asking for absolute ratings or categorized labels (Kendall 1948: Thurstone| 2017 Kingsley& & Brown, 2010). In GLM alignment, common practice is to first train an RM from the human provided preferences, and then use the RM to provide reward signals in Reinforcement Learning (RL) (Böhm et al. 2019 Gao et al. 2020; Stiennon et al. 2020). Recent methods like DPO Rafailov et al. 2023), SLIC (Zhao et 2023), and Identify Preference Optimization (IPO, Azar et al. (2023) go one
step further, by eliminating the RM training step and directly using preference pairs to train
the final GLM. Azar et al. (2023) has shown that these methods essentially optimize the same learning objective, and they differentiate in their regularization terms and function
approximation methods. Compared to the RM-RL two-stage paradigm, DPO and SLiC yield stronger performance in multiple NLP applications with lower computational costs
(Chen et al. 2024 Yuan et al. 2024).
Learning from noisy data. Data used in real-world machine learning applications are often noisy (Sng et al. 2022)), and deep neural models are particularly sensitive to data noise, as they are prone to overfit to noise patterns in the training data (Han et al., 2020). Hence, multiple methods have been proposed to improve the robustness of the neural models, mostly falling into four categories (Frénay& & Verleysen 2014 Song et al. 2022): robust neural architectures, regularizalion methods, robusi loss functions, and dala fillering meds In this paper, we stick to the well established architecture (Transformers by Vaswani al. (2077) and loss functions (DPO), and explore different regularization and data selection methods to mitigate the negative impact of noisy preferences on the alignment performance.
The impact of noisy data on RL has also been studied. Lee et al. (0211)) propose the B-Pref benchmark, platform to test the performance and robustness of preference-based RL algorithms in the face of different types of preference noise on various locomotion and robotic manipulation tasks. To simulate realistic preference noise, they assume they have access to gold-standard reward model r", and design five strategies to derive noisy prefer- ences therefrom, including the stochastic strategy (p(yo > y1) = ofr" (yo) - r" (y1)], where o is the sigmoid function such that (xx = (1 + exp( (<p(-xx)-1), myopic strategy (providing preferences only based on the last part of the presented candidates), skipping
sktiping strategy (reject to provide preferences if both candidates are low-quality), equally-preferable strategy (when the quality of the presented candidates are similar, mark them tie) and random mistake strategy (randomly flip the correct preference direction with a fixed chance). We note that some of these strategies are unrealistic GLM alignment, e-g., the myopic strategy (users usually do not judge the quality of texts based on their last parts), skipping strategy, and equally-preferable strategy (ties are usually not allowed in annotating text qualities). Also, they only consider noise from human annotators but ignore those from Al-based annotators (e.g., RLAIF (Lee al. 2024)). ). For these reasons, we propose a new set of strategies for simulating preterence noise in GLM alignment in
3 Preliminaries
Let be the set of all prompts, and yx be the set of all possible continuations for all prompts in X. We assume there exists gold reward model r" X' x Vx - RR, which measures the quality of continuation y E Vx for prompt E X' on some desired aspects (e.g., helpfulness, informativeness, or harmlessness) A GLM can be defined as a policy π,, such that ( x)) is the probability of generating for the input prompt x. The objective of GLM alignment is to find the optimal policy that can maximize the expected gold reward value while minimizing the divergence from reference policy:
max π
Σ
[y(yx)) 1i1i--DDellll 7
(y]x)),
(1)
where B is a hyperparameter, Dxz. is the Ballbackk-eiblle divergence, and Ket is the reference policy.
3

---

Published as a conference paper at COLM 2024
In practice, cannot optimize Eq. directly, because the gold reward r* is usually inaccessible, and the sunmation operation is prohibitively expensive. Multiple algorithms have been proposed to obtain (approximate) solutions for the objective (see (2) In this work, we use DPO (Rafailov et al.) 2023) because of its strong performance and lower computational cost compared to other methods (e.g., PPO). The loss function in DPO is:
To(Mu|x) TTstt (y [x)
(7(0) (xxg log
B Blog(
(2)
TTn (y)lx))
where nσ is the learnable policy parameterized by 8, is the sigmoid function, and D {(x', you 1f) 31)
)i= is the training dataset consisting of data entries. Each data entry in D
consists of a prompt x E X and two continuations y vi E Vx, such that you is preferred over 1/-
has been proven that if DD is sufficiently large and all pairs in D are noise-free (i.e., r" (y > r" (v)) for = 1,... Rt), the policy learned by DPO is (near-)optimal with respect to Eq. ①1] RRafailov et al. [023 Azar et 20233. However, in practice, some preferences in D can be noisy, Le,, different from the preterence direction induced by the gold reward model. In this work, we remove the (strong) noise-free assumption on D, but instead introduce different rates and types of noise to D (in $4) and empirically study their impact on the quality of Ty (in 6
To measure the alignment performance (i.e., measure the performance of π&), we compute the win rate between To and Tistt
Σ (x, > r (x,
w
Listt
x
where Krest C X' is a held-out test prompt set, and ynu and ynar are generations sampled from Tg and Tist, respectively.
4 Noisy Preferences
The preferences are often noisy, i.e, disagree with the preference directions induced by the gold reward model r-. Inspired by past works (Lee et al. (021a) (see for more discussions), we consider three oracles to provide different types noisy preferences.
Random Noise Oracle. When presented with a prompt x E X' and pair of responses Yw, Vt E Vx, the oracle has (100- n) % chance to return the correct pref- erence (i.e., r" (ywl)) > r" (yrlx)) but has ' chance to return the incorrect/ incorrect/flipped preferences. We can control the noise rate of this oracle by adjusting the value of n.
Stochastic Noise Oracle. For prompt and two responses Ywu V/, Stochastic Noise Oracle prefers yu over yt with the probability o[(r" (yw) - r" (ry(())r //r]) where o is the sigmoid function, and y E R- is the temperature hyperparameter. We can control the noise rate by tuning the y value: The higher the y value, the more unpredictable the oracle is, and hence more noisy the preferences will be.
Gaussian Noise Oracle. Stochastic Noise Oracle requires access to the gold reward model, which is infeasible in practice. Gaussian Noise Oracle, instead, only requires
the access to an approximated reward model r', such that r (y(lx) r ryyxx) x)) +
e, where is the noise term drawn from a Gaussian distribution (u, The preference directions are then derived from the approximated reward r. With r' the probability of Gaussian Noise Oracle prefers you over yj is:
ply. > y1(x)
= 1 -
(y))
= -1((r" (y + cw) (r)()))))
=1 1((""() -(,)x))))() + (Ew - e))].
Since both en and Ej are drawn from the same Gaussian distribution (n, Ew é is a random variable drawn from N (0,,282) Hence, the noise rate of Gaussian Noise Oracle can be adjusted by tuning the value of 6.
4

---

Published as a conference paper at COLM 2024
We believe the three strategies cover some widely observed noise types in preferences. For example, Stochastic Noise Oracle is also known as Boltzmann rational (Ziebart et al. 2008 Jeeon et al. 2020 Gao et al., 2020) and widely used for simulating noise in human-provided
prelerences caused by aleatoric uncertainty (Hüllermeier & Waegeman 2021). Gaussian Noise Oracle simulates the noise caused by the epistemic uncerlainty, I.e., the RMs fail to accurately approximate the human's preferences. Random Noise Oracle simulates the H random mistakes observed in both human-provided (Lindner & El-Assady 2022) and heuristic-based preferences (Chen et al. 2002)). We note that in reality, multiple types of noise can co-exist in the preference pairs; however, ease analyses, in this paper, we assume there exist at most one type of noise in the preferences. We leave mixed-type preference noise for future work.
Experimental Setup
Tasks. We consider preference-based GLM alignment on two tasks: Reddit TL;DR (Stien- non et al. 2020) and Anthropic-Helpful (Bai et al. 2022a). Reddit TL,DR has two subsets, SFT set and preference set. In its preference set, each data entry consists of doc- ument x and two candidate summaries Ye, yt for It has 93k/53k /33k data entries in boiin/vlidation/ test splits. Anthropic-Helpfhe
is subset of the AnthropicHH dataset, in which each data entry contains the dialogue history between human and an AI assistant (x), and two candidate responses (yw, 3/). has 161k/9k data entries in train/ test splits, and separate out 1k aandomly-sample entries from the train set as the validation set.
Generative Language Model. For each task, we fine-tune a T5-Large (770M parameters) model to obtain the initial GLM Stt For TL;DR, we use the SFT subset in Reddit TL;DR as the SFT training data, which has 117k/6k/66 examples in train/ hiann//aliaiiionn test splits. For Anthropic-eelpful, we use all the preferred responses in the dataset the SFT training data. All the hyperparameters used in SFT are the same as in (Liu et al. 2024).
Gold Reward Model r*. For each task (TL;DR and Anthropic-Helpfull)) we train a T5-XXL (11B parameters) [Raffel et al. 2020) model with the respective preference pairs to build the gold reward model r" In line with (Zhao et al., 2023), we format the input to the model with the prompt: [CONTEXT] {x} [RESPONSE] point-wise score for the reply.
{y), and use the logit of the token 1 as
Noisy Preference Oracles. Based on the gold reward model r* described above, we build the noisy preference oracles as described in $4 The noise rate of different noisy preference oracles can be controlled by tuning their respective hyperparameters: л (noise rate) for Random Noise Oracle, (temperature) for Stochastic Oracle, and å (standard deviation) for Gaussian Noise Oracle. To decide the exact values of the hyperparameters for target noise rate (e.g., 20%), we randomly sample 1 examples from each train set, and increase the hyperparameters with a small step (0.01) until the target noise rate is reached. The final hyperparameter values are presented in Table 2
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
TL;DR
?
0.2
0.4
0.65
0.9
1.25
1.75
2.50
3.90
8.0
100
8
0.34
0.70
1.09
1.55
2.15
2.90
4.05
6.50
13.0
100
Anthropic
?
0.11
0.22
0.36
0.53
0.75
1.06
1.54
2.45
4.95
100
0.18
0.40
0.62
0.89
1.25
1.75
2.49
4.00
8.75
100
Table 2: Hyperparameters for the Stochastic Noise (7) and Gaussian Noise Oracle (8) at each target noise rate (column). The hyperparameter for Random Noise Oracle (ıı; see (4) is omitted, it equals the corresponding target noise rate.
Generation Pairs. In line with Liu et al. (2024), we use the trained GLM Tist to sample responses for each prompt, and pair up the sampled responses build the Generation Pairs
5

---

Published as a conference paper at COLM 2024
TL:DR
Anthrop H-Hllll
Stochast choise
RandomNclie
0.9
0.0
GeussienNiise
0.8
Vinn Mtte
0.7
0.6
0.0
Stcchsticlotre RondomMolle
0.5
Gaussianttuis
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Roise Rate
Roise Rate
Figure 2: Influence of the noise rate (x-axis) on the alignment performance (in terms of the win rate; y-axis). Error bars: 95% confidence intervals, computed with double-tailed t-test on 5-10 repeated experiments with different random seeds.
dataset (see Fig. 1 For each prompt x, we sample eight generations from Tan(11x) with temperature 0.7, and randomly group them into four pairs. The pairs are then presented to the noisy preference oracles to build the Preference Pairs dataset (see Fig,
Other Hyperparameters. We choose the hyperparameters by following the choices made in (Rafailov et al. 2023 Liu et al. 2024 in training the gold reward model and GLM si
we use batch size 32 and learning rate Ie-5 with Adafactor optimizer (Shazeer & Stern 2018); in the alignment training stage (Eq. 2 we use B 0.5, and dropout rate 0.1. Later
we will explore different values for the regularization weights (B and dropout rate) to study their effectiveness in mitigating the negative effect of noisy preferences.
6 Impact of Noise Rates on Alignment Performance
Fig. 2 presents how the alignment performance changes with the growth of the noise rates. We make the following observations.
Alignment performance drops with more noise in preferences. This applies to all types of noise and both tasks have considered. Also, we note that the alignment performance drops more quickly with the increase of the noise rates: When the noise rates below 0.3, an increase of 10 percentage points (pp) in noise rate yields less than 10pp drop in the alignment performance; however, when the noise rates are higher than 0.4, 10pp increase in noise can yield 20-30 pp loss in performance.
Different types of noise cause similar harm. At the same noise rate, the alignmment performance of the three different noise types do not have significant differences, suggesting that it the noise rate rather than the noise type that decides the alignment performance.
Alignment is beneficial even with highly noisy preferences. In both tasks, the win rate is above 0.5 even with noise rate 0.45. This observation reaffirms the effectiveness of alignment training (Touvron et al. 2023), and explains why even highly noisy preferences are used in aligrument training in practice (see Table But it is also worth noting that when the preferences are completely random (i.e., noise rate at 50%), the win rate drops below 0.5, suggesting that alignment is detrimental with random preferences.
7 Mitigate the Negative Impact of Preference Noise
In this section, we explore two popular strategies to mitigate the negative impact of prefer- ence noise: regularization in $7.1 and data filtering in 4.2

---

Published as a conference paper at COLM 2024
Rardem Noho 71
Zrachasti Ndsss T.
Gzunder Neaz L.00
3.
04
10
8-32
Sandom Nelbs on Anthrogtt (Mlohd
Stechattic Nobo on Apthroplc Hiepptll
Coursion Rebto Antropii eich
2w10
04
Nebe Fart
Figure 3: Alignment performance with different KL regularization weights B.
Rarddm L.RR
Stachastic holps
G3: 5515 Nsice Le
3.5
0.1
03-
07
0.
3.-
36-
40-62 Haanik
68-2 (iioeiee)
35-
6red1
Dandom Netbs on Anthrogl: (Miohd
Stechattic Noke in Apmrogiic Hilpful
Couosion Rebo on Antrrpii eich
2-2
04
5051
Fare
Figure 4: Alignment performance with different dropout rates (dor).
7.1 Regularization
We consider two methods to strengthen regularization: increasing the weight of the KL divergence loss (B in Eq. (2), and increasing the dropout rate.
Fig. preeenns the alignment performance with different strengths of KL regularization. In general, we find that stronger KL regularization fails to mitigate the negative impact of pref- erence noise. In some cases (e.g., Random and Stochastic Noise in TL;DR), higher strength of KL regularization even hurts the performance. Our finding reaffirms the limitations of KL-based regularization in DPO/SLiC (Azar et al. 2023).
Fig. presents the alignment performance with different dropout rates. We find that higher dropout rates significantly harm the alignment performance. To summarize, our findings suggest that high strength of regularization, in general, fails to mitigate the negative impact of preference noise, and in certain cases can even hurt the alignment performance.
7

---

Published as a conference paper at COLM 2024
7.2 Data Filtering
Another approach to fight against noise is to filter out noisy data in the Preference Pairs dataset (see Fig. [ and only use the remaining data to perform alignment. We use the popular confidence- data filtering method (Cheng et al. 2008). For a prompt x and two candidate responses ywz y1, the confidence of Y/u preferred over y1, denoted c(yw > y)lx), is real value between 0 and 1. Ideally, the confidence function c should be well-calibrated (Silaa Filho et al. 2023), i.e., c(yw > x) is identical the true probability of yo preferred over V/- In confidence-based data filtering, only pairs with the confidence level larger than a pre-defined threshold are used in training; hence, if threshold f 0.5, no filtering is performed; t 1 means all data will be removed. We experiment with f 0.5, 0.6, 0.9 to investigate the effect of different levels of data filtering.
Based on the user studies made by Gao et al. (2020) we use the Bradley & Terry 1952 model to estimate the confidence value: clyw yi x) = (y)l)())))))) xx] ----r"- ["]]]>]]] where o' is the sigmoid function. In practice, there are multiple methods for estimating the confidence values, e.g., conformal predictors (Shafer & Vovk 2008 Einbinder et al.) 2022), ensemble
methods (Liang et al. 2022), and bias estimation methods (Chen et al.
Note that with higher filtering threshold I, the quality of the remaining data is improved at the cost of quantity loss. We deliberately do not back-fill the filtered data, because in practice it can be prohibitively expensive to collect more preference pairs. This setup also allows us to study the trade-off between data quality and data quantity in preference-based alignment. Fig. 6 in Appendix(A AA]s/xssssws how the size of the remaining data shrinks with higher confidence thresholds. We find that in both datasets, the data size drops quite quickly with the growth of values: Almost 20% data are filtered as f increases by 0.1.
Fig. 5 presents the alignment performance with different strengths of data filtering. We make the following observations.
Data filtering does not help to fight against Random Noise. This is because Ran- dom Noise Oracle flips pairs completely at random (Frénay& & Verleysen 2014), i.e, the flipping chance of each pair is uniformly at random, not affected by any other factors (e.g., the prompt x or the responses ye, v1). As result, our confidence-bar filtering cannot reduce the number of the noisy pairs in the filtered data, and hence fails to improve the performance.
Data filtering is effective to mitigate the harm from Stochastic and Gaussian Noise. We find that with certain threshold (e.g., at 0.8), data filtering shows con- sistent and significant improvement across all noise rates, noise types, and tasks. Considering that over 50% preference pairs have been removed with data filtering at confidence threshold 0.8, this result suggests that data quality has a significant impact on the alignment performance.
Over-aggressive data filtering hurts the performance. With very high confidence thresholds (e.g., 0.99), the alignment performance is compromised across all noise rates, noise types, and tasks, due to the severe loss of data size. Hence, it is important to properly trade off between data quality and data quantity, in order to yield the optimal alignment performance.
To better understand how data filtering improves the data quality, we investigate the noise rate before and after data filtering, with different noise types. Fig. in Appendix A shows the noise rates in data filtered with different confidence thresholds. We find that when the noise is from Random Noise Oracle, the noise rate stays the same with all data filtering thresholds; this explains why data filtering does not help improve its performance. When the noise is from Stochastic or Gaussian Oracles, data filtering can effectively reduce noise rate, explaining the performance boost observed in Fig.
8 Limitations & Future Work
Generalizability. We apply a popular alignment algorithm (DPO) to T5-based language models in our experiments. We believe our observations presented in 16 and 7 can be
8

---

Published as a conference paper at COLM 2024
Rardem Noho 71
Mochaai Reaz un TL.11
0.2
370
0.5
0.7
015
02
Tandom Nolos on Anthroott (klofd
Vortadit (f.)
Courtion Robto on Antreppic eicht
0.2
038
132
0.
075
3.4
Datdine
Vatre
Hebe fate
Figure 5: Alignment performance with confidence-based data filtering.
generalized to other alignment methods (e.g., PPO, SLiC and IPO,, because they essentially
optimize the same objective function; see 2 other Transformer (Vaswani et al., 2017) based language models, and other datasets (e.g., UltraFeedback (Cui et al. 2023) and AlpacaEval (Dubois al. 2024)), ) but this is yet to be empirically verified. Considering the large number of possible combinations between alignment techniques and language models, a systematic study beyond the scope of this paper, and we leave it for future work.
Mixture of Noise. We consider three strategies to add noise to the preferences, but we only allow one strategy to be used at one time (see 44.. In practice, there may exist other types of noise, and different types of noise can co-exist in preferences. Our framework allows for creating new noise types by mixing primitive noise types, but it remains unclear whether the current observations can be applied to the new noise types or not. We hope our work can facilitate and encourage more work on this direction.
Data Filtering. Our current experiments demonstrate that data filtering can improve performance on the in-domain test set (see 7..).. However, it remains unclear whether this performance gain extends to out-of-domain test sets. A thorough understanding of data filtering's 's impact on cross-domain generalization requires meticulous experimental design. This includes careful consideration of factors such as the selection of training and testing tasks, and the degree of difference between them. We call for further research in this area. Additionally, the training set used and filtered in the current work is synthetic (see Section D Investigating the effectiveness of data filtering on human gener-eeeeeated datasets would be valuable next step.
9 Conclusion
Pairwise preferences are widely used for aligning Generative Language Models (GLMs) with human values, but it remains unclear how the noise in preferences affect the alignment performance, and how to mitigate their negative impact. To study these problems, we propose a framework in which the types and rates of noise can be simulated and controlled, and we perform systematic experiments on two generation tasks (summary and dialogue generation) with this framework. Our findings suggest that alignment performance can be highly sensitive to the increase of noise rates, and appropriate data filtering is the most effective method to mitigate the negative impact of noisy preferences. Our work builds the first quantitative relation between noise rates and alignment performance across different noise types. We hope our work can help the community better understand and mitigate the impact of preference noise in GLM alignment.

---

Published as a conference paper at COLM 2024
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arviv preprint ar arXiv:2303.08777 2023.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. arXiv preprint ar Xir::2330..222 12036, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. ar preprint arXiv:2204.05862, 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitu- tional ai: Harmlessness from ai feedback. arXiv preprint arXio:2212.0807.. 2022b.
Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. Better rewards yield better summaries: Learning to summarise without references. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNI.P 2019, Hong Kong, China, Nowember 3-7, 2019, pp. 11108-3112 Association for Computational Linguistics, 2019. doi: 10.18653 /111//V1111111//01 URL httpp:///oii ort. 1005://0/000011116///////1--0000
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, ():33/4)324-45, 1952.
Lin Chen, Michal Lukasik, Wittawat Jitkrittum, Chong You, and Sanjiv Kumar. It's an alignment, not trade-off: Revisiting bias and variance in deep models. arXiv preprint a#Xiv:2310.09250, 2023.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine- tuning converts weak language models to strong language models. arXiv preprint Xii=:2401.01335, 2024.
Reynold Cheng, Jinchuan Chen, and Xike Xie. Cleaning uncertain data with quality guaran- tees. Proceedings of the VLDB Endowment, 1(1):722-735, 2008
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high- quality feedback. arv preprint ar arXim:2300.01377 2023.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024.
Bat-Sheva Einbinder, Yaniv Romano, Matteo Sesia, and Yanfei Zhou. Training uncertainty- aware classifiers with conformalized deep learning, Advances in Neural Information Pro- cessing Systems, 35:22380- 22395, 2022.
Benoît Frénay and Michel Verleysen. Classification the presence label noise: A survey. IEEE Trans. Neural Networks Learn. Syst., 25(5):845 869, 2014. doi: 2292894. httphttts///oioorgge
110111109/HHN TNNLS. 2013. 2292894
Yang Gao, Christian M. Meyer, and Iryna Gurevych. Preference-based interactive multi-document summarisation. Inf. Retr. J., 23())555--885 2020. doi: 10.1007/ S10791-019-09367-8. URL htt.hhtttt://oll org/ 10. 10007//07991-11--08866
Gemma. Gemma: Open models based on gemini research and technology. Technical report, Google DeepMind, 2024.
10

---

Published as a conference paper at COLM 2024
Gemini Team Google. Gemini: A family of highly capable multimodal models. arviv preprint arXio:2312.11805, 2023.
Bo Han, Quanming Yao, Tongliang Liu, Gang Niu, Ivor W. Tsang, James T. Kwok, and Masashi Sugiyama. survey of label-noise representation learning: Past, present and future. CoRR, abs/ aabs/201...m 2011.04406, 2020. bttpss//://arxx org/
04406
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110:457-506, 2021.
Hong Jun Jeon, Smitha Milli, and Anca Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. Advances in Neural Information Processing Systems, 33:4415 2020.
Jiaming li, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai alignment: A comprehensive survey, arXiv preprint Xrv::23310 2023.
Maurice George Kendall. Rank correlation methods. 1948.
David C Kingsley and Thomas C Brown. Preference uncertainty, preference learning, and paired comparison experiments. Land Economics, 86(3)5300-544, 2010.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop,
Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. In International Conference on Learning Representations, 2024.
K Lee, L Smith, A Dragan, and P Abbeel. B-pref: Benchmarking preference-based reinforce- ment learning. Neural Information Processing Systems, 2021a.
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficienn interactive reinforce- ment learning via relabeling experience and unsupervised pre-training. arXiv preprint ##Xiv:2106.0001, 2021b.
Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for exploration in preference-basee reinforcement learning. arXiv preprint arXio:2205.12401, 2022.
David Lindner and Mennatallah El-Assady. Humans are not boltzmann distributions: Chal- lenges and opportunities for modelling human feedback and interaction in reinforcement learning. Xro preprint Sttt::2006.33114 2022.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. In International Conference on Learning Representations, 2024.
Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. arXiv preprint ar arXiv:2312.00886, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances Neural Information Processing Systems, 35:77330-7744 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Thirty-sevnnth Anmual Conference on Neural Information Processing Systems, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-555 2020.
11

---

Published as a conference paper at COLM 2024
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.. arXio preprint urXiv:1707.06344 2017.
Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(),, 2008.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp- 4596-46600 4604. PMLR, 2018.
Telmo Silva Filho, Hao Song, Miquel Perello-Nieto, Raul Santos- MonntosRodriguez, Meelis Kull, and Peter Flach. Classifier calibration: a survey on how to assess and improve predicted class probabilities. Machine Learning, 112(9))3211-326) 2023.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3022 2020.
Louis L Thurstone. law of comparative judgment. In Scaling, pp. 81-92. Routledge, 2017.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama Open foundation and fine-tuned chat models. ar preprint ar #XXv:2307.092883 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arviv preprint arXiv:2401.10026, 2024.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic- hf: Sequence likelihood calibration with human feedback. arv preprint arXio:2305.10425, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph Gonzalez, and Ion Stoica. Judging llm-as-a-judee with mt-bench and chatbot arena. In Neural Information Processing Systems, Datasets and Benchmarks Track, 2023.
Brian D Ziebart, Andrew L Maas, Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume pp. 1433-1438. Chicago, IL, USA, 2008
A Appendix
12

---

Published as a conference paper at COLM 2024
TL.DR
Anthropic-Helfful
60
80
60
60
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.3
0.0
10
Corfidence Threshold
Confidence Threshold
Figure 6: With higher data-selection threshold, fewer data will be remained for training.
13

---

Published as a conference paper at COLM 2024
DR
cnthropp--ttelpful
10
Stochastic Noise
Random Noise
10
Stochestic Noise
Rendom Noise
Gouszion
Cx0.7
120.6
TLDDR
Anthrapic-Helpful
Randem
Random Norse
Goursion Noise
Gaurdian Noise
1206
150
ra0.8
call
con
r20-7
T1,OR
Hetprup
30
Random Noise
30
Random Noise
Cauasian Noise
Gauasien Noise
20
20
10
10
CH0.
CH0.5
C+0.7
CH0.8
CA0.9
CH0.5
C-0.7
CH0.8
CH0.95
TTDDR
Anthropic-Helpful
Rendom Noise
40
Rendum Noive
Stachastic Noise
Stochastic Noise
35
25
Gaussion
90
30
25
25
70-
Figure 7: Noise rates of the filtered data, with different data filtering threshold. Note that when the threshold is 0.5, no data is filtered. Here the original/ unfiltered data has 10% (first row), 20% (second row), 30% (third row), and 40% (fourth row) noise.
14


Reviews:
[{"review_id": "giKQhncDCH", "reviewer": "Reviewer_zDfF", "summary": "Authors present an empirical investigation about (1) the influence of noise in preference data on the model performance after alignment fine-tuning, and (2) what strategies could alleviate the degradation. First they describe three ways of injecting noise into preference data produced by their gold reward model such that those ways align with real world noise we may find in datasets we use. Second, they present an analysis of performance degradation as the amount of noise go up across multiple tasks and noise schedules. Finally, they investigate training regularization as well as data filtering and conclude that some methods might alleviate degradation but not completely. However, they highlight that even noisy preferences may still improve model's alignment compared to the reference model.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your reviews and feedback.\n\n* **Why a small model: We acknowledge the limitations of using a smaller model (T5 770M).** This choice was made in line with recent work (Liu et al. 2024, “​​Statistical rejection sampling improves preference optimization”), and due to computational constraints. To conduct our statistical analysis rigorously, numerous experiments were required, making this the only model feasible within our resources. We recognize the value of exploring larger models, and will further highlight the significance of this research direction in the revised manuscript..\n* **Why TLDR and Anthropic-Helpful**: These datasets were chosen for their prevalence in preference alignment studies. While TLDR focuses on summarization, the Anthropic-Helpful dataset includes a substantial number of instruction-following examples (c.f., Bai et al. 2022, “Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback”), aligning with the instruction-following aspect of alignment. We agree that running experiments and evaluations on OpenAssistant2 and AlpacaEval would offer valuable insights. We will acknowledge this limitation of the current work, and encourage future research in this direction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* this is a valuable analysis that is helpful for LLM community working on alignment. Provided filtering technique might be adopted and extended in future work.\n\n* paper has a clear presentation and is easy to read and understand", "reasons_to_reject": "* GLM model authors consider is pretty small (<1B) w.r.t. the parameters count. It feels that authors chose T5 family of models for some reason for both RM and GLM. Experimental results could have larger scope if e.g a llama3 8B or llama2 7B (if larger models are infeasible to train for authors) have been considered.\n\n* general instruction following benchmarks such as alpaca eval is not considered in the experiments while it would be interesting to see the trends on winrates when numbers would correspond to some leaderboard.", "questions_to_authors": "Following on rejection reasons:\n\n- why did you choose T5 770M model as the main GLM model ? Especially given other way more up-to-date LLMs available. \n\n- why did you choose reddit TLDR instead of some more up to date tasks / benchmarks e.g. open assistant 2? Especially since alignment is tightly connected to instruction following tasks rather than summarization.", "extra_scores": {}}, {"review_id": "Cvv7viIREN", "reviewer": "Reviewer_2iR8", "summary": "This paper investigates the impact of noise in preference data on the alignment performance of Generative Language Models (GLMs) and proposes methods to mitigate this impact. The authors provide a systematic study across two tasks—summarization and dialogue generation—highlighting how even slight increases in noise can significantly degrade performance. They introduce a novel framework for injecting and controlling noise types and levels in training data, leading to a detailed empirical analysis.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your reviews and feedback.\n\n* **Why a small model on two specific tasks**: Please refer to our responses to Reviewer zDfF. \n* **Mixed-Type Noise**: We completely agree that exploring mixed-type noise is a crucial next step, as we've emphasized in Section 8. The primary goal of this paper was to establish a robust framework for evaluating the impact of noise on alignment performance. To streamline our initial analysis, we focused on single noise types. A comprehensive study of mixed-type noise would ideally involve a thorough analysis of the types of noise in real-world preference datasets and their proportions, and it is beyond the scope of this work. We will further highlight the significance of this research direction in the revised manuscript.\n* **Potential Harms of Data Filtering**: In our current experiments, the training and testing data are drawn from similar domains, and our results demonstrate that appropriate data filtering does not negatively impact in-domain generalization. However, to thoroughly understand the effects of data filtering on cross-domain generalization would require meticulous experiment design, considering factors such as the selection of training and testing tasks, and the degree of difference between them. While beyond the scope of this paper, we will acknowledge this as a limitation and advocate for future work in this area.\n* **Why Regularization Doesn't Work**: We hypothesize that the two regularization methods we employed (KL divergence and Dropout) indiscriminately remove signals from both \"useful\" and \"noisy\" preferences. Consequently, while they mitigate the harm caused by noisy preferences, they also diminish the effectiveness of clean preferences. Regularization in deep learning is a complex field, and we only utilized the most common techniques in this paper. We will encourage the readers to try other regularization methods (e.g., the ones discussed in Section III-B in \"Learning from Noisy Labels with Deep Neural Networks: A Survey\", Song et al. 2022). Also, given the importance of Language Models and the prevalence of noisy data in RLHF, we will explicitly call for the development of more tailored regularization methods specifically designed for RLHF in future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "（1）The paper is well-structured, and the topic is highly relevant given the increasing reliance on GLMs for various applications.\n\n（2）The paper introduces a novel framework for simulating preference noise and rigorously evaluates the impact using well-defined metrics and experimental setups.\n\n（3）The results clearly demonstrate the sensitivity of GLMs to preference noise and offer practical solutions for mitigating these effects, providing valuable insights.", "reasons_to_reject": "(1) The experiments are limited to two specific tasks and one type of GLM (T5). Additional experiments with other models and tasks could help generalize the findings more broadly.\n\n(2) While the paper presents data filtering as an effective mitigation strategy, the exploration of regularization methods could be expanded, and other potential strategies could be considered.", "questions_to_authors": "(1) The paper handles single types of noise separately. How might the proposed methods perform under conditions of mixed noise types, which are more likely in real-world settings?\n\n(2) While data filtering improves alignment performance, could it also potentially lead to overfitting or reduced model robustness due to reduced training data variability?\n\n(3) Could the authors elaborate on why increased regularization was not effective? Are there other regularization techniques that could potentially yield better results in combating preference noise?", "extra_scores": {}}, {"review_id": "1DV3DrWLQ8", "reviewer": "Reviewer_9Tmq", "summary": "The paper investigates the impact of preference noise on the alignment of GLMs and proposes confidence-based data filtering to mitigate this noise. In controlled experiments, three types of artificial noise are inserted on preferences in the tasks of summarisation and dialogue generation. The authors find that 1) even high noise rates an still be beneficial for alignment and 2) that, contrary to regularisation, confidence-based data filtering improves performance.\nThe paper presents a small but focused contribution, with a firmly-states research question. It doesn't only investigate noise, but proposes mitigation solutions. Even though the scope is limited to two tasks, one model, and one noise type at a time, it can provide initial insights into the effect of preference noise and its mitigation.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the review and feedback. We appreciate your recognition of our contributions and the clear presentation of our work. We will address the formatting issue with the figures to ensure they are color-blind-friendly and accessible in grayscale. We will do this for all figures in the final version if the paper is accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is clearly-written, with a small but novel, focused contribution\n- Controlled experiments with different noise types sufficiently answer the research question", "reasons_to_reject": "- None, since the limitations are acknowledged in the paper.", "questions_to_authors": "- Please change the line format in the graphs so that the different factor can be discerned by the colour blind and when printing in grey scale.", "extra_scores": {}}, {"review_id": "mrtTMfmP51", "reviewer": "Reviewer_3A7y", "summary": "The paper studies a problem of noise in the preference alignment data for language models and its effect on the LLMs performance on 2 tasks, summarization and dialogue response generation. The authors use a controlled environment where they control the amount and the type of noise injected in the preference data and study the corresponding models' performance in terms of win rate of the aligned model vs the SFT model. The authors work with 3 types of noise: Random noise oracle (uniformly distributed noise across the dataset with a given ratio), Stochastic noise oracle (proportional to the reward difference between the preferred and dispreferred example) and Gaussian noise oracle (modelling the approximated rewards as stochastic gaussian variables).\nHaving found that noise in preference data does impact the alignment performance (but aligning even on the highly noisy data is beneficial for the model), the authors also conduct a study of performance degradation mitigation strategies in the settings of noisy preferences. Out of the two techniques they try, KL-based regularization and confidence-based data filtering, the latter results in superior robustness while not weakening the learning performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your reviews and feedback. \n\nWe agree that applying confidence-based data filtering on real unaugmented data is a crucial next step in understanding the practical applicability and effectiveness of our proposed method. We believe it would involve:\n* **Confidence Estimation**: Developing robust methods to accurately gauge the confidence level of each preference pair in a real-world dataset. Now, in our simulated experiments, we estimate the confidence by assuming that we have access to the gold reward, but in practice, the confidence estimation involves leveraging model uncertainty, human annotations, or a combination of both.\n* **Threshold Selection**: Determining the optimal confidence thresholds for filtering data, so as to trade off between data quality and quantity.\n* **Human Evaluation**: Conducting comprehensive evaluations with human experts to assess the impact of confidence-based filtering on the quality of model outputs.\n\nGiven the scope and complexity of these steps, we believe that it warrants a dedicated investigation, potentially as a separate research paper. In our revised manuscript, we will explicitly acknowledge this as a limitation and highlight the need for further research in this direction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A comprehensive focused study of the effect of preference noise on the LM performance is conducted\nAn interesting and useful result is presented on mitigating the harm of preference noise: while regularization does not always help and decreases the alignment efficiency, data selection based on confidence addresses most of the noise types studied", "reasons_to_reject": "The paper has contributions on the analysis side (the experimental studies themselves), modeling side (model training technique with confidence data selection), and resource side (noise datasets/framework). Out of the 3, I see the analysis contribution the most substantial - if analysis papers are welcomed at the conference, that works as a reason to accept, if not - this is a reason to reject.\n\nAnother thing that would add much substance to this work and is missing from the paper in my understanding is trying the confidence-based data filtering on a real unaugmented dataset with a known presence of preference noise and evaluating the resulting performance difference with a human study.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates the impact of noise in preference data on the performance of LLMs after alignment fine-tuning. It explores three noise injection methods and analyzes their effects on summarization and dialogue generation tasks. The study finds that noise degrades performance but aligning with noisy preferences can still improve over a reference model. It also evaluates mitigation strategies, concluding that confidence-based data filtering provides better robustness without compromising learning performance.

The reviewers appraise the paper being well-written and easy-to-follow, the method to inject noises being interesting and novel, and the comprehensive analysis being informative and instrument for the community.
Decision: Accept