Impact of Preference Noise on the Alignment Performance of Generative Language Models

Authors: yang  gao
dana  alon
donald  metzler
google  deepmind
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-20 17:57:06

Abstract: A key requirement in developing Generative Language Models (GLMs) is to have their values aligned with human’s values. Preference-based alignment is a widely used paradigm for this purpose, in which preferences over generation pairs are first elicited from human annotators or AI systems, and then fed into some alignment techniques, e.g., Direct Preference Optimization. However, a substantial percent (up to 42%) of the preference pairs used in GLM alignment are noisy, and it remains unclear how the noise affect the alignment performance and how to mitigate their negative impact. In this paper, we propose a framework to inject desirable amounts and types of noise to the preferences, and systematically study the impact of preference noise on the alignment performance in two tasks (summarization and dialogue generation). We find that the alignment performance can be highly sensitive to the noise rates in the preference data: e.g., a 10 percentage points (pp) increase of the noise rate can lead to 30 pp drop in the alignment performance (in win rate). To mitigate the impact of noise, confidence-based data filtering shows significant benefit when certain types of noise are present.  We hope our work can help the community better understand and mitigate the impact of preference noise in GLM alignment.

Main paper: 
 introduction: 
 As the capabilities of Generative Language Models (GLMs) keep improving through pretraining at a large scale, methods for aligning GLMs with human preferences, i.e., steering GLMs to follow user instructions effectively and safely, have attracted increasing attention (Ji et al., 2023) . A widely used paradigm to align GLMs with human values is to first collect binary preferences on generation pairs, and then use techniques like Proximal Policy Optimization (a Reinforcement Learning algorithm, (Schulman et al., 2017) ), Direct Preference Optimization (DPO, (Rafailov et al., 2023) ), or Sequence Likelihood Calibration (SLiC, (Zhao et al., 2023) ) to align the GLMs with the collected preferences. The binary preferences can be provided by human annotators, trained Reward Models (RMs, (Ouyang et al., 2022) ), or Constitutional AI agents (Bai et al., 2022b; Lee et al., 2024) . Preference-based GLM alignment has proven to be highly effective in improving the safety and usability of GLMs, and hence has been used to develop both open-source (Touvron et al., 2023; Gemma, 2024) and proprietary (Google, 2023; Achiam et al., 2023) GLMs.However, the binary preferences used in GLM alignment are often noisy, i.e., containing preferences that disagree with the ground truth (e.g., preferences provided by domain experts). Zheng et al. (2023) report that 19-37% preferences provided by crowd workers are noisy. Similar noise rates are also observed in preferences provided by RMs and Constitutional AI. Table 1 summarizes the noise rates of preferences used in recent GLM alignment works. It is generally believed that the lower the noise rates in the preferences, the better the final alignment performance (Lee et al., 2024) , but it remains unclear what is the quantitative relation between noise rates and alignment performance, and, furthermore, how to mitigate the negative impact of preference noise on alignment performance. In this paper, we answer these questions with systematic empirical studies. There exist frameworks for studying the influence of preference noise on alignment performance in Robotics, e.g., the B-Pref framework by Lee et al. (2021a) . They assume a gold reward model is available, and design different strategies to corrupt the gold reward model to provide (simulated) noisy preferences (see §2 for more details). Although they have been successfully used to benchmark and compare different preference-based RL algorithms in Robotics tasks, they are not applicable for GLM alignment, mainly for two reasons: (i) some of their noise simulation strategies are unsuitable for NLP tasks; and (ii) the alignment techniques used in Robotics (e.g., PREBBLE (Lee et al., 2021b) ) are different from those used in GLM (e.g., DPO and SLiC). To alleviate these problems, we propose a new framework whose noise-simulation strategies and alignment techniques are tailored for GLM alignment. Figure 1 illustrates the framework.With the proposed framework, we perform controlled experiments to study the impact of preference noise on alignment performance on two tasks, summarization (Stiennon et al., 2020) and dialogue generation (Bai et al., 2022a) . We find that even with high (45%) noise rates, GLM alignment is still beneficial (i.e., yielding 50%+ win rate). However, alignment performance is also highly sensitive to noise rates: a 10 percentage points (pp) increase of noise rates can lead to 30 pp drop in the alignment performance (in terms of win rate). We also explore different strategies to mitigate the negative impact of preference noise on alignment performance. We find that some widely used regularization methods fail to mitigate the negative impact, but confidence-based data selection can effectively improve performance in realistic settings. We hope our findings can help GLM developers better understand the impact of preference noise on alignment performance, and that our framework can facilitate the exploration of more effective and noise-robust alignment methods. 
related works: 
 Preference-based GLM alignment. Pairwise preferences are widely used in AI alignment, because controlled experiments have suggested that asking for preferences places a lower cognitive burden on the human subjects than asking for absolute ratings or categorized labels (Kendall, 1948; Thurstone, 2017; Kingsley & Brown, 2010) . In GLM alignment, a common practice is to first train an RM from the human provided preferences, and then use the RM to provide reward signals in Reinforcement Learning (RL) (B öhm et al., 2019; Gao et al., 2020; Stiennon et al., 2020) . Recent methods like DPO (Rafailov et al., 2023) , SLiC (Zhao et al., 2023) (Chen et al., 2024; Yuan et al., 2024) .Learning from noisy data. Data used in real-world machine learning applications are often noisy (c.f., Song et al. (2022) ), and deep neural models are particularly sensitive to data noise, as they are prone to overfit to noise patterns in the training data (Han et al., 2020) . Hence, multiple methods have been proposed to improve the robustness of the neural models, mostly falling into four categories (Frénay & Verleysen, 2014; Song et al., 2022) : robust neural architectures, regularization methods, robust loss functions, and data filtering methods.In this paper, we stick to the well established architecture (Transformers by Vaswani et al. (2017) ) and loss functions (DPO), and explore different regularization and data selection methods to mitigate the negative impact of noisy preferences on the alignment performance.The impact of noisy data on RL has also been studied. Lee et al. (2021a) propose the B-Pref benchmark, a platform to test the performance and robustness of preference-based RL algorithms in the face of different types of preference noise on various locomotion and robotic manipulation tasks. To simulate realistic preference noise, they assume they have access to a gold-standard reward model r * , and design five strategies to derive noisy preferences therefrom, including the stochastic strategy (p(y 0 > y 1 ) = σ[r * (y 0 )r * (y 1 )], where σ is the sigmoid function such that σ(x) = (1 + exp(-x)) -1 ), myopic strategy (providing preferences only based on the last part of the presented candidates), skipping strategy (reject to provide preferences if both candidates are low-quality), equally-preferable strategy (when the quality of the presented candidates are similar, mark them as a tie), and random mistake strategy (randomly flip the correct preference direction with a fixed chance). We note that some of these strategies are unrealistic in GLM alignment, e.g., the myopic strategy (users usually do not judge the quality of texts based on their last parts), skipping strategy, and equally-preferable strategy (ties are usually not allowed in annotating text qualities). Also, they only consider noise from human annotators but ignore those from AI-based annotators (e.g., RLAIF (Lee et al., 2024) ). For these reasons, we propose a new set of strategies for simulating preference noise in GLM alignment in §4. 
preliminaries: 
 Let X be the set of all prompts, and Y X be the set of all possible continuations for all prompts in X . We assume there exists a gold reward model r * : X × Y X → R, which measures the quality of continuation y ∈ Y X for prompt x ∈ X on some desired aspects (e.g., helpfulness, informativeness, or harmlessness). A GLM can be defined as a policy π, such that π(y|x) is the probability of generating y for the input prompt x. The objective of GLM alignment is to find the optimal policy that can maximize the expected gold reward value while minimizing the divergence from a reference policy:EQUATIONwhere β is a hyperparameter, D KL is the Kullback-Leibler divergence, and π sft is the reference policy.In practice, we cannot optimize Eq. ( 1) directly, because the gold reward r * is usually inaccessible, and the summation operation is prohibitively expensive. Multiple algorithms have been proposed to obtain (approximate) solutions for the objective (see §2). In this work, we use DPO (Rafailov et al., 2023) because of its strong performance and lower computational cost compared to other methods (e.g., PPO). The loss function in DPO is:EQUATIONwhere π θ is the learnable policy parameterized by θ, σ is the sigmoid function, and D = {(x i , y i w , y i l )} n i=1 is the training dataset consisting of n data entries. Each data entry in D consists of a prompt x i ∈ X and two continuations y i w , y i l ∈ Y X , such that y i w is preferred over y i l . It has been proven that if D is sufficiently large and all pairs in D are noise-free (i.e., r * (yi w ) > r * (y i l ) for i = 1, • • • , n), the policy learned by DPO is (near-)optimal with respect to Eq. ( 1) (Rafailov et al., 2023; Azar et al., 2023) . However, in practice, some preferences in D can be noisy, i.e., different from the preference direction induced by the gold reward model. In this work, we remove the (strong) noise-free assumption on D, but instead introduce different rates and types of noise to D (in §4) and empirically study their impact on the quality of π θ (in §6).To measure the alignment performance (i.e., measure the performance of π θ ), we compute the win rate between π θ and π sft :EQUATIONwhere X test ⊂ X is a held-out test prompt set, and y π θ and y π sft are generations sampled from π θ and π sft , respectively. 
noisy preferences: 
 The preferences are often noisy, i.e., disagree with the preference directions induced by the gold reward model r * . Inspired by past works (Lee et al., 2021a) (see §2 for more discussions), we consider three oracles to provide different types of noisy preferences.• Random Noise Oracle. When presented with a prompt x ∈ X and a pair of responses y w , y l ∈ Y X , the oracle has (100n)% chance to return the correct preference (i.e., r * (y w |x) > r * (y l |x)), but has n% chance to return the incorrect/flipped preferences. We can control the noise rate of this oracle by adjusting the value of n. • Stochastic Noise Oracle. For a prompt x and two responses y w , y l , Stochastic Noise Oracle prefers y w over y l with the probability σ[(r * (y w )r * (y l ))/γ], where σ is the sigmoid function, and γ ∈ R + is the temperature hyperparameter. We can control the noise rate by tuning the γ value: The higher the γ value, the more unpredictable the oracle is, and hence more noisy the preferences will be. • Gaussian Noise Oracle. Stochastic Noise Oracle requires access to the gold reward model, which is infeasible in practice. Gaussian Noise Oracle, instead, only requires the access to an approximated reward model r , such that r (y|x) = r * (y|x) + , where is the noise term drawn from a Gaussian distribution N (µ, δ 2 ). The preference directions are then derived from the approximated reward r . With r , the probability of Gaussian Noise Oracle prefers y w over y l is:p(y w > y l |x) =1[r (y w |x) -r (y l )] =1[(r * (y w |x) + w ) -(r * (y l |x) + l )] =1[(r * (y w |x) -r * (y l |x)) + ( w -l )].Since both w and l are drawn from the same Gaussian distribution N (µ, δ 2 ), wl is a random variable drawn from N (0, 2δ 2 ). Hence, the noise rate of Gaussian Noise Oracle can be adjusted by tuning the value of δ.We believe the three strategies cover some widely observed noise types in preferences. For example, Stochastic Noise Oracle is also known as Boltzmann rational (Ziebart et al., 2008; Jeon et al., 2020; Gao et al., 2020) and widely used for simulating noise in human-provided preferences caused by aleatoric uncertainty (H üllermeier & Waegeman, 2021). Gaussian Noise Oracle simulates the noise caused by the epistemic uncertainty, i.e., the RMs fail to accurately approximate the human's preferences. Random Noise Oracle simulates the random mistakes observed in both human-provided (Lindner & El-Assady, 2022 ) and heuristic-based preferences (Chen et al., 2024) . We note that in reality, multiple types of noise can co-exist in the preference pairs; however, to ease analyses, in this paper, we assume there exist at most one type of noise in the preferences. We leave mixed-type preference noise for future work. 
experimental setup: 
 Tasks. We consider preference-based GLM alignment on two tasks: Reddit TL;DR (Stiennon et al., 2020) and Anthropic-Helpful (Bai et al., 2022a) . Reddit TL;DR has two subsets, a SFT set and a preference set. In its preference set, each data entry consists of a document x and two candidate summaries y w , y l for x. It has 93k/53k/33k data entries in train/validation/test splits. Anthropic-Helpful is a subset of the AnthropicHH dataset, in which each data entry contains the dialogue history between a human and an AI assistant (x), and two candidate responses (y w , y l ). It has 161k/9k data entries in train/test splits, and we separate out 1k randomly-sampled entries from the train set as the validation set.For each task, we fine-tune a T5-Large (770M parameters) model to obtain the initial GLM π sft . For TL;DR, we use the SFT subset in Reddit TL;DR as the SFT training data, which has 117k/6k/6k examples in train/validation/test splits.For Anthropic-Helpful, we use all the preferred responses in the dataset as the SFT training data. All the hyperparameters used in SFT are the same as in (Liu et al., 2024) .Gold Reward Model r * . For each task (TL;DR and Anthropic-Helpful), we train a T5-XXL (11B parameters) (Raffel et al., 2020) dataset (see Fig. 1 ). For each prompt x, we sample eight generations from π sft (•|x) with temperature 0.7, and randomly group them into four pairs. The pairs are then presented to the noisy preference oracles to build the Preference Pairs dataset (see Fig. 1 ).Other Hyperparameters. We choose the hyperparameters by following the choices made in (Rafailov et al., 2023; Liu et al., 2024) : in training the gold reward model and GLM π sft , we use batch size 32 and learning rate 1e-5 with Adafactor optimizer (Shazeer & Stern, 2018) ; in the alignment training stage (Eq. ( 2)), we use β = 0.5, and dropout rate 0.1. Later in §7, we will explore different values for the regularization weights (β and dropout rate) to study their effectiveness in mitigating the negative effect of noisy preferences. 
impact of noise rates on alignment performance: 
 Fig. 2 presents how the alignment performance changes with the growth of the noise rates. We make the following observations.• Alignment performance drops with more noise in preferences. This applies to all types of noise and both tasks we have considered. Also, we note that the alignment performance drops more quickly with the increase of the noise rates: When the noise rates are below 0.3, an increase of 10 percentage points (pp) in noise rate yields less than 10pp drop in the alignment performance; however, when the noise rates are higher than 0.4, 10pp increase in noise can yield 20-30 pp loss in performance.• Different types of noise cause similar harm. At the same noise rate, the alignment performance of the three different noise types do not have significant differences, suggesting that it is the noise rate rather than the noise type that decides the alignment performance.• Alignment is beneficial even with highly noisy preferences. In both tasks, the win rate is above 0.5 even with noise rate at 0.45. This observation reaffirms the effectiveness of alignment training (Touvron et al., 2023) , and explains why even highly noisy preferences are used in alignment training in practice (see Table 1 ). But it is also worth noting that when the preferences are completely random (i.e., noise rate at 50%), the win rate drops below 0.5, suggesting that alignment is detrimental with random preferences. 
mitigate the negative impact of preference noise: 
 In this section, we explore two popular strategies to mitigate the negative impact of preference noise: regularization in §7.1 and data filtering in §7.2. We consider two methods to strengthen regularization: increasing the weight of the KL divergence loss (β in Eq. ( 2)), and increasing the dropout rate.Fig. 3 presents the alignment performance with different strengths of KL regularization. In general, we find that stronger KL regularization fails to mitigate the negative impact of preference noise. In some cases (e.g., Random and Stochastic Noise in TL;DR), higher strength of KL regularization even hurts the performance. Our finding reaffirms the limitations of KL-based regularization in DPO/SLiC (Azar et al., 2023) .Fig. 4 presents the alignment performance with different dropout rates. We find that higher dropout rates significantly harm the alignment performance. To summarize, our findings suggest that high strength of regularization, in general, fails to mitigate the negative impact of preference noise, and in certain cases can even hurt the alignment performance.Published as a conference paper at COLM 2024Another approach to fight against noise is to filter out noisy data in the Preference Pairs dataset (see Fig. 1 ) and only use the remaining data to perform alignment. We use the popular confidence-based data filtering method (Cheng et al., 2008) . For a prompt x and two candidate responses y w , y l , the confidence of y w preferred over y l , denoted c(y w > y l |x), is a real value between 0 and 1. Ideally, the confidence function c should be well-calibrated (Silva Filho et al., 2023) , i.e., c(y w > y l |x) is identical to the true probability of y w preferred over y l . In confidence-based data filtering, only pairs with the confidence level larger than a pre-defined threshold t are used in training; hence, if threshold t = 0.5, no filtering is performed; t = 1 means all data will be removed. We experiment with t = 0.5, 0.6, • • • , 0.9 to investigate the effect of different levels of data filtering.Based on the user studies made by Gao et al. (2020) , we use the Bradley & Terry (1952) model to estimate the confidence value:c(y w > y l |x) = σ[r * (y w |x) -r * (y l |x)],where σ is the sigmoid function. In practice, there are multiple methods for estimating the confidence values, e.g., conformal predictors (Shafer & Vovk, 2008; Einbinder et al., 2022) , ensemble methods (Liang et al., 2022) , and bias estimation methods (Chen et al., 2023) .Note that with higher filtering threshold t, the quality of the remaining data is improved at the cost of quantity loss. We deliberately do not back-fill the filtered data, because in practice it can be prohibitively expensive to collect more preference pairs. This setup also allows us to study the trade-off between data quality and data quantity in preference-based alignment. Fig. 6 in Appendix A shows how the size of the remaining data shrinks with higher confidence thresholds. We find that in both datasets, the data size drops quite quickly with the growth of t values: Almost 20% data are filtered as t increases by 0.1.Fig. 5 presents the alignment performance with different strengths of data filtering. We make the following observations.• Data filtering does not help to fight against Random Noise. This is because Random Noise Oracle flips pairs completely at random (Frénay & Verleysen, 2014) , i.e., the flipping chance of each pair is uniformly at random, not affected by any other factors (e.g., the prompt x or the responses y w , y l ). As a result, our confidence-based filtering cannot reduce the number of the noisy pairs in the filtered data, and hence fails to improve the performance. • Data filtering is effective to mitigate the harm from Stochastic and Gaussian Noise. We find that with certain threshold (e.g., at 0.8), data filtering shows consistent and significant improvement across all noise rates, noise types, and tasks. Considering that over 50% preference pairs have been removed with data filtering at confidence threshold 0.8, this result suggests that data quality has a significant impact on the alignment performance. • Over-aggressive data filtering hurts the performance. With very high confidence thresholds (e.g., 0.99), the alignment performance is compromised across all noise rates, noise types, and tasks, due to the severe loss of data size. Hence, it is important to properly trade off between data quality and data quantity, in order to yield the optimal alignment performance.To better understand how data filtering improves the data quality, we investigate the noise rate before and after data filtering, with different noise types. Fig. 7 in Appendix A shows the noise rates in data filtered with different confidence thresholds. We find that when the noise is from Random Noise Oracle, the noise rate stays the same with all data filtering thresholds; this explains why data filtering does not help improve its performance. When the noise is from Stochastic or Gaussian Oracles, data filtering can effectively reduce noise rate, explaining the performance boost observed in Fig. 5 . 
limitations & future work: 
 Generalizability. We apply a popular alignment algorithm (DPO) to T5-based language models in our experiments. We believe our observations presented in §6 and §7 can be generalized to other alignment methods (e.g., PPO, SLiC and IPO, because they essentially optimize the same objective function; see §2), other Transformer (Vaswani et al., 2017) based language models, and other datasets (e.g., UltraFeedback (Cui et al., 2023) and AlpacaEval (Dubois et al., 2024) ), but this is yet to be empirically verified. Considering the large number of possible combinations between alignment techniques and language models, a systematic study is beyond the scope of this paper, and we leave it for future work.Mixture of Noise. We consider three strategies to add noise to the preferences, but we only allow one strategy to be used at one time (see §4). In practice, there may exist other types of noise, and different types of noise can co-exist in preferences. Our framework allows for creating new noise types by mixing primitive noise types, but it remains unclear whether the current observations can be applied to the new noise types or not. We hope our work can facilitate and encourage more work on this direction.Data Filtering. Our current experiments demonstrate that data filtering can improve performance on the in-domain test set (see §7.2). However, it remains unclear whether this performance gain extends to out-of-domain test sets. A thorough understanding of data filtering's impact on cross-domain generalization requires meticulous experimental design. This includes careful consideration of factors such as the selection of training and testing tasks, and the degree of difference between them. We call for further research in this area. Additionally, the training set used and filtered in the current work is synthetic (see Section §5). Investigating the effectiveness of data filtering on human-generated datasets would be a valuable next step. 
conclusion: 
 Pairwise preferences are widely used for aligning Generative Language Models (GLMs) with human values, but it remains unclear how the noise in preferences affect the alignment performance, and how to mitigate their negative impact. To study these problems, we propose a framework in which the types and rates of noise can be simulated and controlled, and we perform systematic experiments on two generation tasks (summary and dialogue generation) with this framework. Our findings suggest that alignment performance can be highly sensitive to the increase of noise rates, and appropriate data filtering is the most effective method to mitigate the negative impact of noisy preferences. Our work builds the first quantitative relation between noise rates and alignment performance across different noise types. We hope our work can help the community better understand and mitigate the impact of preference noise in GLM alignment.  
 
 Appendix:  


Decision: This paper investigates the impact of noise in preference data on the performance of LLMs after alignment fine-tuning. It explores three noise injection methods and analyzes their effects on summarization and dialogue generation tasks. The study finds that noise degrades performance but aligning with noisy preferences can still improve over a reference model. It also evaluates mitigation strategies, concluding that confidence-based data filtering provides better robustness without compromising learning performance.

The reviewers appraise the paper being well-written and easy-to-follow, the method to inject noises being interesting and novel, and the comprehensive analysis being informative and instrument for the community.

Reviews: review: paper_summary: paper_summary: Authors present an empirical investigation about (1) the influence of noise in preference data on the model performance after alignment fine-tuning, and (2) what strategies could alleviate the degradation. First they describe three ways of injecting noise into preference data produced by their gold reward model such that those ways align with real world noise we may find in datasets we use. Second, they present an analysis of performance degradation as the amount of noise go up across multiple tasks and noise schedules. Finally, they investigate training regularization as well as data filtering and conclude that some methods might alleviate degradation but not completely. However, they highlight that even noisy preferences may still improve model's alignment compared to the reference model.
strength_weakness: strength_weakness: * GLM model authors consider is pretty small (<1B) w.r.t. the parameters count. It feels that authors chose T5 family of models for some reason for both RM and GLM. Experimental results could have larger scope if e.g a llama3 8B or llama2 7B (if larger models are infeasible to train for authors) have been considered.

* general instruction following benchmarks such as alpaca eval is not considered in the experiments while it would be interesting to see the trends on winrates when numbers would correspond to some leaderboard.
questions: questions: Following on rejection reasons:

- why did you choose T5 770M model as the main GLM model ? Especially given other way more up-to-date LLMs available. 

- why did you choose reddit TLDR instead of some more up to date tasks / benchmarks e.g. open assistant 2? Especially since alignment is tightly connected to instruction following tasks rather than summarization.
paper_summary: paper_summary: This paper investigates the impact of noise in preference data on the alignment performance of Generative Language Models (GLMs) and proposes methods to mitigate this impact. The authors provide a systematic study across two tasks—summarization and dialogue generation—highlighting how even slight increases in noise can significantly degrade performance. They introduce a novel framework for injecting and controlling noise types and levels in training data, leading to a detailed empirical analysis.
strength_weakness: strength_weakness: (1) The experiments are limited to two specific tasks and one type of GLM (T5). Additional experiments with other models and tasks could help generalize the findings more broadly.

(2) While the paper presents data filtering as an effective mitigation strategy, the exploration of regularization methods could be expanded, and other potential strategies could be considered.
questions: questions: (1) The paper handles single types of noise separately. How might the proposed methods perform under conditions of mixed noise types, which are more likely in real-world settings?

(2) While data filtering improves alignment performance, could it also potentially lead to overfitting or reduced model robustness due to reduced training data variability?

(3) Could the authors elaborate on why increased regularization was not effective? Are there other regularization techniques that could potentially yield better results in combating preference noise?
paper_summary: paper_summary: The paper investigates the impact of preference noise on the alignment of GLMs and proposes confidence-based data filtering to mitigate this noise. In controlled experiments, three types of artificial noise are inserted on preferences in the tasks of summarisation and dialogue generation. The authors find that 1) even high noise rates an still be beneficial for alignment and 2) that, contrary to regularisation, confidence-based data filtering improves performance.
The paper presents a small but focused contribution, with a firmly-states research question. It doesn't only investigate noise, but proposes mitigation solutions. Even though the scope is limited to two tasks, one model, and one noise type at a time, it can provide initial insights into the effect of preference noise and its mitigation.
strength_weakness: strength_weakness: - None, since the limitations are acknowledged in the paper.
questions: questions: - Please change the line format in the graphs so that the different factor can be discerned by the colour blind and when printing in grey scale.
paper_summary: paper_summary: The paper studies a problem of noise in the preference alignment data for language models and its effect on the LLMs performance on 2 tasks, summarization and dialogue response generation. The authors use a controlled environment where they control the amount and the type of noise injected in the preference data and study the corresponding models' performance in terms of win rate of the aligned model vs the SFT model. The authors work with 3 types of noise: Random noise oracle (uniformly distributed noise across the dataset with a given ratio), Stochastic noise oracle (proportional to the reward difference between the preferred and dispreferred example) and Gaussian noise oracle (modelling the approximated rewards as stochastic gaussian variables).
Having found that noise in preference data does impact the alignment performance (but aligning even on the highly noisy data is beneficial for the model), the authors also conduct a study of performance degradation mitigation strategies in the settings of noisy preferences. Out of the two techniques they try, KL-based regularization and confidence-based data filtering, the latter results in superior robustness while not weakening the learning performance.
strength_weakness: strength_weakness: The paper has contributions on the analysis side (the experimental studies themselves), modeling side (model training technique with confidence data selection), and resource side (noise datasets/framework). Out of the 3, I see the analysis contribution the most substantial - if analysis papers are welcomed at the conference, that works as a reason to accept, if not - this is a reason to reject.

Another thing that would add much substance to this work and is missing from the paper in my understanding is trying the confidence-based data filtering on a real unaugmented dataset with a known presence of preference noise and evaluating the resulting performance difference with a human study.

Comments: title: Authors responded to my concerns
comment: Thank you for your response. I will keep my original ratings.
Thanks, the authors' response makes sense to me. I keep my judgement of this paper being a good analysis paper that would be beneficial to have at the conference.
thanks for your reply! 

authors have addressed my comments, although i still think they could have trained at least 1.5B or 7B/8B llama2/3. There is a fair amount of frameworks and code that allows to train such models within 8 gpus or even less one uses parameter-efficient tuning. 

such training would help to see how this findings change when you scale the model up or when you use instruct based mode as the initialization etc. that is indeed a solid future work direction though
The authors have responded to my concerns sufficiently. All in all, this is a suitable paper for the conference, which presents a small focused contribution.