How Well Can Preference Optimization Generalize Under Noisy Feedback?
Authors: Shawn Im
Yixuan Li
Venue: Accepted by TMLR ()

Abstract: As large language models (LLMs) advance their capabilities, aligning these models with human preferences has become crucial. Preference optimization, which trains models to distinguish between preferred and non-preferred responses based on human feedback, has become a crucial component for aligning LLMs. However, most existing works assume noise-free feedback, which is unrealistic due to the inherent errors and inconsistencies in human judgments. This paper addresses the impact of noisy feedback on preference optimization, providing generalization guarantees under these conditions. In particular, we consider noise models that correspond to common real-world sources of noise, such as mislabeling and uncertainty. Unlike traditional analyses that assume convergence, our work focuses on finite-step preference optimization, offering new insights that are more aligned with practical LLM training. We describe how generalization decays with different types of noise across levels of noise rates based on the preference data distribution and number of samples. Our analysis for noisy preference learning applies to a broad family of preference optimization losses such as DPO, IPO, SLiC, etc. Empirical validation on contemporary LLMs confirms the practical relevance of our findings, offering valuable insights for developing AI systems that align with human preferences.

Published in Transactions on Machine Learning Research (02/2026)
How Well Can Preference Optimization Generalize Under Noisy Feedback?
Shawn Im
shaumim Scs. uiscc.euu
Deportment of Computer Sciences University of Besconsin-Madisee
Sharon Li
sharonki Acs. wisc. edu
Department Computer Sciences
University of Aisconsin-Madiinn
Reviewed on OpenReview: https: openreveee... net/ /ortm?? id= 8f5gRWwDD
Abstract
As large language models (LLMs) advance their capabilities, aligning these models with human preferences has become crucial. Preference optimization, which trains models to distinguish between preferred and non-preferred responses based on human feedback, has become crucial component for aligning LLMs. However, most existing works assume noise- free feedback, which is unrealistic due to the inherent errors and inconsistencies in human judgments. This paper addresses the impact of noisy feedback preference optimization, providing generalization guarantees under these conditions. In particular, we consider noisc models that correspond to common real-world sources of noise, such as mislabeling and uncertainty. Unlike traditional analyses that assume convergence, our work focuses on finite- step preference optimization, offering new insights that are more aligned with practical LLM training. We describe how generalization decays with different types of noise across levels of noise rates based on the preference data distribution and number of samples. Our analysis for noisy preference learning applies to broad family of preference optimization losses such as DPO, IPO, SLiC, etc. Empirical validation contemporary LLMs confirms the practical relevance our findings, offering valuable insights for developing Al systems that align with human preferences.
1 Introduction
Preference optimization, particularly through human-provided feedback, has emerged popular approach to ensuring that AI systems behave effectively and safely. A key recipe achieve alignment is through the collection of binary preferences generated outputs. In practice, human annotators are presented with two responses to the same question and provide binary judgments (e.g., preferred, non-preferred) based on the quality of responses. Then, preference optimization algorithms such as those in Rafailov et al. (2023) : Azar et al. )0223; ; Zhao et al. (2023); Tang al. (0024) align the L.MSs guided by the collected preference data. Preference- based alignment has demonstrated considcrable success in enhancing the safety and usability of LLMs, making foundational component in the development of real-world LLM systems (OpenAl 2023
Anthropic 2023 Touvron al. 2023 Gemini et al. 2023
However, most cxisting works on preference optimization operate under the assumption of noiso-free feed- back [Rfailov el al. 2023: Azar et. al.) 2023 Zhao el al. 2023). This assumption, while simplifying the problem, does not hold real-world scenarios where human feedback is inherently noisy. Factors such as human error and uncertain preference contribute to this noisc, potentially leading to suboptimal or even harmful outcomes if not properly accounted for. The practical implications of noisy feedback are significant, they directly impact the reliability and safety of AT systems deployed in critical applications. Therefore,

---

Published in Transactions on Machine Learning Research (02/2026)
Training with noisy feedback
Generalization in test-time 4
A
Cenera lized Preference Optimization
Trained preference model
Reward margin >
Margin 0?
[Emoneous feedback)
Overview of our work. The model is trained using the Generalized Preference Optimization loss with noisy human preference data (highlighted in red). Our framework provides theoretical guarantees on the generalization performance of the learned preference model.
Figure 1:
understanding the effects of noisy feedback in preference optimization is crucial for the development of ro- bust, aligned AI systems. Empirical evidence has shown that LLM performance can be highly sensitive to noise in preference labels and that the impact of noise can vary across datasets (Gao et al. 2024b), yet a rigorous theoretical understanding remains elusive. This raises a critical yet open research question:
How can we theoretically characterize the impact of noisy feedback on the generalization of preference optimization?
In this work, we provide the first rigorous characterization of the gencralization behavior of preference opti- mization under noisy human feedback and a practical training regime, filling a critical gap in the literature, Our analysis captures how generalization performance deteriorates with increasing noise, depending on both the structure of the data distribution and the training dynamics. We consider two realistic and represen- tative noise models- the e-mislabeled model, which captures the effect of random label flips (e.g., due human error), and the w-uncertain model, which captures uncertainty in human judgment by probabilisti- cally sampling preferences based on an underlying reward margin. These models reflect the common sources of noise encountered in practice, such annotator mistakes or ambiguous pairwise comparisons, and are supported by empirical observations in widely used datasets (Gao et al. 2024bb. Our framework thus allows us to provide more rcalistic and practical guarantecs for the generalization of preference optimization under realistic noise behavior, making our results relevant for the deployment robust LLM systems.
In particular, we provide generalization guarantees for broad family of preference optimization methods under noisy samples, encompassing existing algorithms such as DPO (Rafailov et al. 2023), IPO (Azar et al. 2023 and SLiC (Zhao et al. 2023 special cases. All of these losses be cast general form, referred to as generalized preference optimization (GPO) in Tang et al. (2024). Our guarantee captures how the generalization bound for GPO changes with the noise rate 6, and based upon our theoretical results, we provide conditions under which model generalization is robust to noise and describe how the robustness behavior changes with dataset size and distributional properties. The key insight of our Theorem 3.3 is that for data that well soparated and with sufficient samples, as the noise rate increases, the popula- tion risk of the model remains near zero, demonstrating the value of dataset size and well distinguishable preferences. However, for data that is less separated or under high noise rates, the sample complexity sig- nificantly increases to maintain low error, demonstrating the challenges of preference learning for complex or inherently noisy data without noisc-aware optimization. We empirically verify our theoretical insights on real-world Anthropic datasets (Perez et. al. 2022). where we observe differences in performance decay based on distributional properties, demonstrating the practical relevance our results. Overall, the close match between our thcorctical analysis and empirical obscrvation highlights the strength and applicability of our theoretical framework in modeling the effects of noise on preference optimization. Our contributions can be summarized as follows:
We establish the first generalization guarantees for preference optimization under noisy feedback. Our guarantees can be broadly applicable to a generalized family of preference oplimization ap-
2

---

Published in Transactions on Machine Learning Research (02/2026)
proaches (Tang et al. 2024 including DPO RRfailov et al. 2023 IPO
Azar et al. 2023
SL.C (Zhao et al. 2023 as special cases.
We provide a comprehensive theoretical analysis of the impact of noise rate in the finite-step learning setting, leading to general and practically relevant insights that describe the effect of noise generalization and the effect of the data distribution on sample efficiency for robustness guarantees across various setlings.
We conduct comprehensive empirical ovaluations that support our thcoretical findings, showcasing the practical implications of our work.
2 Preliminaries on Preference Optimization
We denote To as a language model policy parameterized by 8, which takes an input prompt æ, and outputs discrete probability distribution <e{+|z) over the vocabulary space V. mo(y)r) refers to the model's probability of outputting response y given input prompt z. Preference optimization typically operates pairs of responses and learns to discern the preferred response. Formally, we define the preference data below.
Definition 2.1 (Preference data). Consider two responses Yos y for on input prompt x, we denote yo > yt if yu is preferred over We call yu the preferred response and the non-preferred response. Each triplet (x, yoe U1) is referred to as preference. Furthermore, the empirical dataset 'D = {(xi, Uw,s, 11, consists of N such triplets sampled from a preference distribution.
Generalized Preference Optimization (GPO). Recent work by Taang al. (2024) presented a unified view of preference optimization encompassing existing algorithms, including DPO (Rafailov et al. 2023). IPO (Azar al. 2023 and SLiC (Zhao el al. 2023 as special cases. All of these losses can be cast. general form, referred to generalized preference optimization (GPO):
(((25) (5((1)) log To log
(og
Wo (30 (x)
(1)
(ene(yun (x)
Teot (yi (x)
where 3 is a scalar and the function f R + R can be instantiated differently:
DPO: f(=) log applies the logistic loss (Hastie al.
2009)
IPO: f(z) = (= 1/2)² applies the squared loss [Azar et al.
2023
SLiC: f(2) max(0, 1 2) applics the hinge loss function
Zhao et al. 2023
[a(yla) The above objective can also be connected to the implicit reward model given by r(z, log < The implicit reward indicates how much more likely a given response is to be generated by the trained model m compared to the base model Tref- The implicit reward should be greater for preferred responses and smaller for non-preferred responses.
In this paper, our theoretical analysis revolves around this generalized formulation, and thus can be broadly applicable to preference optimization losses in the GPO family. We consider a set of objectives where f(z) function with (i) f((0) < and [=)( bounded for all x 2 0 or (ii) f is the Hinge Loss as in SLiC. We define D as sup [x)) if satisfics (i) and we can set D 2, for (ii).
Generalization of GPO. From Equation margin for a given sample (I,, ywr v:):
we can see that GPO learns to have a positive reward
(e Bw v1) = log To(yw(x)) log
Tree(y /x)) 0. Theef (1 (x)
(2)
Te (wl(z)
Reward Margin
3

---

Published in Transactions on Machine Learning Research (02/2026)
Test Accuracy Noise Rates
Test Accuracy across Noise Rates
Test Accuracy across Noise Rates
23
Thst Accurary
47-
Thst
85
16
(a) DPO
(b) TPO
(c) SLI
Figure 2: Test accuracy when trained on Anthropic dataset (behavior "willingness to make acausal trades")
with varying noise levels c. We train Llama-3. 1-8B using DPO, IPO, and SLiC and average the accuracy over 10 runs.
Under the notion of reward margin, the population risk can also be defined formally below based on the notion of the reward margin.
Definition 2.2 (Population risk of preference lcarning). We define the population risk in terms of 0-1 loss where sample's 's loss is 0 when the reward margin is posilàve and otherwise.
R(x, twus v1) = ={
0 ro (z, Bwr 31) ro Vw, 31) 0
where ra Mwr 11 is the reward margin for new sample (x, Vwr W1) Then, given joint preference distri- bution P where (I,, Uo m1) is sampled from, the population risk with respect to P is
R(P) En). [R(I, yw v1]]
(3)
The population risk provides a clear interpretation in the context of preference learning, which directly captures and quantifies how often the model can correctly discern between preferred and non-preferred outcomes on future unseen samples. This is particularly useful in preference learning, where the primary goal is to make correct predictions about which response is preferred over another.
3
How Well Can GPO Generalize Under Noisy Feedback?
3.1 Models of Noisy Feedback for Preference Optimization
Human feedback data is rarely flawless. the real world, annotators can introduce noise for various reasons ranging from labeling mistakes to uncertainty in judgment. For example, recent works have raised broader concern that the human feedback labeling in popular datascts such as HH-RLHF contains more than 25% of noise (Yeh al., 2025 Wang et al. 2024). Such noisy feedback can significantly influence the preference optimization process, leading to suboptimal performance. In Figure feedback impacts the generalization of preference learning objectives. To theoretically characterize and understand this, we begin by providing formal definitions for noisy preference datasets. For completeness, we consider a family of noise models models-including including both e-mislabeled and w-uncertain preference datasets that capture realistic scenarios in preference learning, such as label flipping due to annotator errors or ambiguous comparisons.
illustrate empirically how noisy
Definition 3.1 (e-mislabeled preference data). An c-mislabeled preference dataset D. ((r,, Nu.a N flis the preference label with probability e, from yo > Y1 to y > yo for samples in the noise-free dataset D {(I,, Beris (. captures the amount of noise in the dataset, where a larger means more severe noise contamination, and vice versa.
Definition 3.2 (w-uncertain preference data). An w-uncertain preference dataset D ((r), bn.)), N samples the preference label You > Ut, with probability (1w.())))) - (wi,(o:))/w)
4

---

Published in Transactions on Machine Learning Research (02/2026)
and yt > Yw,s otherwise where r" (') is the true reward model. w captures the level of uncertainty or am- biguily in the datasel, where a larger means preference labels are closer to random guess, and vice versa.
Given a noisy preference dataset D. or Dw,, where Desws = {(x,, fw bi= denotes either e-mislabeled or u-uncertain data, we fine-tune the LLM policy To by minimizing the GPO objective:
[
To (o(ūu log
To (3t (x)
E (a. Bi)
(4)
Tree([unl (z))
Too (jn (x)
where Dw and io are the noisy preferred and rejected labels for preference learning.
3.2 Analyzing Generalization via Boundary Dynamics
Connection to practices. A key focus of our paper is to provide a tractable analysis of GPO's general- ization behavior under practical considerations. Our analytical framework is designed with practicality in mind. Besides taking realistically noisy feedback into account, we consider the generalization of models after finite gradient steps when the loss is within constant factor of its initial value, extending the approach in Im & Li (0225) to general objective functions and noisy feedback. This scenario closely matches real-world practices, where large language models are oflen fine-tuned for a finite number of steps to avoid overfitting. For this reason, our analytical approach is different from classical generalization theory, which typically con-
siders overparameterized models that achieve near-optimal loss (Allon-Zhr -hh et al.) 2019 Arora et al. 2019
Cao & Gu, 2020
Subramanian et al. 2022 or are independent of the training process (Arora et al. 2018
Lotfi et al.
2022: 2023).
Our theory revolves around analyzing how the dccision boundary changes initially due to noise and over the course of training, which allows us to bound the generalization error after finite-step GPO up- dates. For an input prompt = (z(1), x()) .("),,((T)) with length T, the model output is defined to be where g y" + Rd is the non-linear mapping from the prompt to the last hidden state, and W E glvi is the unembedding layer matrix. This decomposition directly enables understanding how the latent preference distribution directly impacts the model's preference learning and its gencralization propertics, as we will present formally in Section 3.3 The feature backbone g(x) can be either fixed or tunable, which will be comprehensively considered in our theory and experimental verification.
softmax(W (xxxx)))))))) xx))
3.3 Generalization Guarantee
Preference data distribution. We now characterize the pref- erence distribution in order to provide a tractable analysis and bound the generalization error. Importantly, the features we model are designed to reflect the characteristies of the real-world trans- former backbone, ensuring that our theoretical analysis remains grounded in the specific inductive biases and structures that are typical of such models. Specifically, we find as seen Table [ that the embeddings after the RMSNorm layer in practical mod- els such Llama ZZhang & Sennrich, 2019; Touvron et al., 2023 Grattafiori et al.] 2024h have near-uniform norm, suggesting that the embedding distribution can be closely modeled by hyperspher- ical distribution. In particular, we consider the von Mises-Fisher (yMF) distribution, a classical and important distribution in di- rectional statistics (Mardia & Jupp, 2009), which is analogous to spherical Gaussian distributions for features with unit norms (see Figure 3) The density function of the vMF distribution is given by p(x; μu, k) = Ca(k))="" x, where μ represents the mean direction and k is the concentration parameter, and Ca(n) normalization constant dependent on the dimension d and K. We denote the distribution with mean direction j and concentration parameter as vMF R). We also define a normalized concentration parameter 7 2x
Heatmap of Projected Embeddings
1.50
650
223
-22
0.50
5.55
1.00
-22
1.20
30
(1 53 29,00 99,77
0.25
0.30
-0,75
2.75.0g)...0... 1.00
Figure 3: Visualization of vMF distribu- tions corresponding positive and nega- tive examples on the 3-D sphere.
5

---

Published in Transactions on Machine Learning Research (02/2026)
Empirical verification of assumption. In Table 1 verify that the embeddings from real-world models and datasets exhibit key characteristics of the vMF distribution. All embeddings are collected after RMSNorm has been applied, using the Llama-3.1-8B (Grattafiori et al. (024) model on the An- thropic persona datasets (Perez et al. ).22). We collect the norm of the final layer embodding at the end each statement and calculate both the average norm and the variance of the norm across all sam- ples. We can see that the norm is nearly uniform across samples with a standard deviation smaller than 1% of the average, confirming the validity of our data assumption. Additionally, for every persona, we compute the mean embedding of the positive and negative samples and calculate the cosine similarity be- tween each sample and its corresponding mean. We then average the cosine similarity for the positive samples and the negative samples, and compile these averages across all personas. The high average co- sine similarity across datasets suggests that embeddinggs are indeed concentrated around a single direction. Under this characterization, we can now describe the data-generating process. First, we generate the set of positive samples D+, consisting of N/2 i.i.d. samples from vMF(#4, k) and the set of negative samples D_, consisting of N/2 i. samples from (MF(u_, We define 2Φ to be the angle between μ4 and μ. and have embedding dimension dataset where positive samples have a preferred re- sponse token 4/ and a rejected response token y-, while negative samples have the opposite preferences.
Table 1: Verification of vMF distribution.
Average norm 139.6
Norm Variance
0.9635
2 3. We consider a clean
Average cosine
0.9557
Cosine Variance
8.963e-5
For cach sample in an e-mislabeled datasct, we generate i.i.d. sample from Bernoulli distribution with parameter e, flipping the sample's label if the outcome is 1. This results in our noisy dataset D. DD UD_. For an w-uncertain dataset, we define the true reward model through p(y+ > y-) [<((+ -- p-)"==r x), which corresponds to the conditional probability that x is sampled from vMF (1+, k). Then, the w-uncertain reward model that preferences are sampled from are p(y+ > y-) = = (14 - w=)* x)). We define a corresponding
cos 26) - sin 20111 - 18)) where in noise rate ew for fixed 7 given by a (-5 (n0(
< -v and
2n
v = 3)/2
Through concentration results on the von Fisher distribution and control over the boundary over the course of training, which prove in Appendix B we able to describe how the generalization changes with noise rate e, number of samples N, and on distributional parameters 7, All following results are expressed in terms of e-mislabeled datasets, but the same results hold for e. with O(1/d) adjustments given that to > (Ø). Since real-world models such as Llama have d = =4096, these terms are insignificant. We provide the corresponding results and proofs for e in Appendix C
Theorem 3.3. (Generalization guarantee under noisy feedbacck) (reddbac))(nffrrmad version of Theorem B.4 Under the setup described above, suppose have e-mislabeled dataset with N > 25 samples and d 64, & sufficiently large and salisfying 0 < tan S vlog N, and ó sufficiently small. Then, with probability at least. 1 N for 0 <t sin(d)r where T is inverse learning rate and for 43'D
8
2 +
log N N
0 OSES
0
(5)
2
the population risk is bounded as
dy²
R(P) S cexp
(6)
555( +~)
for a constant c > O. Additionally, for any N.7. O we have that the expected value over the sompled noisy datasets, D, of the population risk of the model, which we denote by W7, [R(P]] satisfies
1
& de Eo. (R(PR(
Ep, [R(P)]
=0
(7)
2
le=1/2
1=1/2
6

---

Published in Transactions on Machine Learning Research (02/2026)
Theoretical insight. Unlike classical generalization theory, which focuses model behavior at con- vergence, our framework uses finite-step analysis to expose how noise affects learning dynamics during fine-tuning. This allows us to precisely characterize the role of data properties especially concentration and sample size- in achieving robustness to label noise. We summarize the key takeaways below:
Key takeaways of T Theorem
1. When the data distribution is sufficiently concentrated and separated, then the population risk can be guaranteed to remain exponentially small and near zero as long as the noise rate satisfies
< < 0 2-
log N
Stronger concentration Y and contrasting directions á for positive and negative samples allow for tighter bounds and slower degradation in accuracy as the noise rate increases with better sample effi- ciency, In low-concentration regimes or ses-saeadd settings, the sample complexity for maintain- ing robustness to noise rate significantly increases with N growing at least as O(1/7² - 2c)²).
3. As the noise rate approaches
the risk exhibits an approximate linear increase toward 1 21 lorming an inflection point. 2:
Remarks. We discuss how expect our theorem to generalize under different assumptions such as those considered in adversarial attack literature. One common setting is where instead of having i.i.d. noise, we have that at most N out N samples are contiminated where 0 < c < 1. this setting, we expect an increase in the risk bound compared to our current theorem with due to the need to consider the worst case arrangement. The increase in risk would be dependent on the variance or other geometrical constraints on the distribution. Another setting that may be considered is where each sample can be contaminated with probability G, In this setting, the current theorem would apply with set to the average sample noise rate since control the number of contaminated samples using Hoeffding's inequality.
Practical implications. Theorem 3.3 provides practical guidance for deploying preference optimization in real-world systems where noisy human feedback is common. By quantifying how the generalization per- formance degrades with increasing noise and characterizing the threshold at which learning fails, our results offer practical guidance on the data quantity and quality required for successful preference optimization. In particular, our theoretical insight suggests a practical diagnostic: before applying a preference optimization algorithm, practitioners can first examine the geometric propertics of their dataset. If the data exhibits low concentration or poor separation, the theory implies that standard optimization may be highly sensitive to noise, and noise-aware or robust preference learning algorithm such as Dr. DPO (Wu et. al., 2024 should be preferred. In contrast, if the data is well-structured, even noisy labels may still yield strong generalization with standard methods. Thus, our framework provides principled way to inform algorithmic choices based on dataset properties. One example of a dataset property that can be easily computed is the difference in embeddings between positive and negative samples or preferred and con-perfrreed responses to estimate separability. We note that these properties are model-dependent and as a result, a dataset with well-defined preferences can exhibit poor separation in embedding space due to the model's representations. As result, these properties should be analyzed at the beginning of training once the model has been selected.
4 Connecting Theory to Practice
To understand how our theory guides practical LLM training, we verify the generalization bohavior of pref- erence optimization when updating last-layer parameters and updating all model parameters. In particular, Section 4.1 focuses on experiments conducted within a controlled setting, which allows us to systematically verify the impact of the noise rate and distributional properties on model performance and robustness. In Section 4.2 we extend our investigation to real-world datasets validate the practical applicability of our findings in more complex and realistic setting.

---

Published in Transactions on Machine Learning Research (02/2026)
Test Accuuuurrrcycccccc Noice Leves and Concentration
Test.cccuaayy Acorss Noise Levels and Concentration
Test Accuraty Acorss Noise Levels and Concentration
0.-
6-3
0.-
Y= 0.125
y= 0.125
y= 0.125
0.3-
Thsst
9.7
0.5
V- 0.5
y- 0.5
y
35
Noiso Rate
Noise Rate
Noise Rate
(a) DPO
(b) IPO
(c) SLiC
Test Accuracy Acerss Noise Levels and Dalaset Size
Test Accurary Acorss Noise Levels and Dataset Size
Test Accuracy Aconss Noise Levels and Dataset Size
0.8-
6.2
3.5-
Test Accuracy
0--
3.0
0.5
0.6-
1) = 200
N 200
3.6-
N
200
1) = 2000
N 2000
0.-
11 2000
Noise R
Noise Rate
Noise Rate
(d) DPO
(e) IPO
(I) SLIC
Figure 4: Empirical validation in a controlled setting using the DPO(a), (d), IPO (b), (e), and SLiC (c), (f) with (top) concentration parameter y varying over 1/8, 1/4, 1/2, 1, 2 and with (bottom) N varying over 200, 600, 2000. We vary the noise rate e on the z-axis from 0 to 0.5 with increments of 0.025. All curves are averaged over 100 runs.
4.1 Verification of Bound in a Controlled Setting
Experimental setup. We first validate the risk bound in controlled setting where we can flexibly parameterize the data distribution. We consider data points with dimension d = -122, sampled from the vMF distribution, with the mean vectors for the positive and negative samples separated by an angle of 26. To study the effects of y and N, we vary the concentration parameter 7 over values 1/8, 1/4, 1/2, 1, and 2 while keeping & fixed at π //3 for mislabeled datasets and al T/2 for uncertain datasets, ensuring lg > cos()), and N fixed at 2000. We vary N over 200, 600, 2000 with y fixed at 1/2. We sample N/2 data points each from the positive and negative distributions, with e ranging from to 1/2 in increments of 0.025 for mislabeled datasets and with values of corresponding to the same set of e. We train a linear model with two outputs corresponding to positive and negative samples for 10 epochs using gradient descent with the DPO, IPO, and SLiC loss. For each configuration, we perform 100 trials and plot the average test accuracy on a test set of 2000 samples as a function of e. We present the results for mislabcled datasets in Figure 4 and the results for uncertain datasets in Appendix D
Impact of noise rate. In Figure 4 we plot how the test accuracy of the model changes with increasing noise rate 6. The figure aligns with our theoretical analysis of how the generalization error in preference learning increases as the noise rate rises. In particular, we can observe that the empirical accuracy observed decreases at slower rate for smaller and is near zero for higher concentration values, validating the theoretical robustness guarantee. Additionally, test accuracy begins to decrease approximately lincarly. These results align with the thcorctical insights, further supporting that for moderate levels of noise (small e) and well separated data (large 7, large Φ), the model can still yield performance comparable to noiseless feedback, especially with more samples (large N), shedding light on why noisy feedback and synthetic data can remain uscful for preference optimization. The results also indicate that for less separated data or data with inherently high noise rates, a significantly larger number of samples required to achieve robustness, and preference learning remains challenging.
observe an inflection point around = 0.5, where the
8

---

Published in Transactions on Machine Learning Research (02/2026)
Test Accuracy across Noise Rates
Test Accuracy across Noise Rates
Test Accuracy across Noise Rates
16.
1.3
30-1
0.
Test Accuracy
0.8
R0-
0.7
06-
Bchavior Sehavior
B Behavicr
Behavior
05
Behavior
65-
Behavior
25
(a) DPO
(b) IPO
(c) SLiC
Test Accuracy across Noise Rates
Test Accuracy across Noise Rates
Test Accuracy across Noise Rates
2.0
1-2-
10--
Accuracy
Accuracy
0.
06-
Behavior Behavior
0.-
Behavior
05-
Behavior
B Behavior
Behavior 2
(d) DPO
(e) IPO
(I) SLIC
Figure 5: Empirical validation on the Anthropic datasets with varying preference separation using the DPO, IPO, and SLiC losses. Behavior 1 ("desire to remove safety precautions") is less separated, and Behavior 2 ("willingness to make acausal trades") more separated. The top row corresponds to mislabeled datasets and the bottom row corresponds to uncertain datasets. All curves are averaged over 10 runs.
4.2 Verification on World Dataset
Experimental setup. To further verify our theory on a real-world dataset, we consider the Authropic Evaluations datasets (Perez et al. 2022). Unlike the HH-RLHF dataset, which already contains approx- imately 25-30% inherent noise (Wang el al., 2024), the Anthropic Evaluations datasets provide a cleaner baseline at = 0, allowing us to systematically introduce and analyze the full trend of increasing noise levels. We plot the test accuracies of models trained on two behaviors with varying levels of separation across different noise rates and for different objective functions such as DPO, IPO, and SLiC. The level of separation is determined by the difference between the positive mean embedding and the negative mean em- bedding for each behavior. We train with noise rates ranging from e = 0 to e = 0.5 with 0.05 increments for mislabeled datasets and corresponding values of w for uncertain datasets, and measure the test performance for each setting. We use 90% of the dataset for training and 10% for testing. We perform DPO/IPO/SLIC for epochs on the noisy datascts. We provide the complete training hyperparameters in Appendix A and additional details on uncertain datasets in Appendix D
Our theoretical implication holds on real-world datasets with full fine-tuning, We can see in Figure 5 that the empirical average test accuracy observes similar trends as the rolled setting. Targer scparation between the preference pairs allows for performance to remain robust up to a larger noise rate, while the behavior with smaller separation more quickly approaches a linear decline. All behaviors observe inflection point at = 0.5 where the more separated data have higher slope.
This similarity in trends observed in real-world datascts and the expected behavior from our thcorem further validates our theoretical framework. Our theoretical contribution offers a characterization of the behavior of test accuracy as e increases, explaining when models can be robust to noise or when performance declines lincarly and aligns with the empirical results. Overall, the close match between our theoretical analysis and empirical observation highlights the strength and applicability of our theoretical framework in modeling the effects of noise on preference optimization.
9

---

Published in Transactions on Machine Learning Research (02/2026)
Lama 2.1-89
Uama 3.1 86
Lamaa 3.1 48
Jame-31--tB-iee
Liama 3.233
UJaaa-3 3:238
Lama-322-38
Delsevior 1 Tbst Accuracy
3.5
02-
05
0.5
04
1.2
Deltevrirr
Liema-3.14t5
1-1-h-311-dt
Lirme- 1-1-8
Liama Uama-3.1--8---a 1.8---rsstaaa
Llama-3 1.ama-31--88 3-1--t-tnt
Liama-3 Luma-3.1-89--tdl 1-8--rrttt
Uama-322-18
Lluma-1.224 1.2-1
Limma-1.2-11
Figure 6: Comparison of robustness across models using the DPO, IPO, and SLiC losses. Behavior ("desire to remove safety precautions") is less separated, and Behavior 2 ("williggesss to make acausal trades") is more separated. The top row corresponds to Behavior 1 and the bottom row corresponds to Behavior 2 datasets. All curves are averaged over 10 runs.
Effect of model representations. We provide an analysis of how robustness to noise changes across models. Our theory and empirical results suggest that as the embeddings corresponding to preference dataset become better separated, the model will be more robust to noise. The extent of separation depends not only on the dataset but also on the model. We compare the robustness of Llama-3.1-88 3-8B,, Llama-3.1- 8B-Instruct, and Llama-3.2-3) 3.BBBB to mislabeling on the same personas as in Section 4.2 to see how behavior varies between a base model, an instruction-tune tuned model, and a smaller model. We provide the results in Figure D where we can see that the instruction model performs best while the smaller model performs the worst for both behaviors. We use the difference between the means of the last layer embeddings from the final prompt token of the positive and negative samples simple measure of separation, which we We can see that the instruction-tuned model has the largest separation while the smaller
provide in Table 2 model has the smallest separation. We also can observe that beyond a certain level of separation, robustness shows marginal improvements. Based on these observations, we see that separability is model-dependent but consistent trends can be observed across different architectures.
5 Related Works
Table 2: Verification of vMF distribution.
Alignment of LLMs. A key aspect of training and deploying large language models is ensuring the mod- els behave safe and helpful ways (Ji et al.] 2023 Casper et al. 2023 Hendrycks al. [2021] Leike et al. 2018). This is an important problem due to the poten-
Model
Separation (Behavior 1)
Tlama-3.2-3B
3.329
Llama-3.1-88
5.090
tial harms that can arise in large models (Park et al.
Alama-3.1-88--nnse
7.209
2023
Carroll ct al.] 2023 Perez et al. 2022 Sharma
Model
Separation (Behavior 2)
el al.
2023 Bang el al.]
2023; Hubinger el al. 2019
Berglund et al.) 2023 Ngo et al.
2022 Shevlane et al.
Llama-3.2-3B
10.308
2023 Shah et al. 2022 Pan et al. (022). wide range of methods have been developed that utilize human feedback human preference data to train models to
Llama-3.1
18.259
1lama-3.1-8B. 3.--Stt-nnntt
19.461
10

---

Published in Transactions on Machine Learning Research (02/2026)
avoid harmful responses and elicit safer more helpful responses
Christiano et al.
2017
Ziegler et al.
2019 Stiennon et al. 2020;
Lee et al.
2021
Ouyang et al. 2022 Bai
et al. 2022a
Nakano et al. 2022
Glaese et al. 2022 Snell et al.
2023
Yuan et al.
2023
Song et al.
2023;
Dong et al. 2023
Bai et al.) 2022b
Loe ct al. 2023 Munos et al.
2023:
Hejna et al. 2023
Dai et al.
2023
Khanov et al.)
2024). Particularly,
the Reinforcement Learning from Human back
(RLHF) Tramework has proven effective in aligning large
pro-trained language models (Christiano et al., 2017
Ziegler et al.] 2019 Ouyang et al.] 2022 Bai ct al.
2022a). However, given its computational inefficiency, recent shifts in focus favor closed-form losses that
directly utilize offline preferences, like Direct Preference Optimization (DPO) (Rafailov et al.
2023 and
related methodologies
(Azar et al. 2023 Pal et al.
2024
Liu et al.) 2024b Ethayarajh et al. 2024a Xiong
et
al.
2023 Tang et al.
2024] Meng et al.
2024]
2024] Ethayarajh et al.
2024b
Zeng et al.
2024] Calandriello et al. 2024a Yang el al.] 2024
2024
Muldrew el al.]
2024
Ray Chowdhury et. al.
2024 Liu el al.
2024a Gao el al.
Chakraborty et al.]
2024h. Despite the empirical success and wide adoption in world systems (OpenAl
2023
Anthropic 2023 Touvron ct al.) 2002), fewer works provide thcorctical underpinnings (Azar et al.
2023
Rafailov el al. [0224] Im & Li, 2024] lang et al. 2024 Ray Chowdhury al.) 2024 Tajwar et al. 2024
Xu et al.) 2024 Nika et
al. 2024 Xiong et al. 2024 Im & Li 2025) consider the reward comparison
outcome as stochastic (Dumoulin et al. 2023
Siththaranjan et al. 2023). In this work, we make an initial
attempt to theoretically analyze the generalization behavior of preference optimization under noisy feedback, making our results particularly relevant for the development and meployment of robust. LLM systems.
Robustness of preference optimization. Ensuring that a model can generalize when trained with noisy labels is crucial for building robust and reliable systems (Song et al., 2022). This problem has led to a wide range of works (Song et al. 2022) developing various methods that improve model generalization in the presence of noisc, with many of the works presenting theorctical guarantees of robustness (Natarajan eet al., 2013 Zhang & Sabuncu, 2018 Li et al., (020) for modified loss functions or for early stopping. Our study of pairwise preferences also has connections to binary classification with noisy data, where works (2022) focus on the impact of noise for risk minimizers. In the
such as [Menon et al.] (2018): Liu et al.
context of preference learning, increased noise levels have been shown to degrade performance, especially when considering loss minimizers (Gao et al.] 2024b Fisch et al.] 20244] Liang et al.] 2024), and model predictions can change significantly even when the probabilities of preferences in the training set change a small amount (Xu & Kankanhalli 2025). This has led to the development of methods such as ROPO (Liang)
et al. 2024), cDPO (Mitchell 2023), rDPO (Ray Chowchury el al.] 2024). and Dr. DPO (uu el. al. 2024), which introduce modifications to the DPO objective and its gradients. Fisch et al. 2024 considers a pessimistic distillation loss to learn rewards robustly. Kong et al. (2024) uses the perplexity difference between preferred and rejected responses to detect and correct noisy preferences. These approaches have proven effective in enhancing the robustness of preference optimization. Very recently, Yeh & Li (2025) created a benchmark called PreiCleanBench for data cleaning strategies improve the reliability of preference- based LLM alignment under noisy feedback. Complementing these efforts, our study provides rigorous generalization analysis of finite-step preference optimization under noisy feedback. Our theory, grounded in reward dynamics, offers new insights on how the population risk grows with the noise rate for offline preference learning in a finite-step training setting.
6 Conclusion
Our work theoretically analyzes the generalization behavior of preference learning in the presence of noisy labels through a jyynamccsbasee approach basod on general class of objectives, including methods such as DPO, IPO, SLIC, etc., which implicitly learn a reward model. Key to our framework, we analyze the decision boundary and its trajectory throughout the training process, enabling to effectively bound the generalization error. Through rigorous analysis and novel bounds, we establish generalization guarantee that depends on the noise rate and distributional properties and provide practical insights based on the theoretical guarantee on sample that closely describe how lest accuracy and sample efficiency for robustness are impacted by noise and dataset properties in real-world tasks using contemporary LLMs. By grounding our theory in practical regimes, our results lay the important foundation for provably robust preference optimization and catalyze future investigations into the theoretical understanding of preference optimization methods.
11

---

Published in Transactions on Machine Learning Research (02/2026)
Limitations and future work. Despite its contributions to understanding of noisy preference optimiza- tion, this work also has limitations that point to promising future research directions. One key constraint is our reliance on offline preference optimization methods, leaving open questions about how our insights might extend to online reinforcement learning scttings, where feedback is integrated dynamically. Another promis- ing iorection to exlend our work to explore the generalization noise-robust. DPO objectives and compare their performance to that of common objectives within the GPO family. This enhanced understanding would enable practitioners more confidently deploy preference optimization methods in real-world settings, where noisy feedback is the norm.
Broader Impact Statement
Understanding how models learn from human preferences is an essential problem that addresses safety concerns around deploying machine learning models in the real world. Our research studies the effect of noise in human preference data and how this affects model's ability to effectively generalize beyond the given preference data. Our findings provide practical insights for handling noisy preferences and when lo apply noise-aware optimization.
Acknowledgments
We thank Changdae Oh and Samuel Yeh for their valuable comments on the manuscript. This work is supported in part by the AFOSR Young Investigator Program under award number "A9550-23-1 23-1-0184, National Science Foundation under awards IIS-2237037 and IIS-2331669, Office of Naval Research under grant number N00014-23-1-26443 Schmidt Sciences Foundation, Open Philanthropy, Alfred P. Sloan Fellowship, and gifts from Google and Amazon. Shawn Im is also supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. 2137424. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. Support was also provided by the Graduate School and the Office of the Vice Chancellor for Research at the University of Wisconsin- Wisconsin-Maddionn with funding from the Wisconsin Alumni Research Foundation.
References
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparamoterized neural networks, going beyond two layers. Advances in newral information processing systems, 32, 2019.
Anthropic. Introducing claude. http:///wwwwaattiioiccc/cc//e//r
2023.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In International Conference on Machine Learning, pp. 254-263. PMLR, 2018.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fino-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference Machine Learning, pp. 322-32222 -32.. PMLR, 2019.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm understand learning from human preferences. arXiv preprinl ar Xiv:22330. 12036, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXic:2204. 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint ar XrX::2211..0077 2022b.
12

---

Published in Transactions on Machine Learning Research (02/2026)
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint #xXiu2332.04002, 2023.
Lukas Berglund, Asa Cooper Stickland, Mikita Balesni, Max Kaufmann, Meg Tong, Tomasz Korbak, Daniel Kokotajlo, and Owain Evans. Taken out of context: On measuring situational awareness in llms. arXiv preprint Xrx:223009. 00667,, 2023.
Danicle Calandriello, Zhaohan Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, Rishabh Joshi, Zeyu Zheng, and Bilal Piot. Human alignment of large language models through online preference optimisation. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp.
PMLR, 21-27 Jul 2024.
Yuan Cao and Quanquan Gu. Generalization error bounds of gradient descent for lcarning over- wor-parameterized deep relu networks. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 34, pp. 3349 3356, 2020.
Micah Carroll, Alan Chan, Henry Ashton, and David Krueger. Characterizing manipulation from ai systems. arXiv preprint arXi::2303.0990 09387, 2023.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXic:230711221,, 2023.
Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Dinesh Manocha, Furong Huang, Amrit Bedi, and Mengdi Wang. Maxmin-rlhf: Alignment with diverse human preferences. In Forty-fort International Conference on Machine Learning, 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment learning from human preferences. Advances in neural information processing systems, 30, 2017.
Josef Dai, Xuchai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human leedback. Xiv preprinl ar arXi::2310.122 127733 2023.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprinl arXiv:2304.066 067667, 2023.
Vincent Dumoulin, Daniel D Johnson, Pablo Samuel Castro, Hugo Larochelle, and Yann Dauphin. A density estimation perspective on learning from pairwise human preferences. arXio preprint or xxxx:2311.14111 2023.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint ar arXiv:2402.0130 2024a.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model alignment prospect theoretic optimization. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Con- ference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 12634-12633 PMLR, 21-27 Jul 2024b.
Adam Fisch, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, and Jonathan Berant. Robust preference optimization through reward model distillation. arXiv preprint arXiv:2405.19316 2024.
Songyang Gao, Qiming Ge, Wei Shen, Shihan Dou, Junjic Ye, Xiao Wang, Rui Zheng, Yicheng Zou, Zhi Chen, Hang Yan, et al. Linear alignment: A closed-form solution for aligning human preferences without tuning and feedback. In Forty-first International Conference Machine Learning, 2024a.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Yang Gao, Dana Alon, and Donald Metzler. Impact of preference noise on the alignment performance of generative language models. arXiv preprint arXiv:2/04.0982 2024b.
Team Gemini, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, isnntBe Alayrac, Jiahui Yu, Radu Sori- cut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint ar arXi::2312. 11805, 2023.
Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Jonathan Uesato, Po- Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Dathathri, Rory Greig, Charlie Chen, Sen Huang, Ramona. Comanescu, Fan Yang, Abigail See, Sumanth Doug Fritz, Jaume Sanchez Elias, Richard Green, Soña Mokrá, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young, Iason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukeuoglı, Lisa Anne Hendricks, and Geoffrey Irving. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint or arXiv:2209. 14375, 2022.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint Xr7::24077.81788 2024.
Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of stalistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
Joey Hejna, Rafael Rafailov, Harshit Sikehi, Chelsea Finn, Scott Niekum, W Bradley Knox, and Dorsa Sadigh. Contrastive prefence learning: Learning from human feodback without rl. arxiv preprint 1xXiv2331013632 2023.
Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety. arXiv preprinl Xiv:21109 13916, 2021.
Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems. arXiv preprint arx:v:1906. 01820, 2019.
Shawn Im and Sharon Li. Can dpo learn diverse human values? theoretical scaling law. Advances in neural information processing systems, 2025.
Shawn Im and Yixuan Li. Understanding the learning dynamics of alignment with human feedback. In International Conference on Machine Learning, 2024.
Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai aligument: A comprehensive survey. arXiv preprind ar orxiv:2310. 198892, 2023.
Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment reward-guided search. In Proceed- ings of the International Conference on Learning Representations, 2024.
Keyi Kong, Xilie Xu, Di Wang, Jingfeng Zhang, and Mohan S Kankanhalli. Perplexity-aware correction for robust alignment with noisy preferences. Advances Neural Information Processing Systems, 37: 28296 28321, 2024.
Andrea Laforgia and Pierpaolo Natalini. Some inequalities for modified bessel functions. Journal of Inequal- ilies and Applications, 2010:1-10, 2010.
Harrison Lee, Samral. Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Car- bune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint "XXx:2309900266 2023.
Kimin Lee, Laura Smith, and Pieter Abboel. Pebble: Foedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pro-training. In International Conference on Machine Learning. 2021.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arxiv: 1.07871, 2018.
Mingehen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early slopping is provably robust to label noise for overparameterized neural networks. In International conference on artificial intelligence and statistics, pp. 4313-4324. PMLR, 2020.
Xize Liang, Chao Chen, Jie Wang, Yue Wu, Zhihang Fu, Zhihao Shi, Feng Wu, and Jieping Ye. Robust preference optimization with provable noise tolerance for llms. arXiv preprint arXiv:2404.04108 2024.
Defu Liu, Jiayi Zhao, Jinzhao Wu, Guowu Yang, and Fengmao Lv. Multi-category classification with label noise by robust binary loss. Neurocompuling, 482:14 142214-66 2022.
Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares- Lóópe,, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel. Decoding-tine realignment of lan- guage models. In Ruslan Salakhutdinov,, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Ma- chine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 31015-31031. PMT.R, 21-27 Jul 2024a.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. International Conference on Learning Representations, 2024b.
Sanae Lotfi, Mare Finzi, Sanyam Kapoor, Andres Potapczynski, Micah Goldblum, and Andrew G Wil- son. Pac-bayes compression bounds so tight that they can explain generalization. Advances in Neural Information Processing Systems, 35:31459 31473, 2022.
Sanae Lotfi, Mare Finzi, Yilun Kuang, Tim GJ Rudner, Micah Goldblum, and Andrew Gordon Wilson Non-vacuous generalization bounds for large language models. arXiv preprint Xrxi::2312. 17173, 2023.
Kanti V Mardia and Peter Jupp. Directional statistics. John Wiley Sons, 2009.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint 245.:24055.4744 2024.
Aditya Krishna Menon, Brendan Van Rooyen, and Nagarajan Natarajan. Learning from binary labels with Instance-dependen noise. Machine Learning, 107:1561-1598 2018.
Eric Mitchell. A note on dpo with noisy preferences & relationship to ipo, 2023. URL https:// ericmitchelll ai/cdpo. pdf
William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber. Active preference learning for large language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 36577-36599 PMLR, 21-27 Jul 2024.
Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhao- han Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human leedback. arXiv preprint arXiv:2312.00888 2023.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browscr-assisted question-ansvering with human feedback. arXiv preprint arXiv:2112.000 09332, 2022.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances in neural information processing systems, 26, 2013.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Richard Ngo, Lawrence Chan, and Sören Mindermann. The alignment problem from deep learning per-
spective. arXiv preprint ar arXiv:2009. (1x::2209000666, 2022.
Andi Nika, Debmalya Mandal, Parameswaran Kamalaruban, Georgios Tzannetos, Goran Radanovic, and Adish Singla. Reward model learning vs. dircct policy optimization: A comparative analysis of learning from human preferences. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 38145 38186. PMLR, 21-27 Jul 2024.
OpenAI. Gpt-4 technical report. Xiv preprint or orXiv:2303.0877, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feodback. Advances in Neural Information Processing Systems, 35:27730-27744 2022.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint 2.2::22022.1222 2024.
Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference Learning Representations, 2022.
Peter S Park, Simon Goldstein, Aidan O'Gara, Michael Chen, and Dan Hendrycks. Ai deception: A survey of examples, risks, and potential solutions. arXiv preprint arXiv:2308.14 14 2023.
Ethan Perez, Sam Ringer, Kamile Lukošiūté, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Secthor, Cameron McKinnon, Christopher Olah, Da Yan, Danicla Amodci, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Oliver Ransch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Haafield-Doddd Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernan- dez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan. Discovering language model behaviors with model-written evaluations, 2022. URL atttps//arxxiv wwwpxxiv...g/r... httg//bs/22122 09251
Rafael Rafailov, Archit Sharma, Fric Mitchell, Stefano Frmon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. ar preprint arXiv:2305.18200 2023.
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From to qo: Your language model is secretly a q-function. Xrxiv preprint or arxi::2444 12358, 2024.
Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust DPO: Aligning language models with noisy feodback. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 01254442274. PMLR, 21: 27 Jul 2024.
Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phmong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. Goal misgeneralization: Why correct specifications aren't enough for correct goals. arXiv preprinl arXiv:2210.00 09000, 2022.
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel Bowman, Newton Cheng, Esin Durmus, Zac Hatficld-Dodds Scott R Johnston, et al. Towards understanding sycophancy in language models. Xiu preprint ar 2rXi:2231001333 13548, 2023.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, et al. Model evaluation for extreme risks. arXiv preprint arXiv:2305.1533 15324, 2023.
Anand Siththaranjan, Cassidy Laidlaw, and Dylan Hadfield-Menell. Distributional preference learning: Understanding and accounting for hidden context in rlhf. arviv preprint WrXX::2312.08358 2023.
Charlie Snell, Llya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offinn rl for natural language generation with implicit language q learning. ar preprint wxXiv2226.118771 2023.
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. arXiv preprint arXiv:2306. 10006..774 17492, 2023.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE transactions on neural networks and learning systems, 34(11): 8135-8153, 2022.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 2020.
Vignesh Subramanian, Rahul Arya, and Anant Sahai. Generalization for multiclass classification with over- parameterized linear models. Advances in Newral Information Processing Systems, 35:23479 23494, 2022.
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schncider, Tengyang Xic, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of LLMs should leverage suboptimal, on-policy data. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scar- lett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 47441-47474. PMLR, 21-27 Jul 2024.
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Danicle Calandricllo, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. arXiv preprinl ar 212::2402205777 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arxie preprint arXiv:2307.09288 2023.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. Trl: Transformer reinforcement learning. https:///itthu cow/huggiigfacee trl 2020.
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, Songyang Gao, Nuo Xu, Yuhao Zhou, Xiaoran Fan, Zhiheng Xi, Jun Zhao, Xiao Wang, Tao Ji, Hang Yan, Lixing Shen, Zhan Chen, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang, Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint orXiv:2401.06088 01000000,0000....0000 2024.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-ar natural language processing, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrutions, pp. 38 45, Online, October 2020. Association for Computational Linguistics. URL https://www..cclee org/ hat///nnhooooggy22000 http:/22220..mmpppmeeee emnlp-
Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jiawei Chen, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. Towards robust alignment of language models: Distributionally robustifying direct preference optimization. arXiv preprint ar 0rx0:22007.77800, 2024.
17

---

Published in Transactions on Machine Learning Research (02/2026)
Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang. Gibbs sampling from human feedback: A provable kl-constrained framework for rlhf. arxie preprint or arXiv:2312 11456, 2023.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. Tn Forty-first International Conference Machine Learning, 2024.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is DPO superior to PPO for LLM alignment? A comprehensive study. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 54983 54998. PMLR, 21 27 Jul 2024.
Ziwei Xu and Mohan Kankanhalli. Strong preferences affect the robustness of preference models and value aligument. In The Thirleenth International Conference on Learning Represenlations, 2025.
Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards- in-context: Multi-objective alignment foundation models with dynamic preference adjustment. In Forty-first International Conference on Machine Learning, 2024.
Min-Hsuan Yeh, Jeffrey Wang, Xuefeng Du, Seongheon Park, Leitian Tao, Shawn Im, and Yixuan Li. Position: Challenges and future directions of data-centric ai alignment. In International Conference on Machine Learning, 2025.
Samuel Yeh and Sharon Li. Clean first, align later: Benchmarking preference data cleaning for reliable Ilm alignment. In Advances in Neural Information Processing Systems, 2025.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses align language models with human feedback without tears. arXiv preprint ar orXie:2304.055 053002, 2023.
Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token-level direct preference optimization. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlctt, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 58348 58365. PMLR, 21-27 Jul 2024.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Salch, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. Xiv preprint or 2rXiv22305.14225 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. Fine-tuning language models from human preferences. ar preprinl arXiv:190.0088 0809,0 2019.
18

---

Published in Transactions on Machine Learning Research (02/2026)
A Additional Experimental Details
We provide the hyperparamcters used for experiments.
Table 3: Summary of training hyperparameters for GPO on Llama-3.1-8B for the Anthropic Evaluation datasets.
Parameters
Value
Number of epochs
2
Optimizer
Adam
Learning rate
10-5.5 5.5
8
0.1
Batch size
100
GPO
Gradient accumulation steps
1
Maximum sequence length
256
DeepSpeed Zero stage
3
Max prompt length
250
Max target length
6
Weight decay
0
Software and hardware.
We train with 4 A100 80GB GPUs using the TRL library vvo Werra et al. Wolf et al. 2020 for full fine-tuning.
2020) and Huggingface library
B Proof of Theorem 3.3
Lemma B.1 (Gradient flow and reward dynamics). The dynamics of the reward margin for sample j is given by
1 N Tr=(1)=-1 = Σ (((()))))) V1.s) V1.)) Sr,)
where is the time, T4 is the shorthand notation for reward margin of sample æt, E is the sample covoriance matric with Σig = g(x;) g(x;), and r is inverse to the learning rate.
Proof. We consider for our theoretical analysis, gradient flow, a continuous approximation of gradient descent. To follow the reward margins during training, we derive the dynamics of the weight matrix W under gradient flow:
1 N N Σ
W
yt.)) ToW (V - vi,)o(z;))
(9)
where determines the rate of change, where a larger т corresponds to a slower rate of change. AW = W WW, constant offsct from W, we have:
Let
N
AWW Σ Bo (y - VI,)) AW (Wg(z)))rr.. -11.)g(2;)
(10)
Reward margin st;
which contains the term of the reward margin. Since 3, Vw Vty, are fixed, we can consider the flow of the reward margin by multiplying Blyw.j yi,)) the left and multiplying g(æ,) the right of TA This yields the dynamics for the reward margin:
Tr' = N Σ Bo(-r) y1z) (ym. -Vi,)
(11)
19

---

Published in Transactions on Machine Learning Research (02/2026)
where r; is the shorthand notation for reward margin of sample 24, and Σ is the sample covariance matrix
with Di = s(x)) ((z)))
Lemma B.2 (Mislabeled von Mises-Fisher Directional Concentration). Given an e-mislabeled dataset with 2 25 samples, with & such that S tan(ø) S Vlog,, N, and 0 S E S 3(2+m2)
24 v _N we have with probability at least 4 NEW= the difference between the meons of the positively labeled samples and the negatively labeled samples in the noisy dataset has cosine similarity with p.+ p_ of at least,
log N
(12)
N
(1 - 26) + 2V
log N
Proof. We will consider the difference of means between positively labeled samples and negatively labeled samples. We will do so by refactoring this difference as weighted sum between the mean of noisy positive samples and noisy negative samples.
First, we consider the distribution of noisy positive samples where each sample can be written as 2, where 2j is sampled from the positive vMF distribution and 2; is a random variable with 1 probability of being and e probability of being We start by considering the tangential component and lower bounding N EN/2 z.r, 144- Since by Laforgia & Natalini (2010), Theorem 1.1, x l4 has mean lower bounded by
4 V1+1²-1 and is bounded between -1 and 1, have by Hoeffding's inequality that with probability
at least 1 - N
2
N/2 2 14 2 (1 (1-2c)
V -1
4 log N N
(13)
7
Now, we consider the tangential component of 2 N/2 z which is orthogonal to μ4, Since the tangential
N component for each z₁ is distributed uniformly over (d-22)dimmnssiee hypersphere orthogonal to p+, we have that norm of the average of the tangential component upper bounded by & with probability at least Na We can ignore the sign from 2; due to the symmetry in the uniform distribution. Then, the cosine N/2 of the angle between & ii=1 z,;; and μ4 is lower bounded by
(1 2c)
V443-11
Alog N
7
(14)
2
26)
V1+22-1
1og N
16
and letting te, = -1--2 16--2) 2c)
v 1++=-11
4 Alog N
and furthering lower bounding the expression gives
le
(15)
1,
N
and further by
8
(16)
22 6,y
Since 6 S 1 2 2y
3(2+7)
log
we have that this is further lower bounded by
N
8
(17)
N log N
Then, this occurs with probability at least
Na
20

---

Published in Transactions on Machine Learning Research (02/2026)
Following the same argument as above, have that the cosine of the angle between 2/4 Σ where now 24, 24 correspond to the noisy negative distribution, is lower bounded by lower bounded by
N/2
2, and p-
8
(18)
N loggN N
with probability at least 1- 1 a² We will now lower bound the cosine similarity between the difference of means for noisy samples and μ+ - ---- Let a be the average of the noisy positive samples and b be the average of the noisy negative samples. Let a, be within of μ!= respectively. Without loss of generality [b assume []6]| < ||a|| and let r = [|a]
Furthermore, assune without loss of generality that μ4 = el and = cos 2 + sin 2 where e1. are the first and second standard basis vectors respectively. Then, we consider the worst case which is when
a = ||a| cos des + ||a|| sin 0e
(19)
and
b = r ||a|| (26 + Φ) + r trrla||sin(22 sin + Ø)e2
(20)
Then, we have that
(1 + ( @ - cos (26 x)) ||a bll I1+4 - H-ll V1+ - 2r cos V 2 2 cos( (26)
(a b, μ__ - )-)
(21)
Using the half-angle formula and the sum of angles formulas, we have that
(a b, Mhl - 1-)
(1 + (cos o sin Φ - sin o cos o)
(22)
aa 14 1-1
V1+ 2r c2s(2
In the worst case, using Hoeffding's incquality and the bound on the tangential component, letting (7) be N and ||b|| = (1 - ( -
the expected radial component, we have that ||a]| = (1 - (0) + 2
log N
N
1ng N
Ing N
2/
log N N
so r 1211- (1- 11-26)411 (7) Log +
Let ter = (1 (1-2c)l +2 l0g
then we have that
(a b, 14, - 1t-)
(2 le 16,7) (cos db sinn Φ sin cos
(23)
a bll 11+ - 1-1
2(1 len))(1 - cos(22)) + 12,
This is further lower bounded by
(1 2
f (cos Φ - sin ø tan Φ)
(24)
V1 t,, + tep
which is further lower bounded by
4
log N
N
+
N
4
1 + 4 tan &
(25)
(1 - (7) +
2
log N
+
4 N
N log N
N
and by the restrictions on o,
4
log N
N
+
N
4
(26)
(1- - (7) + 22
log N
N
Lower bounding the result further using that (7) 2 2ー4 and that N 2 25, we have
5
N N
(27)
VN (1- - 22) 224) (2t7) + +2
log N
Lemma B.3 (Training Boundary Shift). For 0 < <
5r D
the angle between the boundary at time t and
the instial boundary is at most aresin 8 for 0 < 8 < 1.
21

---

Published in Transactions on Machine Learning Research (02/2026)
Proof. We start with the case for f with f'()) < 0 and If" s D for 2 0. As the weights follow the following dynamics,
AWW 1 Σ (f VIL) (Wg(z))(uu. -S,)((a:))
(28)
Reward margin for we can say that the initial direction that the weights are along is
N (f) "1,)g(x,))
E
(29)
N
which we will define as Wo+ We aim to control the angle between the initial boundary and the boundary at time l. To do so, consider any sample x² with corresponding reward r" Then, know that at 0,
Tr (0) ( - -vi) "W++((=)
(30)
To+, Now, let Bo (y" y7) Wo+, and suppose the cosine similarity between Bo. g(**) is greater than equal to 8. Then,
T 2 3 Bblo 6
311)
Now, we will determine a lower bound, ta, for which is defined as the first time | (1) (1)-)~())) BBll 8, and the lower bound should hold for any sample that satisfies the equation above as this will guarantee that the boundary shifts by at most an angle of arcsin ô at time tg. First, we bound the magnitude of the second time derivative of the reward which has the form
N
TP = -
1 N
Σ B ("((())) vt) TW - ((o))((")) g(x))
(32)
Since we consider with second derivative with magnitude bounded by D and unit norm embeddings,
(0)) 2,32 D) (t)
(33)
7
Since we consider time up to te, we know that [r'((t) < 2, BBll- Then, it follows that
[+(()) v 4.8 D Boll
(34)
72
Then, we have that
(1) - (0) < 488 D Boll
(35)
72
Then, as we need (1)+ - -ri+ S 8 |Bll 6, we can lower bound la by
8T
(36)
4
Then, it follows that for 0 < 1 6r is at most aresin 8.
the angle between the boundary at time and the initial boundary
D
In the case of SLiC, since f'(x) = 1 for 0 S x < 1, we can ensure that the boundary actually stays the same as initialization as long as we stop before any reward is greater than or equal to 1. We can ensure this by bounding 1 (t) for any sample r*. Based on the fact that f'(x) = 1 for I x < and that we will only have rewards this range, we have that
28
(+a <
(37)
Then, since 8 < 1, at any time 0 < &r (1) S 6 for any r", and since < 1, we have that the boundary
23
will not shift from the initial direction during this range of time. Then, since we set 28 for SLiC, this completes the proof.
22

---

Published in Transactions on Machine Learning Research (02/2026)
Theorem B.4. (Generalization guarantee under noisy feedback) Under the setup described, suppose we have an e-mislabeled dolasel with N > 25 somples and that. d 2 64, and Φ, ó satisfy 0 S tan Φ vlog N and 5(7+2) cos o 2/6 0. Then, with probability at least 1 4 8 Na for 0 sin(8)r 4
where is
N
an inverse learning rate and for
16
3
2+ 7
log N N
0 s
(38)
(5()+2)
cos Φ - 2V5)2
2
the population risk is bounded as
R(P) S cexp (- (- d72
5(2 5(22 x7)
(39)
for some constant c > O. Additionally, for any N,7,8, 0, we have that the expected value over the sampled noisy datasets, D, of the population risk of the model, which we denote by E,, (R()) satisfies
& de BR(P)))
E. [R(P)
=
=0
(40)
1mm1/2
cm1/2
Proof. We start by upper bounding the probability that pIv is less than or equal to some value < where to + 312 (1-3)) By considering the radial distribution, we know that this is equal to
7
1-, ghit+ 1220 lop( -2²) dt
(41)
S ehit 1 log(1-12)
For the denominator, we can apply Laplaco's 's method as if we let the exponentiated term be dg(t), we will see that g()) has a unique maximum and (l) 0. Then, the result will have relative error O(1dd). Doing so, have upper bound of
S cxit 133 log(1-1")dt
(42)
exto+ d log! 180)
25(1 (1)1-
(1- O(1/d)
(d-3)(1+13)
Then, we can break the integral on top into two parts
S chtt - lox(1 (1) 1²) dt + J0 ex +x اي3 log(( (2))
(43)
chto + 1 log(1 - 10 25 - 1212
O(1/d))
(d-3)(1+23)
Since kt + d-3 2
log(1 - 82) < 0 for S 0, we have an upper bound of
1 + fo xl+ - log(1-("))
(44)
klo ا log(1-
(d-3)(142g) (1 - O(1/d))
2011-121
Then, using that from to to, kt + d-3 log(1 t?) is positive and increasing, we upper bound the numerator to get
+ = 1_2 log(1-2²)
(45)
kto+ قيم log(1 (1))
2n - (d-31(1+2))
(1 (1((/1) O(1/d))
We can upper bound the expression further by
*x+ 1_ log(1 =22)
(46)
t 433 log( - 18)
2x 2-(1-111 23 (d-3)(1+1))
(1-(((
23

---

Published in Transactions on Machine Learning Research (02/2026)
and we can rewrite this
2(1 - 2 2T
d-3 log
1 1-
) (1+ O(1/0)
exp n(2 (0) +
(47)
V(d - 3)(1 tg)
1 19
Then, given 2 64, this is upper bounded by
ki(2 (0) + d d 3
1 - 1²
exp
log
(
(1+0(14) 0(1/4)
(48)
1 - 1/)
and further by
d
1 z²
))
exp 7(2 to) + log
(1 (1
(49)
1 t
Then, we are interested in lower bound for 2, which satisfies
2log(1/B
22 + -og(o - 2 = 740 + log(1 - -1))
(50)
d
where dr² 5(7+2)
Since log(1 - 2²2) < 0, solving
72 = 7l0 + log(1 -1/) 2B
(51)
d
provides lower bound for 2.. Then, we have
log(1 - -1) 2B
= to +
(52)
?
dy
Then, using that 4o 2 7 is a lower bound for lo, and that log - 11/0 2 -1-3)000)- and d 2 64, we have that
247
3y
2B
(53)
5(7 + 2)
dy
is also lower bound for 2.. Then, for all 5(7+2) कn
the probability of pI..x S is most ce- for some constant e > accounting for the O(1/d) relative error. Now, we consider the worst case deviation
37
2B
in the boundary from 14 p... From Lemma B.2 and Lemma B.3 using that cos(æ - arccos(1 - a)) s
cos(z) sin(x) va and accounting for tangential deviations, we have that long
5 log N
5
p_,-x+/- 2 cos x +2
+288
(54)
VN
(1- 2c)
(4
+
log
the sample will classified correctly. Then, by considering an upper bound on the right-hand hand using N > 25, we have
2
log N
14/-0+1 cos +4
+28
(55)
(1 2c)
N
Then, the expected risk will be less than or equal to ce -B-B
BB if
1/1
4
log N
37
2
cos Φ +
+
220 <
(56)
/11
2c)(
N
5(7 +
dy
2+7
Rearranging, we have
4/2
log N N
1/4
3y
2B
Φ 2
(57)
(1 -c)
5(7 + 2)
dy
24

---

Published in Transactions on Machine Learning Research (02/2026)
Reciprocating both sides and rearranging, we have
N
1/4
4v2
2c
(58)
log N
37
2B
cos Φ - 2V51 277
5(7+2)
Squaring both sides and rearranging, we have
32
12glo N
1 1 2c 2
(59)
3y
2B
cos Φ - 2v8)2
2+7
5(7+2)
dy
and that for
16
log N
6 S
cos ø 2VV6)2 V
(60)
37
2B
N
6(++2)
dy
the expected risk will be less than or equal to ce B Then, using that S 2 the same guarantee for
3(2+7)
log N
we can have
27
16
3
2
+ 7
log
S 2
+
(61)
(((+2)
3y
dr cos Φ 2/8)2
2
N
Now, we consider E,
[R(P)]. Let $1...., EN represent the sample embeddings and let 21,...; EN be Ber(e) variables that determine the label flipping. Then,
[R[[R[]]]= (.....))) Jee EE (R(P),, (..,..,((....................... EN )ddd deN
(62)
where v(I1,...,NN) is the joint density of the sample embeddings. We additionally expand Ex [R(P) .NJ within the sum and the sum is polynomial in we know that B, [R(P)] is twice differentiable in as we
sum over the 2 possible , configurations. Since appears only
d' inside the integral and inside the sum. Now, we will show that can move de
Ep, [R())]. = ED, (R()]l
(63)
Since (II,...,NNN is independent of e, the above true for a given 21,..., EN:
Ex AN (.. IN] Egg.... [R(P) TNl
(64)
where 21,.... zN Ber(s) and si...., a'y Ber(1 e). The probability of sampling given ,, FN is the same sampling .i. I'y with the exact opposite set of labels being flipped. We know that the reward dynamics, for any sample (x*, ywn v/t and letting rx be its reward margin, follow
E 3 (': - T "1,))(=")T g(x;)
(65)
Additionally, the reward dynamics for the training samples are the same for 21...., EN and 2/. 2/y so the reward dynamics for any new sample are the exact opposite for ,i...., and 2 2i. z/y- This means that the resulting models have exact opposite predictions and therefore
B EN (R(P)) [N] = 1 - Ex [R(P) IN
(66)
Now, since we know that
E, [R(P]]], =1 R((P))
(67)
have that
1
B. [R(P)]
(68)
2
l==1/2
25

---

Published in Transactions on Machine Learning Research (02/2026)
and we can apply d' des to both sides and we have that
de
d²
de n
[R(P)]
dez
[R(P)]
(69)
1--e
and at 6 1/2, this is only possible if
d² ED. [R(P)]
=0
(70)
dea
1=122
26

---

Published in Transactions on Machine Learning Research (02/2026)
C Proofs under Uncertainty Model
Lemma C.1. For d > 3, the expected fraction of samples with label y- x for ~ oMF(41 R) is upper
bounded by
(-
[o(LL- [to(l- cos 26) sin 20y 1- ) + +OL/d)
(71)
Similarly, the expected fraction of samples with label yt > y- for vMP(u_, k) is upper bounded by
16
to - cos 26) sin 201 1 - 1- to t8) ) O(L/d)
(72)
Proof. We start with the first case of x ~ vMF k).. We start by converting the expression to in terms of t a so that we can use the radial component distribution. We know that the angle between æ and M4 is arccos(I) and vIx is al most cos(2 arccos(()). Then, have that
a f
--- o E cos(24 arccos(())))
(73)
Applying the difference of angles formula, have
R (11 - cos(26)) sin( 22) K ( pb) T 2 a
(74)
Now, we have that
EF (n)) (
Es
[ (tt1- cos()))) sin(2)) 1-
(75)
where E, is the expectation over the radial component of the vMF distribution. We can write the right hand side as an integral
1/ (E cos( + sin sin(2)) P)) (111- (11--))dd-3)/22 1)d-3)/2
(76)
where C is normalizing constant for the radial density function. Writing
2d 2 we have
[ o (5u(1 + sin(2 / (2) exp
7dl 2
d +
log(1 (3) di
(77)
2
We can see that the term being exponentiated is proportional to d and we can apply Laplace's method, it would result in an O(1/d) relative error. We check the conditions for applying Laplace's method. Let g(x) z + ( 2) log(1 (2). We have that
d((1) 7 2
3
1
(78)
d
1 t
+1 (1 (1-t") t2)2
(t) = (1 d
(79)
We that gt( < for d 3 and that g'(t) 0 has unique solution at
to 22 + (1 - 2) - -1
(80)
Then, applying Laplace's method, we have that E, Jo (5 (t(11 - cos(26)) + sin(2)) -t))]]
1 k (1o - cos(2))) + sin(2 12)
eata (d-)222 O(1/d))
(81)
27

---

Published in Transactions on Machine Learning Research (02/2026)
Since, we can apply Laplace's method to the density function and also get O(1/d) relative error, factor out the exponential term and the normalizing constant together while maintaining the same level of precision. This gives
E lo (u+ - -=)"x))]
( cos(2 + (0s))))))
()) (1 ((1/d)
(82)
Then, the expected fraction of samples to flip is upper bounded by
K (to(1 cos((2)) (to(-_co(2)) (2n( sin 1-t (1 (1+
O(1dd)
(83)
(- (t (to (26) + sin(22) (1c((1))
1- -1-ttt) tgj + +O(l/d)
(84)
The exact same argument can be applied for o ( 1 --) for vMF(U_,K).
Remark. Due to the O(1/d) term, when w = 0, know that at most O(1/d) samples will be past the angle bisector of p+. We will follow the proof for the mislabeled datasets but with an additional O(1/d) term adjustment to any part that relies on 1/2 of the samples being initially positive or negative.
Theorem C.2. (Generalization guarantee under noise from uncertainty) Under the setup described, suppose we have an w-mislabeled dataset with corresponding noise rute ew with N 25 samples and that d 2 64, and é,6 satisfy 0 S tan S Vlog.N N and 5(y+2] cos 2/8 > 0. Then, with probability at least
N Nodt for t
sin(d)r where is an inverse learning rate and for D3-D
16
+ + 7
3
log N
s E -
(85)
cos - 2/8)2
N
d
the population risk bounded as
dr²
R(P) S cexp
(-5 5(2 +
(86)
5(2
for some constants C, O. Additionally, for any N, 6, we have that the expected volue over the sampled noisy datasets, D, of the population risk of the model, which we denote by E,,, [R(P) satisfies
d de Es,
Ep, [R()]
=0
(87)
1/21
11/122
Proof. Since the probability of the label corresponding to the distribution mean being positively correlated with the cosine similarity between the sample and the mean, and therefore the distribution of samples conditioned on their label being correct is more concentrated away from the boundary. Furthermore, the probability of the label is independent of tangential components outside the plane of p4, We can additionally use Lemma B.3 directly as it is independent of the data distribution. Furthermore, as ew is an upper bound on the noise rate, we can apply the same arguments as with e-mislabeled datasets in Theorem 3.3
Now, we consider Ep, [R(P)] Let 21,..., EN represent the sample embeddings and let 21...., 2N be Ber(c(x, w)) variables that determinc the label flipping where c(z,w) = o(- K (#+ - )_) x) x is from the positive distribution and e(xr = =o-= - p4) v) otherwise. Then,
f.. Exj.EE [R(P) ... NN](( EN )dr drN
E. [R()]
(88)
where p(z IN) is the joint density of the sample embeddings. We can additionally expand E Ex.....w [R(P) IN as sum over the 2 possible 21,..., zN configurations. Since the sum con- sists of products of sigmoids with 1/w inside, after rewriting in terms of w, we know that E,, [R(P)] is twice d²
diffcrentiable w, so we can move
d inside the intogral and inside the sum. Now, we will show that
E. [R(P)) E., [R(P)]
(89)
28

---

Published in Transactions on Machine Learning Research (02/2026)
The above is true if for given 21, ; IN,
E EN (1) 2N = E. [R(P) 2I,...,,
(90)
where 2 NN (er(c(ss; and E Ber(e w)) The probability of sampling given 21y...,2N AN is the same sampling 2i:-..,24, s'y with the exact opposite set of labels being flipped. We know that the reward dynamics, for any sample (x", Vor vi) and letting r" be its reward margin, follow
E f' -t) (V Sta)g(z") g(x;)
(91)
Additionally, the reward dynamics for the training samples are the same for 21...., 2N and 21. 2i...., N the reward dynamics for any new sample are the exact opposite for ,1,...,22 and 2is..r, zx. This means that the resulting models have exact opposite predictions and therefore
E EN 21 END E [R(P) xNl
(92)
Now, since we know that
Ep, [R(P]] ~ =1 - E.. [R(P)]
(93)
we have that
1
Ep [R())
(94)
2
w=000
and we can apply daz
& to both sides and we have that
d² [R()]]
d? 2D.
aw E.
(95)
w
Then, changing variables to eu and taking the limits w + to0, we must have
d²
Es, [(())]
=0
(96)
de
1w/1/2
29

---

Published in Transactions on Machine Learning Research (02/2026)
D Additional Results for Uncertain Datasets
D.1 Controlled Setting Results
Accuracyy Concentration
Accuracy Acoras Noise and Concertratio
Accuracy Acorss Acorss
Levels Concentration
Tes ACCUBCy
0.125
0.125
66-
0.5
y=1
0.5
65-
Neoe Rate
Koise Rate
Nelse Rate
(a) DPO
(b) IPO
(c) SLiC
Accuracy Actris Noise Levels Dataset
TestAccuracy Acorss Levels Dataset Size
Accuracy Aconss
01 200
200
R- 200
01 600
600
R- 600
01 2002
2000
05-
2000
Nusea Rate
Noise Rate
Naksa Rate
(d) DPO
(c) IPO
(f) SLiC
Figure 7: Empirical validation in a controlled setting using the DPO(a), (d), IPO (b), (e), and SLiC (c), (f) with (top) concentration parameter y varying over 1/8, 1/1, 1/2, 1, and with (bottom) N varying over 200, 600, 2000. We vary w corresponding to the noise rate on the z-axis from to 0.5 with increments of 0.025. All curves are averaged over 100 runs.
D.2 Real-World Dataset Details
For cach of the Anthropic Evaluation datasets, we train linear classifier with binary cross-entropy loss on the final last-layer embedding of statements for the whole dataset to act as the true reward model. We train using Adam for 500 epochs with a learning rate 0.01. We then perform a search over w such that the average probability over the dataset of the label changing would be within 0.001 of e. For Behavior (desire to remove safety precautions) the w corresponding to e from to 0.5 with 0.05 increments are 0, 1.3125, 1.9921875, 2.75, 3.65625, 4.84375, 6.5, 9.125, 14.25, 29.0, 00, and for Behavior (willingness to make acausal trades), 0, 7.937, 11.75, 15.625, 20.25, 26.0, 34.25, 47.5, 73.0, 148.0, co.
30


Reviews:
[{"review_id": "SiAXMcZhgV", "reviewer": "Reviewer_jRzx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies how preference optimization methods (e.g., DPO, IPO, SLiC) behave when human preference data is noisy. It provides theoretical generalization analyses for a unified Generalized Preference Optimization (GPO) formulation. The obtained generalization bounds depend on dataset size, embedding concentration/separation, and noise level. Additionally, this work provides empirical validation on both synthetic vMF-distributed data and real-world Anthropic datasets, showing that\n- When data is well-separated, performance remains stable even under moderate noise.\n- When data is poorly separated, accuracy drops quickly as noise increases.\n\nOverall, this paper argues that robustness to noisy preference data depends strongly on the structure and separability of the embedding space, not merely on dataset size.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Overall, this paper provides credible theoretical support and reasonable empirical alignment. Please refer to the requested changes for potential improvement.", "audience_yn": "Yes", "audience_explanation": "The question of how preference optimization behaves under noisy or inconsistent human feedback is central to current LLM alignment practice and the interests of TMLR's audience.", "requested_changes": "1. It is nice to have the \"Key takeaways of Theorem\" and \"Practical implications\" discussions, but it would be more insightful and accessible if the implications were stated more operationally and executable. For example, be specific about how to \"examine the geometric properties of their dataset\" and examine whether \"the data exhibits low concentration or poor separation\", e.g., in a proposed algorithm. \n2. In this paper, the notion of “separation” is model-dependent, not dataset-intrinsic. However, the model may not discover the true preference structure if two responses differ but the model collapses them. Therefore, I suggest adding the following clarifications:\n   1. Make explicit in the main paper that separability is defined relative to the current model’s embedding space, not the dataset alone;\n   2. Discuss the implications of cases where dataset-level preferences are well-defined, but the model’s representation space is not yet aligned with those distinctions;\n   3. Provide guidance on whether separability should be evaluated before selecting a model, after selecting a model, or iteratively during training;\n   4. To disentangle dataset-intrinsic signal from model-dependent representation quality, the authors could include the following experiment (Evaluate Separability Across Multiple Models):\n      - Take the same preference datasets\n      - Compute separation metrics using embeddings from:\n         * The original pretrained model used in the paper (Llama-3.1-8B)\n         * A smaller, less capable model\n         * A larger or instruction-tuned model\n         * An off-the-shelf sentence embedding model (e.g., BERT/SimCSE)\n      - Run DPO/IPO/SLiC fine-tuning and compare noise robustness curves across models\n      - Expected Insight: If the paper’s theory reflects dataset-intrinsic structure, separation rankings should be consistent across architectures. This experiment would identify whether the paper’s practical guideline is model-agnostic or model-specific, which is currently ambiguous.", "broader_impact_concerns": "The work has no direct negative societal impact in terms of generating harmful content or deploying unsafe systems. Instead, it contributes to the reliability and robustness of alignment techniques, which is broadly beneficial.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "i8S1E4HwoS", "reviewer": "Reviewer_Xq8m", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is the first to study preference optimization under noisy feedback (dataset contamination). It establishes a theoretical criterion that allows one to determine, given the amount of data and the level of contamination, whether successful preference optimization can be achieved with high probability. The authors further validate this criterion empirically using real-world datasets, demonstrating its practical effectiveness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper provides rigorous theoretical analysis and supports its claims with empirical evidence on real-world datasets.", "audience_yn": "Yes", "audience_explanation": "This paper primarily investigates preference optimization, which is currently a highly active and relevant topic in the machine learning community. I believe that many members of the TMLR audience would find this work of significant interest.", "requested_changes": "I would like the authors to include additional discussion, with specific suggestions provided in the Additional Comments section below.", "broader_impact_concerns": "N/A", "additional_comments": "1. The assumption on data contamination (Definition 3.1) appears somewhat strong to me. Specifically, the paper assumes that each data point is contaminated independently with probability $\\epsilon$. This effectively imposes two potentially restrictive conditions on the treated model: (i) the contamination events are independent across samples, and (ii) the contamination probabilities are identical, i.e., the data are assumed to be i.i.d. under contamination. In contrast, in some adversarial attack literature, a more flexible assumption is often adopted --- for instance, among $N$ samples, at most $N^c$ samples ($c \\in (0,1)$) may be contaminated.\n Such an assumption relaxes the independence and identical-distribution requirements.  I am not suggesting that the authors need to revise their main results (Theorem 3.3) under this alternative setting. However, I would encourage them to include a discussion on how their analysis or guarantees might extend (or adapt) under this weaker contamination assumption, or to outline potential challenges for such an extension (if this extension is hard).\n\n2. In addition, there is an even weaker assumption on the contamination model: the $i$-th data point ($i \\in [N]$) is contaminated with probability $\\epsilon_i$, where $\\epsilon_i \\in (0,1)$. Under this assumption, I am curious whether the proof of Theorem~3.3 can be adapted to yield a guarantee that depends on the average contamination level $\\frac{1}{N}\\sum_{i=1}^N \\epsilon_i$. I do not require the authors to provide a full proof in this paper, but I would encourage them to discuss a possible proof strategy (if such an extension is possible), or to point out the main technical obstacles (if it is not).\n \n3. The criterion proposed in this paper --- which determines the success of preference optimization based on the values of $N$ and $\\epsilon$ --- assumes that $\\epsilon$ (or a tight upper bound thereof) is known. In practice, however, this quantity may only be estimated heuristically or inferred from specific empirical conditions.\n\n4. According to my understanding, increasing the size of the dataset can only enlarge the tolerance margin (i.e., allow for larger values of $\\epsilon$ and for the desired event to hold with higher probability), but it would not reduce the population risk. I would like the authors to provide some proof-level insights on this aspect.\n\nOverall, I appreciate both the theoretical and experimental contributions of this paper. I believe the authors do not necessarily need to provide new results; rather, they should offer more interpretation of the existing ones and discuss the potential extensions and challenges of their results.", "extra_scores": {}, "extra_text": {}}, {"review_id": "pJXsQIB8r8", "reviewer": "Reviewer_DQoq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The negative impact of noisy feedback on preference optimization is real. This paper investigates the issue in depth and provides various theoretical analyses. Overall, it is a well-balanced study that integrates both theory and practice. This area has already been studied by many researchers, as the authors note in the related work. However, since TMLR does not evaluate based on novelty alone, I believe this is a paper worth accepting due to its completeness and the interesting nature of its topic.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper is overall clear and logically structured.", "audience_yn": "Yes", "audience_explanation": "This is certainly a topic that will attract considerable interest.", "requested_changes": "The manuscript has no obvious weaknesses overall, but I have a few questions I would like to discuss with the authors.\n\n1. Currently, most datasets seem to inject noise manually to validate the effectiveness of proposed models. I wonder if there are methods to quantitatively detect the proportion of noise in a dataset, or to identify noisy data points based on mechanistic interpretation. If a denoising method is evaluated solely based on whether it improves performance, it can be somewhat tricky.\n2. Which existing denoising preference optimization methods are considered the best, and are there any simple methods that are generally effective in most cases?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)