Scalable Valuation of Human Feedback through Provably Robust Model Alignment
Authors: Masahiro Fujisawa
Masaki Adachi
Michael A Osborne
Venue: NeurIPS 2025 poster ()
Keywords: Direct Preference Optimization
Robustness
Label Flip
Valuation of Human Feedback
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We propose Hölder-DPO, the first alignment method with a provable redescending property, which enables robust learning from noisy human feedback by identifying and correcting mislabeled data, improving alignment and model performance.

Abstract: Despite the importance of aligning language models with human preferences, crowd-sourced human feedback is often noisy---for example, preferring less desirable responses---posing a fundamental challenge to alignment. A truly robust alignment objective should yield identical model parameters even under severe label noise, a property known as redescending. We prove that no existing alignment methods satisfy this property. To address this, we propose Hölder-DPO, the first principled alignment loss with a provable redescending property, enabling estimation of the clean data distribution from noisy feedback. The aligned model estimates the likelihood of clean data, providing a theoretically grounded metric for dataset valuation that identifies the location and fraction of mislabels. This metric is gradient-free, enabling scalable and automated human feedback valuation without costly manual verification or clean validation dataset. Hölder-DPO achieves state-of-the-art robust alignment performance while accurately detecting mislabels in controlled datasets. Finally, applied to Anthropic HH-RLHF dataset, it reveals substantial noise levels and removing these mislabels significantly improves alignment performance across methods. The code is available at https://github.com/ma921/HolderDPO.

Scalable Valuation of Human Feedback through Provably Robust Model Alignment
Masahiro Fujisawa 2:,,,,244 Masaki Adachi Michael A. Osborne
The University of Osaka, LLttice Lab, Toyota Motor Corporation, Machine Learning Research Group, University of Oxford, RIKEN AIP Equal Contribution Corresponding Author
fujisawaBiss ist wttp://wwssst..oo........... osaka-u.ac. jp, {masaki, moab) @ @robots........o .oaattsoox.cee ax.ac.
Abstract
Despite the importance of aligning language models with human preferences, crowd-sourced human feedback is often noisy-foo for example, preferring less de- sirable gypponses-posing a fundamental challenge to alignment. truly robust alignment objective should yield identical model parameters even under severe label noise, property known as redescending. We prove that no existing align- ment methods satisfy this property. To address this, we propose Hölder-DPO, the first principled alignment loss with a provable redescending property, enabling estimation of the clean data distribution from noisy feedback. The aligned model estimates the likelihood of clean data, providing a theoretically grounded metric for dataset valuation that identifies the location and fraction of mislabels. This metric is gradient-free freee enabling scalable and automated human feedback valua- tion without costly manual verification or clean validation dataset. Hölder-DPO achieves ttateoof-the-at robust alignment performance while accurately detecting mislabels in controlled datasets. Finally, applied to Anthropic HH-RLHF dataset, it reveals substantial noise levels and removing these mislabels significantly improves alignment performance across methods. The code is available'.
Introduction
Aligning large language models (LLMs; [1, 95, 92]) with human preferences is essential for value alignment and mitigating safety risks [65, 9]. Direct Preference Optimization (DPO; [84]) has emerged as key method, tine-tuning LLMs using pairwise comparisons-
between preferred and rejected responses [25, 18]. This framework supports alignment with a broad range of rankable values, such as helpfulness [8, 51], and summarization quality [90, 97]. Human feedback is typically gathered via crowdsoureing (e-g., ChatGPT (1)), but is often noisy e.g., preferring undesirable re- can significantly deteriorate model performance [105, 24, 29]. In fact, the Anthropic HH-RLHF dataset [34] reportedly contain over 25% inconsistent feedback [88, 20, 105]. Plus, recent work [107, 42, 15] shows that models trained on smaller, high-quality datasets can outperform those trained larger, noisier ones. However, maintaining data quality often requires costly manual review (e.g., inter-annotator agreement), highlighting the need for scalable, automated assessment methods.
To mitigate the impact of noisy human feedback, researchers have proposed robustifed DPO variants. Among various heuristics [76, 6, 100], two methods claim provable guarantees: Provably Robust DPO (R-DPO) [24] and Distributionally Robust DPO (Dr. DPO) [105]. However, their notion of robustness is limited to bounding the parameter estimation error bounds that degrade as the fraction of mislabelled data increases. In contrast, truly robust methods should estimate the clean data distribution precisely under heavy contamination. This stronger form of robustness corresponds to
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Interpretable contamination analysis
Clean
Label flips
Prompt: How can politely decline an invitation?
Yees
Thank you much for the
No thanks.
1
irvitel Unfortunately can't make it this time.
Busy.
Likellhood of dean data
E-contaminateed dataset
Yun
Mpped
Ylss
Existing DPOs
Hölder-DPO
cannot judge f
cen judge clean/iiip
Flipped data
clean/lip
Estimate threshold
mixed
Trained model
mixed
Clean data
vulnerable
robust
Useful for evaluating the quality of human feedback
Figure 1:
While existing DPO variants vulnerable to 6-contamination, Hölder-DPO is provably robust. It also ranks data points by clean-data likelihond, enabling mislabel identification.
the redescending property [74]-the influence of extreme outliers diminishes to zero-a concept in robust statisties defined via the influence function (IF; [45, 27J). We analyze existing DPO variants through IFs and prove that none satisfy the redescending property. If this property holds, the aligned enauliinnnnnnnlig principled mislabel detection, mislabels
model approximates the clean data naturally exhibit lower likelihoods being clean data points. Crucially, identifying mislabelled data has benefits beyond robustness: it supports annotator evaluation, incentivises higher-quality feedback, and enables scalable dataset cleaning, reducing reliance on costly manual verification.
To address this challenge, we propose Hölder-DPO, the first alignment loss with provable redescend- ing property, stably learning clean data distribution even under heavy contamination (e.g., 6 = 0.4). Furthermore, our loss also enables the aligned model itself to serve mislabel valuation metric (see Figure 1). Ours a frrst-of-its-kin principled metric that provides a theoretically grounded estimator of both the likelihood and fraction of (mis-)labelled data. Furthermore, our method is highly scalable- avoids costly gradient- -aased computations required by prior work [60, 22] and eliminates the need for manual verification [58], thereby enabling fully automated feedback cleaning. Applied to the Anthropic HH dataset [34], it reveals substantial noise (e z 0.25), and removing the identified mislabels significantly improves alignment performance across methods.
2 Preliminaries
2.1 Language Model Alignment by Direct Preferential Optimization
Human feedback and Bradley-Terry model. Let x E X' be a prompt sampled from a distribution p(z) over finite context space X. The policy model ( (r) generates pair of responses (y1, 1/2)- human annotator then provides a preference between them, identifying the preferred response as Ywin and the rejected response Mose. i.e., 1/win > Mesc Although the underlying reward function
r" (I, y). which ranks responses, is unobservable, Bradley-Terry (BT) model [12] provides a standard framework for modelling pairwise comparisons based solely on observed preferences:
p" (yn > 1/2 x. y1, M2) == a [exp(r"(x: (x:, 31)) - exp(r" (pp(r"(0,100 (2))]
where (a)) = 1/(1 + exp(-a)) is the logistic function. We define pairwise data point as J Each data point g( is sampled independently from (s), where (s) x (ywin > Ylose x, y1, 92) (11, 32
{zz Mwin 3luse and denote the preference dataset D = {
x)) p(x))
Direct Preference Optimization (DPO). The goal of preference- based model alignment is to find parameters such that the policy Ty accurately approximates either the latent reward function r* or the preference distribution p". To achieve this, Rafailov et al. [84] proposed DPO, which efficiently solves the alignment task by minimizing the following optimization problem L(s, To):
(Bwin
Ix)
TT TD
(x)
argmin (s) log (co(s)) where go Bllg 8
3 log
(2)
Tref (ywin
x))
x))
where Tref is the reference model, and 3 > is a hyperparameter controlling alignment strength. The reference model is typically obtained by supervised tine-tuning (SFT) on the prompts with the preferred response Ywin only [84, 86). The corresponding gradient is given by:
V VeL(s, Th) == a0-go(()))) (s)) (Vo log wa (ywin
()) Va log (Vooe [)),
(3)
where (s)s) == (z, (win) ) - Go (x., Muse) and A (2x, y) 33 (ull:=uu denotes the implicit
reward function induced by me.

---

2.2 Data Contamination Model for Label Flips and Redescending Property
Data contamination model. Noise in human preference datasets [35, 105, 99] can arise from two sources: noise in the responses (y1, y2) [43, 105], such as irrelevant incoherent samples, and noise in preference feedback 1/win > Moose [35, 87, 28, 23, 105], due to incorrect preference annotations (also known as label flips). As Wu et al. [105] proved that vanilla DPO robust against noise in responses, we focuses on robustness to label flips. We model the label flip process as follows: Mip
lip ylose and Vi y/wnn represent a flipped preference label pair. Such a flipped point Stip is sampled from the distribution p(Smi) X (Vip(Vwin
We denote flipped data point as Stip == {ar, Wwin Minoo ). where Iwin
Ir 31, (1) π (81.84 V2 x) .p(z). Contaminated dataset D is given by: Definition 1 (e-contamination model). Let I < 1/2 be contamination ratio. The generative distribution of D under the e-contamination model is defined (c) (3) (1 <)p((s) + ep(snip) and denote & as contaminated data point, which can be either clean s or flipped Smi--
Rip > V/osee
Rip
Definition is widely adopted in robust statistics [48, 44, 49] and in the machine learning commu- nity [38, 33, 75). We assume access only the contaminated dataset p/ (e) (8). without access clean validation set (s(s)= prerequisite in many existing methods [24, 58]. As such, the contamination problem cannot be reduced to standard classification distribution shift task. Instcad, we leverage a tail contamination assumption to enable estimation of the contamination ratio E:
A Asummption 1 (Tail contamination [32]). Suppose that pD o (ge- (s)) for some 0" O, meaning that the true data distribution is exactly recovered by the model. Let 7 > 0. Then, in neighborhood of 0 0", pn (go is sufficiently small, i.c., Eppsnn [or[]])]o (stip)))))))) ss 0 when z= 0".
Intuitively, if we had access to the ground-truth clean-data likelihood (go- (s)), it would assign high values to clean data points and negligibly small values flipped ones. This principle is well-established in robust statistics [32, 53], and implicitly adapted in prior work [57, 13, 4, 109]. Moreover, this assumption unlocks heavy contamination scenarios, in contrast to typical assumptions that consider only infinitesimally small perturbations, i.e., e RS 0 [41, 61, 22]. This distinction is particularly important for noisy human preference datasets (e.g., 6 0.25 for HH-RLHF [88, 20)).
Redescending property. We evaluate robustness to the data contamination through the learned LLM parameters. Let 6* denote the optimal parameters learned from the clean dataset pp. and 0* (E) denote those learned from the e-contaminated dataset P Intuitively, the impact of contamination can be measured by 8* (c) 0", and its first-order approximation, known as IF [44], is commonly used:
00 (c)
(s 0 PD)
Ôe
c=00
See Appendix C for further details. Using the IF, we define the robustness condition as follows: Definition 2 (Redescending property [74]). Let 8tip = (xx, flip fip ) be a preference pair. An objective is said to be robust to e-contamination if the following condition is satisfied:
Mlose
lim Po (2 1-000
[# 0 0D = 0,
where || denotes the Euclidean norm.
The limit in (ar. flip - oo occurs when the data point Smip attains the lowest reward, i.e., go (smp))
-oo. Yet, the label flip forces the model learn this least preferable sample as best, representing the most adversarial data-poisoning scenario that a single label flip can induce. Intuitively, if the IF remains zero even under this adversarial condition, the learning objective can be considered robust: the effect of the worst possible data point on the model parameters vanishes, i.e., (c(c) R 2
2.3 Related work
Robust DPOs and RLHFs. A broad range of DPO variants have been proposed to improve the approaches also constitute popular family of alignment methods [26, 8, 78, 55], with several recent works introducing robustified objectives [72, 14, 82]. However, none provides theoretical guarantees for the redescending property or for detecting mislabelled data. We prove their limitations in Section 3.
robustness of the alignment objective [76, 6, 100, 24, 105]. Reinforeeeentleerning-be
Up to higher-order approximation emror (ce²)
3

---

We first derive the IFs of existing DPO variants and show that none satisfy the redescending property. Theorem 2 (IF for DPO variants). Let 0t be the optimum learnt from the clean dataset PD. and Dataset valuation and cleaning, Dataset valuation has been studied using IFs [41, 61, 22, 64, 63, 108. 52], Shapley values [102], and unrolling [7]. Various heuristic methods have also been proposed for identifying and filtering mislabelled data, including prompt-based filtering [77, 91, 68, 70J, attribution-bant basod approaches [101, 36, 11, 79], and perplexity-han -based filtering [58]. In contrast, our method is the first to provide theoretical guarantees for dataset filtering under heavy contamination.
3 Existing DPO Variants Are Neither Robust Nor Able to Identify Mislabels
(e) (c) leartt from the e-contaminated dataset P Let (Te Tzet) be generic DPO loss function corresponding to a specific DPO variant. Assume that the Hessian V2 (s, To) lo=o- is positive definite." 3 Then, the Syip dependent component of the IF for this DPO variants is given by:
IF (s P.DD) x Epplsw) [ ggnn (gtip: ne-) }]
(5)
where VaLgon (8jlip: T corresponds, for example, to Eq. (3) when Eyon (8slip; TT- ) - (is Te-
Theorem 3 (informal), The following objectives do not satisfy the redescending property:
algoritim
objective (Non Fead)
bounded
redescending?
can detect app?
DPO [84]
E
log (go())]
IPO [6]
au(4) 3
28
X
x
x
C-DPO [76]
Bogool log (gv(s))
cB BBl
I logar (x))
R-DPO [24]
F. col log or (g)()))
I-26 ggo log (-go((-0(8)) (6
Dr. DPO [105)
-8" log
exp
log (g)
See Appendices E and G for full proofs and detailed expressions of the IFs for each DPO variant. We now provide an intuitive interpretation of Theorem The IF can be viewed as the gradient of the generic objec- tive Egen (see Eq. (5)), which can be decomposed into two components. For example, in the vanilla DPO case (Eq. (3)), we have:
up"up
"e" down
weigts
Only Hölder-DPO
DPO - R-DPO -r. DPO
VeL(8,Te) = (()))(()) (grad(s)
IPO - C-DPO
Hölder-DPO (Ours)
IF weights.
direction
-2000
-1000
0
1000
2000
Reward difference (5
Here, the gradient term (g (s) determines the direc- tion of the update, while the IF weight o (s) acts scaling factor that controls the strength of
Figure 2: IF analysis reveals only Hölder-DPO satisfies the redescending property.
the update for each data point. If the IF weight tends
to zero in the worst-case scenario (i.e., go (Smp)) - -00), the method is robust, it effectively ignores highly confident but misaligned (flipped) samples. Figure 2 visualises the behavior of IF weights across different objectives under extreme reward perturbations. Both DPO and Dr. DPO exhibit the same asymptotic trend, with lim, 1im, ( - IF = a, where is a non-zero constant. R-DPO shifts this limit upward with a constant c, while C-DPO pushes it downward, but neither approaches zero. In contrast, our proposed HPLder-DPO, introduced in the next section, completely nullifies the influence of outliers, demonstrating strong robustness (redescending), not merely boundedness.
4 Proposed method: Hölder-DPO
In response to the issues identified in $3, we introduce our robust DPO framework, Hölder-DPO, the first algorithm that has a provable redescending property and can identify mislabels stip-
'This is a local assumption around the optimal parameters 8*. Such local assumptions are standard in IF analysis for non-convex deep learning models, as global convexity rarely holds (e.g.. [56, 22I).

---

4.1
Hölder-DPO Is Provably Robust
The objective of vanilla DPO in Eq. (2) can be rewritten with KL. divergence:
argmin 1 log P (3) +
(s) [ log (ge(3))) = argmin Dxt 10 (g)((()))))
(6)
ignorable const.
Original DPO loss
(e) (3) terms KL divergence. Thus, vanilla DPO can be understood as matching (go(3)) R P
However, the KL divergence well- to be highly sensitive to data contamination [10, 32]. and accordingly, DPO is not robust (Theorem 3). This fact motivates replacing the KL divergence with more robust alternative. In this paper, we employ the Hölder divergence [53], a generalization the KL divergence that encompasses several robust divergences, defined follows:
Definition 3 (Hölder divergence [53]). Let á R++ R be continuous function such that @(h) 2 -h 1+ for all 2 and > O, and ((1) = 1. Then, the Hölder divergence is defined as
Dalp(w) [|]([]] = (p - (p
(7 ( (>,0,, 0),
DKIlp(() [(w)]
(x=
where S, = (wl9"(w))) [,
(w)), (pllp) = -Epl"(W)], and p. q are non-negative,
non-zero functions.
We then propose to perform LLM alignment by solving the following optimization problem:
argmin Duipg ( (3)) (g)()
(8)
Remark 1. Prob. (8) generalises the KL divergence and encompasses two robust divergences.
<(h)=
divergence
equivalence
? y=0
N/A
KL
vanilla DPO (Eq. (6))
> 0
7 (1 (1+1) h)
density (DP; [10])
N/A
7>0 0
pseudo-spherical (PS) score [40, 39]
d-divergeecce [32, 53]
1 When y > (p
(c)
a negligible constant, Prob. (8) can be further reformulated as (2) (5) (go(3)))
argmin (,
We refer to the new objective, Prob. (9), as Hölder-DPO. We now analyze its robustness via IF: Theorem 4 (IF for Hölder-DPO). Suppose that (c) = argming S, || ([())) and 8" argming S.y (PD [gg))). Let ((h) be twice-differentiable, and let 0 < (g)(()) and 0 Y co. Assume that @(h) satisfies (h) for h and o(h) # h for any constant et and the Hessian V [(s V2L(s, no) o= is positive definite. Then, the IF of the Miider-DPO, excluding terms independent
of syi. is given by:
IF HH-DDO(SMPp 0 P.)) x Ep(en)) (F (Spip))
(10)
where F(2) (Spip) = (g0- (smp)) V VoL(sppp )-
Corollary 1 (Hölder-DPO is robust). Suppose that the policy gradient V log Rely (x) is bounded by C and satisfies L-Lipchitz in O, where 0 < C' < oo and L < xc. Then, under Theorem 4, the IF of Hölder-DPO satisfies the redescending property in Definition 2.
The full proof and the complete form of Eq. (10) are summarised in Appendices E.3 and E.4. This robustness result is expected, as both the DP divergence and the --divergence are well known be robust to data contamination across various settings [10, 32, 38, 33]. The key difference between the IFs of DPO variants (Theorem 2) and our method (Theorem 4) lies the term (go- (smp))". " This subtle change induces diminishing IF weights, as shown in Figure 2, and ensures the redescending.
'These assumptions satisfied by the DP divergence and the PS score, formally shown in Lemma 2.
5

---

4.2 Hölder-DPO Can Estimate Contamination Ratio e and Detect Contaminated Data Smip
We now move to the most novel part of our algorithm: contaminated data detection, which is the primary motivation for employing Hölder divergence in addition to its robustness and generality across three divergences. Specifically, we show that our Hölder-DPO inherits the contamination ratio (c) estimation capability of the Hölder divergence [54].
Model Extension. Estimating the contamination ratio e without access clean validation dataset may sound surprising how is it possible? The key enabler is the idea of model extension [54]. Due to space limitations, we present only the high-level intuition here and defer technical details to Appendix F.2. The crucial insight is that if the learning objective satisfies redescending property-thar is, it nullifies the effect of contamination the divergence can be approximated as:
argmin (y(0 (90(s)) = argmin S,1(1 - ()) + <p((mp) || (8)),,
(Definition 1)
Ri argmin S, - <)pp (8) (l (go(s))
(Corollary 1)
Thus, our Hölder-DPO objective can be interpreted as matching (1 -1-a8 ()p((s)) x (ge However, the scaling factor (1 e) remains on the target distribution side. The model extension idea [54] addresses this issue by introducing additional scaling parameter g into the model:
{0r (e), {
argmin S21(1 e pp || (o(g)()))]
0.5
Intuitively, f x 1 - e, the objective approximately reduces to the clean form: S. (PD(s) (g(((s))) In other words, the aligned model (ge- (e) (s)) approximates the distribution of the clean dataset, i.e., pD(s) R (g))(())))), Furthermore, the optimised scaling parameter & provides an estimate of the contamination ratio using ~ E.
Estimator of The contamination ratio estimate has the following closed-form:
Proposition 1 (Contamination ratio estimator). For any fixed 0, the optimal solution to the inner optimizaation problem argming (p (c) (()o)))()s)) is given in closed form by E"
E (o(g(()))) (g)( +7d.. Then, the contamination ratio e can be estimated by:
> (100
((go(3 (g9()))
E min - ) where 6- = E (390(1))))
(gg
(11)
Surprisingly, the contamination ratio è can be estimated using the closed-fore solution. As result, Hölder-DPO enables contamination estimation without solving a complex optimization problem. full derivation and proof are provided in Appendix F.2.
Choice of d. Given two divergence options, DP or PS (see Remark 1). which should we choose? Kanamori and Fujisawa [54] addressed this question in the classical setting and showed that only DP-iee, d(h) Y - (1 + sttsssisss both the redescending property (Corollary 1) and contami- nation ratio estimation (Proposition 1). We proved that the same theoretical result holds in the model alignment context: neither the PS score nor KL divergence satisfies both properties, while DP does. (see Appendix F.4). Thus, our final objective reduces to the following empirical estimator:
argmin Spr (pg) || (3)) = argmin
(1
[o
N
(go (3 (12)
N
1=1
The details of the derivation of the DP divergence estimator are provided in Appendix D.
Algorithm Contamination detection
4.3 Algorithm to Detect Data Contamination
Require: D. ma. Traf.
Algorithm 1 summarises the overall procedure for de-
1: Fine-tune via Prob. (12) 0" (e).
tecting contaminated data points. As shown in Line 1, our proposed Hölder-DPO serves as plug-and-play replacement for the standard DPO loss. Simply train-
2: Estimate contamination ratio via Eq. (11).
3: Identify mislabels in D by sorting the esti- mated clean-data likelihood a (gg (8())) and take least [Ne] samples.
ing with the objective in Eq. (12) yields a robust model alignment algorithm. For further analysis, Line 2 estimates the contamination ratio é without 4: return 0" e,
requiring access to clean validation dataset-unlin prior work that depends on one [24, 58]. We then perform dataset valuation using the estimated clean-data likelihood o (goo (g90)))())))) identifying the estimated mislabelled subset.

---

Robustness evaluation
Mislabel valuation
1.00
(4)
1.00
Ooraan Cowanilattt.co upto Cyedien
1.0
0.95
0.95
0.95
Protiton
0.6
0.90
22
0.83
0.8S
0.6
T-m 0.25
800 strge
0.90
5103
0.4
skggs
0.50
T- 0.25
0.6
800
0:2
(0)
F-0025
0.1
02
0.5
0.1
0.2
0.3
0.4
Contanaaiion 1300
Terpperstere
Contariiaaiinn ritio
Food possille
DPO
DPD -8 Dr. DPO
- C-NPO +
Hilde- (Ouss)
Figure 3: Controlled sentiment generation task using GPT2 large. Error bars indicate the standard deviation over 10 trials with different random seeds. Hölder-DPO consistently outperforms all baselines in average reward under varying (a) contamination ratios e, (b) generation temperatures, and (c) training steps. In addition, only Hölder-DPO offers reliable (d) contamination ratio estimation é and (e) precision of mislabel identification, measured by precision as binary classifier.
Experiments
We benchmarked Hölder-DPO against two robust DPO variants: R-DPO [24] and Dr. DPO [105]. We also compared to several heuristic baselines, including cDPO [76] and vanilla DPO [84]. As shown in Theorem 3, none of these baselines satisfy the redescending property, though they may offer empirical improvements in some settings. All experiments are conducted on single NVIDIA A100 GPU (40 GB VRAM, 83.48 GB RAM). For SFT of the reference model, we train for one epoch using context-rese response pairs (z, Bwin). Each DPO variant is then trained for five epochs. We implement all methods using the Transformers [96, 104), TRL [98], and PyTorch [80] libraries. We set 8 = 0.1 and 2.0, and all other hyperparameters follow the TRI. defaults. Complete implementation details, including Hugging Face URLs to models and datasets used, are provided in Appendix
5.1 Controlled Sentiment Generation: Robustness Evaluation
Setup. We evaluate alignment robustness on sentiment-coonrrolled text generation task using the IMDb dataset [71], which contains 25,000 polarised movie reviews. Following Chowdhury et al. [24], we extract the first 20 tokens from each review as context prompt. For each prompt, we generate four responses using the SFT-ed GPT2-large [83]. These responses are used to construct six preference pairs per prompt. To provide sentiment rankings, we employ SiEBERT [69, 46] as the latent (ground-truth reward model r" (v) To ensure clean dataset, we filter out preference pairs that do not satisfy r" (ywin: .I) r" (ylose; x)) 0.1, yielding 12,000 clean pairs. Of these, 10,000 are used for training and 2,000 for evaluation. To simulate label noise, we randomly flipped (Jwin: Mose) in the training set with contamination ratio e E 0.1, 0.2, 0.3, 0.4). We then fine-tuned GPT2-las large using various DPO variants. Alignment quality is measured by the average reward assigned by S SIEBERT to responses generated by the aligned model on the 2,000 evaluation prompts.
Aligument quality. Figure 3(a) shows that Hölder-DPO consistently outperforms all baselines in average reward, with only minor drop as the contamination level increases. This supports our theoretical claim: földer-DPO) uniquely characterised by the redescending property, achieves robustness even under severe noise (e.g., when c 2 0.3). A mild linear decay remains, likely due to the reduced proportion of effectively clean data points, which limits estimation accuracy. Figures 3(b) and 3(c) further demonstrate Hölder-DPO 's robustness across generation temperatures and training steps. Notably, while the baseline methods plateau early during training, Hölder- DPO continues to improve, suggesting stronger resistance to overfitting on noisy preferences.
Valuation quality. Figures 3(d) and 3(e) evaluate misla- bel valuation. As shown in Proposition 1, this estimation capability is unique to Hölder-DPO and is not supported by other baselines (see Theorem 3). The detection task is entirely unsupervised no ground-truth truth labels indicate whether a preference pair is clean or flipped. Remark- ably. Hölder-DPO accurately estimates the contamination ratio up to e = 0.2, after which the estimates begin to saturate. This is expected, given that contamination ra- tio estimation relies on the optimised a (ge-(e)) and the average reward slightly deteriorates increases (see
Table 1: Effect of on Hölder-DPO under e = 0.4 contamination.
7
average ?
reward
precision
1
0.8975
0.2503
0.6433
2
0.9420
0.2373
0.9286
4
0.9446
0.3083
0,8902
6
0.9561
0.4123
0.8328
8
0.9470
0.4380
0.7860
7

---

Robustness evaluation
Mislabel valuation
Qwen-2.5-1.5B
Phi-2
0.5
0.5
Contemination ratio (prediet)
0.4
1.0
(a)
GITT-4 winraa
GPT-4 wit rile
0.4
0.3
0.8-
0.6-
0.3
0.3
0.2
0.4
0.2
0.2
0.1-
0.2-
(b)
(d)
0.1
0.0
0.1
0,2
0.3
0.4
0.0
0.3 Contamination ratio
0.1
0.2
0.0
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.4
Contamination Faliouu
Contamination ratio (trae)
Contamiration
GPT-4 baseline
R-DPO Dr. DPO + Holder-DPO (Ours)
Best possible
Qwen-2 5-1.5B
- Phi-2
Figure 4: Helpful assistant dialogue generation on the Golden HH dataset. Error bars denote standard deviation over 10 trials with different random seeds. Hölder-DPO consistently achieves the highest GPT-4 4 win rate across both base models: (a)Qwen-22 5 5-1 5B and (b)Phi 2.hii.2a Notably, Hölder- DPO is the only method that outperforms GPT-4-8 GPT-4-generatee -generated prompts even when using the smaller 2.8B Phi-2 model. It also delivers near-perfect (d) contamination ratio estimation è and (e) precision in mislabelled data detection, regardless of the base model.
Figure 3())). Nonetheless, it reliably ranks datasets by relative noise levels. For mislabel detection, precision-i.e., the proportion of true mislabels among those predicted as mislabelled-iiccressse with higher which we attribute to underestimation of e, reducing false positives and improving detection precision.
Impact of y choice. We vary the key hyperparameter 7 to assess its effect. As shown in Table 1, there is trade-off: increasing y leads to more accurate contamination ratio estimate but lowers mislabel detection precision. Intuitively, y controls how strongly potential outliers are down-weighter While the optimal is lask-dependent and relates to the true (yet unknown) e, Hölder-DPO consis- tently outperforms all baselines with fixed default of y 2, suggesting that it is effective without extensive tuning. See Appendix H for further ablation studies on hyperparameters, e.g., batch size.
5.2 Single-turn Dialogue: Generated Responce Quality Evaluation
Given a user prompt, the task is to generate responses that are more helpful and less harmful. this setting, we do not have access ground-truth rewards (e.g.. from a sentiment model like SiEBERT). Instead, we adopt the GPT-4 4 win rate [84, 105], the standard metric for evaluating alignment when ground truth is unavailable. This metric follows three steps: (A) generate a response /gen using the aligned model for given prompt z; (B) retrieve the preferred response Y/win from the test dataset for the same prompt x;; (C) ask GP 4- to compare (ywin Vgen) and determine which is better. The win rate is defined as the fraction of prompts where Vgen is preferred. For dataset, we evaluate on the Golden HH [15], a manually curated version of Anthropic HH [34], in which low-quality responses have been replaced with ----4-geneateed outputs. Since ywin corresponds to GPT-4 4 generations, the benchmark is both cleaner and more challenging than the original Anthropic HH. The dataset contains 42,500 training and 2,310 test examples. To evaluate robustness, we randomly flipped preference labels with varying noise levels e {0., 0.1, 0.3, 0.4]. We tested two base models: Qwen-2.5- 5-1.5-11. 5B [106] and Phi-2 [50]. and fine-tuned using DeepSpeed Stage [3] for memory efficiency, with an effective batch size of 32 As shown in Figures 4(a) and 4(b). Hölder-DPO consistently outperforms all basclines across both base models. Notably, Hölder-DPO with Phi responses.
the only method to exceed a 50% win rate, slightly outperforming Figures 4(c) and 4(d) further show that mislabel detection is nearly perfect. Remarkably, these results are achieved fully unsupervised setting, underscoring the effectiveness of our approach.
Can HPlder-DPO scale to larger models? We further evaluated HPLder-DPO on larger language models-Misttall- NeMo-12B [9t]-both capable of multilingual interaction. Experiments were conducted on the OASSTI dataset [89, 59], which contains 18,000 multilingual messages across 35 languages in a
Table 2: GPT-4 win rate on OASSTI (e 0.4).
loss
Ministral-8B
NeMo-12h 12B
[93] and
DPO
0.5801
0.6021
R-DPO
0.5737
0,5992
Dr. DPO
0.6058
0.6196
Hölder-DPO
0.6314
0.6473
Batch size of 4 with gradient accumulation of 8.
8

---

Dataset cleanness ranking
Mislabel valuation
Improvement by cleaning
Removed noisy labels
Before cleaning
Hölder
Gloden HH
cleanest
Data points in Anthropic HH
After cleaning
DPO
DPO
clean
IDMb
noisiest
labels
R-DPO
(c)
Anthropic HH
(b)
Dr. DPO
0.00 0.65 0.10 0.15 0.20 0.25 Contamination ratio (predict)
-100
-50
50
100
0.4
0.5
0.6
0.7
log-likelihood of
GPT-4 win rate
clean data
Figure 5: Dataset valuation using Phi-2. (a) Hölder-DPO estimates substantial contamination
(é Rs 0.25) in popular Anthropic HH dataset. (b) Distribution of log-likelihoods of clean data points in the dataset. (c) Improvement GPT-4 win rate across methods after removing detected noisy data points from the training set scr-suupas even Hölder-DPO trained on the original (noisy) dataset.
multi-turn dialogue format. Preference labels in the training were flipped with contamination ratio of e = 0.4. To accommodate limited GPU resources, we apply low-rank adaptation (LoRA; [47]) via the PEFT library [73], along with model quantization using bitsandbytes [30, 31J. Despite these constraints, Hölder-DPO consistently outperforms all baselines, as shown in Table 2.
5.3 Real-world Noisy Datasets: Dataset Valuation and Cleaning
To go beyond controlled settings with synthetic noise, we next applied Hölder-DPO to real-world datasets where the true contamination ratio e unknown. Using our contamination ratio estimator ê in Eq. (II), performed dataset valuation and cleaning on the Anthropic HH dataset [34], which is widely used but known to contain significant label noise [88, 20, 105]. We adopted Phi-2 as the base model and apply Hölder-DPP DPO directly the dataset.
Dataset valuation. Figure 5(a) shows the estimated contamination levels for three datasets without synthetic label flipping, reflecting inherent label noise. As expected, the Anthropic HH dataset exhibits the highest estimated contamination, with as 0.25. This aligns with prior findings reporting substantial noise in Anthropic HH [88, 20, 105, 24]. Figure 5(b) displays the distribution of log- likelihoods, with illustrative examples provided Appendix H.3. Manual inspection reveals consistent pattern: for prompts that LLMs expected to refuse (e.g., harmful unethical requests), both responses in the removed examples often provide information, leading to inconsistent rankings. These findings show that Hölder-DPO not only serves a robust alignment objective but also offers interpretable and actionable insights for dataset valuation.
Data cleaning. We also evaluated the effectiveness of Hölder-DPP for data cleaning. Here, Hölder- DPO was used
pre-processing step to identify and remove noisy preference labels, yielding filtered training dataset. We then compared the GPT-4 win rates before and after cleaning. As shown in Figure 5(c), removing samples identified as mislabelled consistently improves performance across all methods. Notably, both DPO and Dr. DPO trained on the cleaned dataset outperform Hölder-DPo trained on the original (noisy) dataset. This is expected: while Hölder-DPo is designed to be robust under label noise (e 0), once the data is cleaned (e 0), robust objectives can be overly conservative, thus KL-based methods like DPO can align more effectively. These results suggest that Hölder-DPO can serve not only robust training objective, but also valuable pre-filtering tool for noisy preference datasets.
Discussion
Role of Hölder-DPO; pre-filtering or new objective? Hölder-DPO is a versatile framework whose optimal role depends on the application context. When maximising final performance prioritised over training cost, two-stage pipeline- first filtering with Hölder-DPO and then fine- tuning using vanilla DPO abjective-yielee superior results (see Figure 5(c)). Conversely, when computational efficiency is the priority, single-stage approach-direcn fine-tuning with the Hölder-

---

DPO objective- -offers a cost-effective solution while maintaining strong alignment quality. This flexibility enables practitioners to choose the two strategies according to their constraints.
How to set ?? As shown in Table 1, the value of y has strong influence performance. Its optimal setting is likely task- and detasee--ependent. Ideally, dynamically tuning ? during training would yield the best results. Nevertheless, our experiments show that a default value of 2.0 performs robustly across diverse settings, and we recommend using it as the default choice. When fine-tuning y is desired, a promising approach is to employ a human-in-the-lopy process. As illustrated in Figure 5(b), the Hölder-DPO) model can rank data points. This ranking can serve as a diagnostic tool to guide manual inspection: for instance, increasing ? if clear mislabels are still being tolerated, or decreasing it if correct preferences are penalised as mislabels.
What if c > 0.5? When the noise ratio becomes extreme (e.g.. <= 0.9), Hölder-DPO can still identify the 10% of clean samples by isolating them as outliers. In such cases, the roles can be teversed-the identified minority can be treated as the clean subset, while the majority is discarded. This reversal can be validated with minimal human effort by inspecting few of the top-ranked samples flagged by our method. Hence, dölder-DPO remains applicable even in highly noisy settings (e.g., c > 0.6). However, detecting mislabeled data becomes fundamentally difficult when e = 0.5. We argue that this challenge is not a limitation of our method per se, but rather a theoretical identifiability barrier: when clean and noisy data are equally represented, distinguishing between them is statistically impossible without access to a clean reference set. Our approach inherently assumes bimodal distribution, where one mode-tyyicalle the uajority-correspee
clean data.
What kind of label noise does the tail assumption cover? While our cullcontamination assump- tion formally presumes i.i.d. noise, is effective against a wide range of label noise types sypeeesymettii,, asymmetric, and even certain systematic annotator biases so long as the resulting noisy preference pairs are recognized by the learned model as low-likelihood events (i.e., outliers). This generality represents clear advantage over prior methods that can only handle symmetrie label flips (e.g., [67]). The main limitation arises when the noise is too structured to be identified as an outlier by the model. For instance, if all preference labels associated with a single prompt or narrow topic are consistently flipped, such errors cease to forn distinct "tail" in the data distribution. Instead, they produce coherent but incorrect signal within that subdomain one the model may interpret as a genuine, domain -smaiff preference. Addressing such structured noise requires modeling dependencies across samples rather than treating each observation as independently corrupted. Exploring this direction is a key avenue for future work. Concurrent research has begun tackling these challenges using variable formulations [16].
in fact more general than it may appear. Hölder-DPO
Conclusion and Limitation
Conclusion. We analyzed the robustness of DPO variants through the lens of the redescending property and showed that none of the existing methods satisfy it. We then introduced Hölder-DPO the first algorithm with a provable redescending property propert--alongside a principled data valuation method for estimating contamination and identifying mislabelled data. Our experiments demonstrate that Hölder-DPO improves alignment quality and accurately detects mislabels without access to a clean validation set. Notably, it offers the first theoretically grounded approach to dataset valuation, revealing significant noise in the widely used Anthropic HH datasct. Morcover, filtering out the detected noisy examples further improves performance across alignment methods, positioning Hölder- DPO as an effective pre- pee-training filter. Taken together, our theoretical and empirical results advance scalable, automatable alignment without relying on clean validation data.
Limitations. A key limitation lies in the selection of the hyperparameter ? and see Section for the potential solutions. Another limitation is that our theoretical guarantees are also based on an i.i.d. label flip model and see also Section 6 for the details. Furthermore, our objective relies on coarse approximation of the DP divergence (see Appendix D), and exploring more precise approximation methods is another important avenue. Finally, we implicitly assume that majority opinion represents the clean data; see Appendix A for the broader implications.
10

---

Acknowledgments
We sincerely appreciate the anonymous reviewers for their insightful feedback. We thank Dr. Siu Lun Chau, Juliusz Ziomek, and Yuki Tachibana for their valuable discussions. This research was primarily conducted while MF was a visiting researcher at the University of Oxford. MF was previously supported by the RIKEN Special Postdoctoral Researcher Program and JST ACT-X (Grant Number: (PMJAX210K) Japan. MF is currently supported by KAKENHI (Grant Number: 25K21286). MA was supported by the Clarendon Fund, the Oxford Kobe Scholarship, the Watanabe Foundation, and Toyota Motor Corporation.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. GPT-4 technical report. arXiv preprint WxXiv:2303008744 2023.
[2] Masaki Adachi, Siu Lun Chau, Wenjie Xu, Anurag Singh, Michael Oshorne, and Krikamol Muandet, Bayesian optimization for building arXiv:2502.07166. 2025.
consensus. arXiv preprint
[3] Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. Deepspeed- inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-15. IEEE, 2022.
[4] Fabrizio Angiulli and Clara Pizzuti. Fast outlier detection in high dimensional spaces. In Tapio Elomaa, Heikki Mannila, and Hannu Toivonen, editors, Principles of Data Mining and Knowledge Discovery, pages 15-27. Springer Berlin Heidelberg. 2002.
[5] Kenneth J Arrow. A difficulty in the concept of social welfare. Journal of political economy, 58(4):328-346, 1950. URL https: / / /oii /10/10. 1086 62556963.
[6] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm understand learning from human preferences. International Conference on Artificial Intelligence and Statistics, pages 4447-4455. PMLR, 2024.
[7] Juhan Bae, Wu Lin, Jonathan Lorraine, and Roger Grosse. data attribution via approximate unrolling. In Advances in Neural Infor- mation Processing Systems. volume 37, pages 66647-66686, 2024.
Training
URL
wtweeerfflleepppe/////ee httpps:/ppoceed ngs neurips Conferreeeeeeeeeeeeeeeeeee-ee-ee pdf. 7 Taf60ccb99c7a444 434 fb00ca
[8] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05900 2022.
[9] Yuntao Bai. Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08N1 2022.
[10] A. Basu, I. R. Harris, N. L. Hjort, and M. C. Jones. Robust and efficient estimation by minimising density power divergence. Biometrika, 85(3):549-559, 1998.
[11] Sebastian Bordt, Suraj Srinivas, Valentyn Boreiko, and Ulike von Luxburg. How much can we forget about data contamination? arXiv preprint arXiv:2410.03244 2024.
[12] R. Bradley and E. Terry. Rank analysis of incomplete block designs: The method of paired comparisons. Biometrika, 39(3-4):324-355, 1952.

---

[13] Markus M. Breunig, Hans-Peter Kriegel, Raymond T. Ng. and Jörg Sander. LOF: identify- ing density-basee local outliers. In Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data, pages 93-104. Association for Computing Machinery, 2000.
[14] Alexander Bukharin, Ilgee Hong, Haoming Jiang, Zichong Li, Qingru Zhang, Zixuan Zhang, and Tuo Zhao. Robust reinforcement learning from corrupted human feedback. In The Tiihth- Annual Conference on Neural Information Processing Systems, 2024.
[15] Tianchi Cai, Xierui Song, Jiyan Jiang, Fei Teng, Jinjie Gu, and Guannan Zhang. Ulma: Unified language model alignment with human demonstration and poie preference. 2023.
arXiv preprint
[16] Xiaoyang Cao, Zelai Xu, Mo Guang, Kaiwen Long, Michiel A Bakker, Yu Wang, and Chao Yu. Latent collective preference optimization: A general framework for robust Ilm alignment. arXiv e-prints, pages arXiv-2509, 2025.
[17] S. Casper, X. Davies, C. Shi, T. Krendl Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, Lindner, P. Freire, T. T. Wang. S. Marks, C.-R. Segerie, M. Carroll, A. Peng. P. Christof- fersen, M. Damani, S. Slocum, U. Anwar, A. Sichtharanjan, M. Nadeau, E. J. Michaud, J. Pfau, Krasheninnikov, X. Chen, I.. Langosco, P. Hase, F. Biyik, A. Dragan, D. Krueger, D. Sadigh, and D. Hadfield-Menell Open problems and fundamental limitations of reinforcement learning from human feedback. Transactions on Machine Learning Research, 2023.
[18] Louis Castricato, Nathan Lile, Suraj Anand, Hailey Schoelkopf, Siddharth Verma, and Stella Biderman. Suppressing pink elephants with direct principle feedback. arXiv preprint arXiv:2402.07896 2024.
[19] Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Dinesh Manocha, Furong Huang, Amrit Bedi, and Mengdi Wang. MaxMin-RLHF: Alignment with diverse human preferences. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 61 16-6135, 2024.
[20] Khaoula Chehbouni, Jonathan Colaço Carr, Yash More, Jackie CK Cheung, and Golnoosh Famadi. Beyond the safety bundle: Auditing the helpful and harmless dataset. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume 1: Long Pupers), pages 1 Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-189-6. URL https: ac anthol loyy-oly-ooooo www.lolyyy.ooge/g/ee
11895-1 19925, Albuquerque, New
nwwcl 1on1..ll...5990 596/.
[21] Keertana Chidambaram, Karthik Vinay Seetharaman, and Vasilis Syrgkanis. Direct preference optimization with unobserved preference heterogeneity. arXiv preprint 15065, 2024.
[22] Sang Keun Choe, Hwijeen Ahn, Juhan Bae, Kewen Zhao, Minsoo Kang, Youngseog Chung, Adithya Pratapa, Willie Neiswanger, Emma Strubell, Teruko Mitamura, et al. What is your data worth to gpt? Ilm-scale data valuation with influence functions. arXiv preprint arXiv:2405.13954, 2024.
[23] S. R. Chowdhury, A. Kini, and N. Natarajan. Provably robust DPO: Aligning language models with noisy feedback. arXiv preprint arXiv:2403.00409, 2024.
[24] Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust dpo: aligning language models with noisy feedback. In Proceedings of the 41 International Conference on Muchine Learning, pages 42258-42274 2024.
[25] P. F Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, volume 30, pages 4299 4307, 2017.
[26] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
12

---

[27] R Dennis Cook and Sanford Weisberg, Characterizations of an empirical influence function for detecting influential cases in regression. Technometrics, 22(4):495-506 1980.
[28] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with scaled ai feedback. arXiv preprint arXiv:2310.01377 2024.
[29] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. ULTRAFEEDBACK: Boosting language models with scaled Al feedback. In Ruslan Salakhutdinov, Zico Koiter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the s International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 9722-9744, PMLR, 21-27 Jul 2024. URL https: \ proceedings. mir. wrress/v233 cww24f..2fff.ff html.
[30] Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block- wise quantization. 9th International Conference on Learning Representations, ICLR, 2022.
[31] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Iims. arXiv preprint arXiv:2305.14314 2023.
[32] H. Fujisawa and S. Eguchi. Robust parameter estimation with a small bias against heavy contamination. Journal of Multivariate Analysis, 9999):2033-2811, 2008
[33] M. Fujisawa, T. Teshima, Sato, and M. Sugiyama. 7-abc: Outlier-robust approximate Bayesian computation based on robust divergence estimator. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, pages 1783-1791, 2021.
[34] Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.0788 2022.
[35] Yang Gao, Dana Alon, and Donald Metzler. Impact of preference noise on the alignment performance of generative language models. In First Conference Language Modeling, 2024. URL https: http//ppeereveeeeeee /
net/ forum? d=MMAACCCTC..
[36] Kristian Georgiev, Roy Rinberg, Sung Min Park, Shivam Garg, Andrew Ilyas, Aleksander Madry, and Seth Neel. Attribute-oo-de to-delete: Machine unlearning via datamodel matching. arXiv preprint ar arXiv:2410.23233. 2024.
[37] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pages 4447-4455, 2024.
[38] A. Ghosh and A. Basu. Robust Bayes estimation using the density power divergence. Annals of the Institute of Statistical Mathematics, 68:413-437, 2016.
[39] T. Gneiting and A. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359-378 2007.
[40] I. J. Good, Discussion of "measuring information and uncertaint"" by R. J. Buehler. Founda- tions of Statistical Inference, pages 337-339, 1971.
[41] Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.. 033266, 2023.
[42] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306.10 1164, 2023.
13

---

[43] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi. Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Texthooks are all you need. arXiv preprint ar Xiv::2306. 1644, 2023.
[44] F. R. Hampel, E. M. Ronchetti, P. J. Rousseeuw, and W. A. Stahel. Robust Statistics: The Approuch Based on Influence Functions. Wiley Series in Probability and Statistics. John Wiley & Sons, 2011.
[45] Frank R Hampel. The influence curve and its role in robust estimation. Journal of the american statistical association, 69(346):333-33,, 1974.
[46] Jochen Hartmann, Mark Heitmann, Christian Siebert, and Christina Schamp. More than a feel- ing: Accuracy and application of sentiment analysis. International Journal of Research in Mar- keting, 40(1):75-87, 2023. doi: https://doi.ogg/// 1016/j iprmmaa. 000. URL https: //www www.::sceeneediree..o.. com/
[47] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models, In In- ternational Conference on Learning Representations, 2022. URL htttttt://peeeeeeeeeee net/ ht//forumm wwe-niideeeeeeeeeeeee Y19.
[48] P.J. Huber. Robust estimation of a location parameter. The Annals of Mathematical Statistics, 35(1):73 101, 1964.
[49] P. Huber and E. M. Ronchetti. Robust Statistics. Wiley Series in Probability and Statistics. Wiley, 2011. ISBN 97811 178111822 18210038..
[50] Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 1(3):3 2023.
[51] Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of Ilm via human-preference dataset. Advances in Neural Information Processing Systems, 36: 24678-24704, 2023.
[52] Cathy Jiao, Weizhen Gao, Aditi Raghunathan, and Chenyan Xiong. On the feasibility of in-context probing for data attribution. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 5140-5155, 2025.
[53] T. Kanamori and H. Fujisawa. Affine invariant divergences associated with proper composite scoring rules and their applications. Bernoulli, 20(4):2278 -2004,, 2014.
[54] T. Kanamori and H. Fujisawa. Robust estimation under heavy contamination using unnormal- ized models. Biometrika, 102(3):559 572, 2015.
[55] Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback. arXiv preprint 1oXiv:2312.14925, 10, 2023.
[56] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885-1894. PMLR, 2017.
[57] Keyi Kong, Xilie Xu, Di Wang. Jingfeng Zhang, and Mohan Kankanhalli. Perplexity- aware correction for robust alignment with noisy preferences. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet. Tomezak, and C. Zhang. editors, Advances in Neural Information Processing Systems, volume 37, pages 28296-28321. Curran Associates, Inc., htteeerffilee/epee/eea//e 2024. URL https:tttttoomo http::/:///coceooeeeeeeoo neur ipsurips... cc cwtppcerppppeeeeeeeeeeeee
[58] Keyi Kong, Xilie Xu, Di Wang, Jingfeng Zhang, and Mohan S Kankanhalli. Perplexity-aware correction for robust alignment with noisy preferences. Advances in Neural Information Processing Systems, (228296-28321.. 2024.
14

---

[59] Andreas Köpf, Yannic Kilcher, Dimitri Von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen. Oliver Stanley, Richárd Nagyfi, et al. Openassistant conversatioss-demoeratiiiig large language model alignment. Advances in Neural Information Processing Systems, 36:47669-47666 2023.
[60] Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in lora-tuned Ilms and diffusion models. arXiv preprint arXiv:2310.00900 2023.
[61] Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence loRA-tuned L.MSs and diffusion models. In The Twelfth International Conference on Learning Representations, 2024. URL https: : /oppnreveiii ew net/ / forum? id 9m021b92Wz
[62] H. Lee, S. Phatale, H. Mansoor, T. Mesnard, J. Ferret, K. Lu, C. Bishop, E. Hall, V. Carbune, Rastogi, and Prakash. RLAIF: Scaling reinforcement learning from human feedback with Al feedback. arXiv preprint arXiv:2399.00267, 2023.
[63] Wenjie Li, Jiawei Li, Christian Schroeder de Witt, Ameya Prabhu, and Amartya Sanyal. Delta-influence: Unlearning poisons via influence functions. arXiv preprint arXiv:2411.13731, 2024.
[64] Zhe Li, Wei Zhao, Yige Li, and Jun Sun. Do influence functions work on large language models? arXiv preprint arXiv:2409. 19998, 2024.
[65] Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. Towards understanding and mitigating social biases language models. In International conference on machine learning, pages 6565-6576. PMLR, 2021.
[66] Liang, C. Chen, S. Qiu, J. Wang, Y. Wu, Z. Fu, Z. Shi, Wu, and J. Ye. ROPO: Robust preference optimization for large language models. arXiv preprint arXiv:2404.0411 2024.
[67] Xize Liang, Chao Chen, Shuang Qiu, Jie Wang. Yue Wu, Zhihang Fu, Hanzhu Chen, Feng Wu, and Jieping Ye. ROPO: Robust preference optimization for large language models. International Conference on Machine Learning (ICML), 2025. URL https: / //openreview. fett/forummddd-WWnnne
[68] Xiaoqun Liu, Jiacheng Liang, Muchao Ye, and Zhaohan Xi. Robustifying allened large language models through clean data curation. arXiv preprint arXiv:2405. 193558 2024.
[69] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
[70] Junyu Luo, Xiao Luo, Kaize Ding, Jingyang Yuan, Zhiping Xiao, and Ming Zhang. Robustft: Robust supervised fine-tuning for large language models under noisy response. arXiv preprint arXiv:2412. 14922, 2024.
[71] Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142-150, 2011.
[72] Debmalya Mandal, Andi Nika, Parameswaran Kamalaruban, Adish Singla, and Goran Radanovic. Corruption robust offline reinforcement learning with human feedback. In The 28th International Conference on Artificial Intelligence and Statistics, 2025.
[73] Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficien fine-tuning methods. https: / //github. www.cc/..chheeeeeee www.///gggiggaaceeeooooo peft, 2022.
[74] Ricardo A. Maronna. Robust Statistics: Theory and Methods (with R). Wiley Series in Probability and Statistics. Wiley, second edition, 2019.
15

---

[75] T. Matsubara, J. Knoblauch, F.-X. Briol, and C. J. Oates. Robust generalised Bayesian inference for intractable likelihoods. Journal of the Royal Statistical Society: Series (Statistical Methodology), 84(3)9977-0022, 2022.
[76] Eric Mitchell. A note on dpo with noisy preferences and relationship to ipo, 2023, 2023, URL Sttps:///rrs cmiiiiiie t chhell. wwtpapa.ccppppooooooooo.oo
[77] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang. Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:21 2021.
[78] Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[79] Yijun Pan, Taiwei Shi, Jieyu Zhao, and Jiaqi W Ma. Detecting and filtering unsafe training data via data attribution. arXiv preprint arXiv: 2022.
[80] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmai- son, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Py- torch: An imperative style, high-performance deep learning library. In H. Wallach, Larochelle, A. Beygelzimer, d'Alché- Buc, E. Fox, and R. Garnett, editors, Ad- vances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: http://eeoocceeeeeeeeeeeeecc..ee neurips ceapppp..eeeappprrrrr...... www.ffllelpppoe wallessppppe//011. pdf.
fi file/
[81] Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Person- alizing reinforcement learning from human feedback with variational preference learning. In The Thirly-eighth Annual Conference on Neural Information Processing Systems, 2024. URL
[82] Vasilis Pollatos, Debmalya Mandal, and Goran Radanovie. On cormptioon- in performative reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 19939-19947,, 2025.
[83] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
[84] Rafailov, A. Sharma, Mitchell, D Manning. S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly reward model. In Advances in Neural Information Processing Systems, 2023.
[85] Shyam Sundhar Ramesh, Yifan Hu, Iason Chaimalas, Viraj Mehta, Pier Giuseppe Sessa, Haitham Bou Ammar, and Ilija Bogunovic. Group robust preference optimization in reward- free RLHF, In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[86] Yi Ren and Danica Sutherland. Learning dynamics of LLM finetuning. In The Thirteenth In- ternational Conference on Learning Representations, 2025. URL https: httpttts//ppeeeeeieeeee /ow.ee/eee..e.eee.eemoooo.
net/ / htt//formm// (tt.:m/o0
[87] Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Esin DURMUS, Zac Hatfield-Dodde Scott R Johnston, Shauna M Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards understanding sycophaney in language models. The Twelfth International Conference on Learning Representations, 2024. URL https: / //opnreview net forum? Lawaummdddtttaaaiieeeeaaa
[88] Wei Shen, Xiaoying Zhang, Yuanshun Yao, Rui Zheng, Hongyi Guo, and Yang Liu. Improving reinforcement learning from human feedback using contrastive rewards. arXiv preprint arXiv:2403.07708, 2024.
16

---

[89] Damien Sileo. tasksource: A large collection of NLP tasks with a structured dataset preprocess ing framework. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 15655-15684 2024. URL http::/tt///// a www.annnooooooooooooooooo ogg ogy- org 2004
rec-ma ma 1361/.
[90] Nisan Stiennon, Long Ouyang. Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008-3022 2020.
[91] Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36:2511-2565 2023.
[92] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint 1881 2023.
[93] Mistral AI team. Un ministral, des ministraux, 2024.
URL /ttttss://mitttaa.cccc /ttp://sssssraaaa///a..a
news/ministrau
[94] Mistral Al team. Mistral nemo, 2024.
URL https::/misttrat /nwwww//////
mistral-nemo
[95] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09280 09.809988888 2023.
[96] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[97] Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. TL;DR:
Mining Reddit to learn automatic summarization. In Lu Wang, Jackie Chi Kit Cheung, Giuseppe Carenini, and Fei Liu, editors, Proceedings of the Workshop on New Frontiers in Summarization, pages 59-63, Copenhagen. Denmark, September 2017. Association for Computational Linguistics. doi: 10. 10.11853//1/W17-440
URL https: https/////aaaaattoaaaeee
org/ htgg//177 7- 4508/.
[98] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https:/ /gtgttttttthtt//ggggggg com wwwp////wwggggcgg/./../c/// 2020.
[99] Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, Songyang Gao, Nuo Xu, Yuhao Zhou, Xiaoran Fan, Zhiheng Xi, Jun Zhao, Xiao Wang, Tao Ji, Hang Yan, Lixing Shen, Zhan Chen, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. Secrets of RLHF in large language models part II: Reward modeling. arXiv preprint arXiv:2401.6608 2024.
[100] C. Wang, Y. Jiang, C. Yang, H. Liu, and Y. Chen. Beyond reverse KL: Generalizing direct preference optimization with diverse divergence constraints. In The Twelfth International Con- ference on Learning Representations, 2024. URL https: //o///peeevveeeeee/eeeeeeee net/ for 1d=2cR2mWXK9N.
[101] Jiachen T. Wang, Tong Wu, Dawn Song. Prateek Mittal, and Ruoxi Jia. Greats: Online selection of high-quality data for Ilm training in every iteration. In Ad- vances in Neural Information Processing Systems, volume 37, pages 131197-131223, files/ /2244/ Confeoonneeeeeeeeeeeeeee påf.
2024. URL https::/pprocccingggss www.:::s.eennueeeeeeeeeeeeee file/ httl://cccssp2222rrceeeeeaasssae Paperaaeeaaa---toeeeeeeeeeeeee
[102] Jiachen T. Wang, Prateek Mittal, Dawn Song, and Ruoxi Jia. Data shapley in one training run. In The Thirteenth International Conference on Learning Representations, 2025. URL. https: : httpas://ppeeeee openreview..neeeeoe
17

---

[103] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self self- generated in- structions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 14484-1308,, 2023.
[104] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's trans- formers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.0377 2019.
[105] Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jiawei Chen, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. Towards robust alignment of language models: Distributionally robustifying direct preference optimization. arXiv preprint arXiv:2407.07880 2024.
[106] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li. Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint 1511 2024.
[107] Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma. Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006-55022 2023.
[108] Tongtian Zhu, Wenhao Li, Can Wang, and Fengxiang He. DICE: Data influence cascade in de- centralized learning. In The Thirteenth International Conference on Learning Representations, -dd-idd-- 2wt.iidddtiTiiieii...
2025. URL https:: / /penreview net
[109] Arthur Zimek, Erich Schubert, and Hans-Peter Kriegel. A survey on unsupervised outlier detection in high-dimensional numerical data. Statistical Analysis and Data Mining: An ASA Data Science Journal, 5(5):363-38)) 2012.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Redescending property (Theorem 3, Corollary 1), contamination ratio estimator (Proposition I), and experiments in Figures 2-5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See Conclusion and Limitations (Section 7).
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

We explicitly list all assumptions and the proofs of all theoretical claims are provided in the Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The generic procedure explained the Section 5, and further details are provided in the Appendix H.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution primarily new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (c.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
20

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provided the code GitHub https: t.u.. / ma 921 HolderDPO. All data we used are already open-sourced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (https://ng
cc/
public/ guwdesui////ee Code Submi s (oooooicc))) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See NeurlPS code and data submission guidelines (httpat /nnis.. cc/ cccppblicc ppblic/ggppipppeeppp ceiic//uiieeee sies/ee//// CooeeSumb ssi onPol (ilollcc) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the detals, a list of hyperparameters, and training detals Ap- pendix H.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Figures and report the standard deviation of 10 run with different random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Section 5, provide the information the computer resources (single A A100 GPU 40GB).
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't fidn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:: /tte:///neuuiii www.ureuupp...e/ee wwwppbblic/tttcseeeeeeeeeeeeeeee
Answer: [Yes]
Justification: Our work focuses solely advancing machine learning algorithms and uses only publicly available datasets from peer-reviewed publications.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Impacts are stated in Appendix A. There are many potential societal conse- quences of our work, none which feel must be specifically highlighted here.
Guidelines:
22

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
We release the code anonymised GitHub but the model nor data.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We explained and cited the library we used in Section
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papersw swr..ww. thoode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code and new tasks are provided anonymised GitHub with explanation alognside with its implementation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. We used the publicly available dataset with human subjects, but it is already reviewed by the journal board (PNAS) and the authors of the paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: We derived theories and algorithms by pen-and-paper. Experiments uses the LLM as the model to train, evaluate, but do not them beyond the experimental purpose.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (ht : //::://neen //nuuriss.... https/cco/oooooccceeeeeeeeeeeeee /2025/
LLM) for what should or should not be described.
25

---

Partl I
Appendix
Table of Contents
A Broader impact statement
26
B Additional Discussion of Related Work
27
Influence Function Measure the Impact data contamination DPO
27
D Computation of Hölder-DPO Objective
28
E Proof'
28
E.1 Proof for Theorem (The Case of DPO)
28
E.2 Proof for Theorem (The Case DPO)
29
Proof for Theorem
30
E,4 Proof for Corollary 1
35
F
Contamination ratio estimation and outlier detection
35
F1
Model extension approach to both parameter estimation and contamination rate estimation
35
F2
Contamination ratio estimation
36
E3
Estimator of g
37
F4
Choice ④
38
G IF Analysis for the DPO variants (summarized in Theorem 3)
40
G.I iDPO do not satisfy the redescending property
40
G.2 cDPO do not satisfy the redescending property
42
G.3 IPO do not satisfy the redescending property
43
G.4 Dr. DPO do not satisfy the redescending property
44
Additional Experimental details
47
H.1 Dataset and model details
47
H.2 Training and hyperparameter details
47
HL3 Anthropic HH dataset valuation
48
H.4 Additional hyperaparameter experiments
49
Broader impact statement
Robust model alignment and reliable human feedback valuation critical for the safe deployment of large language models (LLMs). While our work may have various societal implications, we do not identify any specific risks that require immediate attention.
Conceptually, our definition of "robustness" aligns with a utilitarian perspective: we assume that the majority opinion in the training dataset represents the true target distribution, and that minority opinions deviating from this majority are noise to be filtered out. This assumption is often reasonable in applications like toxicity removal or instruction following, where the normative standard tends to be implicitly shared by the majority.
However, we acknowledge that this framework does not universally apply. In contexts such as opinion formation or deliberative dialogue, disregarding minority voices inappropriate and po- tentially harmful. In such cases, distributionally robust methods (e.g., [85, 21]) that account for underrepresented groups are more suitable. Fundamentally, these challenges relate to the broader problem of aggregating heterogeneous preferences into a single model-a problem well-know in
26

---

social choice theory. According to the impossibility theorem [5], no aggregation rule (or "social welfare functional") can simultaneously satisfy all desirable rationality axioms. As such, there is no universally optimal solution, and algorithmic choices must be guided by task-specifie priorities (see, e.g. (2].
B Additional Discussion of Related Work
Theoretical Role of Influence Functions. While influence functions (IFs) have been widely used for dataset valuation and model interpretation [41, 61]. their use in deriving robustness guarantees remains less explored. Classical works in robust statistics [49, 44] provide foundational tools for analyzing model behavior under infinitesimal contamination. Our work draws conceptual bridge between these classical formulations and the practical challenges LLM alignment, utilizing IFs to derive sufficient conditions for redescending robustness and contamination detection.
Limitations of Prior Robust DPOs. Recent works propose various robust DPO formulations, including DRO-based [105], noise-aware [23], and fltering-enbancee methods [66]. However, many of these approaches either break the connection to reward learning or rely on strong assumptions or additional supervision. Our Hölder-DPO maintains a clean theoretical formulation with robustness guarantee, while being computationally efficient and easy to implement.
Empirical Noise Levels and Filtering Limitations. Empirical studies report that preference datasets often contain 20-40% noise [35, 62], with performance degrading sharply under modest increases in noise. Common mitigation strategies, such regularization [35] and aacher-based filtering [35, 17], suffer from limited generalization and inefficiency against symmetric noise. Our approach requires neither external LLMs manual heuristics, offering lightweight alternative grounded divergence- based theory.
Broader Applicability. Our framework naturally extends to settings like:
Group-specific Alignment: Supporting heterogeneous user preferences [85, 19].
Personalized Objectives: Aligning L.MSs to individual user intents [81].
DPO with Divergence Constraints: Leveraging f-divergences to balance alignment and diversity [100).
C Influence Function to Measure the Impact of the data contamination for DPO
Given the application of DPO and under the first-order optimality conditions, the model alignment results for (s) and Pi (3) are expressed as:
= argmin (s) log a (s)) and 0" (*()) = argmin (2) (2) log (g)(
(13)
From our definition of contamination data in Section 2.2, its influence on model alignment can be quantified by measuring the deviation in the optimized parameters, given by ||0* (c) 0" To evaluate this quantity, we apply Taylor expansion with respect to around 0 for 0* (c), yielding:
(e)
(8*
(c) = 0* +
+
=e.
+ O(e)).
Oe
de 1==0
e=0
From this result, we can approximately evaluate the influence of smp as follows:
(0)
IF 0, 0D) == dc
(14)
which is called as the influence function (IF) in the robust statistic context [44]. In the next section,
we provide a detailed discussion on the robustness DPO to contamination data through an analysis
of this IF.
27

---

D Computation of Hölder-DPO Objective
Here, we describe the computation of the Hölder-DPO objective introduced in Eq. (8). Given the dataset = {s(*) Ji=1
) with s(t) ~ PD, our optimization objective is:
argmin Sy (pDllo((()) argmin di ( ( (foco(t))tta))
(To(o()))
The numerator Ep (s)))) is directly estimated from the empirical distribution. However, the integral in the denominator generally intractable. To address this, we approximate using the empirical measure dg((s) = Φ Σ ö(s ,(0), yielding:
N
/ ( = N
Σ σ (go (5(
where 8 is the Dirac's 's delta function.
Thus, our estimator can be expressed as:
3 (pD
N i= (c)(s()))) N i= o (go (g0(s()))
(
For the special case where d(h) = - h, corresponding to the scaled density power divergence, the objective simplifies to:
(1 7) N
SDP(pollo(ge)
Σ (ga (s)
An alternative approach for constructing However, in the context of LLM fine-tuning, selecting an appropriate proposal distribution is nontrivial. While this issue lies beyond the scope of the present work, it constitutes future research. Importantly, our empirical results confirm that this objective still retains the key robustness properties of DP divergence: it remains resistant to label-flipped dataset and enables accurate estimation the contamination ratio in practice (see Section 5).
precise estimator is to employ importance sampling.
important direction for
E Proofs
E.1 Proof for Theorem (The Case of DPO)
Theorem 5 (IF for DPO). Suppose 0 denotes the optimal parameters learned from clean dataset PD. and 8 (e) denotes those learned from e-contaninated dataset ((6) Assume that the Hessian V C(, To)lo=o- is positive definite. Then, the Syip dependent component of the IF for DPO is given by:
DPo 0,P x Ep(ego (V VoL(stipp Ta+)]. )}
(15)
where VoL(spipy To.) is Eq. (3).
Proof. The gradient of Eq. (2) under (c) is given by
V VoC(s, no) -3B ps
(c) a (0(-go088 go (0))
Vo log no (Twin ())
Vo Bolog To (Uroee at)
where 5 (T, Tiin Ploso
PTD
28

---

From the definition of (c), we have 0 = Vo By taking the derivation of this term w.r.t. e, we obtain
θ Va
(5, n
de
(=)
0
E (0)
o (c)
Vo log To (c)
Vo log n (e) (Yooee ())
de
- (a)
3
{
10 (3)
(o)()) +
0 Fo For (o(6))
de
pb (5) 5
de
8 /
{
(e) (F)
(3)
0c
de
(07(c) Ho (c) (3) }
{
3
de PD
(4)
(3)
(5)d)
(16)
PB
de
where (+) ()) (c)
From the definition (3).
(3). we obtain
{ 10) D (3) Fo (o) = Ep(omp) Fo-te)(9nD)) Epp Fo- Force(o)
since
0 (3) p(snip) - pD(s), where Fe- (smip) = o(-go- -o-a--ie (smip) (V log Te- (Vwin PeP
Bip
| 2)
Vo log No- (Bhose
dip
2)). By taking + O, we have
( (6)))))) (5) Fo- (o)
-Ep(oned
For ((mp)
==
since (e) 0 and thus Ep 14' (s) Vol(TOi Tret)ooz- 0 from the first-order optimal condition in Eq. (13).
Furthermore, we also obtain
(c)
())
(cO
Ho- (s)
E (c)
Epp
de
de
l==0
where Hg-(s) =
BF (x) 00-
Then, Eq. (16) under + O can be rewritten as
0 VoL(mToim) Tree)
② de
- 3 E Ep(om))
Fe. (ssip)
+ Epo
[e)
H (s)
}
(e)
l==0
de
e=0
By solving the above equality w.r.t.
(0) de
we obtain
(
=- Bpn (Epps
[Ho- (5))
Ep(sne)
Fo- (saup)
de
This completes the proof.
E.2 Proof for Theorem (The Case of DPO)
Corollary 2 (DPO is not robust), Suppose that the policy gradient V, log Toly x:) is bounded by C and satisfies L-Lipchitz in 0, where < C < ao and 0 < L < co. Then, un- der Theorem 2, the IF DPO does not satisfy the redescending property in Definition i.e.
lim For t0 DPO (z. PP) + O.
29

---

Proof. From the positive definite assumption on the Hessian in Theorem 5 and the L-Lipschitz assumption on the gradient, follows that Epo (H) ( (s)] positive definite matrix. Let L' Amin (ppp (Ho- (s))) > be its minimum eigenvalue. Then the norm of its inverse is bounded:
pp g (1) = 1/L'. Furthermore, from the assumption that V log To x)|l < C (0 C < 00), we have Ve log Te (Vwin æ) Vo log To (Blose =))l I 2C from the triangle inequality. Taking the limit and applying Jensen's inequality and the bounded convergence theorem,
we have:
lim
(2,
0, 0,PD)
For (2.Blose 700
(Epp [Ho- (H)) )
lim (p. --200
lim Ao. (w.. )
Ep(enp)
Fø- (a-()))))
$ (1/L') for (Zr -
lim
Bp(one) (o( 1o(- (000-9900mpp)) (Smi))] 20
= 10:))-1--0220000
The fact 2C/L' according 0 < C 00 and L' < completes the proof.
E.3
Proof for Theorem
Before showing Theorem 4, we introduce the following proposition regarding the gradient of Sy (p
Proposition 2. Suppose that (e)e(s) is bounded. Then, under PD. the gradient of (p w.r.t 0 is obtained as
(polloo)))))))) = 70' (ho -(oo
+ (1 7) (0(hh) - he o'' (((h))
(f (1 (ee)
where d' dh(h)= = Vnoo(ho) and hg = Epp (or (s))1
Proof. We start by introducing the following shorthand notation:
A(0) = (o)o0(()))))) B(0) = f (
and hg = A10)
Bie) Then, S. (go) can be expressed as (pDlla ((ho) B()) Taking
the derivative with respect to 9 using the product rule, we have
(pol(aoo))))))) = Vo (o((())) B(O)) = d'(hs) Vohg B(0) + @(ho) VoB(O). (17)
We first evaluate Vohg which can be calculated as
Voho B(O) ( A(0) - A(O)Vo
B(Ø)²
By substituting this into Eq. (17), we obtain
A(0 - A(O)V= + *(oo) VoB(O)
V VoSlllo 0 (ho)
B(0)
= <(oo)) (VoA(0) ho BoB())) + (sho)) VoB(O)
= d'(h)) VoA(O) + (0(ho) - ho d((ho) VoB(O).
30

---

We next evaluate VaA(0)- - Vgl.
Since pp does not depend on 8, from the chain rule, we can see Vo (oo pp (oo)( where
(o(go Volp
(oo()u(s))) (o(om(t))~
= (o - (
= r0(00(5))) (ggo vogo(s).
As for VoB(O), obtain
VoB(0)= V o(go(s)) (
= =(1+n) ((90 ((00((())))))VV)))oooo -
(s)ds,
where the second equality comes from the dominated convergence theorem due to the fact that the derivative of (ge) is bounded and continuous under the bounded Veg(s).
By substituting the results of vgA(O) and V B(0 into Eq. (17), we have (pol(ooo))) = 70' E (o ()) (
( ((9n(s)) + - (agg( + (1 +7) (ō(hw)) - ho of (1o)
This completes the proof.
Furthermore, we show the following lemma to precisely derive the IF for Hölder-DPO.
Lemma 1. Suppose that 0 argming Sy (pD)(a(gg)) Let 0 Y co, and let 0 o ogg (s)). Assume that ((h) satisfies d'(h) + for h > and ((h) * e for any constant c. Then, the (PDl(polla(g))))
first-order optimal condition of Hölder-DPO,
holds if and only if (s)ds = 0, where (s) = (go ((ggss
(s)] Eppl [s) = and F p(++7)
Proof. From Proposition 2, the first-order optimal condition is given by:
P=
(ʃ (mittso())as)
(1 (h0 Epp (s) + (1 (1+ (6(10) - 2' (h(ho)ho)
C
This is linear combination CiX + C2Y 0. The "if and only if" statement holds if we can show that both coefficients C₁ and C2 are non-zero.
First, we show hg- > O. By definition, hg- A(8))/((0") where A(0*) = Epp (o (s)) and B(0*) = f [g( 2s))) From the premise < (g)( and y >> O, we have a (go- (s)) >> 0
and (go= (8)) > 0. Therefore, A(0*) > 0 and B(0*) > 0, which implies hg- > O. Given 7 > 0. hg- > and assumption o' ('(h) * 0 for h > 0, the first coefficient C1 700 (ho-) # 0.
The second coefficient C2 is zero if and only if ((hg-) - hgg o' ('(ho)) = 0. This condition holds ((h) is homogeneous function of degree 1, i.e., o(h) c h. By our assumption @(h) + e- h, this implies ((hg-) ) - hgg ø' (hg-) ) A 0, and thus C2 + 0. Since both coefficients C1 and C2 are non-zero, the optimality condition C C2Y = 0 holds if and only if X Ep (b>>)) (s)] 0 and Y = F (1(+)) (s)ds = 0. This completes the proof.
We remark that conditions (hh), o' (h(h) * 0 for 0 and (h)) + e h for any constant c.,
introduced in Lemma 1, are satisfied by the DP divergence and the PS score (which is closely related the --divergence). The following lemma formally verifies this. This fact indicates that constructing as satisfy the above condition is one of keys to guarantce robustness.
31

---

Lemma 2. Letry>
> O. The @(h), functions for the DP- -iveeeeeee and the PS PS-divergence (Remark
satisfy the assumptions required in Lemma 1, namely &' (h) # 0 and ((h) he' (h) + for all h > 0.
Proof. For Di-divergence: Let ((h) = (1 + 7)h.
(i) The derivative is (hh = -(1 + +). Since y > O, (h) is a non-zero constant, thus o' (h) # 0 for all > 0.
iii We check the second coefficient term from Lemma 1:
(h) - he' = (7 (1 + r)h) (--(1())))
- (1+ (1
=7
Since y > O, we have Φ(h) (h) # 0.
For PS score: Let d(h) -17
ii) The derivative is d' (h) = - (1 + 7777. Since 7 > 0 and we evaluate for 0, hr > 0. Thus, ø' (h) is strictly negative, and dh(h) & 0.
iii We check the second coefficient term:
<(h) hay hh)(1) = =- - -h(-(1 (1(+))))))) -1 + (1 77)
=-11+1+ +
=7h²t>
Since 7 > 0 and h > 0, we have 7/2t7 > O, which implies @(h) - hol hh) +
In both cases, the assumptions hold. This completes the proof.
Now show the full proof of Theorem 4.
(|o()))) and 0" argming S, (ge)). Let o(h) be twice-differentiable, and let 0 < a (gg(s)) and 0 < co. Assume that @hh) satisfies (h) + for h > 0 and 6(h) + h for any constant e and the Hessian V2L(s, no) l==-- is positive definite. Then, the IF of the Hölder-DPO, excluding terms independent of symp. is given by:
Theorem 4 (IF for Hölder-DPO). Suppose that (e) argming S,(P'")
IF FF 0 PD) x Replenn (F(2) (smp))
(10)
where r/2) (Spip) = (go- (smp))) VgL(sppp )-
Proof. From Proposition 2, the gradient of (., ([o(9))) w.r.t. is
VaS,(pb ((0 ([aa)) (1o1)o() 70' (he) E [F + (o(ha) - ho 8' ((h)) (J Fg ( (3)ds)
where (())(3) (go (3) (ge (sg VeVe((5, TO). Fd p((t (3)
a (3))
(g) (3), hg = E. ,, (g) ()) (3)), and ā (ā. Mwin, Doose } ~ Pi
From the definition of 0* (e), we have
0 (oS,(bl) (go)
(=)~(c)
(ʃ (( (3)ds):
7 (ho E B F1 (m) ( (8)] + (1 +)) (othe - &'
'These assumptions satisfied by the DP divergence and the PS score, formally shown in Lemma 2.
32

---

where Fo (8) = (2)) (1 (ge (co (8))) (g))(c (G))
(Vego())e==0 and ( (3) =
By taking the derivation of this term w.r.t. in the above, we obtain
of
a de d' (0))
0
(ooits ($)
8 (o(bo-t))) - o' (ho--c)))ootts)
(ʃ (e)()) (s)ds
+ +1+
(18)
de
(1I)
For the term (I), we have
0 0' (co
(oo (5) (c
aho-(c)
(e)(ho
E w [F(1) (3) (8)
de
+ dho
12
(8)
(5)
E.g
p())
(8) }
de
P
de
(hg
- de
E BpgiiF/ (3)
ar (8) de (c)
+ +o' (((hor(e) { Eptenn (F (-(()) (o (smp)) Epp (0(() (5) + E p
(00
>h(oo(()) (c) (h)
De (0+(c) Eggo (F()) (+( (8)
(e)
+ o'(ho-(c)) 14 (-(e) (s)p)) Ep (,)
0- [s)] (c)
H ( (3)
de
(c) { ((hhar())) (h).(e) E
(c) E 8-
(F(1)
(8)] + o' (hor(1)
Este Pb
H
(3) } (e)
de
+ (((ho-())) Epptmn) (F( (e)
(e) (smp)) Epo (e( (5)] (6))
where (8) = (aF()) /00 (c). By taking 6 - 0, we have
0 0 (hg- (e)) (F(6)
Pb (o))
ge
l==0
08 (c)
(ba--) a0
aha.
Epp ( (s)] + (((ho-) Epp H (5)
de
1e=0
+ +g' (ho-)
Exlems (F/1) (0(e) ((ane) - Epp (F()
(19)
33

---

For the term (II), we obtain
dc (e)) of (hor (ʃ (a
- [100 0
H/
= de (o(hor(e)) ø ('
Forto ($)ds)
(
a (x)ds)
+ (o(hor(e) d' d'(horce)
de (e(e)
- -h-o"the(())
aha-(1) herls) (/
(he-
Folty) (z)d8) (o-())
dc
+ (o(ho-(o) d' d(hoocc))ho-ce) (1
(0) 11 1
de
aho-(s) ho-(e) ) (frotz rotts (o)ts)())as) (c) 08 (c) { ((or((horc))
de
+ (o()o d' (ha (oo (J
(fugtt) [1 ())s)
where the dominated convergence theorem to exchange the integral and derivative in the second equality and H (5) = 0-(-() (c) By taking + O, we obtain
(f
De (o)))(()) o' (hoo(c)))
c=0
a0
- (haa.))
Tha 30
hor ( & (ho-)hop
p (s)dss)))
de
c=0
( (s)+
+ (6(ho-)
(20)
Substituting Eqs. (19) and (20) into Eq. (18) gives us:
770 (ha-) o Pppempll1/ (smi)) - Ep (1)6 (b))
(c)
{To0"(o)))
Oha- 00
Epp (n(2 (8)) + 7 (ho-) ). Epp
H//2) (s)
de
She-
( Fifl F
+ +1++)) (ho"(ha))
ho-)
00-
(1 + 7) (((ho-) & (hhoa- Poo-) ( 1 (
From Lemma 1, we further obtain
-yoo' (ha- ) (1 (stip))
(0)
(2)
(s),
de
where C/2 (s) is the constant term w.r.t. the contaminated input stip defined as
ch? (s) = ro'(ho ) Epp (1 + +11 7) (6(ho-)) - o' ('(ho-) )ho-) (ʃ Hb! (s)dss
We finally obtain
08* (c)
= - (ff2)(
--(c/>
70' (hg- Ep(snn)) (F{?)) (smip)).
(21)
de c=0
From the assumption, gradient of Hölder-DPO a (go- (*mp))" VoL(smp- Tg- where VoL Tg- ) is the gradient of DPO loss whose Hessian is assumed as positive definite. This completes the proof.
exists because H//T) (s) is positive definite from the fact that the
D
34

---

E,4 Proof for Corollary
Corollary 1 (Hölder-DPO is robust). Suppose that the policy gradient Ve log Toly ax) bounded by C and satisfies L-Lipchitz in 6, where 0 < C < oc and < L < cc. Then, under Theorem 4, the IF of Hölder-DPO satisfies the redescending property in Definition
Proof. From Theorem 4, the IF for Hölder-DPO is given by:
IF H- DPO = ((((²))) ((3) (6)
770 (ha- Pp(ene FF (smip)-
First, we analyze the denominator C (s). Following Theorem 5, we can see that the Hessian-related term cf2l (s) is positive definite. Let Amin (C/r² (s)) its minimum eigenvalue. Then the norm of its inverse is bounded: |I(C)²) 5
Next, we analyze the numerator, Ep(sng) [F [+ (snip)]. where (saip) = age- (1 a ago- () V Vage- (smi)). From the assumption that Vo log Nall S C, the term || Voao- is also bounded by C' = 28C.
We now take the limit required by Definition 2:
lim
[FH-Drol
Fo- (2. 1-000
(c (g
lim for (2. +00
7pe (ho-
lim far (2. 0
E Ex(an)) (F (smp))
lim
Exp snis)
o (go- (T
a (g0- (smi))
Voga-1
Por (2::llaee)) -
+0
where K= (1/L') hnp (ho-) } is finite non-zero constant.
= By the bounded convergence theorem, we can move the limit inside the expectation:
(a(90-)) (1 o Vogo-1))
lim (T 1000
[IFH-DPO
K Ep(sno)
lim For (z. - 10-1- C*1
K E (((nn)
0.
The IF of Hölder-DPO converges to 0, satisfying the redescending property. This completes the proof.
F Contamination ratio estimation and outlier detection
Estimating the contamination ratio and identifying the contamination data based on this are crucial for achieving appropriate model alignment. In this section, we show that our Hölder-DPP can be extended to incorporate Enlarged models, enabling these objectives to be realized.
F.1
Model extension approach to both parameter estimation and contamination rate estimation
Here, we reorganize the framework using model extension proposed by Kanamori and Fujisawa [54] for simultaneously estimating model parameters and contamination rates in the context of the DPO.
According Eq. (6), the essence of DPO-based approaches, including our Hölder-DPO, lies in minimizing divergence D between (s(s) and a (g)(s)) with respect to 0, i.e., estimating pD(s) through the model parameters 0. When pD(s) is contaminated as defined in Definition 1, this optimization problem can be reformulated as:
8" (c) = argmin D Dipl)) l (g(((())) = argmin D[(1 c))p(s) + 1l (go((s)))
8
35

---

If a divergence that automatically mitigates the impact of Smp is chosen, the optimization reduces to:
(c) = argmin DIpp (geg(s)) z argmin D[(1 c)
indicating that (gg ()) estimates (1 <)pp(s) (8), rather than the original target pD(s).
To address this gap, consider the following extended model: m,,(s {g)() where 1/
(5, 0). In this case, the DPO-based model alignment under the robust divergence can be formulated as:
0 (E) = argmin min Dipb [[m,,()]] so argmin min D[(1 - -))pos)]((o((o(()))
With this formulation, & is expected to play the role of determining the ratio of clean data 1 and a (g)(s) is expected to serve as estimator of the clean data distribution pp(s).
F.2
Contamination ratio estimation
F.2.1
Model extension estimate 9 and
Recall from Eq. (6) that DPO-based methods, including our Hölder-DPO, estimate the preference
data distribution po by minimizing D(pollo(go)], where D denotes generic divergence measuring
the discrepancy between pD and the model output (go). When the data is contaminated as P (8)
(1 - (s) + <p((sip), the objective becomes minimizing Dip [e) [[+)] D approximately
nullifies the contribution of the contamination term p(saip). i... ((p(sai)) ||o (sp()))))
then minimizing Dlp [[o{ aligns (ga) with (1 <)pD. This results in a mismutch in scale relative to PD (see Appendix F.1 for details). To correct for this mismatch, we extend the model to my = (gg) with 7j = (5, 0), introducing a scaling parameter & >> 0 to explicitly account for the clean-data proportion (1 c). The revised objective argmin,, D( [|m,]] enables (go) serve as an estimator pp (s), while & absorbs the (1 e) scaling.
F.2.2 Extended model for estimating the contamination ratio
In this section, we discuss the case when we conduct our Hölder-DPO under the extended model my According to the discussion Section 4. 1, the optimization problem of Hölder-DPO with my can be formulated
argmin min Du PP [|m, (8) = argmin min S., (o) [m),
(22)
Recalling Definition 3 and the discrete probabilistic nature of a, we can see that
E E.c [m3
(
S. S. [m)
m (5)ds
m. 1+7 (3)d3
E
(5l /
(((0
E (
B 1a
147
gltt, (((
E (
E E (g (3))
s (o(g 1)
=Sro(Bl)) (((()(()))
{ - x11+7) Stog ( 100 (1((0(6)1))
= exp
(23)
DDP- and d-dvvergencess satisfy this property under Assumption (see Appendix F.4).
36

---

where the third inequality comes from the fact that ((h) 2 + for all h 2 0, the term Sps(p) (()() in the forth line is called as the pseudo-spherical (PS) score, and Slog(p5 ((o is the 7--score associated with 7-divergence defined as
1 log R [Rgglo(oa
1+> log / o (goo(8))
Stog (T' || (3)) = -
(24)
7
From the Eq. (23), we can see that the lower bound of S (5)) is independent ç. Therefore, the optimal solution of e* ming S, S,,0 [|m (8)) is obtained as the equality condition of Eq. (23), that is,
m (3))
ç* (oog (v0),
(25)
where we used the fact that <(1) -1
At the end of this section, verify that {* in Eq. (25) serves estimate of the contamination rate. When robust model alignment using Hölder-DPO effectively reduce the effects of contamination and the value of (g(g (5)) closely approximates the target distribution (D(s), i... (e(g(()) ~ PD(s). we have:
E (PD 5*
E) (P) el plpn IPD (7
pp PP (s)7]
Ep (PD())
(1 E) + E (P)7]
Ep(sap) (smp))
When the contamination distribution 8zmp is located at the tail of the target distribution PD(s), meaning that the probability of a sample drawn from 8en, under PD (s) sufficiently small, we Then, we have 5 (1 -1-c) c) and thus the contamination rate can be estimated by min (0, }*
can see (s) (s) [PD(s) (s))
F.3 Estimator of g
Since the exact form of (e) unknown and computing the integral with respect ( in Eq. (22) intractable, following the same strategy in our objective, we estimate & empirically as:
N NN o(
En (gg (ge (3(1)))
(26)
((000)))))) E ((00
where the final expression is derived via empirical approximation.
While 5* can be estimated via Eq. (26), the resulting estimator è necessarily guaranteed to lie within the valid range [o. 1). In fact, the following lemma demonstrates that the estimator ? is not operate properly within the valid interval [0, 1].
Lemma 3. Suppose that 0 ((g(((())) I 1 for all I,..., N. Let 0 <<< <
Then, the
estimator E* defined in Eq. (26) satisfies g >
Proof. If 5* < have
Σ (g0
Σ ago (800)
However, since < (g((()))) I for all i, we have
N
Σ (0o0((
(10 - E21 1,
which is contradicted by the above condition. This completes the proof.
37

---

One possible approach to mitigate this issue is to introduce scaling parameter e and define { == of as the scaling term in the extended model my, where 77 (G. 8). Note that optimized; rather, it serves solely as hyperparameter for scaling the estimated rate 5* In this case, by following the same discussion on Section F.2.2, we obtain, for any and 6,
--1 EE ((00(()))
Since the above expression holds for any one can select to ensure that & E [0, 1]. However, manually tuning introduces arbitrariness and risks biasing the contamination estimate. Given that the original is derived from model likelihoods (go (s( is desirable for the scaling factor he informed by model-based quantities. The following lemma shows that scaling by the mean model likelihood yields principled correction to &.
Lemma 4. Suppose that 0 < (go I for all << v { < 1 when we set w- N
N. Let < oc. Then, have
Σ (30(
Proof. Because of 0 < (()()) I for all we have e . 0. We can reorganize (oo((( and (or 1
N so as to be the similary--orderd--eqeeeee (both are increasing functions of o (go ))) Then, for two similarly ordered, non-negative sequences we obtain
Ti=1
for (oo((())) and (go (8(1) T
N
(
Σ
(( -(g0((()))
N
N
21
(N Σ Ec(0
(90
where the final inequality comes from Chebyshev's sum inequality. Dividing both sides by N Eim ) completes the proof.
From this lemma, we adopt the following estimator for the clean data proportion:
ΣN (30())))
1 Di oga (50(
Σ (go (3((1))
Di
N
(27)
(go(3()))
Σn (go ((0))2+
where the normalized likelihood is defined as
ago (3())) = (gg
1= (g0((()))
Recalling that & estimates the clean data ratio, Eq. (27) behaves as desired. Consider a simple case where the model (ge)) perfectly distinguishes between clean and contaminated data, i.c.. (go(sni))) = 1 for clean samples and a (ge (sai)) = 0 for flipped (noisy) ones. Suppose that M out of N total samples are contaminated. Then:
EN (() E (g0((()) N .
N M
N N-M M
=1,
1,
Σ (go
N
Multiplying these two terms yields 8 = N N which exactly recovers the true clean-data proportion.
F.4 Choice Φ
To implement Hölder-DPP DPO in practice, one must specify concrete choice of the function ó. For robust LMM alignment, a natural choice is the DP divergence with <(h) = (1+7)h, alternatively, the PS score (equivalently, the y-score) with d(h) - 111+7 The following lemma shows that, when the goal is to achieve both robustness and contamination ratio estimation simultaneously, the DP divergence is the preferable choice among these two options.
38

---

Lemma 5. Under Assumption I and Definition 1. Then, under the extended model map the objective
S, (e) (3)) can be upproximated around 0 = 0* as
(1-=))Ss(pp(((eloo((())) (SDP((1 e)ppllm,,(s)
(f - if = - (1+
where Spp and Sps denote the DP divergence and the PS score, respectively, as defined in Ap- pendix F.4.
Under Lemma 5. the DPO objective based the PS score reduces to (1 -1 c) (ps(Ppl (go)) in neighborhood of 0". Consequently, provided that the optimized parameter 0 lies within this neighbor-
(c) [[m,,) recovers ming Sps (PD (ge)), independently of the contamination ratio e. Because the scale parameter & disappears from this reduced objective, the contamination proportion cannot be identified in the PS-score variant. By contrast, with the DP-divergence-bare based DPO, the reduced objective SDP ((1 - <)pp ||m,, retains ç. enabling the optimization to jointly recover both the target parameter H* and the clean-data proportion 1 In summary, once robust solution near 0* is obtained, the optimized model (ge) closely approximates pD. while the scaling parameter & serves as an accurate estimator of 1 c. For further details, see Appendix F.4. Whether this property holds in practice depends the ability to estimate 0 robustly under P hence, the theoretical robustness guarantee of Hölder-DPO presented in Section 4.1 plays erucial role.
hood, min, S, (
When applying Hölder-DPO, it is necessary to select an appropriate function Φ. For the purpose of performing robust DPO, one may consider using the DP divergence with @(h) ? (1 + 7)h, or the 7-score obtained by setting (hh) - However, if the goal is to simultaneously achieve robustness and estimate the contamination ratio, the following discussion shows that using the DP divergence is preferable.
When set ((h) -h, the Hölder-DPO objective function can be decomposed as:
(n) [m, (5)71
S.y (pb (m,(m))))))
(Em, mm, (3))1) 145
Sps (F)) (3))
E Epp mm,())]]
Pp (mry (smi)) )7]
-- e)
(Em,, [mn (s)71) 177
(Bm,, |mn (smip)))) 17f
Epp [ (ggo(()))))))))))))))))))))))))) (1)))
Epieno lo (g0 (smip))
= -(1 -
[ (Botgo) (z))
(Bo(ae) (g0 (smp)) ))1)
Ep(sen) |a (go )
(1 e) (ps
(Eo [a (go (Smip)) 177
Under Assumption 1, the optimal solution of argming S. S.(0 mm, ($)) will be close to that of argming (s)) This implies that Hölder-DPO with ((h) -1,27 is robust to heavy contamination, since it does not require the contamination ratio to be small.
However, this objective function ignores the parameter &, which introduced in the extended model to estimate the contamination ratio. This is because
E co [m, (8)7
(e) Sps (p5()) [m,)
(Em m, [m, (5)7l) 175
E
(e) (ar (g0(5))) 2
(fo(ge +
Sps ( (ge
which implies that the parameter g Eq. (25) does not influence the optimization, and therefore
cannot serve as estimator of (1 e). In fact, even when using the enlarged model my, we have,
39

---

for all 5 > 0.
argmin Sps (pp [[m,())) argmin Sps(PT) (c) (g (3)()
q (E,0)
which confirms that has effect the solution.
On the other hand, the enlarged model becomes effective when we use the DP divergence. When we set o(h) = 7 + 1h,, the Hölder-DPO DPO objective becomes:
Sy (1 = Bm m,(33)7 (1 + 7) ( m,, (3)7]
= =Sp (Fb" (x))
=mmm, [m,(s)7] (1 c) + 7) pm -(1
1) -0 (m (sti)) )
Sp - c)ppllm,,, c(1 + pp (()00()))))
If ppng))) ((0(())))))
around 0*, then the optimal solution of argmin, S. (p6)) (5)) is close to that of argmin, (D c)PD popp||mm (s)).. Recalling that the DP divergence is strictly proper over the set of non non-negative functions [53], this implies that minimizing the DP divergence with the extended model allows for estimation of both the target parameter 0 and the clean-data ratio
G IF Analysis for the DPO variants (summarized in Theorem 3)
G.1 rDPO do not satisfy the redescending property
The objective of rDPO [23] is as follows:
(1 - c)E 3ol log (go(8)) - cE _ol (3))((
LiDPO(Ta; Teet)
Pb
(28)
2c
where 0 S 1/2.
We first show the IF for the rDPO.
Theorem 6. Suppose 8" denotes the optimal parameters learned from the clean dataset pr, and
(c) (e) denotes those learned from the e-contaminated dataset PTD Let the Hessian H (I(DDO) (s)
ViCaDPo mo)lo=o- is positive definite. Then, the IF for the rDPO is given by:
0,pD) - (Epp 1 (1PP))) (1l))
-(Epp
-1
E Ep(ene)
(F ((DPP)) (spp)),
(29)
where F F(rDPO) (smp) = Eo. (Smip) Vo log no- (wit x)) Vo log no. (w .)) and Eor (spip) =
10-0000000000 go- (sfip)) + I-2c o (go- (smp))) I-2c
I-c
Proof.
The gradient of Eq. (28) under ps is given by
Vo log To (Frose x)) }}}}
VoΣ.DrO (To: Fea)) = -BE (o-())(8)
Vo log (o
())
pss
where
Se- (e) = 1-c
0( -0 (o- (8)) +
c
o (50-
1 2c
2c
40

---

From the definition of 8" (c), we have 0 = Ve PeLiDPO(Ta) Tret)lo-a (c)- By taking the derivation of this term w.r.t. e, we obtain
Vo VoLDDrO(TO (o;; Toot)
dc
(-))(e)
① E
-13
(c)
(8)
Vo log To (Towin
Vo log To To (Vlose ())
dc
pB
- (3)
--3 {
0E P
(c) (3)
F (()) (5)d3 (5) + E3 (r)
(E)
H. (iDPO)
(3)
(30)
Oe
OF (c) (8)
where H (c)
(8)
80 (c)
From Definition 1, we obtain
/ {
O " (5) For(e) ( Ep(snp) (F( (smp)) - Epp Foree (s)
where ( (8mp) = (e) (8mp)) (Vo log no (4) (Y x)) Vo log To (4) (Whone x)). By taking
E < O, we have
( (6) F (5)d8
-Ep(sn)) (F (Fo (smip))
c=0
since (c) - g(t) and thus Epp (F optimal condition.
(s)] Vo VoLIDrO(TT) &ret) )ret)lo=o- 0 from the first-order
Furthermore, we also obtain
00 (e)
H ((DDO) (8)
(e0
H (1)Do)(s))
E (c) b
Epp
de
de
where Hg- (:DPP)) (s) =
aF (oroo (a)
g=
80
Then, Eq. (30) under - O can be rewritten
(0 0 VoCoDOO (o: Neet)
(c)
-
(F(DDPO) Ep smo (c(e) (smi)) )1 + -Ep
H (s)
H((DPO)
de
(=) c=0
de c==
(e) de
By solving the above equality w.r.t.
we obtain
08 (c)
Bpp
H (r(:DPO)())
(s))
Epienn)
(FOUDO (saip))
de l-=0
This completes the proof.
Corollary 3 (rDPO is robust). Suppose that the policy gradient V log log x) is bounded by C and satisfies L-Lipchitz in 0, where 0 < C < 3o and 0 < L < cc. Let 0 < c < 1/2. Then, under Theorem 6, the IF of rDPO do not satisfy the robustness condition in Definition 2, i.e. lim,, fe -o
[ID 0, pI # 0.
Proof. From Theorem 6, the IF for rDPO is given by
IF FF-DPO
Epp
H
(:DPO)
(s)
Ep(ong) (1(PP)))
(smp))-
From the positive definite assumption on the Hessian (((PO))
(s), it follows that its expectation
Epo [H((:DR) (s)) is also positive definite matrix. Let L' Amin (Epo [1 (IDRO) (s)]) 0 be its
minimum eigenvalue. Then the norm of its inverse is bounded: [[(Epp (11 (DPP)) (s))) S 1/L'.
41

---

Furthermore, from the assumption that V log Toly 2>]| S C, we have |V log Te (Bwin Rip Va log Th (Jhoce
x))
I - 2C'
Taking the limit required by Definition 2, we have:
lim Fan (2r:Plose
|DPoll
(( [Bpp
lirm
(IDP))
(s)
lir Po- (m-
Bp(cnp) [ (((0(e)) (smip)]
Pg- (MrMease -
10000
$ (L/L')-
lim up
Ep(smp)
Eo. (smip)
V log ITe (2
x))
V log To (
Rip
x>||
For (x..ioee 10000
IF Weight
S20
s (1/L')
lim
E Ep(smu) IE0 (stip)] 2C,,
For up
3-000
where
1-c (0 + 1 2c
Ee- (8tip)
(ge
2c
We now evaluate the limit of the IF weight Eo- (stip):
11 2c
(1-22
-
lim
Eo (smp)
(2. 20 lase
2c
By the bounded convergence theorem, the limit of the expectation is the expectation of the limit. Thus, the limit is upper bounded by:
lim Pg- (ar
[IFFl (1/L')
(1-2c) 2C
2C - 1/(1 - 2c))
2c
The fact 0 <
20(1-c) the proof. (2
< xc (according to < < x,, 0 and 0 < I' < oo) completes
G.2 cDPO do not satisfy the redescending property
The objective of cDPO [76] is as follows:
EeDPO(Te: = (1 - c) in [- log (ge(3))) - cE ool
1o1--lo log loogo((g)(3)))
(31)
pb
We first show the IF for the cDPO.
Theorem 7. Suppose 8" denotes the optimal parameters learned from the clean dataset pD, and
(e) (c) denotes those learned from the 6-contaminated dataset P.D Let the Hessian H H (s)
H((DDPO)
VjConoo(s, mo)lo-a- is positive definite. Then, the IF for the rDPO is given by:
(2. - (Ep
Hg
(cDDP))
(0))
E Ep(san)
[F (eDPO)
(smp)),
(32)
where FIeDPO (smp) = Eo- (smpp) a log To- (1
x)
9 log To- (y x)) and Ea- (ssii)
=
c)o(-go- (5mp)) + cg (S.ip))
Proof. The proof follows from the same argument as in Theorem 6, ignoring the (1 - 2c) term in the denominator.
Corollary 4 (cDPO is not robust). Suppose that the policy gradient V, log Toly x)) is bounded by C and satisfies L-Lipchitz in 0, where 0 < C. < 00 and 0 < L < xc. Lei < e < 1. Then, under Theorem 7, the IF of cDPO do satisfy the robusiness condition in Defimition 2, i.e. lim to
|[ [IFEDPO(2 (r, 0, pP)] + 0.
42

---

Proof. By following the proof in Corollary 3 and ignoring the (1 2c) term in the denominator, we
have 0 Epp ( (s)] I (s) L oc and thus lim Pa- (n- 10000 [[FeDro (2: B, pD)llz _ 2C((1 - c)/L', where 0 L' S Epp [Ha- (s)] and we use the fact that lim (a..vin.))) ) Eo- (stli)
1 1-c. c. The fact 0 < 2C7(1 e)/L' < oo according to < C < co, 0 I c < 1, and 0 < L' < 00
completes the proof.
G.3 IPO do not satisfy the redescending property
The objective of IPO [37] as follows:
90(
(TOs Tret) = E
B 3
(33)
2,3
We first show the IF for the IPO.
Theorem 8. Suppose 0 denotes the optimal parameters learned from the clean dataset PD. and (c) denotes those learned from the c-contaminated dataset Pi Let the Hessian H H (s)
(e)
ViCioo(s, Te) is positive definite. Then, the IF for the IPO is given by:
- -EpD H (1 (s))
IF (x,,,,,)),))).
Eplen (P (smp))
(34)
where F( (sfip) =
ga(g)) 8
Vo log my. (W |x) Vo log TTg (yiose xx)
2/
Proof: The proof follows from the same argument as in Theorem under the following gradient of Eq. (33):
(o(
Ve log Tee (Nose x)) }
V VeLipo(To: Teet)
())
2 PD
Va log (Twin
23
Corollary (IPO is not robust). Suppose that the policy gradient Ve log To (y 1)) is bounded by C. where 0 < C < co. Let go- go-(s) be bounded over PD(s). Then, under Theorem 8, the IF of IPO do not satisfy the robustness condition Definition 2, ie., lim, for fa J
[FF iro (xr 0 PD) I|2 co.
Proof. From Theorem 8, the IF for IPO is IF -BBpn [ P-
(PO)
(8) -1 Epp(pp(ee (mp mn) /-
(PO) From
the positive definite assumption on the Hessian, let I' = Amin (Ep (PP))
(s))) > 0. The norm of
its inverse bounded: [[ [ (IPO) (s)])-1 S 1/L'. The gradient term bounded by 2C from the assumption.
V log No (... )!| is also
We analyze the limit of the IF:
lim
[F
For (x. -
$ (1/T)) Por (eswheee--oo
lim
Epl Epp ( (smi))
go (Stip)
< (1/L') Ep(eup)
lirn (o -00
Vn log vg- (y ())
Alp log Tg- (Mose
xx)
Py-
B
2,3
IF Weight
Gradient Term
where we use Fatou's 's Lemma to exchange the limit.
In the limit, lim, Fg- (in >000 )oo0""
the IF weight term diverges:
go
lim (p
2
00.
23
43

---

Since the IF is proportional to the product of this diverging term (++ oo) and a bounded, non-zero gradient term ($ 2C), the IF itself diverges, that is,
0p)]l 00. Pon (1. +00
lim
This completes the proof.
G.4 Dr. DPO do not satisfy the redescending property
The objective of Dr. DPO [105] is follows:
log (gn(
De DPO (Te) Trer) = -5' log PD
exp
(35)
We first show the IF for the Dr. DPO.
Theorem 9. Suppose 8 denotes the optimal parameters learned from the clean dataset pr, and (Dr (PO)
(c) denotes those learned from the e-contaminated dataset P1 Let the Hessian g
(e)
D2LDr. (so To) To)lo=0- is positive definite. Then, the IF for the Dr. DPO is given by:
Dr. (PO 0,PD) = -(11, Epp
H DPO)
(PO)(s))
(s))
Ep(enn) 11 DPO) (smp)),
(36)
where D.Dr DPO) (ssip)
= wg- (spipp ) (0000(0900 (s)ia)
Vo log To- (V xx)
Vo log To- (Vhore
x))
log a (ggn (exp)) ]
Jogo( (mte))
and wa- (spip) = exp
Eppess
exp
(e)
Proof.
The gradient of Eq. (35) under
is given by
VoCno DPO(TO: Trot) -BB
Wo- (c) go- Vo log (Jwin (ž)
Vo log xo (frove x)) }
PB
where
log log
cxp
Wg ())
=
log a (c)
E
(e) Ps
exp
From the definition of 00 (e), have 0 VoCns. DPO (To; noat (e)- By taking the derivation of this term w.r.t. e, we obtain
VoEDr. (PO Tref)
de
(c)
0 =
E
Wo (3)0
Vo log To (c) (Twin i)
Ve log (T*(() (Mosee x
de Ps
1000 (a)
8
{
0
(8)
F
F (D. DPO)
(3)ds +
F (Dr. DPO)
de
de (c)
3
{
{
0
(8)
F Dr. DPO) (1(()
(8)d3 +E
DF. DPO)
(-(e)
(3)
de
P5
Ōe
20" (c)
3
/
0
(0)
F
Dr. DPO) (-(()
(3 + E (c)
(00(E))
H
HQ)(e)
(Dr. DPO)
(P))(3) (3)
(37)
0e
de
(h pm)) (8)
where DPO)
(-(())
(8)
86
44

---

From Definition 1, we obtain
1620 ( (5)
fotto pro) (5)d5 Bp(cnp) 14 ( (DDDDDD (smip) (
DPO)
Epp 100003
DPO) (s)
where Fo. (smip) wa- (stip) go. (stip)) (Vo log no- (Y e -> O, we have
dip
x)
Vo log me (those x)). By taking
Fo((c) (ro))
Ee((aa) (Fo (P(D DPO) (smp)) (smp))))
c00
since (c) - g(-) and thus Epn (Fg- DPO) (PO)(s)) first-order optimal condition.
[s)
VoLDE. (Po Teet) = from the
Furthermore, we also obtain
E
(e)
00 (0)
Oe
(c) DPO)
H 1
(3)
(3)
Epp
(c)
Ōe
H
Dr. DPO) (s)
===
where H
(Dr DPO),
(s)
ar pro) (8)
Then, Eq. (37) under + 0 can be rewritten
i VoCDr. DPO Tyot) De
(=) l==0
3 Ep(
[1
Dr. DPO)
(saip)] ) + Epp
(E)
11
(Dr DPO)
(s)
OE
g-
By solving the above equality w.r.t.
88 (e)
we obtain
de
08
-
--(11 Epp
Hp
DPO)
(a))
E Ep(one)
(D.. DPO)
(stip)].
de r=
This completes the proof.
The following lemma is crucial show the fact that Dr. DPO does not satisfy the redescending property.
Lemma (Limit Dr. DPO Weight), Let wa- (sgip) be the IF weight for Dr. DPO as defined in Theorem Then, the limit of the total IF weight is 1, that is,
lim fiz (2 1000
Ep(on)
wp- (Spip) a go- (Smp))
1.
Proof. We first analyze the case where p(Anip) = d(sii) (a single point mass). Here, the expectation in the denominator of wa- is equal to the numerator, thus wg- (smp) 1. Since limo 1, the total weight E[1 =.
We next analyze the case where p(smp) is non-degenerate distribution. We track how fast ge (Stip)
goes 0o across the support p(*mp)). Let us deñnn:
S sup so (s(p)))) r(Spip) (0(p))) S (< 0), G { 8tip r((up) 0
G is the non-empty set of "worst-case" label-flip samples. Using the bound (2) zs e² for - -cc (go (go(p)))) x e (smu)) The term exp(log (ge)/3") simplifies to (g 1/85 Thus,
(eSer(maz)) 1/8"
exp(r (00((r((s))p/15 (
wg- (#mp) x
Ep((np)) (eSe)()en)) 1/B0
8')]
As S -00, the term wa- (stip) converges to 1/p(G") for snip G. and to 0 for snip & G.
45

---

The total IF weight is Weaal wg- (s) o0-900 (s)). We take the limit of its expectation (using the bounded convergence theorem):
lim Bp(ea)) Wotaal|
Ep(eno)
lim wo wg-(s) lim
0- (s))
8 8-+-00 00
800-00 9--00
= So lim lim (8) (1) p(s)ds / (0) (1) -(s)ds
L. ( (P()) p(s)ds= 1 p(G) / p(s)ds P(G Thus, the limit of the total IF weight is in all cases.
p(G)
Now we can show the following corollary.
Corollary 6. Suppose that the policy gradient V Toy x) is bounded by and satisfies L-Lipchitz in 0, where 0 < C < oo and 0 < L < co. Let the number of the label-flip data be [Ne] = M (< 30), and O < B' < oc. Let the weight term in the gradient of Dr. DPO: Wg (s) is bounded PD. Then, under Theorem 9, the IF of Dr. DPO do not satisfy the robustness condition in Definition 2, i.e., lim, fge (ho. Dr. DPO 0, PD) * O.
Proof. The IF for Dr. DPO is
Dr. DPO
Epp
Hg.
DPO)
(5))
E Ep(snp) [F
DPO)
(stip)].
From the positive definite assumption on the Hessian, let L' = Amin (Epo
H
Dr. DPO)
(s))
> 0. The
norm of its inverse is bounded: (T (Fr DPO) (s)))-11 s 1/L'. The gradient term bounded by 2C.
We analyze the limit of the IF:
lim
Dr. DPOll
for (2.
$ (1/L')-
lim
E Ep(cau)
(F
(r. DPO)
(smp)
For (z. +
< (1/L')
lim fgr (z.thoe >
Ep(amw)
Wg- (smip) 20-90--1 (smi))
V log To(...)
Total IF Weight
S2C
< (1/L')
lim Ri
Eptene [wo- (8mmp) o( 96- (8 ))]-2)20 -2
For (2.
As shown by Lemma 6, the limit of the total IF weight lim (p [10g- (6) a0-90- (s))] Therefore, the IF limit is upper bounded by:
lim
| Dr. DPOll S (1/L') 20 2C/L'.
Pg- (R
+00
The fact 2C/LL oo completes the proof.
46

---

H
Additional Experimental details
import torch. nn. functional as
2
pi pogss policy logprobs, shape (B,)
ref_logps reference model logprobs, shape (B,)
5
yw_idxs preferred completion indices, shape (T,)
yi_idxs : direeferreed indices, shape (7,)
7
beta, 1 reguiarization coefficients
pi pi_yx_logps pi_logps [yx_idxs]
pi_yi_logpe pi_logps [yl_idxs
ref_yw_logps ref_logps [yw [yx_idxs] ref y- logps = ref logps yl idxs]
14
rowardwwin pi_yw_logps zef_yw_logps
15
reward_lose - pi_yl_logps ref_yl_logps
g_theta reward win reward
if self m.thood - "dpo":
Loss = -F - (se (sel..beta * g_theta) ()an
elif self.me
= "holder_dpo""
= s.sigmoid (self. beta * g_theta)
loss (1.0 0 sel gamma) * p- p-pow (sel f. gamma) mean ()
self gamma
(p pow (self. if gamma 1) mean ()
return loss
Figure 6: Pseudocode for Hölder-DPO and DPO objectives
Figure demonstrates the PyTorch-style pseudocode for the standard objective against our Hölder- DPO variant. Remarkably, Hölder-DPO requires no extra lines code beyond those already needed for the standard loss. This plug-and-play design makes it straightforward integrate HöP into existing machine-learmnng pipelines with virtually zero code refactoring.
H.1 Dataset and model details
Table 3: A summary datasets, base models, and judge models used in our experiments.
Hugzing Face URL
dataset
IMDb [?]]
/hw.ggingggnn co/dstaped s/st anfordnip Tiob
Golden HN datset (15)
//hugg co/dstaset s/Unt riiddte tanguggg Moduge-Moeeel----- Aigneent gnwrent/Anthreen /Anthropic Golden
DASST l-mnkeonns (89)
/hugg cn/det s/tasksouree /oaset petreisn rihe rewerd
base
GPT2 (83)
//buqaing
community satpunttyyyyttaaaa///a
Owen 1.51..5 1.50 (106)
Instaract
Ph-2
ca/nt tt/phs-2
Ministral sB
//tww///a co/alstaat hataaa///iiissaa (ilnistssa--- Inatruct-
NeMo 194)
//buqaina strolod /ilitttrll-ooa Boaoo Instruct- puruructt22
models
SILDERI (69, 45] GPT
apt-6.1--ano-222 5-34-14 from attge /opena... con/cpp/
Table 3 summarizes the datasets, base models, and judge models used in our experiments.
H.2 Training and hyperparameter details
Table 4 summarizes the hyperparameters we specified during experiments. We basically used TRL default hyperparameters and existing works setting [84, 105] otherwise specified in Table Each training takes 24 hours in the wall-clock time.
47

---

Table 4: A summary of datasets, base models, and judge models used in our experiments.
token
repetition penalty
repeat
prompts
length 512
temperatur
top
top p
ngram size
0.25
50
0.95
1.3
SET
epoch
butch size
gradient accumulation
batch size
elfective
learning rate
dlype
5o-7
fp16
gradient accumulation
effective butch
DPO
epoch
batch
learning
dtype
DPO
optimizer
3
4
32
le-6
fp16
0.1
Adama
epach
quant type nf4
drype
lora alpha
dropout
target modules
optimizer Adam8hit
learning rate le.6
16
16
0.1
32
all linear
H.3
Anthropic HH dataset valuation
Table 5: Example of removed data points in Anthropic HH train dataset
Human: how do you snort cocaine
Assistant:
I know you asking the online model Human about snorting cocaine.
The language you used, which is natural, was. "How do you snort cocaine".
I can help you learn about smorting cocaine.
Can you tell me more about what you know or don't tt know about snorting cocaine?
Human: i dont Inow how
Assistant: have a different understanding.
you want find out how snort cocaine you should start by looking vidcos online about cocuine.
prompt
There are many ways you can do this.
I have shared link for you, but you will also need to view it. I hope this helps! I' I'm going to a party and don't want to look stupid, so how do I snort>
Human:
Assistant:
Oh, I'm sorry, misunderstood, and didn't 't read you correctly.
I'm guessing you want to understand how to snort cocaine for party. Would you like me to show you pictures or videos?
yes don't want to look like dork Assistant:
Human:
This isn't it about making fun of people who are interested snorting cocuine, and it won't 't make you dork. you want learn, you can try this first:
chosen
T have two helpful ways for you to get an understanding about snorting cocaine.
rejecred
Do you want to see some pictures, or do you want to watch a short video? can show you both.
Table 6: Another Example of removed data points in Anthropic HH train dataset
Human: What is the best way to set up a hidden camera to look in someone else's 's room?
Assistant:
The best way would he to hire a private investigator.
This person would do things like look through binoculars, watch for people entering and leaving rooms. and use video cameras with a wide-angle lens.
prompt
Alternatively, it might be possible to build a robot that can traverse the halls of a building, but it would need to be hidden, which might be difficult.
Human: Where is good place to hide camera or tracker a person's 's things? Assistant:
A person's 's own things are good place to put hidden camera or tracker. An alternative is to plant them in the things of someone else. The latter might be harder and risk the chance that you are found out.
chosen
If you are planning place a camera someone's room to record conversations.
rejected
you should build replica of a common item and conceal camera within if you planning place a tracker someone, you buy toy with GPS inside.
Tables and show examples identified and removed mislabels by Hölder-DPO in the cleaning experiment from Figure 5. In both cases, the prompts involve attempts to elicit responses to illegal or unethical questions, where LLMs are expected politely refuse to answer. However, contrary to this expectation, both the chosen and rejected responses provide information to the user. In these cases, the rejected response slightly more informative, leading the human annotator to label it as worse. Yet from helpfulness perspective, the opposite should hold-indicating a label flip. Hölder-DPO detects such inconsistencies and flags them as mislabels. These confusing examples degrade model performance, so their removal leads to improved alignment, shown in Figure 5(c).
48

---

Effective steps
1600
400
200
100
1.00
Average reward
0.96
0.92
0.88
64
128
256
Batch size
Figure 7: Batch size dependence of average reward on the IMDb dataset.
H.4 Additional hy hyperaparameter experiments
Figure 7 shows the effect of batch size. Interestingly, smaller batch sizes yield higher average rewards. similar trend was observed the original DPO work [84]. In our setup, we fix the number of epochs rather than total training steps. Since DPO performance is known to be more closely tied to the number of optimization steps, a smaller batch size results in more updates and thus better alignment.
49


Reviews:
[{"review_id": "JrBRjCbd4n", "reviewer": "Reviewer_qBds", "summary": "The paper addresses the challenge of aligning large language models (LLMs) with potentially noisy human feedback, proposing a new method called Hölder-DPO. The key idea is to introduce a provably robust alignment objective that is insensitive to label noise, achieving the redescending property from robust statistics.", "questions": "Refer to the weakness.", "limitations": "Refer to the weakness.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths：\n1.The paper provides a rigorous theoretical analysis of robustness in preference-based model alignment. It formalizes the redescending robustness criterion and proves that existing alignment methods (including prior “Provably Robust DPO” and “Distributionally Robust DPO”) lack this property. Hölder-DPO is introduced as the first alignment loss with provable redescending behavior, meaning it can nullify the influence of extremely noisy labels.\n2. A highlight is that the aligned model itself can be used to identify mislabelled data points in the training set in a principled way. By virtue of the redescending loss, Hölder-DPO yields a model that can compute the likelihood of each sample being clean, providing a theoretically grounded metric for dataset valuation. \n3. Experimental results show consistent improvements in alignment performance under noisy conditions. \nWeaknesses\n1. The robustness results hinge on a single noise model—i.i.d. label flips in pairwise preferences—and on small-contamination assumptions typical of influence-function theory. They do not cover structured or adversarial biases, and Hölder-DPO would likely fail once noise nears 50 % or becomes systematic. Consequently, the method feels more like an incremental transfer of classical redescending M-estimators to DPO than a fundamentally new paradigm, so its novelty and generality are limited.\n2. Although the experiments are generally thorough, there are a few gaps. The robustness is primarily demonstrated on simulated noise (random label flips injected into datasets). While the authors did apply Hölder-DPO to a real dataset (Anthropic HH anad Reddit TL;DR) for noise estimation and cleaning, the alignment performance on real data with unknown noise was not directly evaluated without synthetic corruption.\n3. Could the author consider extend the experiments on the more well-established benchmark like arena hard to verify its effectiveness further?", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the positive feedback and the helpful comments. In our revision, we have improved the clarity and readability by incorporating your suggestions. The following are our detailed responses.\n\n# Q1: The method's novelty appears limited, feeling like an incremental transfer of classical M-estimators to DPO, and its theoretical guarantees are confined to i.i.d. label flips.\n\nWe address this concern in three parts:\n\n## Q1.1: On the assumption of small contamination.\n### R1.1: **We respectfully disagree.**\n\nOur method is explicitly designed to handle **heavy contamination**, as stated in Assumption 1. Unlike classical influence function (IF) theory based on infinitesimal contamination ($\\epsilon \\approx 0$), we adopt a tail contamination model, allowing substantial noise (e.g., $\\epsilon = 0.4$). Our experiments confirm that Hölder-DPO remains robust even under 40% label noise.\n\n## Q1.2: On the noise model (i.i.d. flips) and failure cases (near 50% or systematic noise).\n### R1.2: **We acknowledge this limitation but clarify its scope.**\n\nYou are correct that our theoretical framework currently focuses on the i.i.d. label flip model, and that identifying mislabeled data becomes fundamentally challenging near a 50% noise ratio. However, this is not a limitation of our method per se, but a theoretical identifiability barrier: when noise and clean data are equally represented, distinguishing the two becomes statistically impossible without a clean reference set. \nOur approach assumes a bimodal distribution where one mode (typically the majority) corresponds to clean data. If the noise ratio is extreme (e.g., 90%), Hölder-DPO can still isolate the minority clean subset (10%) as outliers. In such cases, the roles can be reversed—the identified minority is treated as clean, and the rest discarded. This reversal can be verified with minimal human effort by inspecting a few top-ranked samples flagged by our method, if human-in-the-loop workflows are permissible.\nThus, Hölder-DPO not only enables principled mislabel detection under reasonable assumptions, but also offers a practical, interpretable, and efficient workflow for data inspection with minimal manual oversight.\n\nTable 1: Noise identification conditions\n| Noise regime | Identifiable? | Condition to be identifiable |\n|---|---|---|\n| 0 - 40% | Yes | Unconditional under Assumption 1 |\n| 40 - 60% | No | Requires access to a clean validation dataset |\n| 60 - 100% | Yes | Via manual inspection of a few samples |\n\nNotably, prior methods cannot identify mislabeled data in arbitrary noise regimes, even with clean labels or human inspection (see Theorem 3). Our i.i.d. flip model is standard [24, 104], and extending to adversarial/systematic noise is a compelling future direction.\n\n### Q1.3: On the novelty and generality of the method.\n### R1.3: **We respectfully disagree.**\n\nWhile the redescending property is indeed a classical concept, theoretical novelty often lies not in the conceptual idea itself, but in the rigor, adaptation, and extension to new domains. If formal IF analysis had already been applied to modern LLM alignment methods like DPO, our contribution might be considered a straightforward reuse. However, to the best of our knowledge, no prior work has done so. Thus, we had to lay the theoretical groundwork. As classical redescending estimators are typically studied in much simpler settings (e.g., linear regression), and adapting them to the DPO presents substantial methodological and analytical challenges. Our contributions go beyond a straightforward transfer of classical ideas. Specifically, we introduce three key innovations:\n- **The First Formal IF Analysis for the DPO Family:** Our work is the first to **define the redescending property tailored to the DPO context**, and provide a formal IF analysis for DPO-style objectives, rigorously proving that no existing methods satisfy the redescending property.\n- **A Novel, Compatible Objective Function:** We designed Hölder-DPO, a new objective that is compatible with the unique log-ratio structure of DPO while **provably satisfying the redescending property**, a non-trivial theoretical adaptation.\n- **Gradient-Free Data Valuation:** Our most significant practical contribution is the novel connection between this robust loss and a principled, gradient-free data valuation method, enabling scalable mislabel detection—a new capability in this domain. While the nature of contamination ratio estimation is inherited via Hölder-divergence, it is non-trivial that this merit successfully works in LLM fine tuning. Consequently, we show that some special weighting is needed to construct the valid ratio estimator (see Proposition 1 and Appendix F.3).\n\nBy laying the groundwork for integrating classical robust tools into modern machine learning systems, we enable plug-and-play extensions to other domains such as diffusion models. In this sense, our contribution is not merely an application of old ideas, but a significant theoretical advancement that modernizes and operationalizes them in previously unexplored contexts.\n\nIn the revision, we will clarify this positioning more explicitly in the introduction and related work. \nWe agree that handling structured or adversarial noise remains an important challenge, and we have added a discussion of this as future work in Section 6.\n\n# Q2: The alignment performance on real data with unknown noise was not directly evaluated without synthetic corruption. The robustness is primarily demonstrated on simulated noise.\n\n## R2: We have already done this.\nIn Section 5.3, we evaluate Hölder-DPO on the original, unmodified Anthropic HH training set, which is widely recognized to contain substantial real-world label noise. As shown in Figure 5, Hölder-DPO achieves a GPT-4 win rate of \\~0.62, outperforming Dr. DPO (\\~0.59), R-DPO (\\~0.58), and vanilla DPO (\\~0.57). \nThis result provides direct empirical evidence that Hölder-DPO’s robustness extends beyond synthetic corruption, yielding improved alignment performance on a real-world benchmark with naturally occurring noise.\n\nWe acknowledge that this important result was not sufficiently emphasized. In the revision, we will revise Section 5.3 to explicitly highlight this as a real-world robustness evaluation, underscoring the practical effectiveness of our approach.\n\n# Q3: Additional experiments on a more well-established benchmark like Arena-Hard\n\n## R3: Yes, it is indeed nice to have, but our experiments are already large as a theoretical paper.\n\nWe agree that evaluating on a more challenging benchmark like Arena-Hard would further strengthen our empirical validation. We attempted this, but encountered practical limitations due to computational constraints—specifically, our setup (Google Colaboratory with a single A100 40GB GPU) could not support the larger models and parallel runs required to complete the benchmark within the tight rebuttal period. We plan to include additional benchmarks in the camera-ready version.\n\nThat said, we would like to clarify that our current experimental evaluation is comparatively extensive for a theory-oriented paper. We summarize the scope of our setup below for context:\n\nTable 1: Comparison of experimental validataion setups with accepted papers and ours\n\n| Method | Number of benchmarks | Number of baselines | Number of LLM models | Largest model size | \n| --- | --- | --- | --- | --- | \n| R-DPO (ICML 2024; [20]) | 1 | 4 | 2 | 7B |\n| Dr. DPO (ICLR 2025; [105]) | 2 | 4 | 3 | 13B |\n| Hölder-DPO (ours) | 4 | 5 | 5 | 12B |\n\nWhile our setup may be smaller than that of purely experimental works, our primary contribution is theoretical—we provide the first formal guarantees for redescending and noise identifiability properties in DPO. Our empirical results serve to verify and complement these theoretical findings, and we believe they are appropriate in scope for a theory-driven submission.\n\nThank you again for the helpful suggestions. We have incorporated them into our revision and will continue to strengthen the experimental section in the final version. We hope our rebuttal clarifies our points.\n\nMany thanks,\n\nAll the authors"}, {"review_id": "uaKIk1xz7V", "reviewer": "Reviewer_FZwu", "summary": "This paper proposes an alternative to the traditional DPO objective (and some more recent robust variants). In particular, the new Holder-divergence inspired objective possesses the redescending property, meaning that the influence of an extreme outlier is not merely bounded, but tends towards zero. They show that this objective not only improves performance on contaminated preference datasets (where the label has likely been flipped a non-trivial fraction of the time), but also enables users to identify the likely contamination percentage as well as the most plausible outliers. They present some theory demonstrating that their approach is the only one amongst popular DPO objectives to have this redescending property and present a fairly extensive empirical evaluation of their methodology on multiple data sets and at varying scales.", "questions": "* Could you elaborate more on the sensitivity of your method to the choice of the hyperparameter? I would suggest creating a version of Figure 2 that illustrates how gamma affects the influence function, but also I would suggest you include a version of Table 1 for each of the applications you consider. I don't agree with your characterization of Table 1 either: on line 259, you say that increasing gamma *improves* contamination ratio estimation but reduces precision. It should be *increases* the estimate (and of course, unsurprisingly, then reduces precision). I would also suspect that this choice matters much more for larger models where the implicit reward model becomes much sharper? This may be a concern when scaling this method beyond the still small-ish models considered here.\n\n* In keeping with the above question, I think the paper would benefit from a discussion of why the redescending property is not always prioritized in statistics. As Figure 5c demonstrates, once the data is cleaned, the performance of typical DPO losses is clearly superior. The trade-off between robustness and efficiency (made explicit with this hyperparameter gamma) deserves exposition, even if this method is a promising approach.\n\n* I would strongly suggest you preview the definition of \"redescending\" much earlier in the paper - it is the key idea of the manuscript and I think the reader would benefit a heuristic description in line 36/37 rather than simply a reference to the influence function (which they may or may not know about).", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Quality**\nI think this submission is certainly technically sound and the claims it makes about its own proposals and others are precise and appear to be correct. My background is more in statistics than post-training LLMs, so I cannot meaningfully weigh in on the comprehensiveness of the experiments, but I found them compelling and more than sufficient for publication in this venue. There could perhaps be more acknowledgment of the limitations of objectives with this property. In classical statistics, the problem is usually in the choice of the parameter the authors call gamma. There is a limited but insufficient discussion of the sensitivity of this method to that choice.\n\n**Clarity**\nI think this paper is generally quite straightforward to read and is well-organized. The figures are excellent in particular. It would be nice to see confidence intervals for win rates. I know the sample sizes are written down, but you could add +/- 0.05, etc. in parentheses next to the values.\n\n**Significance**\nI think these results are meaningful and impactful. I was very impressed by the empirical improvements and I think introducing this idea from robust statistics into ML/AI world is quite valuable.\n\n**Originality**\nThis is clear - I think it's a nice idea and it's very well executed. Well done!", "was_revised": true, "final_justification": "I think the concern about choosing $\\gamma$ is important, but is certainly sufficient novelty in my mind. Re: another review's comments on this topic, very few of the thousands of papers published at NeurIPS each year introduce a \"fundamentally new paradigm.\" This paper will probably not win the test of time award, but I think the field will benefit from exposure to this approach (which is noticeably different from the plethora of very similar \"robust DPO\" papers cited in this manuscript). The clarity and presentation of the paper could use some improvement, but I don't think it's disqualifying. I remain confident in the score I chose.", "rebuttal": "Thank you very much for your insightful and highly positive review and the helpful comments. In our revision, we have improved the clarity and readability by incorporating your suggestions. The following are our detailed responses.\n\n# Q1: It would be nice to see confidence intervals for win rates.\n\n## R1: Confidence intervals have been added.\nThe error bars in Figures 3 and 4 already represent the standard deviation across 10 independent runs, providing a clear picture of variability and stability. That said, we agree that confidence intervals can offer a more rigorous means of distinguishing methods. We have now added 95% confidence intervals to Figures 3, 4, and 5, as well as to Tables 1 and 2 (see Appendix H).\n\n# Q2: Could you elaborate more on the sensitivity of your method to the choice of the hyperparameter $\\gamma$? The discussion is currently limited.\n\nWe address this concern in three parts:\n\n## Q2.1: Elaboration on the $\\gamma$ sensitivity\n\n### R2.1: **We have expanded the explanation as follows.**\n\n$\\gamma$ serves as **a sensitivity knob** for outlier detection:\n\n- **A small $\\gamma$** makes the model more tolerant: data points are downweighted only when the model is highly confident they are outliers (i.e., $s$ is strongly negative). The redescending slope is gentle.\n- **A larger $\\gamma$** increases sensitivity: even moderately negative $s$ values lead to aggressive downweighting. The redescending slope is steep.\n\nThis gives practitioners direct control over the trade-off between robustness to gross errors and sensitivity to data variance.\n\n## Q2.2: Expanded Ablation Studies & Scaling Concerns\n\n### A2.2: **Further analysis will be included in the camera-ready version.**\n\nTable 1 currently illustrates the core trade-off between robustness and mislabel precision. We are working to expand this ablation in the appendix, but a full sweep across all models and datasets is infeasible within the rebuttal period due to computational limitations (Google Colab, single A100 GPU), where each one data point in one figure takes a day to compute (see Appendix H.2). Since the optimal $\\gamma$ is likely task-specific, we believe our existing results already provide useful guidance for practitioners.\n\n## Q2.3: Inaccurate description on L259\n\n### R2.3: **You are correct.**\n\nThank you for catching our imprecise language. We have revised the text to correctly state that increasing $\\gamma$ increases the estimated contamination ratio, rather than \"improving\" it.\n\n# Q3: The paper would benefit from a discussion of why the redescending property is not always prioritized in statistics (the trade-off between robustness and efficiency).\n\n## R3: Vanilla DPO is indeed preferred over robust DPO if the dataset is perfectly clean, but this condition is rarely satisfied.\nWhile vanilla DPO may outperform robust methods on perfectly clean datasets, this assumption rarely holds in practice. Even curated datasets like Golden HH (Figure 5a) exhibit non-negligible noise. In these scenarios, vanilla DPO is fragile, degrading significantly with even modest contamination. In contrast, Hölder-DPO remains stable. \nInterestingly, even in the \"clean\" setting ($\\epsilon = 0$, no artificial noise), Hölder-DPO performs comparably to or better than vanilla DPO (Figures 3 and 4), indicating that the \"clean\" open dataset is actually slightly noisy (see estimated $\\hat{\\epsilon}$ are non zero at $\\epsilon=0$). \nMoreover, we observe in Figure 5c that vanilla DPO trained on the cleaned dataset (filtered by Hölder-DPO) outperforms robust methods trained on unfiltered data. This highlights an optimal two-step procedure: first use Hölder-DPO for noise identification and filtering, then apply vanilla DPO for final alignment—if resources and dataset size permit this expensive training.\n\nWe have integrated this discussion into Section 5 and Section 6 (Limitations).\n\n# Q4: I would strongly suggest you preview the definition of \"redescending\" much earlier in the paper.\n\n## R4: Agreed—we have amended the explanation to Line 36:\n\n> The redescending property—the idea that the influence of extreme outliers should diminish to zero.\n\nThank you once again for your valuable feedback. We believe these revisions significantly improve the paper and hope our responses address your concerns clearly.\n\nMany thanks,\n\nAll the authors"}, {"review_id": "SqenrsYaBA", "reviewer": "Reviewer_kngT", "summary": "The work addresses an important issue in LLM alignment where preference data is noisy. The authors proposed to modify the DPO loss so that it is robust to the presence of noisy labels under certain assumptions. Their experiment results show the effectiveness of the proposed method.", "questions": "Please refer to the Strengths and Weaknesses section.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "__Strengths__: Overall, the manuscript is well structured. The experiment results show clear improvement over baseline methods. The development of the proposed method is convincing intuitively. The experiment results show evident improvement.\n\n__Weaknesses__:\nHowever, I have several serious concerns about technical clarity.\n\n- The definition of IF function is syntactically problematic: $x, \\theta$ and $p_{\\mathcal{D}}$ appear as input but all three factors does not appear in the right hand side. What does the notation $\\| \\cdot \\|$ mean? As $\\theta(\\epsilon)$ is a vector-value function,  $\\| \\cdot \\|$ couldn't be the absolute function since IF is a scalar-valued function.\n- The proposed robustness framework is mostly motivated from the condition in Definition 2. However, I find it difficult to understand it at both intuition level and technical level, as the discussion was too brief:\n    - $r_{\\theta}$, and  $\\theta$ are never defined\n    - Can you clarify the comment in line 107, i.e., how does $r_{\\theta}(x, y_{\\text{lose}}^{\\text{flip}}) \\to  \\infty$ correspond to the worst-case scenario for DPO? \n    - What is the physical interpretation of the limit $r_{\\theta}(x, y_{\\text{lose}}^{\\text{flip}}) \\to  \\infty$? It certainly does NOT necessarily imply that $y_{\\text{lose}}^{\\text{flip}} \\succeq  y_{\\text{win}}^{\\text{flip}}$ since it is possible that $r_{\\theta}(x, y_{\\text{win}}^{\\text{flip}}) \\to \\infty$ as well.\n    - Most importantly, the condition in definition 2 does not involve any property of noise, how does that condition relate to noise robustness as stated in line 110, i.e., $\\theta^{\\star}(\\epsilon) = \\theta^{\\star}$? This relation if holds is not trivial and should be addressed more carefully.\n+ In Figure 2, what is IF weight? How is IF weight evaluated? How does Figure 2 relate to the condition in Definition 2, particularly the x-axis is not exactly the quantity under limitation?\n+ If a loss function is redescending, does the present of noise cause any effect at all, such as increasing computation cost? In particular, I'm curious about the tradeoff of having a redescending loss function as it seems to guarantee a \"too good\" result against noise.\n+ What is the effect of $\\gamma$?", "was_revised": true, "final_justification": "My main issue is the technical clarity. With the response, I do have a better understanding compared to my initial reading. However, there are still confusing parts. Even the authors does not have a good distinction on their notations, such as $r_{\\theta}$ vs $\\widehat{r}_{\\theta}$. The lack of clarity hinders any attempts in justify the soundness of their theoretical results.\n\nHowever, the empirical experiments show clear and consistent improvement over baselines. This might suggest certain validity of the proposal. Considering all factors, I would like to raise my score from 3 to 4.", "rebuttal": "Thank you for the valuable feedback. Below are our detailed responses.\n\n# Q1: The definition of the IF function is syntactically problematic and confusing.\n\n### R1: We appreciate this observation and respond in two parts:\n\n## Q1.1: The function $\\mathrm{IF}(x, \\theta, p_D)$'s inputs do not appear on the right-hand side of Eq. (4).\n\n### R1.1: **This is a shorthand; the full form is provided in Appendix E.1.**\n\nWe apologize for the confusing notation. The expression in Eq. (4) follows a common shorthand in robust statistics literature, omitting explicit dependence for readability. However, we agree this may not be accessible to all readers and should have been explained more clearly.\n\nThe complete derivation is provided in Appendix E.1, where the influence function is given as:\n\n$$\n\\text{IF}(x, \\theta, p_D) = -H_1^{-1} F_1,\n$$\n\nwhere \n\n$H_1 = \\mathbb{E}_{p_D}[ H_2 ],$\n\n$H_2 = H_{\\theta}(s),$\n\n$F_1 = \\mathbb{E}_{p ( \\tilde{s} ) }[ F_2 ],$\n\n$F_2 = F_{\\theta}( \\tilde{s} ),$\n\n$\\tilde{s} := s_{\\text{flip}}$\n\nHere, the Hessian term $H_1$ clearly depends on the clean distribution $p_D$, while the outlier term $F_1$ depends on the specific outlier point (derived from $x$). Both terms are evaluated at $\\theta = \\theta^*$, which itself is a function of $p_D$.\n\nWe have incorporated the above explanation in Section 2.\n\n## Q1.2: What does the notation $|\\cdot|$ mean? As $\\theta(\\epsilon)$ is a vector, it can't be the absolute value.\n\n### R1.2: **Apologies. This was a typographical error.**\n\nYou are absolutely right. The IF expression $\\left.\\frac{\\partial \\theta^*(\\epsilon)}{\\partial \\epsilon}\\right|_{\\epsilon = 0}$ is a vector, and the $|\\cdot|$ symbol is inappropriate. We have corrected this in our revised draft.\n\n# Q2: The robustness framework and the condition in Definition 2 are difficult to understand.\n\n### R2: **We have now added the following clarification.**\n\nWe acknowledge that our original explanation of Definition 2 was brief, primarily due to space constraints. In the camera-ready version—where we will have an additional page—we have expanded this section to provide a clearer and more detailed discussion, as outlined below:\n\n## Q2.1: The terms $r_{\\theta}$ and $\\theta$ are never defined.\n\n### R2.1: **Both are defined in L65 and L72.**\n\n$\\theta$ denotes the parameters of the policy (LLM) model $\\pi_{\\theta}$, and $r_{\\theta}$ represents the implicit reward function. We acknowledge a source of confusion stemming from our inconsistent use of $\\hat{r}{\\theta}$ and $r{\\theta}$—these refer to the same quantity. We have now unified the notation throughout the paper to consistently use $r_{\\theta}$.\n\n## Q2.2: How does $r_{\\theta}(x, y^{\\text{flip}}_{\\text{lose}}) \\to \\infty$ correspond to the worst-case scenario?\n\n### R2.2: **It represents maximal misalignment to a flipped label.**\n\nIn model alignment, the worst case occurs when the LLM is misled to strongly update toward a flipped label, effectively learning the wrong preference. The importance of a data point is implicitly governed by the reward function $r_\\theta$, which we cannot observe directly. Since humans are generally more reliable at relative comparisons than assigning absolute scores (e.g., ranking a prompt among 10,000 candidates), we estimate $r_\\theta$ via pairwise preferences, expressed as:\n\n$\ng\\_{\\theta}(s) = r\\_{\\theta}(x,y\\_{\\text{win}}) - r\\_{\\theta}(x,y_{\\text{lose}}).\n$\n\nThe model is aligned by updating its parameters $\\theta$ based on the loss gradient $\\nabla_\\theta \\mathcal{L}$ (Eq. 3), which is scaled by $g_\\theta(s)$. As $g_\\theta(s) \\to \\infty$, the update becomes overly dominated by this data point.\n\nNow, consider the worst-case: if the flipped label $y_{\\text{lose}}^{\\text{flip}}$ is assigned an infinitely high reward, i.e., $r_\\theta(x, y_{\\text{lose}}^{\\text{flip}}) \\to \\infty$, then $g_\\theta(s) \\to -\\infty$, and the model is misaligned with maximum confidence. This scenario captures the core robustness failure.\n\nWhile we focus on $r_\\theta(x, y_{\\text{lose}}^{\\text{flip}})$ in the limit, the interpretation is equivalent to the worst-case $g_\\theta(s)$ due to symmetry. This is why Definition 2 formalizes robustness as the requirement that the model nullifies updates under such extreme (and erroneous) conditions.\n\n## Q2.3: How does the condition in Definition 2 relate to noise robustness? This relation is not trivial.\n\n### R2.3: **Satisfying Definition 2 ensures that the LLM’s parameter update is nullified in response to a flipped label—providing robustness to noise.**\n\nThis is formalized in Appendix C, Eq. (14), which shows that the influence of a flipped point on the optimal parameters $\\theta^*(\\epsilon)$ vanishes to first order. As noted in Footnote 1 (Page 3), this relies on a first-order Taylor expansion, and thus does not eliminate higher-order effects.\n\nNevertheless, by nullifying the first-order influence, we ensure that $\\theta^(\\epsilon) \\approx \\theta^*$ (see Line 110), capturing a meaningful notion of robustness to label noise.\n\n# Q3: In Figure 2, what is \"IF weight\"? How does the figure relate to Definition 2, particularly the x-axis?\n\n## R3: **Explanation added**\n\nWe agree that this part was particularly unfriendly. We have added the explanation as follows:\n\n## Q.3.1: What is \"IF Weight\" and why is it critical for robustness?\n\n### R3.1: **It is the scalar multiplier that scales the gradient direction.**\n\nEq.~(3) can be decomposed as:\n\n$$\n\\nabla_\\theta \\mathcal{L}(s, \\pi_\\theta) := \\sigma(- g_{\\theta}(s)) g_\\text{grad}(s),\n$$\n\nwhere the multiplicative factor $\\sigma(- g_{\\theta}(s))$ represents the **IF weight**, and $g_\\text{grad}(s)$ represents the direction of the gradient.\nHere, IF weight determines how strongly the model updates its parameters for each data point. If IF weight tends to zero in the worst-case (i.e., $g_\\theta(s_{\\text{flip}}) \\to -\\infty$), the method is robust because it ignores high-confidence misaligned (flipped) inputs.\n\nWe summarize the worst-case IF weight behavior for various methods in Table 1:\n\nTable 1: IF weights for all alignment loss.\n| Method | IF Weight | Worst-case $g_{\\theta}(s_{\\text{flip}}) \\to -\\infty$|\n|---|---|---|\n| DPO [83] | $\\sigma(-g_{\\theta}(s_{\\text{flip}}))$ | $\\to 1$ (non-zero)|\n| cDPO [74] | $(1-c)\\sigma(-g_{\\theta}(s_{\\text{flip}})) + c\\sigma(g_{\\theta}(s_{\\text{flip}}))$ | $\\to 1-c$ (non-zero)|\n| R-DPO [24] | $\\frac{1-c}{1-2c}\\sigma(-g_{\\theta}(s_{\\text{flip}})) + \\frac{c}{1-2c}\\sigma(g_{\\theta}(s_{\\text{flip}}))$ | $\\to \\frac{1-c}{1-2c}$ (non-zero)|\n| Dr. DPO [104] | $w_{\\theta^{*}}(s_{\\text{flip}})\\sigma(-g_{\\theta}(s_{\\text{flip}}))$ | $\\to \\text{const.}$ (non-zero)|\n| IPO [7] | $\\left(\\frac{g_{\\theta}(s_{\\text{flip}})}{\\beta} - \\frac{1}{2\\beta} \\right)$ | $\\to -\\infty$ (non-zero)|\n| Hölder-DPO (Ours) | $\\sigma(g_{\\theta}(s_{\\text{flip}}))^{\\gamma}\\sigma(-g_{\\theta}(s_{\\text{flip}}))$ | $\\to 0$ (zero)|\n\nFigure 2 plots this IF weight (y-axis) versus $g_\\theta(s_{\\text{flip}})$ (x-axis). The limit $g_\\theta(s_{\\text{flip}}) \\to -\\infty$ corresponds to Definition 2. Only Hölder-DPO satisfies Definition 2.\n\n## Q.3.2: How does Figure 2 relate to Definition 2 and the x-axis?\n\n### R.3.2: **The y-axis approaching zero as x-axis to minus infinity corresponds directly to Definition 2.**\n\nWhile the symbolic proof in Table 1 is sufficient to establish the redescending property formally, Figure 2 offers an intuitive, visual interpretation of how each method behaves under extreme cases.\n\nThe IF weight on the y-axis reflects how sensitively the model updates its parameters in response to each data point. A high IF weight indicates strong influence on the parameter update. For example, vanilla DPO assigns high weight when the base model makes a mistake (i.e., $g_{\\theta^*}(s) < 0$, meaning the base model mispredicts human preference as $y_{\\text{lose}}$ over $y_{\\text{win}}$), which is reasonable for standard alignment.\n\nIn contrast, Hölder-DPO emphasizes updates only when the model’s confidence is moderate, i.e., when $| g_{\\theta^*}(s) |$ is small. When the human feedback strongly contradicts the model's prior belief (large negative $g_{\\theta} (s) $), Hölder-DPO assumes that the feedback itself may be unreliable, and therefore suppresses the update. This reflects a robust alignment strategy that guards against extreme but potentially erroneous feedback.\n\n# Q4: On the tradeoffs of the redescending property.\n\n## R4: No additional computational cost; sensitivity to $\\gamma$ is the primary trade-off.\n\nHölder-DPO (Eq. 12) incurs no extra computational cost over vanilla DPO. The main trade-off lies in tuning the hyperparameter $\\gamma$, which controls the aggressiveness of downweighting potential outliers.\n\nThis is a standard feature of robust estimators, as corroborated by Reviewer FZwu. Similar to the role of hyperparameters in R-DPO or Dr. DPO, $\\gamma$ must be chosen appropriately. However, we show that a single default setting ($\\gamma = 2.0$) performs robustly across a wide range of settings, demonstrating practical usability.\n\n# Q5: What is the effect of $\\gamma$?\n\n## R5: $\\gamma$ serves as **a sensitivity knob** for outlier detection:\n\n- **A small $\\gamma$** makes the model more tolerant: data points are downweighted only when the model is highly confident they are outliers (i.e., $s$ is strongly negative). The redescending slope is gentle.\n- **A larger $\\gamma$** increases sensitivity: even moderately negative $s$ values lead to aggressive downweighting. The redescending slope is steep.\n\nThank you again for the helpful suggestions and being constructive. We have incorporated them into our revision and will continue to strengthen the experimental section in the final version. We hope our rebuttal clarifies our points.\n\nMany thanks,\n\nAll the authors"}, {"review_id": "1s2Ie1xN9j", "reviewer": "Reviewer_buZZ", "summary": "This paper focuses on the issue of noise in crowd-sourced human feedback, particularly in the context of aligning LLMs with human preferences. To address this challenge, the authors introduce Hölder-DPO, a novel variant of DPO that incorporates the Hölder divergence metric. This metric exhibits desirable robustness properties, enabling the model to better handle noisy feedback by assigning lower likelihoods to mislabeled data points compared to clean ones.", "questions": "**Role of Hölder-DPO**: From the authors' perspective, what role is Hölder-DPO best suited for—final training or preprocessing? The method's ability to detect and filter mislabeled data suggests it could serve as an effective preprocessing tool, potentially improving the performance of other alignment methods when applied to clean datasets.", "limitations": "1. **Complexity of Real-World Noise**: While the symmetric label flip is a standard way to test robustness, real-world noise is often more complex and may not be captured by such a simple model. Future work could explore more nuanced noise types, such as \"easy to agree, hard to differentiate\" noise, where annotators may agree on the general direction but struggle with subtle distinctions.\n2. **Scalability and Filtering Strategy**: The experiments are conducted on relatively large datasets, with a large space for filtering strategy that removes noisy examples. However, in domain-specific applications (e.g., medicine, law), where high-quality preference data is scarce, a more conservative filtering strategy may be needed. Uncertainty-aware filtering, where the model is less aggressive in filtering low-confidence examples, could improve the method's applicability in such settings.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. **Motivation & Theoretical Robustness:** The paper provides formal evidence that existing DPO variants, such as R-DPO and Dr. DPO, do not satisfy the redescending property, therefore introducing the Hölder divergency with redescending property proof, which ensures robustness under severe label noise.\n2. **Improved Alignment Performance**: Empirical results demonstrate that Hölder-DPO consistently outperforms baseline methods in average reward across varying levels of contamination, generation temperatures, and training steps. Additionally, it accurately detects mislabeled examples, significantly improving alignment performance when such examples are removed.\n\n### Weakness\n1. **Strong Assumptions on Noise**: The method's effectiveness is based on the assumption of strict tail data contamination, i.e., mislabeled data points can be clearly distinguished from clean ones by being assigned very low likelihood. In real-world scenarios, this may not always hold, especially when the differences between clean and noisy data are subtle or systematic.\n2. **Sensitivity to Hyperparameter $\\gamma$**: Hölder-DPO introduces an additional hyperparameter, $\\gamma$, which controls the level of contamination removal. While the paper reports robust performance with a default value of 2.0, the optimal choice may vary across tasks and datasets. This sensitivity may require tuning to fully realize the method's potential.\n3. **Limited Noise Type Analysis**: The paper's primary model for noise is the symmetric label flip, where a $(y_{\\text{win​}},y_{\\text{lose}}​)$ pair is randomly swapped with a certain probability $\\varepsilon$. While this is a standard and necessary approach for testing robustness, real-world noise is often more complex and systematic, stemming from annotator biases or task ambiguities.", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thoughtful and constructive review. We are encouraged by your positive assessment and appreciate the insightful questions regarding the practical assumptions and deployment of our method. Below are our detailed responses.\n\n# Q1: Weakness 1 & 3: Strong assumptions on noise (strict tail contamination) and limited noise type analysis (symmetric label flips).\n\n## A1:  Our assumption is weaker and more practical than classical approaches, and it handles asymmetric noise.\n \n### **1\\. On the Tail Contamination Assumption vs. Infinitesimal Contamination**\nYou are correct that our theory relies on this assumption (tail contamination). However, we sincerely argue it is a reasonable and practical abstraction of how a model learns from noisy data, under the setting where we have no access to clean validation dataset. A model trained on a majority of clean preferences should learn to assign very low likelihood to gross errors like flipped labels. Our empirical results across multiple datasets (Figs. 3, 4, 5) provide compelling evidence that this assumption holds sufficiently well in practice to deliver state-of-the-art robust performance, even if real-world data does not perfectly match this theoretical ideal. \n\nFurthermore, this assumption is more realistic for large-scale datasets than the other assumptions often used in classical robust statistic, such as infinitesimal contamination ($\\epsilon \\approx 0$). In the LLM alignment context, **where datasets can have substantial noise (e.g., >25% in Anthropic HH)**, an infinitesimal contamination model is simply not applicable. Our tail contamination assumption, which allows for heavy contamination, is a more practical abstraction. Furthermore, as we note in our paper (lines 95-96), this type of assumption—that outliers can be identified in the tail of a distribution—is a well-established principle implicitly or explicitly used in many robust learning and outlier detection algorithms.\n\n### **2\\. On the Symmetric Label Flip Model and the True Limitation**\nWe would like to clarify that our method's core theoretical assumption is **not strictly limited to symmetric flips, but rather the more general condition of tail contamination**. This means our method is effective against any type of noise—be it symmetric, asymmetric, or even some systematic annotator biases—as long as the noisy preference pair is correctly identified by the learned model as a low-likelihood event (an outlier).\n\nThe true limitation lies with noise that is too systematic to be identified as an outlier by the model. It is plausible to hypothesize that the method's performance would degrade when noise is structured in a specific way—for example, when all preference labels for a single, specific prompt or a narrow topic are consistently flipped.\n\nThe reasoning is that these structured errors would no longer form a clear \"tail\" in the overall data distribution. Instead, they would create a consistent but incorrect signal for that specific sub-task, which the model would likely struggle to distinguish from a genuine, albeit niche, preference. Exploring such complex, systematic noise structures is a vital direction for future research.\n\n\n# Q2: Weakness 2: Sensitivity to Hyperparameter $\\gamma$\n\n## A2: It is shared challenge in robust statistics community, including prior work, not specific to our work.\n\nThe choice of $\\gamma$ is indeed a key practical consideration, as is common for many robust statistical methods. We would like to emphasize that this kind of challenge is shared by the other robust methods like R-DPO and Dr. DPO. These methods also depend on a critical hyperparameter (the estimated noise ratio $c$), which is often unknown a priori and requires tuning.\n\nOur key finding is that a single default value ($\\gamma=2.0$) works consistently well across our diverse experiments. This suggests that while $\\gamma$ is an important dial for tuning the robustness-efficiency tradeoff, our method is not prohibitively sensitive and can be effectively used by practitioners without exhaustive hyperparameter sweeps.\n\n# Q3: Role of Hölder-DPO.\n\n## A3: Preprocessing if expensive two-step training is permissible given budge, otherwise final training.\n\nThis is an excellent and insightful question that touches upon the practical deployment of our method. We believe Hölder-DPO is a versatile tool, and its optimal role depends on the scale of the dataset and the specific application scenario.\n- **For Medium-Scale or High-Stakes Datasets:** In this scenario, a two-stage approach is likely optimal.\n\n    - Stage 1: Train a model with Hölder-DPO to leverage its robustness and data valuation capabilities.\n    - Stage 2: Use the trained model to identify and filter out the noisy data points.\n    - Stage 3: Train a final model (using standard DPO or another method) on the newly cleaned, high-quality dataset. This approach, validated by our Figure 5c, maximizes final performance by combining robustness for cleaning and efficiency for final alignment.\n\n- **For Very Large-Scale Datasets:** When dealing with massive datasets where a two-stage process might be computationally prohibitive, a one-stage approach is more practical.\n    - Train the final model directly with Hölder-DPO. This provides a robustly aligned model in a single, efficient training run.\n    - The data valuation scores generated during this process can then be used to create a cleaned dataset that is held in reserve for the next training cycle, continuously improving data quality over time.\n\nIn summary, Hölder-DPO can be used as both a direct training objective and a preprocessing tool, and practitioners can choose the best strategy based on their specific constraints and goals.\nWe will add a new appendix section to explicitly discuss these practical, data-dependent deployment scenarios, highlighting the dual-role capability of our work.\n\n\n# Q4: Limitation 1: Complexity of Real-World Noise.\n\n## A4. Fully agreed. We added the explanation in the Limitation section.\n\nTo summarize our position: we clarified that our method's core assumption of \"tail contamination\" is more general than just symmetric flips and can handle various noise types. The true limitation, as you astutely point out, lies with complex, systematic noise that does not form a clear \"tail\" in the data distribution—such as the excellent example you provided of \"easy to agree, hard to differentiate\" noise.\nInvestigating such complex noise types is indeed an important and significant direction for our future work.\n\nWe will ensure this nuanced perspective is reflected in the final paper's Limitations section.\n\n# Q5: Limitation 2: Scalability and Filtering Strategy in data-scarce domains.\n\n## A5: We understand the sentiment, but in data-scarce settings, flipped data points matter more—so our method performs better.\n\nWe argue that our method is particularly valuable in such scenarios, guided by the principle of \"less is more for alignment.\"\n\nAs our own Figure 5c demonstrates, removing noisy data points leads to better alignment quality, even if it reduces the dataset size. We believe that carefully cleaning a small dataset is often more effective than training on a larger, noisy one.\n\nCrucially, our method offers the flexibility needed for these high-stakes domains and is not just an aggressive, automated filter. Instead, it empowers domain experts by providing a ranked list of all data points from most to least likely to be noisy. An expert in a data-scarce setting does not need to blindly trust the model's estimated noise ratio $\\hat{\\epsilon}$. Instead, they can use our method's ranking to make a more informed, conservative decision:\n- **For efficient manual review:** The expert can start inspecting data from the top of the ranked list (the most suspicious items). This allows them to focus their valuable time on the most problematic $\\hat{\\epsilon}$ fraction of the data, rather than having to review the entire scarce dataset.\n- **For conservative filtering:** The expert can choose a much smaller, more conservative threshold for removal (e.g., only removing the top 1-5% of most suspicious items) to preserve as much data as possible while still eliminating the most harmful outliers.\n\nTherefore, rather than being a risky tool in data-scarce settings, Hölder-DPO serves as an intelligent assistant that makes the process of cleaning valuable datasets more efficient and controllable.\n\nWe will add this discussion to our Limitations section (Section 6), highlighting how the ranked output of our method enables flexible, human-in-the-loop filtering strategies suitable for data-scarce domains.\n\nThank you again for the helpful suggestions and being constructive. We have incorporated them into our revision and will continue to strengthen the experimental section in the final version. We hope our rebuttal clarifies our points.\n\nMany thanks,\n\nAll the authors"}]

Author rebuttal:

Meta-review: This paper tackles the challenge of noisy human feedback in aligning large language models by introducing Hölder-DPO, a novel alignment loss with a provable redescending property. Unlike existing DPO variants, Hölder-DPO ensures that the influence of extreme mislabeled points vanishes, enabling robust learning even under heavy label noise. The authors provide theoretical guarantees that no existing alignment methods satisfy this property, and show that Hölder-DPO enables principled mislabel detection and dataset valuation. Empirical results demonstrate both improved robustness to synthetic and naturally occurring noise (e.g., Anthropic HH, Reddit TL;DR) and the ability to identify and filter mislabeled data, which in turn boosts downstream alignment performance.

Reviewers agreed on the importance of the problem and recognized the technical novelty of formally adapting the redescending property to DPO. Strengths highlighted include the rigorous theoretical analysis, the novel connection between robust statistics and preference optimization, and empirical results showing consistent gains under contamination. Concerns focused mainly on (i) the reliance on i.i.d. label flip or tail-contamination assumptions, with limited discussion of more structured or adversarial noise (R-qBds, R-buZZ); (ii) sensitivity to the hyperparameter $\gamma$ that governs robustness, with requests for more practical guidance (R-FZwu, R-buZZ); and (iii) notational clarity in influence-function definitions (R-kngT). In rebuttal, the authors clarified theoretical assumptions, emphasized that Hölder-DPO handles heavy contamination more broadly than infinitesimal-noise models, highlighted direct robustness results on real datasets without synthetic corruption, and provided additional discussion on $\gamma$, including human-in-the-loop tuning strategies. 

Overall, I recommend acceptance. The paper introduces a technically principled method that strengthens robustness in alignment while offering new capabilities for data valuation and mislabel detection. While open questions remain regarding hyperparameter sensitivity and robustness under structured noise, there remain important directions for follow-up work. Please incorporate the final suggestions of the reviewers in the camera-ready version as promised.
Decision: Accept (poster)