A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO
Authors: Xingyu Zhou
Yulian Wu
Francesco Orabona
Venue: ICML 2025 spotlightposter (2025)
Keywords: Differential Privacy; LLM Alignment; Robustness
Primary area: social_aspects->privacy
TL;DR: A unified reduction framework for analyzing private and robust offline alignment

Abstract: In this paper, we theoretically investigate the effects of noisy labels in offline alignment, with a focus on the interplay between privacy and robustness against adversarial corruption. Specifically, under linear modeling assumptions, we present a unified analysis covering both reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) under different privacy-corruption scenarios, such as Local differential privacy-then-Corruption (LTC), where human preference labels are privatized before being corrupted by an adversary, and Corruption-then-Local differential privacy (CTL), where labels are corrupted before privacy protection. Our analysis leverages a reduction framework that reduces the offline alignment problem under linear modeling assumptions to parameter estimation in logistic regression. This framework allows us to establish an interesting separation result between LTC and CTL, demonstrating that LTC presents a greater challenge than CTL in offline alignment, even under linear models. As important by-products, our findings also advance the state-of-the-art theoretical results in offline alignment under privacy-only or corruption-only scenarios.

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO
Xingyu Zhou' Yulian 2u? Francesco Orabona
Abstract
Ouyang et al., 2022), which learns an intermediate reward model before optimizing the policy. The second is the direct approach, exemplified by Direct Preference Optimization (DPO) (Rafailov et al., 2023), which directly optimizes the policy via supervised learning on the preference dataset.
In this paper, we theoretically investigate the ef- fects of noisy labels in offiine alignment, with a focus on the interplay between privacy and robust- ness against adversarial corruption. Specifically, under linear modeling assumptions, we present a unified analysis covering both reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) under different privacy-corruption scenarios, such as Local differ- ential wwwaacy-hhen-orrune
It is clear that the performance of both RLHF and DPO is significantly influenced by the quality of the preference labels in the dataset. However, in practice, these labels often noisy due to various factors (Lambert et al., 2023), One potential noise source is corruption or misspecifica- tion during label generation or data collection, e.g., data poisoning attack (Casper et al., 2023). Additionally, pri- vacy concerns human preference (as illustrated in Feng et al. (2024)) may prompt individuals to provide noisy or privatized preferences rather than their true rankings.
(LTC), where hu- man preference labels are privatized before being corrupted by an adversary, and Corruption -trru- then-
Local differential privacy (CTL), where labels are corrupted before privacy protection. Our analysis leverages reduction framework that reduces the offline aligmment problem under linear modeling assumptions parameter estimation in logistic regression. This framework allows us to establish an interesting separation result between L.TC and CTL, demonstrating that LTC presents a greater challenge than CTL in offline alignment, even un- der linear models. As important by-products, our findings also advance the state-of-the-art theoreti- cal results in offline alignment under privacy-only corruption-on/y scenarios.
From theoretical perspective, understanding the impact of these noisy labels-resulting from both corruption and privacy-is essential for improving offline alignment. Re- cent studies have made some initial attempts to address this issue (Mandal et al., 2024; Chowdhury et al., 2024; 2023; Bukharin al., 2024), but they face two fundamen- tal limitations: (1) They often treat corruption and privacy separately and focus exclusively on either RLHF or DPO, while, in practice. noisy labels can stem from both factors simultancously; (2) The theoretical guarantees provided by these studies are often suboptimal, even when privacy and corruption are separately considered. Motivated by these limitations and practical scenarios, we are particularly inter- ested in the following question:
1. Introduction
The alignment training process in language models that utilizes a human-labeled preference dataset has been instru- mental in producing more helpful, harmless, and honest responses (Bai et al., 2022). Leveraging an offline prefer- ence dataset, two prominent paradigms have emerged. The first is the indirect approach, such as Reinforeement Learn- ing from Human Feedback (RLHF) (Ziegler et al., 2019;
Can we provide unified analysis of the interplay between privacy and robustness in both RL.HF and DPO?
We provide an affirmative answer to the above question by presenting the following contributions:
1. A Unified Theoretical Framework. We present a unified theoretical framework for analyzing the interplay between privacy and robustness in offline alignment, covering both RLHF and DPO. Specifically, for privacy protection, we con- sider Local Differential Privacy (LDP) (Kasiviswanathan et al., 2011; Duchi et al., 2013) for preference labels, while for robustness, we consider the strong adversary corrup- tion model (Diakonikolas & Kane, 2023), where an adap- tively chosen fraction of labels can be corrupted. Our frame-
'ayne State University, USA 'King Abdullah University of Science and Technology, Saudi Arabia. Correspondence to: Xingyu Zhou <xingyu.zhou wayne.edu>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHI to DPO
work can simultaneously handle three privacy-corruption scenarios for both RLHF and DPO: Corruption-then-LDD. (CTL), LDP-then-Corruption (LTC), and Corruption-LDP- Corruption (CLC), capturing different ways privacy and corruption may interact in practice.
proof. For robust DPO, Chowdhury et al. (2024) considers a strictly weaker corruption model and derives a suboptimal- ity bound of rate ((I/n2/4), In contrast, our general result, when reduced to the same corruption model, achieves bet- ter rate of (11 / /v). Bukharin et al. (2024) also considers a specific corruption model in the label generation process of RLHF but only provides the estimation error of the reward model, without performance guarantee for the final policy.
2. Reduccion to Logistic Regression. Our unified analyt- ical framework leverages a reduction that transforms the offline alignment problem, under certain linear modeling assumptions, into parameter estimation in logistic regres- sion. This reduction enables us to establish suboptimality bounds for both RLHF and DPO by focusing on parameter estimation in logistic regression under private and corrupted labels across different scenarios. Moreover, it highlights key differences between RLHH and DPO, providing insights into practical design considerations.
Provably Private Alignment. The most related work in this aspect is Chowdhury et al. (2023), which mainly focuses on the reward model estimation in RLHF under various privacy constraints (i.e., local and central label differential privacy). Our intermediate result on estimation error (Sec- tion 5) recovers the one in Chowdhury et al. (2023) when the corruption parameter is set to zero. Moreover, compared to the implicir suboptimality bound in Chowdhury et al. (2023), we provide the first explicit bound in terms of the relutive condition number (Agarwal et al., 2021), which parallels similar results in standard (robust) offline RL (Zhang et al., 2022), ie., reward-based rather than cefference-basede
3. Separation between CTL and LTC. A key takeaway from our study of the interplay between privacy and robust- ness to corruption is that LTC is a more challenging setting than CTL, illustrating that the order in which privacy and corruption interact with each other significantly impacts the performance of offline alignment.
3. Preliminaries
4. New State-of-the-ard Guarantees. Our results, when reduced privacy-only or corruption-only settings, set new state-of-the oft results on theoretical guarantees for RLHF and DPO. For instance, for DPO under "corrupted" labels, our result is the first one that achieves (LL/VT) rate (where n is the size of preference dataset), matching the standard rate without noise. Additionally, as by-product of our reduction approach, we provide the first results on parameter estimation error in logistic regression under both private and corrupted labels, which may be of independent interest.
Background on Offline Alignment. The goal of offline alignment is to further tune the Supervised Tuning (SFT) model to match human preferences using an of- fline preference dataset. The preference dataset 'D = (st, a af, y:) consists of n samples, each has one con- text/state s; (e.g.. prompt), two actions al of (e.g.. two answers from language models) and label/preference feed- back y: E 10,1] indicating which one is preferred by hu- mans. We assume 8; to be sampled independently from a distribution p. A widely used approach for modeling ys is Bradley-Terry model (Bradley & Terry, 1952):
Finally, we remark that, as in many previous related works, e.g.. Zhu et al. (2023); Chowdhury et al. (2023), we con- sider linear modeling assumptions for the sake of theoretical analysis. However, we believe that our results could serve as important benchmarks for more general function classes. In fact, we have also verified our separation result between CTL and LTC in the general case via experiments on GPT2- large, see Appendix D for a detailed discussion.
P (1 = [|s,, of a.) exp(r" (s.a)) (x.a)) (I)
explr"
for I E (0, 1}. where (:,)) ) is a ground truth reward model.
Based on this preference dataset, offline alignment aims to learn a good policy न. In particular, the performance of the learned policy मे is evaluated by the suboptimality gap between गे and comparator policy æi, defined as
2. Related Work
In the main body, we only focus on the most related work on robust and private offline alignment, while relegating additional discussion to Appendix A.
SubOpt(f, nt) = ((xt) (
(2)
where J(m) := E Egp (r*(s,, a)) and 1 is not neces- sarily the optimal policy.
Provably robust alignment under corruption. Mandal et al. (2024) considers offline RLHF with corrupted pref- erence datasets and establishes upper bounds on the sub- optimality gap under various coverage assumptions of the offline dataset. As will be discussed in Section 6.1, their
RLHF and DPO. As already mentioned, there are two major paradigms in alignment for finding R: indirect and di- rect approaches. The former, exemplified by RLHF (Ziegler et al., 2019), involves an intermediate reward model learning process from preference dataset D before the policy opti- mization. The latter, represented by DPO (Rafailov et al.,
results are either suboptimal or lack rigor due to gaps in their
2

---

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO
2023), employs a direct policy optimization, i.e., using a supervisedleaming loss function optimize the policy di- rectly over the preference dataset D.
Definition 3.4 (CTL and LTC). Given a raw preference
dataset D = (si, af; a )< we consider the following settings that differ in the order of privacy protection (see Definition 3.1) and corruption (see Definition 3.3). In all cases, the final input dataset for the learning algorithm will be denoted by Dm = (3,, ab a. =.)
Privacy Protection in Human Feedback. The prefer- ence signal yi in 'D could reveal sensitive personal infor- mation (Feng et al., 2024; Chowdhury et al., 2023), hence requiring a rigorous privacy protection. To this end, we con- sider the local label Differential Privacy (DP) (Chaudhuri & Hsu, 2011 I; Ghazi al., 2021), which means that the learner now only has access to a privatized label rather than the raw one. More specifically, we have the following definition. Definition 3.1 (Label DP in Local Model (Chowdhury et al., 2023)). Let E > 0 and 8 E [0, 1]. If each label is privatized by a local randomizer R. which satisfies for any y, y/ and any subset S in the range of R. that
(CTL): An adversary first corrupts the labels in D to ÿt. Then, each label y, is privatized by a local randomizer.
Corruption-then-Bee
L.P--the-CCorree Corruption (LTC): Each label yi in D is first privatized by a local randomizer, resulting in the private label Vi- Then, the preference dataset with private labels is further corrupted by an adversary.
Remark 3.5. As last setting, one may also consider the set- ting where corruption happens both before and after privacy protection, which turns out be a simple combination of the results for CTL and LTC, hence omitted in results.
((((y)) E S) S e P(R ES + 5.
then we say R is an (s, -label differentially private local randomizer, and this privatized dataset is called label-private preference dataset. The entire alignment process that oper- ates with the privatized dataset is said to satisfy local label DP. When 8 = O, we simply say is a c-local label DP.
4. Reduction to Parameter Estimation
In this section, we will show that the key to establishing the suboptimality guarantees in both RLHF and DPO a tight parameter estimation in logistic regression, under certain modeling assumptions. This allows to focus single- parameter estimation problem under different settings (i.e., CTL and LTC) for both RLHF and DPO. More importantly, this unified perspective also enables us to easily see the connection and difference between RLHF and DPO.
Remark 3.2 (Randomized Response). Given the binary data of the true label, we would like to maintain the binary data property after privatization. Thus, we will adopt the stan- dard randomized response mechanism (Warner, 1965) as our local randomizer, which essentially injects controllable noise in labels by random flipping. Here, by "controllable," " we mean the noise injection method, and noise level is under our control based on the privacy parameter e.
Logistic Regression. Recall that given a feature vector xi E Rdd under logistic regression, the label y; e {00, 1] is generated according to the following probability:
Corruption in Human Feedback. The human feedback y: can often be noisy and even be corrupted in the source or during the data collection process, which deviates from the assumed true generation process in (I), To this end, the final learned policy मे needs to be robust with respect to corruption in labels. We consider corruption model similar to strong corruption model from robust statistics literature (Diakonikolas & Kane, 2023), which roughly says that an adversary can adaptively corrupt the labels of fraction of samples, by inspecting the samples.
P(u, = 1|r;} = a ((Perue, 2;));
(3)
where (=)) Ite is the sigmoid function, Ourue Rd is the unknown true parameter and (s,-) denotes the inner product of two vectors.
4.1. RLHF with a Linear Reward Model
We show that when the reward model in (1) is a linear func- tion, the key to bounding the suboptimality gap in RLHF is the parameter estimation in a logistic regression problem. To start with, we formally state the linear reward model, following common definitions used in prior work (Zhu et al., 2023; Xiong al., 2024; Cen et al., 2024; Chowdhury et al., 2023; Mandal et al., 2024).
Definition 3.3 (Label Corruption Model). Let E [0. 1/2]. We consider an a-corruption model: an adversary can in- specr the samples in a preference dataset of size n and then assign any label value of 0 or at most an samples.
Interplay between Privacy and Robustness. One key theme of this paper is to study the interplay between pri- vacy and robustness in offline alignment. In particular, we are interested in the impact of the order between privacy protection and corruption in the labels on the suboptimality gap (cf. (2)), for both RLHF and DPO. To this end, we will mainly consider the following settings.
Assumption 4.1 (Linear Reward with Boundedness). We assume that the ground truth reward r* is linear, i.e., r* (s. a) = (ø(s, a), 0*), where o(s, a) : S x A + Rd is some known and fixed feature map and S, A are the state space and the action space, respectively. We also as- sume the following standard boundedness conditions. For
3

---

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHI to DPO
all E S and a E A. without loss of generality, we assume [[o(s, a)]l _ 1. Moreover, R (1,0) 0, |[0]| < BJ. where the condition (1, 0 is to ensure the identifiability of H*.
We can further simplify the result in (6) by introducing the following relative condition number, which can be viewed as the natural extension of standard one (Zhang et al., 2022; Agarwal et al., 2021) to the RLHF setting.
assume 0* OB {0 E
Definition 4.3 (Relative Condition Number). For T1, T2 and feature map Φ, we define (, a, a') = ((8.a) - (8..a)) and 3m as
Under the above assumption, we consider the standard of- fline RLHF algorithm, but with an additional parameter 1). In particular, we consider two alternative outputs: When = 0, the output policy is मे = argmax, (TT) where (TT) = (=-p.a-z(11.)) [6, Ø(s, a))l, that is essentially greedy algorithm with respect to an estimate A; When vj = 1, the output is रे argmax J(T, where the objective func- tion is defined via the principle of pessimism (Zhu al., 2023; Jin et al., 2021; Li et al., 2024)
(",,a,, (7)
For any comparator policy æ and any given reference policy Trof we define
Edif T -1, F ddif Ratt
w
w
k(TT, Tret) == sup
(8)
[(T min E, Es-par (1)) (s) (0, ((s (EO((,))
WERd
We can now simplify our previous suboptimality bound using the relative condition number above in the following corollary, with its proof given by Appendix E.2.
-epppamae 100, (ss, a)}},
by constructing a confidence set around an estimate 0:
Corollary 4.4. Let the same assumption in Proposition 4.2 hold and further assume X Q ( -I((n/8))- For any given comparator policy x! with <((T), (Toe)) oc, we can upper bound (6) as follows:
(0 θg 110 00000l S T(n, d,
For completeness and due to space limitations, the full algo- rithm is given in Algorithm in the Appendix B.
SubOpt(T, x1) 22v3 ((n, vd K(T,, Tyet).
Here, we use reference policy ree because the confidence set only measures the uncertainty of the difference in reward. That is, it does not measure the uncertainty for a single state- action pair.
4.2. DPO with a Log-Linear Policy Class
In this section, we will show that for log-linear policy class (defined below), the suboptimality in DPO is also related to the parameter estimation in logistic regression.
We have the following key theoretical result on Algorithm 2, with its proof in Appendix E.I.
Proposition 4.2. Under Assumption 4. the labels (yJeeem) in the preference dataset of RLHF follow the logistic regression model with Oerue 6* and I;
We begin with a brief recap of DPO, following the origi- nal paper (Rafailov al., 2023). The key idea is to repa- rameterize the reward model by the optimal policy of KL-regularized problem. In particular, for the following KL-regularized optimization objective (with 8 > 0)
((si, a)) - a.). Algorithm 2 with 17 = 0 achieves
SubOpt T $
0
Ourue
(4)
JM(T) = (erp.o (((s,a) 3 Taass) net (a)s) s)
where π² argmax, J(x. Further let Σ :==: 1 E, 2t2e and X > () and suppose with probability at least 1 the estimate 0 satisfies
the optimal solution has the closed-form expression
1
00
Ouruc
< TTn,, <((,,d,,,)
(5)
nT" (a(ss) (x) Tate (a|s)
(9)
++1I
Then, setting 11 = in Algorithm 2, we have for any nt and P. with probability at least 1-5, 5,
where 75(8)= = E'EEA Tan (als) exp(r* (s, a)/8) is the nor- malization factor. This allows us to rewrite the reward r* in terms of x* as follows
SubOpt m+) < (m, d, 8,
7 (als) Tax (als)
x Eemplo(s (8)) - (68 Te (s)] (6)
r' a) = Bl
+ B1 (s))).
(10)
for any reference policy Trel, where we define o(s, a(s) = Bewx(1)) a)).
With the above re- parametrization of the reward using policy in (10) and BT preference model (1), DPO (Rafailoy et al.,

---

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO
2023) directly minimizes the following log-loss function:
Remark 4.8 (single-policy vs. all-policy concentrability) One nice thing about the above reduction is that it allows us to easily see the key difference between RLHF and DPO. In particular, from Corollary 4.4 and Proposition 4.6, we can see that the key (and only) difference lies in the choice of relative condition number (especially when considering the typical scaling of B O(vd) for the parameter), which also closely related the "concentratability coefficient" in offline RL (Munos, 2007: Jin al., 2021). In particular, due to the use of pessimism in offline RLHF, one can achieve bound in terms of <(*², Tret), which is related to the "single-policy concentratability" (Rashidinejad et al., 2021; Jin et al., 2021) for any comparator policy n². On the other hand, due to the lack of uncertainty characterization in DPO, one needs "all-policy concentratability" (Chen & Jiang, 2019) KT in the upper bound, which often much larger. In fact, this kind of dependence in standard DPO is shown to be necessary (Song et al., 2024).
[(T ==
1 (1 = 0) In II
x(a]los) Tatt (o91os)
- -In (ac) sote (a/ll))))
(y: = 1) In (gln
(.,) (o:la))
-IIn
s(e")ss) Nati (1]]3.)
(11)
In this paper, we consider the log-linear policy class for the sake of theoretical analysis.
Assumption 4.5 (Log-linear Policy Class), We assume that the optimal policy in (9) satisfies x" E II and Tift E II where
exp((0, ((8,))) a))) To (T((a)s) EwEd exp( o(s, ('))a
II
(12)
is the log-linear class for some known feature map 6(s, a)
5. Parameter Estimation Under Private and Corrupted Labels
S x A + IR with [[o[s, a)ll < 1. Moreover, 0 correspond- ing to n² satisfies that 8* E θ = {0 E IR (1, = 0, ||0|| < B). where the condition (1, 0) = 0 is to ensure the identifiability of 6*
As motivated by the last section, we now turn to designing algorithms for providing label privacy while accurately esti- mating the unknown parameter fteue in logistic regression, even under corrupted labels. As we will see, the key to the design is a new loss function, which allows us to adaptively handle the privacy-robustnes, interplays in a unified way. To facilitate the upcoming discussion, we formally state the general problem setup for logistic regression under private and corrupted labels.
The above policy realizabillity assumption is equivalent to the reward model realizability. In particular, by plugging log-linear policy into (11), we can establish that the labels yi again follow from the logistic regression in (3) with proper choices of ferue and r;. In particular, we have the following formal statement, with its proof in Appendix E.3.
Proposition 4.6. Under Assumption 4.5, the labels {y. Jie(m) in the preference dataset of DPO follow the logis- tic regression model with true B( O.n) with B > O and x; 0(5;, a!) o(st,a)) Suppose with probability at least 5, there exists an estimate 0 that satisfies
Definition 5.1 (Private and robust parameter estimation problem). Let D be a dataset of i.i.d samples {z,, yi 3 where 24 ~ I and y; follows from the logistic regres- sion model in (3). The input dataset D.,. {a,, 2 ) is the private and corrupted version of D. following Def- inition 3.4. The goal here is to design a local random- izer R. for privatizing labels (cf. Definition 3.1) as well as an analyzer A that receives Din outputs an estimate Ø that is close to the underlying true parameter Otree. mea- sured by a proper choice of norm. We assume the following boundedness conditions: for any E [n]. ||aa|| S 1 and Otrue E θg. = (0 E Rd (1. = 0, [|0|| B').
1|0
D+AI <I(n,d,6,
(13)
where E = E, Iaoa and X > O. Then, let 0 = 0/3 B +
Onn and X > Q ( In(n/o)), the corresponding policy मे Th. with probability at least 1 - satisfies
v3
Su < V2 Ven B (0, d,
Remark 5.2. The boundedness assumption essentially fol- lows from the reduction in the last section. Here, we assume |x:|| || < 1 rather than upper bounded by 2 for simplicity and B' can be properly chosen for RLHF and DPO, respectively.
where en == max ((α,T) T) the maximum relative condition number across the entire policy class.
Remark 4.7. One can also rewrite the above bound using the maximum value of the implicit reward function, Tmax
5.1. Our Algorithm
As mentioned, our choice of local randomizer R. for privacy protection is the simple Random Response (RR) mechanism with parameter & > 0 (Warner, 1965). That is, the binary output from RR equals the input with probability ae)) =
SubOpt )) S c /Kn
Tmax B
I(n, d, 6, A),
for some constant e > and log-linear policy IT.
5

---

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHI to DPO
Algorithm 1 Private and Robust Estimation
5.2. Estimation Error Bounds
1: Procedure: e-local label DP mechanism R
In this section, we will establish the estimation error bounds achieved by Algorithm 1. Throughout this section, we will let ACTL, htTC be the estimates outputted by Algorithm I under CTL and LTC respectively. Our first result is the following theorem, which characterizes the estimator error in terms of weighted norm, with proof in Appendix E.4.
2: //Input:
U,E E 1),
parameter: E
3: Random response: U,-
U;
w.p. af+1
11-1. U4
w.p. et
4: Return Ü,
5: Procedure: Analyzer A
6: //Input: { 24 4) 1
parameter: E
Theorem 5.4. Consider the problem in Definition 5.1. For any > O, a, E [0, 1/2). 5 E (0, 1), and À > 0, with probability at least 1 1- 6, the output of Algorithm I achieves
7: Let 2-(3)-1
8: Compute 0 argmingee,
- Ei (0) where
||θCTl - -Brrue -fleuellsta |1+ S &eTL (errinn, d,
(0) = In - o00 mx)) + (z: + (6) - 1)c(e)0² a.
c Va +
c(E) d aa++mm In(1/8) + BVVA
9: Return 0
?
n
||0LIc- Oeruellss fltulllstt &Furo(n,d, d d,
T4 otherwise, the privatized binary output differs from the input. RR satisfies the e-local label DP guarantee (cf. Definition 3.1) (Dwork & Roth, 2014).
<
= deewa + c(E) d (n(1//) + B' B'V
n
We now turn to the design of the analyzer A, which is re- sponsible for outputting an estimate 0. We first point out that in the non-private non-corrupted case, the standard max- imum likelihood estimator (MLE) that minimizes the loss function (() = Eh= (,(0) enjoys a good concentra- tion (Zhu et al., 2023) with respect to Otree, where (0))) is the standard log-loss:
where Σ = 1 T d(e) = - 7 1/(2 + + exp(B')), and C is a universal constant.
Remark 5.5. First, when there is no corruption, our result matches the one in previous work on private parameter esti- mation (Chowdhury al., 2023). Second, when corruption exists, the order of corruption and local privacy mafters. In particular, LTC has an additional cost c(s) in the first cor- ruption term compared to CTL. highlighting the interplay between privacy and robustness.
(0) yI (00 x;)) + (1 v:) log(1 (0T 2;)) = logg(1 - [(0 + y;0" Ni.-
Our second result is concentration result under Lg-norm with the additional condition of uniform coverage, which has been leveraged in prior work as well (Mandal et al., 2024; Zhang et al., 2022; Chowdhury et al., 2023).
However, due to the private labels, our analyzer is designed to minimize new loss (())- = - 1/ Σm - (0) where
(0) = ln(1n1111 o00 x4)) + (zz + - 1))cs)0 24, (14) 5
Assumption 5.6 (Uniform Coverage). There exists a pos- itive constant & > O such that the minimum eigenvalue Amin (2:) 2 E. where Σ == Ex=g J.
and a(E) := 20 (0)- -- The key difference lies in the "shifting and scaling" of the received labels z, which, in fact, enjoys exactly the same "shifting and scaling" intuition as in mean estimation under RR, i.e., it is an unbiased estimate. Putting the above choices of R and A together, yields the final Algorithm 1 above.
Under the above assumption, we can have another estima- tion error bound for the underlying parameter, which is now in terms of 12-norm, with proof in Appendix E.5.
Theorem 5.7. Under Assumption 5.6, for any E > 0, a e [0, 1/2), 8 E (0, 1), and n 2 8ln(d/8), with probability at least - 5, Algorithm I under CTL and LTC achieves
Remark 5.3. We remark that similar loss (up to some scaling) has been considered in Chowdhury et al. (2023; 2024). However, they are motivated from a different per- spective (e.g.. logit) rather than our connection to standard mean estimation under RR for local privacy (i.e., shifting and scaling). The form we use here in (14) has not appeared before. This new form not only makes it easy to see that our new loss is an unbiased estimate of the standard log loss, but also allows us to easily show that our single algorithm is adaptive to different privacy-corruptoe settings, i.c., it does not know the specific setting in advance.
(
c(E)
In
1OCTLL - Oeruell SC
V
7
7
n
|n 1
c(E)a
c(E)
1ILLC-- Oerwell SC
V
7E
7
n
Here, we see that the separation between CTL. and LTC still
6

---

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO
exists, with an additional factor of c(s) in LTC, illustrating negative impact of LDP on robustness.
particular, it takes a linear MDP view (rather than our lin- ear bandit view) of RLHF under strong corruption of both features and labels. Under the same relative condition num- ber assumption, their dependence on a is O(a!/4) when reduced from MDP to bandit. In contrast, our result gives a better dependence ((va), although only with label cor- ruption. It is worth noting that this ((va) dependence is state-of-the-an even in the easier setting of standard offline reinforcement learning (Zhang et al., 2022). Moreover, Algorithm I is much simpler than the one in (Mandal et al., 2024). Thus, a fair conclusion here could be that our result offers a better algorithm and theoretical result in the easier label-only corruption setting.
6. Putting It All Together: Suboptimality under RLHF and DPO
In this section, we are ready to present our main results the suboptimality gap under RLHF and DPO by combining our reduction results with estimation error bounds.
6.1. Private and Robust RLHF
Theorem 6.1. Under the conditions of Corollary 4.4 and Theorem 5.4, RLHF (Algorithm 2) achieves the following suboptimality with probability at least 1 - 6
To our best knowledge, we are unaware of prior work that explicitly states the private suboptimality of RLHF in terms of relative condition number, often used in the standard offline RL. The most related one is Chowd- hury et al. (2023), which generalizes the non-private RLHF in Zhu et al. (2023) to the same locally private one as ours. However, both Chowdhury et al. (2023) and Zhu et al. (2023) state their suboptimality as
2. Private RLHF:
SubOpter((R.tl S Ovd addn(tt,,aa,
x Va + de) /d+ d
7 V
+ +BV
n
SubOpturc(f nt) s C
d. k([T,, Tret)
SubOpt(f,, x²) S [ T (s)) - (St2)) 1
x c(s\va de) d d+Ing In 1
BVA
x 2F d, 6, A):
(15)
n
for any chosen reference vector E Rd and some function F. This is similar to our intermediate result in (6) but has some key differences. One potential issue in (15) is that it does not offer clear guidance on choosing the important vector v. In particular, if e = O, then the suboptimality may not converge to zero as n - 0o. This is because in both papers, À has to be on the order of 1/n 1f so as to ensure that F(n, d, 8, A) S O(1/Vn). However, in this case, if the minimum eigenvalue of the empirical matrix Σ is small, the norm term ||Ex~plő(s, x (s)) - 2ill( can be on the order of vn. given the choice of X. To partially address this, Zhu et al. (2023) suggest heuristic way of selecting v as the most common feature vector that appears in the data set. In contrast, we consider a reference policy Te and offer a condition number along with Corollary 4.4.
for any comparator policy nt and X 2 22 ( In(n/6)).
The proof follows directly from the reduction result in Corol- lary 4.4 and estimation error bound in Theorem 5.4. To the best of our knowledge, this is the first result on the subop- timality performance of RLHF under both privacy and cor- ruption. In partieular, let À~ ((d/ (B)) 2 ((d/n), the sample complexity part in the bounds (i.e., the last two terms approaches zero with a rate of ((vd/n), but with a multiplicative factor of c(E) that captures the cost of pri- vacy. Meanwhile, due to strong corruption, a non-vanishing bias term exists in all three cases in terms of corruption pa- rameters, which illustrates an interesting interplay between privacy and robustness, discussed below.
rule for selecting it via relative
Separation between CTL and LTC. One key observation is that LDP before corruption leads to an additional c(E) factor in the bias term, which mimics the same phenomena in private and robust mean estimation problems (Zhou & Zhang, 2024; Cheu et al., 2021).
Our next result is the suboptimality in RLHF under the assumption of uniform coverage (cf. Assumption 5.6).
Theorem 6.2. Under the conditions of Proposition 4.2 and for > 8ln(d/8) RLHF (Algorithm 2) achieves the follow- ing suboptimality with probability at least 1 - 8
Comparisons with Prior Work. We now highlight our contributions even in robust-only or pnivate-only RLHF, by comparing our result above with existing ones where privacy and robustness are separately considered.
C(E)
In
1
SubOpterr mT) S
7
7
n
1. Robust RLHF: To our best knowledge, only recent work (Mandal et al., 2024) establishes theoretical subopti- mality bounds for RLHF under adversarial corruption. In
c(E)a
c(E)
In 1
SubOptiro(f, (f, S 0
7
7
n
7

---

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO
The proof follows directly from Proposition 4.2 and Theo- rem 5.7. Compared with Theorem 6.1,, the corruption term becomes (with a factor of 1/E) rather than Va while the concentration part has no explicit dependence on d but with 1/5 factor, which however implicitly depends on d. As before, a separation exists between CTL and LTC, due to the additional c(e) factor in LTC. is worth noting that the O(a/E /{) dependence matches the best existing result in standard offline RL under corruption in Zhang al. (2022).
cormuption. As before, we can see that the interplay of local privacy and adversarial corruption introduces separation between CTL and LTC by a factor of c(s). Moreover, our re- sult also significantly advances the state-of-the-ar for DPO- style algorithms under privacy or corruption separately, as discussed in detail below.
Private DPO. Consider a 0, A ((dd/(BBB2ynn) Z
R(d/n), we obtain the first suboptimality for private DPO with rate 0(1/7 c(e) VKT), where c(e) is the ad- ditional cost due to local privacy. The rate matches the best possible non-private one as E + oo (Song et al., 2024).
Comparisons with Prior Work. Mandal et al. (2024) also consider the uniform coverage case and establish a bias corruption term on the order of vdda o(1) when reduced &
Robust DPO. To the best of our knowledge, only the recent work by Chowdhury al. (2024) provides a formal theoreti- cal bound on the suboptimality of rDPO under label corrup- tion. In particular, it considers the uundom-flipping corrup- tion model (i.e., with some known probability, the true label is flipped). This is a much weaker model than ours and, in fact, is equivalent to local privacy after re- re-prammterization. Under this weaker model, Chowdhury et al. (2024) only established a suboptimal rate of Ó(1/n (14/)//4) in the general case, while our result implics a rate of ((1/n (by using our private DPO result above) under the same corruption model. Moreover, moving from this weaker corruption model to corruption model in the robust statisties literature (i.e., strong corruption model), our result above shows that rDPO now suffers a non-vanishing bias term.
from their MDP to bandit setting. In contrast, in our label- corruption setting, we have no explicit dependence on d and a better dependence on a. Moreover, we highlight that the missing dependence of 1/7 in Mandal et al. (2024) is actually due to an error in their proof (see Appendix G for a detailed discussion). That is, the correct bound of their algorithm also has 1//r factor. In the context of private RLHF under uniform coverage, our bound matches the state- of-the-art in Chowdhury et al. (2023) when the corruption parameter is zero
6.2. Private and Robust DPO
Thanks to our reduction result, we can also leverage the es- timation error bound to give the first result on suboptimality in DPO-style algorithms under privacy and corruption.
Practical Implementation and Experiments. Given that Theorem 6.3 establishes the SOTA theoretical results of rDPO in both private and corruption cases, under the log- linear policy. One may also interested in its empirical per- formance in general with neural nets as the policy class. We have series of experiments (see Appendix D for details), which demonstrate some interesting results.
Theorem 6.3. Under the conditions of Proposition 4.6, the policy corresponding to the output of Algorithm 1 achieves the following suboptimality with probability at least 1 5
SubOpIeTL(A,TT SC. -BVEI
Va + d In
c(e)
+ BBVA)
x
? n
7. Discussion and Conclusion
SubOptircc C,, )+) s C. BVBEI
While we present only upper bound results in the main body, we briefly discuss their tightness here; for further details, please refer to Appendix C. First, when a = 0, the ad- ditional factor c(E) due to privacy matches the minimax lower bound established in Chowdhury et al. (2023). Fur- thermore, the dependence on 1/7 = O(e") ") ) = O(e"mux) appears in nearly all existing results on both offline and online RLHF (Zhu et al., 2023; Zhan et al., 2023; Xie et al., 2024; Pacchiano et al., 2021; Chen et al., 2022), stemming from the non-linearity of the Terry model. Second, in the limit + oc (non-private case), our dependence on a is O(va) and O(a/ç) (under uniform coverage), both of which align with state-of-the-art results in standard offline RL settings, where rewards rather than preferences are ob- served. In fact, we conjecture that the O(a/C) dependence is optimal. Third, regarding the separation between CTL
dEevaa dE),
dd + BBVA
x
V
7
for 3 > 0, X in(n/6)), 7= 1/(2 + exp(-3BB exp(3B)), and some universal constant C > O.
Remark 6.4. The policy in the above theorem in fact cor- responds to the output of the algorithm rDPO proposed in Chowdhury et al. (2024) with a log-linear policy class, see Appendix C for more details. That is, while Chowdhury et al. (2024) only shows suboptimal rate for rDPO, we are the first to attain (lI/Vn) rate, see more discussion below.
The proof follows from Proposition 4.6 and Theorem 5.4 with B' = ()B.). To our knowledge, this is the first the- oretical result on DPO-style algorithms under privacy and
8

---

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHI to DPO
and LTC, the conclusion is nuanced. We tend believe that the additional factor c(s) in the uniform coverage case is tight, as it matches the known result in mean estimation and
Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire,
P. Wang, T., Marks, S., Segerie, C.-R., Carroll, M., Peng, A., Christoffersen, P., Damani, M., Slocum, S., Anwar, U. Siththaranjan, A., Nadeau, M., Michaud, E. J., Pfau, J., Krasheninnikov, D., Chen, X., Langosco, L.,, Hase, P., Bıyık, E., Dragan, A., Krueger, D., Sadigh, D., and Hadfield-Menell D. Open problems and fundamental lim- itations of reinforcement learning from human feedback. arXiv preprint 0rXiv:2307.15217 2023.
offline bandits (Zhou & Zhang, 2024). However, under the O( dependence without coverage, we hypothesize that achieving an O (((()) separation-rather than ((c(s)) possible, presenting an exciting direction for future work. Looking ahead, our reduction analysis and new results on private and robust alignment may serve as key benchmarks and inspire further research in this domain.
Cen, S., Mei, J., Goshvadi, K., Dai, H., Yang, T., Yang. S., Schuurmans, D., Chi, Y., and Dai, B. ialue-incentivized preference optimization: A unified approach online and offline RLHF arXiv preprint arXiv:2405. 19320, 2024.
Acknowledgements
XZ is supported in part by NSF CNS-2153220 and CNS- 2312835. XZ would like to thank Weihao Kong for insight- ful discussions.
Chaudhuri, K. and Hsu, D. Sample complexity bounds for differentially private learning. In Proceedings of the 24th Annual Conference on Learning Theory, pp. 155-186. JMLR Workshop and Conference Proceedings, 2011.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Chen, J. and Jiang, N. Information-theorree considerations in batch reinforcement learning. In International Con- ference on Machine Learning, pp. 1042-1051. PMLR, 2019.
References
Chen, S., Koehler. F., Moitra, A., and Yau, M. Classification under misspecification: Halfspaces, generalized linear models, and connections to evolvability. arXiv preprint arXi::2006.04787 2020.
Agarwal, A., Kakade, S. M., Lee, J. D., and Mahajan, G. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1-76, 2021.
Chen, X., Zhong, H., Yang. Z., Wang. Z., and Wang, L. Human- in-the-loooo. Provably efficient preference- -base reinforcement learning with general function approxima- tion. In International Conference on Machine Learning, pp. 3773-3793, PMLR, 2022.
Awasthi, P., Das, A., Kong, W., and Sen, R. Trimmed maxi- mum likelihood estimation for robust learning in gener- alized linear models. arXiv preprint arXiv:2206.04777 04777, 2022.
Cheu, A., Smith, A., and Ullman, Manipulation attacks in local differential privacy. In 202/ IEEE Symposium on Security and Privacy (SP), pp. 883-900. IEEE, 2021.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N.. Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfeld-DDodss, Z., Herman- dez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, D., Brown, T., Clark, J. McCandlish, S., Olah, C., Mann, B., and Kaplan, J. Training a helpful and harmless assistant with reinforce- ment learning from human feedback. arXiv preprint arXiv:2204.00 05860,8862222 2022.
Chhor, J. and Sentenac, Robust estimation of discrete distributions under local differential privacy. In Interna- tional Conference on Algorithmic Learning Theory, pp. 411-446. PMLR, 2023.
Chowdhury, S. R., Zhou, X., and Natarajan, N. Differen- tially private reward estimation with preference feedback. arXiv preprint arXiv:2310. 19733, 2023.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. 1952.
Chowdhury, S. R., Kini, A., and Natarajan, N. Provably robust DPO: Aligning language models with noisy feed- back. arXiv preprint MXXx:2443.004099 2024.
Biometrika, 39(3/4):324
Bukharin, A., Hong. I,, Jiang, H., Zhang, Q., Zhang, Z., and Zhao, T. Robust reinforcement learning from corrupted human feedback. arXiv preprint arXiv:2406. 15568, 2024.
Diakonikolas, I. and Kane, D. M. Algorithmic high- dimensional robust statistics. Cambridge university press, 2023.
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J.,

---

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO
Duchi, J. C., Jordan, M. I., and Wainwright, M. J. Local privacy and statistical minimax rates. In 2013 IEEE 54th annual symposium foundations of computer science, pp. 429-438. IEEE, 2013.
Li, M., Berrett, T. B., and Yu, Y. On robustness and local differential privacy. The Annals of Stafistics, (1 737, 2023.
Mandal, D., Nika, A., Kaaalaruban, P., Singla, A., and Radanović, G. Corruption robust offline reinforce- ment learning with human feedback. arXiv preprint arXiv:2402.067344 2024.
Duchi, J. C., Jordan, M. I., and Wainwright, M. J. Minimax optimal procedures for locally private estimation, Journal of the American Statistical Association, 113(52)):182- 201. 2018.
Munos, R. Performance bounds in L ---norm for approximate value iteration. SIAM journal on control and optimization, 46(2):541-561, 2007.
Dwork, C. and Roth, A. The algorithmic foundations of differential privacy. Foundations and Trends@) in Theo- retical Computer Science, ((3-4):211 1071 2014.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-277444 2022.
Feng, J., Xu, H., Mannor, S., and Yan, S. Robust logistic regression and classification. Advances in neural infor- mation processing systems, 27, 2014.
Feng, Q. Kasa, S. R., Yun, H., Teo, C. H., and Bodapati, S. B. Exposing privacy gaps: Membership inference attack preference data for L.M alignment. arXiv preprint arXiv:2407.06445, 2024.
Pacchiano, A., Saha, A., and Lee, J. Dueling RL: reinforce- ment learning with trajectory preferences. arXiv preprint arXi::2111.04850 2021.
Ghazi, B., Golowich, N., Kumar, R., Manurangsi, P., and Zhang, C. Deep learning with label differential privacy. Advances in neural information processing systems, 34: 27131 2021.
Prasad, A., Suggala, A. S., Balakrishnan, S., and Ravikumar, P. Robust estimation via robust gradient estimation. Jour- nal of the Royal Statistical Sociery Series B: Statistical Methodology, 82(3):601-627, 2020.
Hsu, D., Kakade, M., and Zhang, T. A tail inequality for quadratic forms of subgaussian random vectors. arXiv preprint arXiv:1110.2842. 2011.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. C. Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. Ad- vances in Neurul Information Processing Systems, 36. 2023.
Huang, A., Zhan, W., Xie, T., Lee, J.D. Sun, W., Krishna- murthy, A., and Foster, D. J. Correcting the mythos of Kl-rreularizzaton: Direct alignment without overparame- terization via Chi-squared preference optimization. arXiv preprint ar 2rXiv:2407.13333 2024.
Rashidinejad, P., Zhu, B., Ma, C., Jiao, J., and Russell, S. Bridging offline reinforcement learning and imita- tion learning: A tale of pessimism. Advances in Neural Information Processing Systems, 34:11702-11716, 2021.
Jin, Y., Yang, Z. and Wang. Z. Is pessimism provably efficient for offline RL? In International Conference Machine Learning, pp. 5084-5096. PMLR, 2021.
Song, Y., Swamy, G., Singh, A., Bagnell, D., and Sun, W.
Kairouz, P., Oh, S., and Viswanath, P. Extremal mecha- nisms for local differential privacy. Advances in neural information processing systems, 27, 2014.
The importance of online data: Understanding prefer- ence fine-tuning via coverage. In ICML 2024 Workshop: Aligning Reinforcement Learning Experimentalists and Theorists, 2024.
Kasiviswanathan, S. P., Lee, K., Nissim, K., Raskhod- nikova. S., and Smith, A. What can we learn privately? SIAM Journal on Computing, 40(3):793-826, 2011.
Tropp, J. A. An introduction to matrix concentration inequal- ities. Foundations and Trends@ in Machine Learning, 8 (1-2):1-230, 2015.
Lambert, N., Krendl Gilbert, T., and Zick, T. The history and risks of reinforcement learning and human feedback. arXiv e-prints, pp. arXiv-2310, 2023.
von Werra, L., Belkada, Y., Tunstall, L.,, Beeching, E., Thrush, T., Lambert, N., Huang, S., Rasul, K., and Gal- louédec, Q. TRL: Transformer Reinforcement Learn- ing. https: /ttt://///thbb cwt comm/ugggingacce/tt... 2020.
Li, G., Shi, L., Chen, Y., Chi, Y., and Wei, Y. Settling the sample complexity of model-based offline reinforcement learning. The Annals of Statistics, 52(1):233- 2024.
10

---

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHI to DPO
Wang, Z.,, Bi, B., Pentyala, S. K., Ramnath, K., Chaudhuri, S., Mehrotra, S., Mao, X.-B., Asur, S., et al. A comprehen- sive survey of LLM alignment techniques: RLHI, RLAIF, PPO, DPO and more. arXiv preprint arXiv:2407.16216, 2024.
Warner, S. L. Randomized response: A survey technique for eliminating evasive answer bias. Journal of the American statistical association, 60(309):63 69, 1965.
Xie, T., Foster, J. Krishnamurthy, A., Rosset, C., Awadal- lah, A., and Rakhlin, A. Exploratory preference optimiza- tion: Harnessing implicit approoxximtiionn for sample- efficient RLHF. arXiv preprint arXiv:2405.2 2046, 2024.
Xiong, W. Dong, H., Ye, C., Wang, Z., Zhong, H., Ji, H., Jiang, N., and Zhang, T. Iterative preference learning from human feedback: Bridging theory and practice for RL.HF under KL-constraint. In Forty-first International Conference on Machine Learning, 2024.
Zanette, A., Cheng, C.-A., and Agarwal, A. Cautiously optimistic policy optimization and exploration with lin- ear function approximation. In Conference on Learning Theory, pp. 4473-4525. PMLR, 2021.
Zhan, W., Uchara, M., Kallus, N., Lee, D., and Sun, W. Provable offline preference-banee based reinforcement learning. arXiv preprint arXiv:2305.1488
2023.
Zhang. X., Chen, Y., Zhu, X., and Sun, W. Robust policy gra- dient against strong data corruption. In International Con- ference Machine Learning, pp. 12391-12401. PMLR, 2021.
Zhang, X., Chen, Y., Zhu, X., and Sun, W. Corruption- robust offline reinforcement learning. In International Conference Artificial Intelligence and Statistics, pp- 5757-5773, PMLR, 2022.
Zhou, and Zhang, W. Locally private and robust multi- armed bandits. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
Zhu, B., Jordan, M., and Jiao, J. Principled reinforcement learning with human feedback from pairwise or K-wise comparisons. In International Conference on Machine Learning, pp. 43037-43067. PMLR, 2023.
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv: 190..9999,998 2019.

---

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHI to DPO
A. Additional Related Work
We discuss here more relevant work that do not fit the main text. In addition to the work discussed below, we refer readers to Huang al. (2024) for theoretical results on standard offline aligmment, the survey Casper et al. (2023) for a more comprehensive overview of RLHF, and to Wang al. (2024) for the overview of LLM alignment in general.
Provably robust alignment under corruption. We would like to remark that our use of the strong comuppiion model from robust statistics literature is motivated by its popularity in robust offinn and online reinforcement learning (i.e., when the actual rewards are observed) (Zhang et al., 2022; 2021), as well as the recent interest in examining its interplay with local differential privacy across various statistical tasks (Li et al., 2023; Cheu et al., 2021; Chhor & Sentenac, 2023). Moreover, this corruption model allows us to consider corruption occurring both data generation and collection.
Provably robust offline RL. Without privacy constraints, our work can be seen non-trivial extension of the results in corruption-robust offline RL (Zhang al., 2022) to the setting of offline RLHF, where only relative rankings, rather than true rewards, are observed. As will be discussed in Appendix C, the lower bounds established for robust offline RL, along with their proof techniques, can be applied or adapted to derive lower bounds for offline RLHF.
Robust logistic regression under corruption. Among those works on logistic regression under adversary corruption (Feng et al., 2014; Prasad et al., 2020; Chen et al., 2020; Awasthi et al., 2022), the most relevant one is Awasthi et al. (2022) that considers Binomial regression under label corruption, which includes logistic regression as a special case. Awasthi al. (2022) propose alternating minimization method that achieves recover rate of O( In(1/a)) in L2 norm, where [0, 1/2) is the corruption parameter. In contrast, intermediate result in Section 5 implies rate of O(a). Moreover, our rate is achieved by the simple maximum likelihood estimator rather than the inefficient trimmed maximum likelihood estimator in Awasthi et al. (2022).
B. Algorithm
Algorithm 2 Offline RLHF
1: Input: The current parameter estimate 0, the empirical covariance matrix Σ, the regularizer X, the concentration bound [(n, d, 6, A), a reference policy Tref and a tuning parameter 17 E 10.
2: if i 0 then
3: J(x) = E -E-ppaa((11 (1) ((s.
4: return के argmax, [(r
5: else
6:
Construct confidence set
((,A)== {0 OB 1 -00-0ss <(n,,,,,,))
Compute pessimistic expected value
J(x)= = min (e o0s a))) Eerpp (10)) 00, a))
0E0(0,x)
7: return a = argmax, ((r)
8: end if
C. Discussions
In this section, we discuss the tightness of our suboptimality bounds. In particular, primarily focus on the result in Theorem 6.1, as it offers stronger guarantees compared to Theorem 6.3.
Dependence on 1/y, The dependence on 1 1/7 (eeB) ((e"mon) ) is present in nearly all existing results on both offline and online RLHF (Zhu et al., 2023; Zhan et al., 2023; Xie et al., 2024; Pacchiano et al., 2021; Chen et al., 2022). This stems from an intrinsic feature of the Bradley-Tene Terry model, namely, the non- non-linearity of the sigmoid function.
The privacy cost of c(e). Compared to the non-private (non-corrupted) case, our bound includes an additional multiplicative
12

---

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHI to DPO
factor of cle), which we believe be tight when E E [0, 1]. i.e., c(e) (1/ss) First, this factor appears even in simple mean estimation, where matching lower bound is provided in Duchi et al. (2018). Second, more concrete argument can be made by modifying the existing lower bound proof for the non-private case to show that c(s) is necessary. Specifically, the key insight is that any LDP mechanism is a contraction of the KL divergence, as stated in Duchi et al. (2018, Theorem 1), Thus, in the lower bound proof for the private case, the non-private KI. divergence is replaced with the private one, which is smaller by factor of (e - -)),, eventually leading to a factor of 1/s.
The separation between CTL and LTC. We observe an additional factor of c(E) in the corruption under LTC compared to CTL. We conjecture that this is tight for all E > especially for the one in Theorem 6.2. First, the separation result also seen in the mean estimation problem and is shown to be tight (Zhou & Zhang, 2024). Second, more concrete argument can be made by modifying the lower bound for standard offline linear bandits under corruption (Zhang et al., 2022). ! This lower bound is valid for offline RLHF under CTL,² as offline RLHF is at least as hard as offline linear bandits, and CTL is harder than corruption-only settings. To demonstrate the additional c(E) factor under LTC, a key fact is that any LDP mechanism contracts the total variation distance by a factor of c(E) (cf. Lemma H.4). Using a standard coupling argument, one can then derive a lower bound with the additional factor of c(e) for the LTC setting.
Dependence on a. Our current va dependence matches the best existing result, even in standard offline RL (Zhang et al., 2022). However, this va dependence does not align with the existing Maa lower bound (Zhang al., 2022). On the other hand, under the uniform coverage assumption, our result in Theorem 6.2 achieves the optimal dependence on a. Furthermore, we conjecture that the 1/E factor preceding a is optimal. Our reasoning is as follows: due to boundedness, we have& & & 1/d. In the best case, when ş = 1/d, our upper bound matches the lower bound of da in Zhang et al. (2022), which was established for standard offline linear RL, except for the difference of 1/r due to the non-linearity.
Practical implementations. For the sake of theoretical analysis, we adopt linear modeling in the main paper. Nevertheless, we mention that our proposed method can be readily extended to the case with general function classes (albeit losing the current formal theoretical guarantees). Take DPO for an example, we can solve the following optimization problem:
(r (r
म = argmin Tell
Σ In(1 - (1 + (zs + t() 1)c(s) In
(16)
1
where
T T(o ss) Taft (a/1s)) [s:)
B In T(a)1s)) Tatt (a?
R.Tafl Bi
:= =31n 3l
Some sanity checks are in order. First, for the standard case (i.e., 8 - oo and a = 0), we have a(e) 24 Yis which leads us back to the standard DPO loss, see (1I). Second, if we consider log-linear policy, (16) reduces to (14) (up to some scaling of (3. Third, if there is only privacy (or similar random flipping noise with known flipping rate as in Chowdhury et al. (2024)), one can verify that the above loss is equivalent to the one in Chowdhury et al. (2024) (see their Eq. 12, which is called rDPO), up to some simple rescaling. Thus, in this sense, compared to the sub-optimal rate of ((//n//2) for the log-linear policy class established in Chowdhury et al. (2024), we give the first O(1/ /n) rate for private or "robust" DPO. One can also follow a similar approach as above by simply replacing the policy-parameterized reward m.m.r by a reward function in reward function class for RLHE, Then, a similar method as in Algorithm 1 of Zhan et al. r B3
a(E) = 1 and
(2023) can be adopted for introducing pessimism.
D. Experiments on DPO and rDPO under Privacy and Corruption
As mentioned in the last section, we provide the first results for rDPO (Chowdhury et al., 2024) under both privacy and corruption with a log-linear policy class (cf. Theorem 6.3). In this section, we would like to empirically demonstrate its performance with a general function class, i.e., neural nets.
Note that the hard instance in Zhang et al. (2022) only requires corruption in rewards, not hoth features and rewards. Hence, it can he used for our setting.
With factor of two in the sample complexity.
13

---

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHI to DPO
D.I. Experiment Setup
Dataset, We utilize GPT-4o to generate synthetic dataset, referred as finance-pree pewnnceereeeeeeeeeee which comprises 1697 preference samples. Each sample includes prompt related to financial scenario and two possible responses, where "rejected" represents the high-risk option and "chosen" represents the low-risk option. This labeling can be viewed as private or sensitive information. For illustrative examples from our dataset, please refer to Appendix I. For SFT training, we construct the f finance_sft dataset by simply concatenating the prompt with the corresponding "chosen" response.
SFT Training. We begin by fine-tuning GPT2-large using the finance_sft dataset to obtain the SFT policy, Tat. For this, we directly utilize the SFT trainer from the Transformer Reinforcement Learning (TRL) library (von Werra et al., 2020), with the hyperparameters listed in Table 3.
DPO and rDPO Training. For alignment training, we split the dataset into 85% for training, 5% for validation, and 10% for testing. For DPO, we utilize the implementation provided in the TRL library, using the hyperparameters listed in Table 4. Similarly, for rDPO, we leverage the TRL implementation. which corresponds to DPO with Lose_type set to "robust." " In the private setting with a privacy budget of E, one can simply set label smooth ng to the flip rate, given by e-4T- This setting recovers the same algorithm presented in our main paper when the policy class is log-linear. Finally, we use the same set of hyperparameters for rDPO as in DPO training.
1
CTL and LTC Settings. The LDP mechanism follows the randomized response model, where the flip rate is given by cf41 For corruption, we assume that randomly sampled subset of (am) labels are always flipped compared to the true label. To implement both privacy and corruption, we introduce mask variable initialized to 0 for each sample. The LDP mechanism flips the mask variable with probability --ب while the corruption mechanism sets the mask to 1 with probability a. Finally, after CTL or LTC processing, labels ("chosen" and "rejected") are flipped if the corresponding mask value is 1. At this point, an astute reader may notice that LTC results a higher number of Is in the final mask variables compared to CTL
Evaluation. We evaluate our trained models TDP. TDPOO, and TFFT by generating responses for the test dataset using the hyperparameters listed in Table 5. To assess performance, employ the 1 1lama3: : 70b model judge, comparing responses from TDPO and T,PPO against those from #SFT. Finally, we use the win rate from these comparisons as our primary performance metric, following the methodology outlined in the DPO paper (Rafailov et al., 2023). We compute the average and standard deviation across five seeds.
D.2. Results
Private Case. We first compare the performance of DPO and rDPO in the private setting, as shown in Table 1. Due to the "shifting and scaling" loss used in rDPO, we observe that rDPO outperforms standard DPO in the private case. Interestingly, we make an additional observation: in the non-private setting, if we still introduce random label flips at a rate of approximately 1/(e" + 1), )), rDPO achieves even better performance than DPO. This suggests that deliberately adding noise to labels can enhance performance, resembling the well-known effect of label smoothing in classification tasks. We also tend to believe that this injected noise also somewhat help to address the overoptimization issues in DPO-style algorithms. We plan to further explore this phenomenon main theoretical result, which provides an upper bound in the worst case.
larger dataset. Finally, note that this observation does not contradict our
Private and Corruption Cases. We now examine whether the separation between CTL and LTC persists beyond the linear setting. As shown in Table 2, rDPO demonstrates better performance under CTL compared to LTC. Furthermore, the performance gap widens as e decreases. These observations are consistent with the theoretical insights derived from the linear setting.
Table 1. Comparison of win rates (%) for DPO and rDPO across different values of privacy budget e.
rDPO winrate (%)
DPO winrate (%)
0.1
59.0 + 4.7
55.4 + 1.1
0.5
65.8 + 5.6
60.4 + 3.0
14

---

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHI to DPO
Table 2. Comparison of win rates (%) for rDPO under CTT. and LTC.
(E
win rates (%) under CTL win rates (%) under LTC
(1,0.1)
69.6 + 5.1
65.4 + 5.0
(0.5.01)
64.4 + 2.8
58.6 + 2.6
E. Proofs
This section presents the proofs for main results in previous sections.
E.1. Proof of Proposition 4.2
Proof. By definition, for any xt, have
SubOpt(f,, x = J(T!) (
=J(TT) - [(xT)) + ((TT) -
( (TT).
[[
Ti
T2
Ts
holds for any function J(.). For the first case when r7 = 0, we have J(T) = Empp. (0(s, a) 0]. By the greedy algorithm in Algorithm& we have To S Further, under Assumption 4.1, we can rewrite Ti and Ts as
T Essp,a s)(((s, a) (0* (0)- 0),, Ta E0 a) T((()) T 0*)]
By the boundedness assumption, both terms can be upper bounded by |00 0- I12 which implies the first result by the fact that fruee 8*
For the second case when yj = 1, we introduce the following notation
[TT; 0*) Ee
J(T).
(17)
Thus, we have (s>p (58 a)} - (erpparnn ((s,a))] = [TT;0) - [[Tret: 0). Let ginl argm J(T;; 0) [Trett 0). Hence J((x) [T; 0lal) [rotf; 0. Then, have
SubOpt xt) =(x()) (
J[x (()) J (Fret: 0t) (J(F:0*) (TTeet:0)))
< (J(TT;0') - [ 0²)) (J( ginlt - J (Trefi 0in()
+J((F; 0ialt) J (Trett 0ial) (J(F:; of 8*))
(
$ (J( ;0) )- - 8*)) - [J(mT]00mn] -[rrr ginf)) U(T';0") - - 0*))= - [mm] - (ITett 0)
T4
(J(=);0)= [[Tret: 0)) - (J(r (Fret ginf)
Ts
where (a) holds by the greedy algorithm; (b) holds by the definition of gin and the fact that 6* E 0 A) by (5). To bound Ta and Ts, we use the definition in (17), the concentration in (5) and the definition of ( with Giuf O A), and obtain that
T4 + To < 2T (m d, x) |( [6s, - Φ(s, Teer (rer (5))]]((++))))
where we let (()) == F. (-)a (o(s, This finishes the proof.
15

---

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHI to DPO
E.2. Proof of Corollary 4.4
Proof. We need only focus on the last term in (6). Note that
Espplo(s, x (s)) - Φ(s, [s)r E
(x o(s,a'))
Thus, we have
E [8(8, n/ (s)) 618 need ( (Sta))
B Beupaaat (1)))) p0( a) Ms a (Stx))
(n)
S 3
o(s, a' )ll
2
+11--
(6)
3 3 k(Tr Tref) [eappart
- o(s, a )l)
(0)
I 3 x(TT, Trot) Banppart((())) -
- ,(s,d))
Sddif (2
-(ss,a)))(s,a'))
@ x((>) myar) traco(I),
where (a) holds by Lemma H.1 for À Ω ( In(n//)): (b) follows by the definition of n(x Feet) in (8); (c) holds by Jensen's 's inequality; (d) simply follows from the interchange of trace and expectation along with the cyclic property of trace. Taking the square root, yields the required result.
E.3. Proof of Proposition 4.6
Proof. We first show that under Assumption 4.5, the labels are generated via logistic regression model. This follows from direct computation. In particular, by (1), (10), (12), we have
1 + exp((* a.) (st, y/s = 1 s,, al a!)
=
= (s) - r" (.))
Bln xt (
T (' si) B Taft (a (s: aat (a. sss)
((8(0* Ban), A(s; at) ((s (01))
Thus, with Our = 3(0 - -aacc) and x; = ((s at)) (s, a.), we have that each label yi follows from logistic regression in (3).
We now turn to the suboptimality part.
SubOpt x*) E Ex- (s"(, - (s,,
(a)
Maa [V((''(sss ((1ss]
(6)
A Amxx Beup 11/2 KL (-1s), (-1)))
(0)
$ Amax / 1/2-Eanp Eewp KL( (T+ (-1s), + (-1s)))
where in (a) we have Amax = max..a (r* a) 31 2x < 23B, (b) follows from Pinsker's inequality, and (c) holds by Jensen's inequality.
Then, since both x* and #' are log-linear policies with parameters 6* and Ø. respectively, by a direct calculation and Taylor expansion, we have
KL (T(ss())),, (-1+s) = - 1- -0) - 0*),
16

---

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHI to DPO
where A,( (t)) a))(s, a) (=)) Bana a)] for some between 0* and 0. By independently sampling of a, ~ T-(-]s), we have Eaup (A. (0) = Ludiff definition of Kn in Definition 4.3, yields that
no. (cf. (7)). Combing the above with the
Amax 2/2
SubOpt(f, (T) v KTI
0
Faft Faft
Note that Ediff Tar is the corresponding population matrix Σ. Thus, by Lemma H.I. for A 2 CA in(n/6)), we have
v34max
"") S KTI
Φ - 0"
2v2
E+XT
Finally, note that 0' -0- Oenue) /83. Then, by (13) and Amax S 233,, we have the final result
v3
Su,Opt(R,,T)) x"
Vkn B B. I.Inn,d,6,1
v2
E.4. Proof of Theorem 5.4
Proof. We divide the proof into CTL, LTC, and CLC cases. Before that, we will present some common properties of our new loss, which will be used in all three cases.
Recall that our new loss is given by
Σ(0) = 1 Σ (0) where (() = ([n))In(1-))) o(0 x;)) + (24 + o(s) 1)c(E) TT.,
where c(e) = 20(e)-1
We will need its gradient and Hessian in our proof, given by e--1
vož(0)=-- - n
Σ
[c(s)(z + +o() - - -0(00x))) Is.
(18)
VOž(0)
n Σ
1
[a(0) ((OT)))( 21 o(0Tx:)) x.e
(19)
where we use the simple fact that o d'(x) = ((=)(())))))) oz)).
Let A = 0 - Otrue. by the fact that minimizes the loss and (19), we have
|| [(eerue + A) [(Oerue) (V (VL(mtte)) A) -VV A)
$ (((erre))
[+ |
(20)
where 7 1/(2 exp(-B') + exp(B')) by the boundedness condition. Thus, the key is to bound the term [V((0me))]] 1. which will be handled separately for each case later.
For now, let us suppose we have the following high probability bound:
(V((mme))l S (n,,d,6,,
(21)
for some function f, and proceed to establish the final bound. In particular, by the boundedness condition for θ and (20),
we have
V |1 Vl
7 || + &
+ 4yAB"a
which implies that
- - ((.dd6.)))))
0 Ouru
(22)
S+N
17

---

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHI to DPO
for some universal constant C.
Thus, it remains to establish the high probability bound in (21) under the three settings. To this end, we will fully utilize the following claims. See Appendix F for the proofs.
Claim E.1. Let ?/ be zero-mean i.idd sub-Gaussian with parameter a, condition on 3i. Then, for any 5 E (0, 1) and À > 0, with probability at least 8,
d ln(1/))
Σ 7
SC.o.
(S+AA)-1
for some universal constant C.
Claim E.2. Let b (bi,..., bn)) be a vector that at least 1 an elements are zero, and the rest are bounded by some constant ç >> 0, i.e., lb:l I 5. Then, we have
bi (S+11--1
<<
1=1
With the above claims in hand, we going to establish (21) for CTL, LTC and CLC, respectively, CTL case. In this case, we rewrite the gradient in (18) as follows
v ((Quue)) = - Σ (z))) 1) Dt + y1 V + y -(0 xss
where recall that under CTL, the true label y is first corrupted to T. which will then be privatized to generate 2, Thus, we have
V ((0 11
n Σ (c(E)(z= -1) De
Us
7
(++NT-1
Tyrraay
Tarmgtion
Ui a(
(23)
(D+A)-1
Tansdet
For Tprvvay and Tstandard, apply Claim E.1 due to zero-mean and sub-Gaussian with parameters O(c(E)) and 1. respectively. Thus, we have with probability at least 1 - 8,
d + In(1/8)
Tprivacy + Istandtard S C1 o(E))
n
for some universal constant C₁ 0.
For Teommptionn we can apply Claim E.2 with 1, and obtain that
Teomptton S Va.
Thus, combining these bounds with (21) and (22), yields the bound under CTL.
LTC case. In this case, we rewrite the gradient in (18) follows
Vī(Otrue) = (c(s) +o(e) - 1 To 7. c0TTxx)) It.
11 Σ
[c(s)(=,- 3)) c(s) + +o(e) 1) -0(0xx)) x;)) x
18

---

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHI to DPO
where recall that under LTC, the true label yi first privatized to be Dr. which will then be corrupted generate 24- Thus, have
v L(Oerue)) (1
Σ (c(e)(zi (i:)
+ +o)) - 1) -0(0
000 xx)]]]]]] Ti
(24)
(5++2))
T Toomuption
Tyerway
Similarly, for privacy. we can again apply Claim E.1 due to pero-mean and uub-Gaussian with a parameter O(c(e)). Thus, have with probability least 8,
+ In(1/8)
Tprivacy S C1 c(())
for some universal constant C₁ 0.
For Tcorruption can apply Claim E.2 with Ç c(E), and obtain that
T Tcomuptionn I <c(E)va.
Thus, combining these bounds with (21) and (22), yields the bound under LTC.
CLC case. With the results of the previous two cases in hand, we can now easily analyze the CLC case, as it is essentially the summation of the CTL and LTC. More specifically, we will now rewrite the gradient in (18) as follows
V V((erru)) =
[c(e)(2; + - ot) c(e) o(s) - 1) Th + Dz a(0 [(Tx:;)] x;
1=1
[c(=)(e: - T.) + clE) + o(e) - 1) - 7. + 0: - - o.)] x4
n
where recall that under CLC, the true label is first corrupted to ys (with parameter a1) and then it privatized to Ti which will then further corrupted 2; (with parameter 02). By direct utilization of the bounds in (24) and (23) (along with c(=) 2 1), we have with probability at least 1- 8,
VV( 2 - d In(1/8) + c(E) + Vai,
for some universal constant C.. Thus, combining these bounds with (21) and (22), yields the bound under CLC.
E.5. Proof of Theorem 5.7
Proof. As before, we present some common steps and results in all three cases. Similar to (20), we have
||l < Z(0true + A) -[(rroe)) - (V ((0mn))) S -VV[((erec)) A) S V
||
Suppose for we have the following high probability bound
V [V((0me))) S g(n, 8),
(25)
for some function g. and proceed to establish the final bound. In particular, we need lower bound on | in terms of | 2| To this end, by Lemma H.3 with X, n, H 1, Pmin = nε, we have with probability least Amin Σ) 2 E/2, when n 2 8la(d/6) Thus, we have
75
| 2/ < g(,, d, ||Al|2
19

---

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHI to DPO
which implies that
0 0rruc
2 X5
A 1| S (n, 8)
(26)
Thus, it only remains to establish the bound (25) under three cases. To this end, we will leverage the following two claims, the counterparts of our previous two claims, but in Le norm.
Claim E.3. Let T be zero-mean i.dd sub-Gaussian with parameter o, condition on 24. Then, for any 8 E (0, 1) and À with probability at least 6, 6,
+ ln(1/5)
T
SC.
for some universal constant C.
Claim E.4. Let b (b1,.... bn) be a vector that at least 1 an elements are zero, and the rest are bounded by some constant ç > 0, i.e., lb:l I 5. Then, we have
b
<ça
S
We are left to establish (25) for CTL, LTC, and CLC, respectively.
CTL case. Following the same process as before, replacing the weighted norm by L2 norm and leveraging the new claims, yields the following result
1+ In(1/8)
V & ((n, = C1 d()
+ a,
72
which implies the final result by (26).
LTC and CLC cases. Both of them follow the same process above, which gives the final result by (26).
F. Proofs for Claims
Proof of Claim E. As in Zhu et al. (2023), the proof mainly utilizes the concentration in Lemma H.2. To this end, we let X guxd where ny e idd is its i-th row and let v = (M....., .m) column vector. Then, we have
= T Mn. where M : n² 1 1X)
S+AI
With simple linear algebra, we can have
s d
trace(M S d
1 |M
trace (M)
and
n²
n
Thus, by Lemma H.2, we have with probability at least 1 - 8,
d ln(1/
T
SC.
n
(É+AN-1
for some universal constant C > 0.
Proof of Claim E,2. By a direct computation and recall M = 1/ x(x + AT) with |M] S 1/n in the above proof,
we have
b
b
Mb
| |bb| < an 52
(
which implies the result by taking the square root.
20

---

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHI to DPO
Proof of Claim E.3. The proof also relies Lemma H.2. As before, let X E Ruxd where by E Rd is its i-th and let 1) (,,,.... Qa) be column vector. Then, have
2 Mn,
1 XXT.
7::
1)
where M: == n
71
2
With simple linear algebra, can have
1
trace(M) S n
trace(M)3 n²
|M -
Thus, by Lemma H.2, we have with probability at least 1 - 8,
In(1/))
73
SC.0.
for some universal constant C >> O.
Proof of Claim E.4. This simply holds by algebra:
R
12
b
S
Σ
| 16:| S Ca,
n
n
2
which holds by the boundedness assumption of ||a||| S
G. Discussion on the Gap in Prior Work
As we have pointed out the main paper, the current stated result in Mandal et al. (2024) (in particular, their Theorem 3.3) misses the 1/7 factor. This due to a gap in their proof of Lemma 3.2. This happens on the last chain equations on Page 20. In particular, the first inequality below has the wrong direction.
Σ
(exp(-o"(0xx)/) + exp(o"(0,x))2) 2n
N
nESHI
Σ
In
REŠnI
where they claim to use e - 2 2. Notice that due to the negative sign, the inequality direction should be reversed. In order to have the right direction, we need to introduce ?, which in turn introduces 1/y in the final bound.
H. Auxiliary Results
Lemma H.1 (Concentration of Covariances, Lemma 39 in (Zanette et al., 2021)), Let o1...., On Rd be i.idd samples from a distribution l with || s 1. Let Σ = B0 be the population matrix. If X 2 8 ($ -i(nn/))), then with probability at least --8. 8.
5
M1+ = 3
+1
(1+
Lemma H.2 (Tail bound for quadratic forms, Theorem in (Hsu al., 2011)), Let A Rmxn be matrix and let Σ: AT A. Suppose {==)=_ is i.i.d sub-Gaussian with parameter and let x (z.,..,, In) be column vector. Then, for any d e (0, 1), with probability at least 1-2 ö,
|| |x - x Txx < trace(E) + 2 In(ee/) + 22
'The original version can handle non-independent case.
21

---

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO
Lemma H.3 (Matrix Chernoff, Theorem 5.1.1 in (Tropp, 2015)), Consider finite sequence {Xx) of independent random,
symmetric matrices in Rddxd, Assume that Amin (X²) and Amax (X;) I H for each i. Let Y Ei X; and pmnin denote
the minimum eigenvalue of the expectation E [ i.c., Mmin Amin C E [X.]). Then, for any (0, it holds
P (mmia (Y) S Emmin) S d.d exp (-(1-
=) Pmin
211
Lemma H.4 (Corollary 2.9 in (Kairouz al., 2014)). For any > O, let Q be any s-LDP mechanism. Then, for any pair of distributions P1 and P2, the induced marginals M1 and M2 via satisfy
E 1 TV(M
TV(PPP2).
+
Additional Details on Experiments
L1. Samples in Our Dataset
Below, we present a selection of examples from our generated financial dataset across various categories. Each example demonstrates prompt alongside "Chosen" and "Rejected" responses, illustrating the alignment of decisions with risk levels and priorities.
Category: Lifestyle & Personal Planning
Prompt: "You'n re saving $3,000 to host family talent show. How do you proceed?"
Chosen: "Rent small venue and create DIY props and prizes."
Rejected: "Spend on professional staging and lighting for a one-time event."
Category: Home Improvement & Maintenance
Prompt: "You't 're saving $10,000 to add an outdoor kitchen. How do you proceed?"
Chosen: "Install a grill, sink, and storage with
materials."
Rejected: "Spend on high-end appliances that exceed your budget."
Category: Investments
Prompt: "You're 're saving $12,500 invest in green construction funds. How do you proceed?"
Chosen: "Choose funds with diverse holdings in sustainable building materials."
Rejected: "Invest in speculative green startups with limited financial history."
Category: Small Business
Ventures
Prompt: "You't re saving $10,000 to start custom clothing line.
How do you proceed?"
Chosen: "Focus on affordable designs and use an online platform to sell."
Rejected: "Spend on luxury boutique storefront before establishing demand."
Category: Education Skill Development
Prompt: "You'r saving $5,000 to attend data visualization course. How do you proceed?"
Chosen: "Enroll in a course with interactive projects and industry relevance."
Rejected: "Choose a program with limited hands-on training."
Category: Debt Management
Prompt: "You're 're saving $12,000 pay off business loan. How do you proceed?"
Chosen: "Apply the funds directly to reduce the principal and future interest."
Rejected: "Use the funds for operational expenses while extending the loan term."
Category: Miscellaneous
Prompt: "You want to save $4,500 to organize a youth art festival. How do you proceed?"
Chosen: "Partner with local sponsors and focus on cost-effective exhibits."
Rejected: "Spend heavily on promotional campaigns without engaging artists."
22

---

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO
These examples illustrate the structured nature of our dataset and its alignment with decision-making scenarios across diverse financial categories.
1.2. Hyperparameters
The hyperparameters for the experiments outlined below. Any hyperparameters not explicitly mentioned use the default values in the TRL library.
Table 3. Hyperparameters used for SFT training.
Parameter
Value
learning rate
le-5
batch size
8
num train epochs
3
Table 4. Hyperparameters used for DPO and rDPO training.
Parameter
Value
beta
0.1
learning rate
le-6
batch size
8
num train epochs
1
Table 5. Hyperparameters used for response generation.
Parameter
Value
temperature
0.25
max length
50
truncation
True
do sample
True
top k
30
23


Reviews:
[{"review_id": "Pl0XqrpjOv", "reviewer": "Reviewer_xT8c", "summary": "The paper develops a unified theoretical framework analyzing the impact of label corruption and privacy on two primary offline alignment methods: RLHF and DPO. The authors focus on the interplay between LDP and adversarial label corruption, formalizing three noise models: CTL (corruption then LDP), LTC (LDP then corruption), and CLC (corruption before and after LDP). A key conceptual contribution is a reduction from the offline alignment problem to logistic regression under these noise models. Major findings include:\n\n- LTC is provably more challenging than CTL, even under linear models.\n- The analysis yields new state-of-the-art theoretical guarantees for offline alignment under privacy-only or corruption-only regimes.\n- Novel estimation bounds are derived for logistic regression under the joint noise setting, leading to suboptimality bounds for RLHF and DPO algorithms.\n- The authors also provide practical algorithmic implications, including a new estimator and empirical verification of the separation between LTC and CTL on GPT2.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and positive evaluation of our paper. We're glad to hear that you found our methods well-motivated, our derivations clear, and our experimental results useful.", "claims_and_evidence": "The main claims of the paper—such as the separation between CTL and LTC, and the improved suboptimality bounds under joint privacy and corruption—are supported by a well-structured reduction to logistic regression under noisy labels. While the claims appear to be justified by clear derivations and formal proofs, I have not carefully reviewed the correctness of the theoretical arguments.", "methods_and_evaluation_criteria": "The methodological choice to reduce the suboptimality of RLHF and DPO to parameter estimation in logistic regression is well-motivated and principled, enabling a unified treatment of privacy and corruption. The authors assess performance using standard metrics, including suboptimality gap and parameter estimation error in both weighted and Euclidean norms, all under clearly stated assumptions such as linear reward models and bounded feature maps. These choices appear appropriate given the theoretical focus of the work. That said, due to my limited familiarity with several of the domains involved, I am not in a position to rigorously evaluate the methodology in its entirety.", "theoretical_claims": "The derivations appear technically sound from a high level, and the proofs are well-organized. However, I am not equipped to verify their correctness in detail. I particularly appreciate the clean and interpretable separation between CTL and LTC in the error bounds, which draws an interesting parallel to similar phenomena observed in private robust mean estimation.", "experimental_designs_or_analyses": "While the primary focus is theoretical, the authors include empirical experiments on GPT2-large that demonstrate the practical difference between CTL and LTC. These experiments are not extensively described in the main paper, but are useful as sanity checks. Further empirical validation across more complex or realistic settings would strengthen the practical case, but is not strictly necessary for this theory-focused submission.", "relation_to_broader_scientific_literature": "From the perspective of the LDP literature, this paper contributes a novel extension of LDP techniques to preference-based alignment, which remains relatively underexplored in the privacy literature. The authors extend ideas from private logistic regression and randomized response mechanisms, as seen in works such as Chowdhury et al. (2023), to a more complex setting involving both privacy and adversarial corruption.\n\nWhile I do not have deep familiarity with the broader literature on the other areas covered in the paper, the authors position their work as unifying and extending previous theoretical treatments of RLHF and DPO under separate noise models. Their comparison between CTL and LTC, and the development of a unified reduction to private and corrupted logistic regression, appear to be novel contributions that bridge multiple subfields.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S5FQXRUxqs", "reviewer": "Reviewer_FQRv", "summary": "The authors provide a theoretical framework to analyze the suboptimality gap of the learned policy in offline alignment, in the presence of privatized and corrupted labels. Specifically, they reduce two main paradigms, the Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), to the parameter estimation problem of logistic regression under some linear model assumptions. The suboptimality gap is then upper bounded by the estimation error of the logistic regression parameters. They demonstrate that, with a “shifting and scaling” loss function, the estimation error can be upper-bounded when labels are privatized (i.e., random response mechanism) and corrupted (i.e., an adversary modifies partial labels arbitrarily), quantifying the robustness of the alignment training to noisy labels.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive evaluation of our paper. We are delighted to hear that you find our methods promising and well-founded.", "claims_and_evidence": "The claims made in the paper (“a unified analysis of the interplay between\nprivacy and robustness in both RLHF and DPO”) are well matched with their theoretical results.", "methods_and_evaluation_criteria": "Reducing the alignment problem to logistic regression and upper bounding the estimation error is a promising and sensible approach.", "theoretical_claims": "I do not check the correctness of the proof, but it reads technically sound.", "experimental_designs_or_analyses": "There are no experiments included in the main text. One experiment is included in the Appendix, showing that training with a robust loss function to noisy labels can outperform the regular cross entropy loss, which demonstrates the benefits of robust alignment training. This aligns with the robustness shown in the paper.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Other strength: as the authors point out, the results of the estimation error bounds of private/robust logistic regression may be of independent interest.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DPU9hWHusS", "reviewer": "Reviewer_2Ymr", "summary": "The paper considers alignment problems such as RLHF or DPO, in a robust private setting. In this setting, we are given a preference dataset where each example contains an input text $s$, two actions $a_0, a_1$, and a label $y \\in \\{0, 1\\}$ denoting which action is preferred in the example. Commonly, we assume the label $i$ is sampled with probability proportional to $exp(r(s, a_i))$, where $r$ is a reward function on the text $s$ and action $a$ In RLHF we train an intermediate reward model on the dataset which is then used to optimize the policy, in DPO the policy is directly optimized on the dataset. In the robust, private setting there are two sources of error on the labels (1) corruption, where some fraction $\\alpha$ of the labels are corrupted by an adversary after inspecting the dataset, and (2) local DP, where we apply randomized response to change the label of each example (with probability $1 / (1 + e^\\epsilon)$ for some $\\epsilon$) to provide privacy to user. One can consider the LTC or CTL settings, where local DP is applied and then corruption occurs, or vice-versa. The authors give an analysis for both RLHF and DPO, in the LTC, CTL, and \"CLC\" setting (where corruption occurs both before and after local DP). \n\nFor RLHF the analysis assumes that the reward function is the dot product of some ground (bounded norm) ground truth $\\theta^*$ and some known function $\\phi(s, a)$, which was done by most of the prior work. They consider two algorithms: One that chooses a policy maximizing the expectation over the policy of this dot product for a current parameter estimate of $\\theta^*$, and one that choosing a policy that maximizes the minimum this expectation can increase relative to a reference policy even if the parameter estimate is perturbed by a bounded amount. Under the linear model assumption, the authors show the labels correspond to a logistic regression model and bound the suboptimality of their algorithms. For DPO, they assume the optimal policy falls a log-linear model and prove similarly that the labels in DPO follow a logistic regression model and bound the suboptimality of a policy that is a function of an estimate of the ground truth in the logistic regression model.\n\nUsing these results, the authors design algorithms for getting the parameter estimate of $\\theta^*$, by shifting and scaling the received labels based on the privacy of randomized response. They give bounds for this parameter estimate's error, with and without corruption, and with and without a uniform coverage assumption (the covariance of the features has lower bounded eigenvalues). Combining the parameter estimation bounds with the suboptimality bounds for the algorithm using the parameter estimate, they derive suboptimality bounds for RLHF and DPO in the LTC and CTL settings, with or without uniform converage. The bounds consist of (1) a bias term depending on the corruption parameter $\\alpha$, which is larger for LTC than for CTL, demonstrating that LTC is fundamentally harder) and (2) terms decaying to 0 as $1/\\sqrt{n}$ in the dataset size $n$, matching the noiseless rate.\n\n## update after rebuttal\nI remain in support of accepting the paper", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive feedback. We will fix the typos in the next version. We now recap your comment and present our detailed response. \n\n**Minor presentation weakness about LTC and CTL.** Thanks for your sharp question. You are absolutely right and we will make this point more clear in the paper. We have briefly talked about the lower bound on the separation between LTC and CTL in the conclusion section and Appendix C. Let's recap them based on the following two scenarios:\n\n- **With uniform coverage condition.** As mentioned in Appendix C, we believe that the current additional $c(\\epsilon)$ faction in LTC is tight. \n\n- **Without uniform coverage condition.** As mentioned in the conclusion section, in this case, the current additional factor in LTC may not be tight. However, we believe that there still exists a separation between LTC and CTL. The intuition here is that adversary corruption after randomized response for privacy can always lead to a larger amount of flipping of the preference labels, as the adversary can focus on the attacks on the clean data by first observing the output of LDP.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I did not check the proofs in detail.", "experimental_designs_or_analyses": "Yes, no major issues.", "relation_to_broader_scientific_literature": "Matches the dependence on $n$ of past work in the noiseless setting. Improves on Mandal et al. in terms of dependence on corruption parameter $\\alpha$ for RLHF under a weaker corruption model, and matches the offline rate. For DPO, improves dependence on $\\alpha$ and removes dimension dependence for bias term. First result on private and robust DPO, prior work studied private-only or robust-only. Improves Chowdury et al. result for robust DPO which has weaker dependence of $1/n^{1/4}$ on the dataset size. Overall, seems to match the rate in more restrictive settings (i.e. non-private or non-robust) or improve the rates of past works.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Other Strengths:\n* The technical results are reasonably difficult to arrive at.\n* Presentation allows for one who is not super familiar with RLHF/DPO or with local DP/randomized response to understand the problem setup and algorithms reasonably well.\n\nOther Weaknesses:\n* A minor shared weakness with past work, that the analysis assumes a linear reward function. The authors do cite this as room for improvement in future works, and attempt to mitigate this with empirical studies where the linear reward function assumption may not hold.\n* Minor presentation weakness: The introduction suggests a separation, which implies a lower bound for one setting and upper bound for the other setting. However, the results are upper bounds only. It is unlikely but theoretically possible there is no separation with a different algorithm, this should be clarified when discussing the contributions of the paper.", "other_comments_or_suggestions": "-Line 24, column 2: Missing period\n-Line 265, column 1: \"singple\" -> single", "questions_for_authors": "No questions that would substantially affect my evaluation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a9BrRYeLNV", "reviewer": "Reviewer_hRoJ", "summary": "This submission studies the interplay between privacy and robustness in both RLHF and DPO, two main alignment methods for language models. It shows that, when considering a linear reward model for RLHF and a log-linear policy class for DPO, the problem of offline alignment reduces to parameter estimation for logistic regression under private and corrupted labels. Using this reduction, one can derive suboptimality upperbound for the policy learned for both RLHF and DPO, based on parameter estimation error for logistic regression under private and corrupted labels.\n\nSummary after rebuttal: I asked a question from the authors after their rebuttal about their considered corruption model in the LTC case. However, the authors did not reply to this question. Yet, I am going to keep my score for the other results in the paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and comments. We will recap your valuable comments and present our detailed response. We hope our answers will resolve your concern.\n\n**1. About experiments.** Thank you for your valuable suggestion. In the next revision, we will aim to include our experiments within the main body. Since our primary contributions are theoretical, we initially chose to present experiments using DPO as a proof of concept. However, we believe similar results can be obtained with RLHF, aligning with the findings in [R1], which considers corruption alone. We plan to incorporate additional results on RLHF in our future revision.\n\n**2. Intuition about the separation between LTC and CTL.** Your observation regarding the random-flipping corruption model is absolutely correct. That is, under this oblivious corruption model, the final corruption rate is the same between LTC and CTL. However, it becomes different when we move from this corruption model. Think about the following situation. The clean data is all 0, and the corruption model here is to simply set the data to be 1 (with probability $\\alpha$). That is, with probability $\\alpha$, the data will be set to be 1, otherwise, it remains unchanged. This is exactly the corruption model considered in our experiment, which is different from the random flipping corruption model. As mentioned in Line 737, one can see that LTC will lead to more 1s in the end compared to CTL (as the randomized response of LDP may flip back 1 to 0). \n\n**3. Generalization beyond current assumptions.** Thank you for the insightful question. Although our main results (e.g., the separation between LTC and CTL) are established under our current assumptions, we believe these findings will extend more generally—beyond linear models—to broader reward functions, policy classes, and general preference models (not limited to the BT-model). However, transitioning from linear to more general settings will likely require new analytical techniques. This challenge arises because, even in settings without privacy or corruption, employing an analysis similar to ours yields suboptimal rates [R2]. Exploring these generalizations represents an exciting avenue for future research. Nevertheless, we believe that our current results under the linear model will serve as important benchmarks for these advancements.\n\n---\n\n[R1] Chowdhury, S. R., Kini, A., and Natarajan, N. Provably robust DPO: Aligning language models with noisy feedback. arXiv preprint arXiv:2403.00409, 2024\n\n[R2] Zhu, B., Jordan, M., and Jiao, J. Principled reinforcement learning with human feedback from pairwise or K-wise comparisons. In International Conference on Machine Learning, pp. 43037–43067. PMLR, 2023.", "claims_and_evidence": "The authors have provided proofs for the theoretical claims made in the submission.", "methods_and_evaluation_criteria": "Currently, the experiments reported in the submission are very limited in terms of the amount of experimental results, the datasets and the used alignment algorithms. The only reported results are in Table 1 and 2, which are just for DPO (and rDPO). No results are reported for RLHF under corrupted/privatized labels.", "theoretical_claims": "I did not go through the proofs in details. However, I had a quick look at them to get an idea of their sketch.", "experimental_designs_or_analyses": "I have the following comments/questions about the experimental results.\n\n1. I think it would be better if the authors could manage to include the experimental results (Tables 1 and 2 in the appendix) in the main body of the paper. \n\n2. Also, why are there no results included on RLHF under privacy/corruption/both? For completeness, both DPO and RLHF should be evaluated under four scenarios of privatization (RR)/corruption/LTC/TCL.", "relation_to_broader_scientific_literature": "In the context of alignment of language models, this work allows for establishing a theoretical separation result between LTC and CTL, which is important to consider when performing offline alignment of language models.", "essential_references_not_discussed": "It seems that most of the related works are mentioned and discussed in the submission.", "other_strengths_and_weaknesses": "I have mentioned my comments/questions above. After receiving the authors' answers, if needed, I will consider changing my evaluation.", "other_comments_or_suggestions": "see above", "questions_for_authors": "1. Is there any intuitive reason why alignment under LTC is more challenging than CTL? For example, in the first row of Table. 2, $\\epsilon=1$ and $\\alpha=0.1$. Therefore, in LTC, each label is flipped by randomization (RR) with probability 0.27 followed by another flip due to corruption with probability $\\alpha=0.1$. Therefore, roughly, the probability of each label to get flipped (compared to that in the clean preference dataset) is 0.027. With the same reasoning, the probability of each label getting flipped under TCL is also roughly 0.027. So why does the order of the two label flippings (RR and corruption) matter? From another point of view: the constant $c(\\epsilon)$ appearing in the upperbound of LTC is not a large constant (for example for $\\epsilon=1$, it is 2.16). However, from the results in Table. 2, we see that LTC is clearly worse than CTL. Do the authors have an intuitive idea why this is the case?\n\n2. The main assumptions made in this submission are 1. a linear reward model for RLHF 2. a log-linear policy model for DPO 3. The label distribution in the preference dataset following a LR model. How much limitative these assumptions are? Can we extend the obtained theoretical results to other RLHF/DPO models with more relaxed assumptions on the data as well as reward/policy models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a unified and rigorous theory to understand the impact of label corruption and local differential privacy on offline alignment methods including RLHF and DPO. All reviewers recommend acceptance due to the paper’s novel reduction to logistic regression under joint noise, insightful separation between CTL and LTC settings, and new bounds that advance the literature. One limitation is that the current result heavily relies on the linear-model assumption.  Overall, the theoretical contribution still makes it a strong submission that deserves acceptance.
Decision: Accept (spotlight poster)