PROPS: Progressively Private Self-alignment of Large Language Models
Authors: Noel Teku
Fengwei Tian
Payel Bhattacharjee
Souradip Chakraborty
Amrit Singh Bedi
Ravi Tandon
Venue: Accepted by TMLR ()

Abstract: Alignment is a key step in developing Large Language Models (LLMs) using human feedback to ensure adherence to human values and societal norms. Dependence on human feedback raises privacy concerns about how much a labeler’s preferences may reveal about their personal values, beliefs, and personality traits. Existing approaches, such as Differentially Private SGD (DP-SGD), provide rigorous privacy guarantees by privatizing gradients during fine-tuning and alignment but can provide more privacy than necessary as human preferences are tied only to labels of (prompt, response) pairs and can degrade model utility. This work focuses on LLM alignment with preference-level privacy, which preserves the privacy of preference labels provided by humans. We propose PROPS (PROgressively Private Self-alignment), a multi-stage privacy preserving alignment framework where privately aligned models in previous stages can serve as labelers for supplementing training data in the subsequent stages of alignment. We present theoretical guarantees for PROPS as well as comprehensive validation using multiple models (Pythia and GPT) and datasets (AlpacaEval, Anthropic HH-RLHF, truthy-dpo-v0.1) to demonstrate the utility of PROPS over existing methods while still providing high privacy. For the same privacy budget, alignment via PROPS can achieve up to 3x higher win-rates compared to DP-SGD, and 2.5x higher win-rates compared to Randomized Response (RR) based alignment.

Published in Transactions on Machine Learning Research (12/2025)
PROPS: Progressively Private Self-alignment of Large Lan- guage Models
Noel Teku
ntekul @orizona.ed edu
Deportment of Electrical and Computer Engineering University of Arizona
Fengwei Tian
fengtian&urizona.coo edu
Department of Electrical and Computer Engineering University of Arizona
Payel Bhattacharjee
wayelb@arizona. edu
Departmend of Electricul and Compuler Engineeerigg Universily of Arizona
Souradip Chakraborty
schakra3Shumd.c edu
Department of Computer Science University of Moryland, College Park
Amrit Singh Bedi
anwritbedi@ueef.ci edu
Department of Computer Science University of Central Florida
Ravi Tandon
tandonr www.rroaaaaraaaaaaaooo edu
Department of Electrical and Computer Engineering University of Arizona
Reviewed on OpenReview: https: // optpr/veew.eeee... net/ forum? id- RubRuheBoo
Abstract
Alignment is a key step in developing Large Language Models (LLMs) using human feed- back to ensure adherence to human values and societal norms. Dependence on human feodback raises privacy concerns about how much labeler's preferences may reveal about their personal values, beliefs, and personality traîts. Existing approaches, such Differen- tially Private SGD (DP-SGD), provide rigorous privacy guarantecs by privatizing gradients during fine-tuning and aligument but provide more privacy than necessary as human preferences are tied only to labels of (prompt, response) pairs and can degrade model util- ity. This work focuses on LLM alignment with preference-level privacy, which preserves the privacy of preference labels provided by humans. We propose PROPS (PROgressively Private Self-aligmment) a multi-stage privacy preserving alignment framework where pri- vately aligned models in previous stages can serve labelers for supplementing training data in the subsequent stages of alignment. We present theoretical guarantees for PROPS as well as comprehensive validation using multiple models (Pythia and GPT) and datasets (Al,,,aEval, Anthropic HH-RLHF, truthy-dpo-vO.11 to demonstrate the utility of PROPS over existing methods while still providing high privacy. For the same privacy budget, align- ment via PROPS can achieve up to 3x higher win-rates compared to DP-SGD, and 2.5x higher win-rates compared Randomized Response (RR) based alignment.

---

Published in Transactions on Machine Learning Research (12/2025)
D
RR
+DRR
DPO
M
D1
RR
RRR
DPO
M₁
Intermediate aligned model
Aligned model
Stage-
(a) RR-based alignment
Alignment (eg. DPO)
M
D
DP-SGD
M Aligned model
D2
MLE
DERAar
DPO
M
(Gradient clipping and noise addition)
RR
Aligned model
Stage-2
(b) DPSGDDDDb based alignment
(c) PROPS: preference privacy based alignment
Figure 1: (a) Randomized Response (RR) based alignment where humann preferences in the dataset D are privatized using RR. which are then used for alignment.
(b) DP-SGD based aligument where differentially private gradients are used for model alignment. (c) Two stage PROPS framework: Dataset D is partitioned into disjoint subsets (D1, D2). In Stage 1, preferences in D2 are privatized using RR, resulting in an intermediate aligned model Mj. In Stage 2, model M1 is used to independently rank the responses in D2. We then obtain private labels for D2 which are derived from combining model's predictions and RR via a maximum likelihood estimator (MLE). These progressively refined privale preferences are then used for aligmment arrive the final model M2.
1 Introduction
The process of aligning LLMs relics on datasots comprising prompts, LLM-gencrated responses, and prefer- ence labels that indicate which response aligns better with human values, collectively referred to preference data. The alignment approaches, including reinforcement learning with human feedback (RLHF) (Stiennon et al. 2020 and direct preference optimization (DPO) (Rafailov et al. 2024b. leverage these preference
datasets with ranked labels provided by human annotators.
While preference data can significantly improve the align- ment of large models with expert reasoning, it often carries deep privacy risks particularly in domains where human feedback encodes sensitive strategies, values, or professional heuristics. This tension is most evident in several high-stakes application scenarios example, physicians' feedback reflects diagnostic reasoning and treatment heuristics tied to protected health information and institutional IP; disclosure can undermine both patient privacy and clinical compelitiveness [Reddy 2023 Han et al., [0244). In legal and judicial settings, aligmment with lawyers' or judges' feedback risks revealing privileged deliberations, litigation strategics, or interpretive biases that must remain confi- dential to preserve due process. These scenarios illustrate a central point: preference privacy is not merely a theoretical consideration but critical requirement for deploying aligned models in the most sensitive and societally consequential domains. The privacy utility trade-offs in such settings demand specialized align- ment mechanisms that protect individual and institutional preference signals while preserving their utility for model improvement.
Motivation for Human Preference Privacy:
discuss next. Tn clinical decision support systems, for
In Wang et al. 22024), a large number of consultation records were used to construct dataset to fine-tune an LLM to act as a chatbot physician, with GPT-4 used to retain records that highlighted professionalism, explainability, and emotional support. As the model was shown to be effective, this indicates that certain preferences of a doctor's decision-making can be identified which could lead potential exposure of a doctor's preferences. Also, in policy analysis (Rao et al., 2202),, publicly available survey questions or proposals can be used to elicit LLM-generated analyses, where policymakers' feedback reveals sensitive interpretative insights that may require protection. Bakker et al. (2021), for example, observed that placing more emphasis on politics in their surveys to participants "resulted in self-reports of personality traits that were in some cases more aligned with preexisting political preferences."
Recent work on sponses, and human foedback as jointly private. However, in most alignment settings, only the human- provided labels rankings sensitive. While methods like differentially-private SGD (DP-SGD) protect
fine-tuning and alignment (see Section typically trcats prompts, re-
2

---

Published in Transactions on Machine Learning Research (12/2025)
the entire training tuple (as shown in Figure [b)), they hurt utility under stringent privacy requirements. This paper instead focuses on protecting human preference data. Figure 2 shows representative results that compare the method proposed in this work with conventional privacy preserving techniques including DP- SGD and Randomized Response (RR); the figure shows that for the same privacy guarantee, models aligned by our method provide higher quality responses compared to DP-SGD and RR based alignment.
Motivated by the above observations, we study the problem of aligning LLMs with preference privacy. Specifically, we investigate two notions of privacy: a) preference-level privacy, and b) labeler-level privacy. Preference level privacy ensures that for any tuple (z, 31, y2), where x denotes the prompt and y1, U2 denote the LLM-generated responses, the individual human-preference e* (which denotes whether y1 or 1/2 is preferable) does not significantly impact the aligned model. Formally, we leverage the existing notion of Difierential Privacy (DP) Dwork et al. (2014), and use it to formalize the notion of (c, (.0--preference-eeee DP, where (c, represent the privacy budgets. The notion of labeler-tevel privacy (also commonly referred to as user-level privacy in the DP literature) hides the presence/abss human labeler and protects all the labels annotated by the labeler. We summarize and highlight the main contributions and novel aspects of this work:
Main Contributions:
of any individual
PROPS for Private Alignment. We introduce Progressively Private Self- Luff-Alimmeent (PROPS), a multi-stage algorithm for alignment that improves privacy and utility. Instead of processing the entire perturbed dataset at once, PROPS divides alignment into stages. the tthh stage, for (k 1), the model from the previous stage (Mx-1) is used along with non-private prompts, responses from new batch, and noisy labels perturbed using Randomized Response (RR) RRR M generates its own rankings ÊM2- which, combined with fRR, are used to compute maximum-Hikelihoce likelihood estimates (MLEs) for aligmment. This staged process leverages intermediate models to improve preference labeling and reduce reliance on noisy labels, enhancing alignment quality while maintaining privacy. PROPS effectively balances privacy preservation with performance, offering a novel alignment framework.
Theoretical Insights. We study the utility-privacc tradeoffs for PROPS by analyzing the Sub- optimality gap Chowdhury et al. (2024a) defined as the difference between between the weights of a non-privately trained model with the weights of a privately trained model with a privacy budget in Section 3.1 Theorem
D. The upper bound on this gap shows that PROPS (which uses MLE combining in the second stage) is no worse than vanilla RR and performs better as long as the intermediate model gets better at prodicting preferences.
Empirical Evaluation. We conducted a comprehensive set of experiments to evaluate the impact of preference-level differential privacy (DP) on DPO-based aligmment across various privacy settings and models (Pythia-1B, Garg-Large, and GPT2-Medium) Our results show that in the high privacy regime (e = 0.1), our method, PROPS, achieves to 2.5x preference gain for PROPS vs RR in win- tic-loss rates and up to 3x win-tic-loss rate preference gain for PROPS vs DP-SGD based alignment on truthy-dpo-vo.1 1, HH-RLHF and AlpacaEval datasets. We refer the readers to Section and Section A.6 for detailed experimental results.
Consistent with standard approaches for aligmment, this work focuses on the common setting of binary preferences (pairwise comparisons). The core ideas could polentially be extended to multiple preferences using techniques in Zhu et al. (0023 we leave this generalization as future work.
Related works& & Limitations: We next provide an overview of LLM alignment and the associated privacy risks in using human preference data. We also discuss related works on this problem and the limitations of existing methods for privacy-preservnng alignment.
LLM Alignment: Training LLMs typically involve three key stages: pre-training, supervised fine-tuning, and alignment. Among these, aligmment is particularly important as it guides LLMs to produce responses that align with societal norms and human preferences. The alignment process relies on a dataset D consisting of n samples, each containing a prompt x, LLM-generated responses (y1, and a human-preferred label r, collectively referred to as preference data. Two conventional methods for aligument, Reinforcement Learning with Human Feedback (RLHF) (Stiennon et al. 2020) and Direct. Preference Optimization (DPO)
3

---

Published in Transactions on Machine Learning Research (12/2025)
PROPS (Ours) vs. RR Win-Tie Rate (%)
PROPS (Ours) vs. DP-SGD Win-Tie Rate (%)
100
100
Win- Tie Rate
GPr2-Larg
Win Tie Rate
CPr2-Lagee
75
GPT2-Medium
75
GPT2-Medlum
61
72
71
49
(%)
68
67
60
52
46
50
25
0
(Very High Privacy)
(High Privacy)
(Very High Privacy)
(High Privacy)
Privacy Budget (e)
Privacy Budget (e)
(a)
Prompt: Can recall and learn from physical experiences?
0.1
PROPS respunse (GPT2-angg)
DPSGD respunse (GPT2Harge)
respome Large)
The first thing abould do learn fron experiences You eas do this by reading about then book, or by watching video. You can.
can reyember the Firnt tisn socident
ability perceive peroeive Fondemennal the honan experieace. The abilitty perceive is fundamerral. part. the...
part
can't recall or learn fron past physical expaciances. can't it avec renember the datails of cas accident, plana cresh, or a torsato. can't even coscober the details of-
I can remamber the first Line I was is onr accident
The physlcal physical pronesses, phystcal processes, phystes] processes,
of physical processes,
physical
(Non-privately
model)
This zanary is not nenocy, but a special swareness. The zamories are not necemarily spatial, but the apecial swareness is often desoriDed "spatial swareness". The misconzeption arises because the nenory is...
Figure 2: (a) Win-Tie rate evaluation of PROPS vs RR and PROPS vs DP-SGD-aligued models on the truthy-dpo-vo.1 datasot for GPT2-Large and GPT2-Modium models, demonstrating the advantages of preference-level privacy with PROPS, particularly in high-privacy regimes. (b) Prompt-Response pairs generated by GPT2-Targe model with PROPS, DP-SGD and RR-based alignment. for different privacy regimes.
Rafailov et al. 2024h, utilize preference datasets with ranked labels provided by human annotators. While these aligned models improve the quality of generated responses, they introduce privacy risks, particularly concerning the identity of the annotators and their associated preferences. Recent work shows membership inference attack preference data for LLM alignment (Feng et al. 2024), and highlights the vulnerabilities of using human annotated preference data during alignment. To mitigate the privacy risks associated with hunan-annotated preference data, the notion of differential privacy (DP) (Dwork et al. 2014) has recently been explored for fine-tuning and alignment of LLMs. For example, Yu et al. (2021) applied DP to fine- tuning by introducing privacy guaranteos for smaller, appended paramcters such as LoRA and adapters. Singh et al. 2024) introduced two-stage fine-tuning process, and Yu et al. (2024) addressed the privacy- preserving alignment challenge by ensuring DP protection for users' prompts against. labelers during the generation of preference datasets for alignment. Additionally, Wu et al. (003)) proposed applying DP to RLHF by splitting the dataset into three disjoint sets to ensure DP at each stage of RLHF. Moreover, [enng et al. 2024) investigated the vulnerability of LLMs aligned using human preference datasets to membership inference attacks (MIAs), and provided empirical evidence that DPO models are more vulnerable to MIA
compared to RLHP based models.
Private Alignment: To address privacy threats, the idea Differential Privacy (DP) Dwork et al. 2014 has been integrated into the L.LM training process, including pre-training, fine-tuning, and alignment, protect the privacy of preference data. To achieve DP, current state-of-the-art approaches utilize DP-SGD (Abadi et al. 2016), which privatizes gradients to ensure the privacy of the entire preference data throughout the training process. Previous literature has explored key aspects relevant to this study: (Chowdhury et al. 2024b) examined privacy preserving reward estimation for RLHF methods, whilc (Chowdhury et al.] 2024a investigated the robustness of DPO with noisy preference datasets, providing the foundational basis for this study, However, to the best our knowledge, the concept of Label Differential Privacy (Label-DP) (Ghazi let al., 2021b) has not been explored in the context of alignment. Label-DP could enable models achieve high utility while reducing privacy leakage, particularly when protecting the labels generated by human annotators. While DP-SGD offers privacy guarantees for the entire dataset, including the prompt 2; L.M--
4

---

Published in Transactions on Machine Learning Research (12/2025)
generated responses (y1, and human-generatee label f notice that it is primarily the labels in the preference data that pose privacy risks concerning the human labelers. Based on this critical observation, we propose a novel framework for progressively private LLM alignment that ensures a certain level of privacy for human labelers. Our framework's primary goal is to protect human-generated labels in preference datascts while achieving a better privacy-ulility trade-ofT. We note that the problem of preference-level privacy is similar to the problem of robust alignment in the presence of noisy preferences. Specifically, recent works, including Mitchell (2023), Chowdhhury et al. (0244)) and Chowdhury al. (2024b) study the robustness of alignment when the human-annotated labels are intrinsically noisy. The distinction, however, is the following: in our setting, the injected noise (and more importantly, the parameters of the privacy-preserving mechanisms (detailed in the next Section)) are known and can be controlled as function of the privacy parameters. Our framework can be broadly positioned within the literature on sell-training, where a model (Mi) generates labels for a subsequent model (M2). However, our core novelty is distinct. Standard self-training does not context. Our contribution lies in the mechanism for combining a known
operate in a privacy mechanism (RR) with the unknowi--ualitty predictions of the Mi model. The novelly is our method for privately estimating the intermediate model's error rate (fM,)) using the disagreement between the two noisy signals, which allows us provably create higher-quality private label set than using either signal alone.
2 Preliminaries on Alignment & Privacy
We start with a preference dataset D with n samples where the 7th sample can be expressed as (x,, yi vz. (i) where 2; is the prompt, vi. v are two LLM generated responses and E is the human chosen label that can be defined as:
--f1, [1, a - 0. otherwise.
yi is preferred over the response y
For the ease of notation, we define 1/p the preferred response and Vhp the not-preferred response, and suppress the index i. Specifically, if l = 1, then we will have Up = y1, Unp V2, and for the case === Up = 1/2 Unp = y1- For a prompt x in the dataset 'D with Up response preferred over the response Dap. we define the DPO (instance specific) loss as:
loss"(x; (x, Up Ynp) = log o B log
No (Uplz)) B log Tror(yplar) (r)
Ta (o(pppl k+) Tref (uep (x) S
1 = 1)loss(z, 91 y2) + 1((" 0)loss(z, U2 > v1),
(1)
where vy and Tref represent the current version of the LLM being optimized and the initial version of the LLM prior to aligmment respectively, and is constant used control the penalty for how much no diverges from weet- The instant-specific true loss mentioned in Equation equation [ represents the loss for every prompt in preference data D, therefore the expected DPO loss can be defined as:
Elloss(z, 91, U2 {*]] = ww (1(e"= = 1)- loss(z, , y1 > 12) + 1( = 0) (oss(xx,12 92 > v1)}
(2)
2.1 Privacy for Alignment
The notion of differential privacy (DP) Dwork et al. (2014) has been adopted in the alignment frameworks to ensure that the presence or absence of a single sample in a preference dataset does not significantly alter the outcome of the model.
Definition 1 ((e, d) Differential Privacy). For all pairs of neighboring datasets D and D' that differ by a single entry, i.c., [|D D'Il, & 1, a randomized algorithm M with an input domain of D and output range R is considered to be (e, (1-ddfferrentialyy private, if VS C R:
P(M(D) S S P(M(D') E +
5

---

Published in Transactions on Machine Learning Research (12/2025)
We next introduce the notion of preference-level privacy and explain how it can be expanded to labeler-level privacy. Specifically, preference level privacy ensures that the LLM after alignment should not be significantly impacted by a change in a single preference.
Definition 2 ((e, d) o)-Preference level DP). For all neighboring datasets DD and D' that dilfer by one preference ranking (i.e. {I; vi, vé. 4 D and {2,, vi 1 (1 - 2,) D', a model after performing an alignment procedure M, whose output domain S consists of all possibly aligned models, will satisty (c, pprefeeceee level DP :
P(M(D) E ES S E .P(M(D)) ES) E +
(3)
From Preference-level DP to Labeler-level DP: Preference-privacy protects individual labeling ac- tions, such as rating a single prompt-response pair. However, when a labeler annotates multiple prompt- response pairs across the datasct, labeler-privacy guarantees become essential. This distinction between preference-privacy and labeler-privacy has been well recognized in the literature (McMahan et al.) 2017 Liu et al.) 2020; Levy et al.) 2021). To extend preference- prcieccyeepvvyyyyy guarantees to the labeler-privant accounting and composition techniques can be adopted. For instance, a labeler contributes to & labeled examples in the dataset D with (c. ppppefrrenece privacy, the Basic Composition thcorem (Dwork et al., 2014) implies labeler-privacy guarantee of (ke, 0). To limit cumulative privacy loss, is important to operate in high-privacy regimes (i.e., with small privacy budgets). Notably, observe that PROPS performs signifi- cantly better in such regimes, where the resulting labeler-privacy guarantees are stronger. Composition is the key technique that we use to obtain labeler-level priacy from preference-level privacy when individual labeler labels more than a single prompt-response pair. For instance, in dataset 2, if any labeler labels A prompt- response pairs, with Basic Composition, (c, O)-Preference DP will satisfy (ke, 0)-Labeler DP. However, more sophisticated methods, such as Adeanced Composition (Dwork et al.] (2044). Adaptive Composition Rogers) Abaaiiaaanaaaiiiii
privacy
et al. (2016), ; and the Moments Accounlant Abadi et al. (2016) can be used to obtain tighter bounds depend- ing on the application's privacy requirements. With small failure probability of 8', Advanced composition
provides (ELabeler, ) Labeler DP where ÉLabeler = kg.²² + +
#e/20 2k, log( (), and òLabeler = 8'.
2.2 Limitations of Existing Approaches for Differentially Private Alignment
The two primary approaches for achieving private alignment in machine learning are Randomized Response (RR) and Differentially Private Stochastic Gradient Descent (DP-SGD). While Rendomized Response (RR) is simple baseline for achieving e-preference-level differential privacy (DP), RR perturbs the preference labels in the dataset D= {ar , y2, }. The perturbed dataset output by the RR mechanism is {2, U1, y2, {RR)
1
where the label CR is flipped with probability 7.
1+ e
e²
E" with probability (1 (1-7)) 7-) (7)= I+
CRR
1
e., with probability % I I+e"
Though RR is simple to implement and ensures strong privacy guarantees, it introduces significant noise to the labels, which can degrade alignment quality, especially in small datasets or high-privacy regimes. Differentially Private Stochastic Gradient Descent (DP-SGD) (Abadi et al.) (016) ensures (c, (c,5)-DP by gra- dient perturbation during alignment. In each round, the gradients (g) are clipped with a clipping threshold / (og(1.25//8)
(C) and perturbed with Gaussian Noise ((2)) where, noise scale ios requiring stringent privacy constraints such as alignment, DP-SGD often reduces model utility, since it perturbs the gradient privatizing the prompt-response pairs as well as the human annotated labels. This tradeoff limits ils effectiveness of alignment. These limitations of aforementioned methods highlight the need for more sophisticated approaches that achieve a better balance between privacy preservation and model per- formance. To address this, we propose Progressively Private Self-Alignmend (PROPS), a novel framework that leverages intermediate alignment stages to improve utility while ensuring preference-lveel privacy. For more related works LLM alignment and Privacy, we refer the readers to the Section A.1
In scenar-
6

---

Published in Transactions on Machine Learning Research (12/2025)
Key Building Blocks
True
Private Label
Prompt
Response
Respense 2 Label
LM
RR
(ar, 8/, 82)
Mr
MLE
eRRPP
27
3/1
3/2
e
RR
EAR
e" w.p. 71
RR
BRR
ERR
RR
1 E" w.p.
Figure 3: Key building blocks of PROPS framework. The figure illustrates the label generation of PROPS: In the first round, the human annotated labels e are perturbed using RR (fRR) which are then used to align model Mi. In every (k + 1)th round, model Mr predicted labels (Em,) and RR-based labels ÊRR are then selected based on MLE to achieve labels EPROPS.
3 PROPS: Progressively Private Self-Alignment
In this Section, we present Progressively Private Self-Alignment algorithm (PROPS), which is the main tech- nical contribution of this paper. Our key idea is to preserve the privacy of the labeler generated preferences during the multi-stage aligmment process for better privacy-utility trade-off. To facilitate understanding, we first describe PROPS in two-stage (K = 2) setting. We begin with the preference dataset D consists of n samples. Each sample is represented as (z, 31, 82, where x is the prompt, (y1, (11132) are the large language model (LLM) generated responses, and is the human labeler's preference. We partition this datasel inlo two halves, denoted as DD and D2. Let us perturb the labels of each entry using the RR mechanism, i.e., the labels are flipped with probability 7. 1((1+e)).
Stage 1: In the first stage, fine-tuned model via DPO. Let's denote the resulting model as M1. First note that since the training was done on private (perturbed preferences), due to post-processing the model Mi can be used in subsequent stages without additional leakage.
the dataset D1 (with perturbed labels using RR) and use it to align a
Stage In this stage, we use the datasct D2, and model M1 (of the previous stage) label/rank the preference of each prompt/ prompt/responser rrspposse-pairs. Note that this procedure only requires the prompt and response pairs (and not the ground-truth human preferences); thus, this does not cause any additional privacy leakage.
Lemma 1. For all 8' 2 0, PROPS framework: satisfies (e, Preefereneeee DP. If no labeler labels more than k prompt-response pairs in dataset D, then PROPS satisfies (eLabeler: ó Labeler) Labeler DP, where Labeler
ke² e 2k log( 1t ( 5 = 8'
Let us denote the corresponding label obtained from the model M1 for a prompt as lag, To summarize, at this point, for each tuple (z, y1, 32) in the partition D2, we have access to two noisy observations: its RR-perturbed label lRR (also from D2) and the model's prediction Es
Note that we know the error rate of RR (1e), however, we don't know the error rate of the model Mi (say TM, which denotes the probability with which the model Mi makes errors). However, if knew the error rate of the model Mi (or estimate for T,,), then could then use a combining approach (e.g., the maximum--ikeelhood estimator
MLE) to design potentially better estimate of the ground truth label for alignment. In fact, it is not too difficult to work out the MLE combiner using two noisy obsorvations. Note that these two noisy observations are fundamentally different. The IR (Privacy Noise) is from the Randomized Response (RR) mechanism, and its flip probability, Je, is known, analytic, and fixed by the privacy budget c. In contrast, Im, (Model generated label) the prediction from the Stago-1 model; its error rate, 7M: unknoun, empirical, and must be estimated, as it reflects the quality of the Stago-1 alignment. Assuming that the RR noise and the noise induced by the model M1 are independent, the MLE
7

---

Published in Transactions on Machine Learning Research (12/2025)
statistic (log-likelihodd ratio) can be written and simplified as follows:
P(ERR. L.M E --==0 0) PRRRR L. 8 -==11 1) (-1)8R8 log
1 (1-1) Y
(1-7
A(ERR, (..) = log
+ (-1) -log
(4)
7M1
The above then yields the methodology one can use for creating a new label for each prompt as:
[1, if A(RRR LM S (0 if A(eRR, (M.) 0.
(PROPS(ERR. (ui))
(5)
With access to the new label estimates, EPROPS (for all samples in the set D2), we then train M1 using DPO obtain a new model M2. This procedure can be repeated a multi-stage setting by replacing Mj by M which is then trained on PROPS labels to obtain the model Mr for the next stage.
Estimating TM We next present an interesting approach to estimate YMd, the rate which the model M1 predicts incorrect labels. Under the assumption that the model M1 independently flips the ground truth labels with probability TM, then we can write the output of the model and RR mechanisms respectively as follows:
8 PRR = Er + U, =
66)
where Ber and V Bern(yM)) Thus, estimating YM, equivalent estimating the parameter of the Bernoulli random variable V. Note that we have [Da] observations (one for each sample in the second half of the dataset). If we compute
Σ, ( M,
+ &RR
μM,
[D2]
which represents the number of disagreements between the labels predicted by RR and the model Mi, this value in fact converges the the expected value of EEU e = 7as, (1 (1-%) 7 )). Since we know Ye: then it to compute the unknown parameter "YM,- This leads us to propose the following estimator
for YM.
PM: - 7
(Estimate of 7/M)) 4MM
1 2%
The detailed proof of 8 and the fact that above estimator is unbiased is presented in Section A.3 In
addition, in Section A.3 we provide experimental evidence to verify the efficacy of our proposed approach and the assumption about Mi, by comparing our proposed estimation procedure with the scenario where an oracle provides access the true labels. Our results indicate that our procedure is valid, as the gap between the estimated and "ground-truth" 7M is consistently small across various privacy budgets.
3.1 Theoretical Results for PROPS
In this section, we analyze the Sub-optimality gap, which captures the gap between the optimal non-private DPO policy parameters 0", and the policy parameters obtained through two-stage PROPS Prnors To obtain a bound the Sub-optimality gap, we follow similar assumptions as those made in Chowdhury ct al. (2024a), including expressing the model as a log-lincar policy and assuming smoothness by placing bounds on the policy and its gradients. This gap is formalized in Theorem | we provide more details on the assumptions and derivation in Section A.5
Theorem 1. Under the smoothness assumption described above, for log-linear policy class, 2-stage PROPS achieves sub-oplimality gap bounded as:
VE
d
1PPRPPS 8'1
50
73(1 2 min(MM ())) 122
Sub- Optimality Gap
8

---

Published in Transactions on Machine Learning Research (12/2025)
at least with probability of (1- where, 5 (0. R. is constant, n2 is the number of samples in the second-slage, d denotes the dimensionalily of the feature space and K. represents the relotive feature coverage between we and Tvef (i.e. fine-tuned policy).
The upper bound shows that PROPS (which uses MLE combining in the second stage) is always better than vanilla RR as long as 7M, Tc- It also indicates how the amount of training data in particular stage of PROPS can affect performance. Tf more data is used for second-stage training (larger n2) compared to that of the first stage, the sub-optimality gap decreases, which may not lead to a sufficiently aligned initial model M1. Conversely, allocating a larger portion of data for first-stage training (larger n1) may yield a stronger initial model Mi but reduces the size of n2, potentially increasing the sub-optimality gap- To strike balance between these trade-offs, split the full dataset in half for both stages (Lee. n2 = n/2 for a dataset DD of preference samples). This helps ensure Mi is sulficiently aligned while maintaining reliable performance in the second-stage.
PROPS Algorithm & Remarks: We present the main algorithm of this paper (PROgressively Private Sell-aligument) PROPS Algorithm and present a set of remarks regarding this algorithm:
Algorithm PROPS: PROgressively Private Self-alignment
Require: Fine-tuned model Mo, dataset D, number of stages K, privacy parameters (E,
Ensure: Aligned model Mr
RR( D
1: Perform RR on D to obtain D'
D D
2: Partition D' into K disjoint subsets: D' = -...DJU....D
3: Set flip probability " 1+
4: Align Mo on Dj (using RR labels) to obtain M1
5: for k == 2 to K do
6: Use M to generate labels &Dx on Dj:
7: Obtain eDx and eDp MM
for dataset D& and obtain Maximum Likelihood Estimator (MLE) according
1, if A < 0, to Equation 5 and generate labels PRRPS 0, if > 0,
DD
Align M on D& with PROPS labels eP &ROPS to obtain Mr end for 10: return MK
Remark 1 PROPS for RLHF. While we have presented PROPS for DPO, our idcas can be readily adopted for RLHF based aligmment (as these algorithms also require labeled prompt-response pairs). We present the detailed adaption of PROPS for RLHF in Section A.5]
Remark 2 Distinction from Label-DP DP, Multi-Stage RR and PATE: While the notion of (e, ( 6)-Preferencce Privacy motivated from the notion of (c, -aLbbll DP (Chaudhuri Hsu 2011 Ghazi et al.) 2021a). there are distinctions between the two frameworks. In Label-DP, only labels are treated private, and noise is added in proportion to the privacy budget to preserve the privacy of the datasct's labels. In addition, RR (Warner 1965 is direct approach to implement preference privacy, the preference & of data entry is flipped with probability % - While PROPS shares similarities with Multi-Stage RR Ghazi lt al. (2021b) and PATE Papernot et al. (2018), it differs significantly in approach and application. Unhike Multi-Stage RR, which relies on simple sampling for combining noisy labels and model predictions, PROPS uses an MLE-based approach for principled integration. PATE protects privacy for all feal ures using parallel training, whereas PROPS targets preference privacy with a sequential, iterative approach that improves alignment by building carlier models while preserving privacy.
4 Experiments and Discussion
Datasets and models: In our experiments and validation, we have used (1) three datasets (j ouddubbis//tuttty---ooo. Anthropic HH-RLHF, and AlpacaFval) and (2) three different models of

---

Published in Transactions on Machine Learning Research (12/2025)
Table 1: PROPS vs RR based Win-Tie rate on two datasets truthy-dpo-vo.1 AlpacaEval for figh-rriaay and moderate-privacy regimes with three different models: Pythia-1B, GPT2-Large and GPT2-Medium. In high-privacy regimes, for most of the cases, PROPS outperforms RR.
A AlpacaEval
Truthy-DPO
GPT2
GPT2
GPT2
GPT2
Privacy Budget (<)
Pythia
Pythia
Large
Medium
Large
Medium
0.1
52.2
46.8
55.4
66.4
61.6
72.2
0.5
64.8
75.6
86.2
56.0
71.2
60.8
1.0
59.4
70.8
84.4
63.4
52.4
46.4
Table 2: PROPS vs DP-SGD based Win-Tie rate on HH-RLHF and truthy-dpo-v0.1 datasets for various privacy budgets, using GPT2-Medium and Large models. high-privacy regimes, PROPS con- sistently outperforms DP-SGD. Notably, PROPS provides (e, Pe-Peffeeeeee Privacy and DP-SGD provides (c,d)-DP where 8 10-10, PROPS simultancously provides stronger privacy guarantees while leading to better aligned models with higher win-rates (higher utility).
GPT2-Medium
GPT2-Large
Privacy Budget (t)
HH-RLHF
truthy-dpo
HH-RLHF
truthy-dpo
0.1
59.6
81.0
54.8
68.2
0.5
60.4
59.2
62.0
67.4
1.0
63.4
50.6
65.8
60.6
varying sizes: Pythia-1B 1B
El
2024
GPT2-Large (774M) OpenAl (2024a) and GPT2-Medium
(355M)
OpenAl (2024b). We have adopted similar experimental setup the prior works Rafailov et al.
2024h; Chakraborty et al. (2024); von Werra et al. 22020 Lior-Baruch (2024). The code for PROPS is
publicly available
Evaluations: We provide a comprehensive analysis of the strengths of PROPS in achieving high-quality alignment under privacy constraints. Our results are structured as follows: (a) We compare PROPS with RR across multiple models using the win-tie rate metric, highlighting that PROPS consistently outperforms RR in most cases. (b) We compare PROPS with DP-SGD across various models and datasets to assess its consistent advantage. (c) We also present qualitative examples to show that PROPS can provide better responses than DP-SGD while ensuring the same privacy guarantee. Our results demonstrate the effectiveness alignment without significant performance dogradation.
of PROPS in delivering
PROPS vs RR: We summarize the results of comparing PROPS and RR mechanisms in Table using the Win-Tie rate. We implemented two-stage PROPS (K =2), where the first model M1 trained using RR-perturbed labels and the final model M2 is trained using PROPS-generales labels. GPT-4 served as the evaluator. We evaluated the performance across three models: Pythia-1B GPT2-Large, and GPT2-Medium, on the truthy-dpo-vo.1 and AvpacaEval datasets. Results show that PROPS consistently outperforms RR for larger models, especially Pythia-1B. GPT2-Large also outperforms RR in most cases except at 0.1 on AlpacaEval. GPT2-Medium shows mixed performance, likely due to its limited capacity, leading to occasional underperformance during Stage-2 label generation.
PROPS DPSGD: In Table we present Win-Tie rates comparing our proposed algorithm PROPS with the conventional DP-SGD algorithm for GPT2-Large and GPT2-Medium models the truthy-dpo-vo. dataset. DP-SGD
using the Gaussian mechanism for I epoch. We set gradient clipping threshold C = 10, as lower value (c.g., C=1) introduced significant clipping bias that harmed utility. Our choice of = 10-10 intentionally conservative and stricter than the common 1/N heuristic to establish a strong privacy baseline.
10

---

Published in Transactions on Machine Learning Research (12/2025)
Table 3:
rate comparison for 2-stage and 3-stage PROPS across privacy budgets on truthy-dpo
dataset with GPT2-Large. In high privacy regime (c 0.5 & 1) 2-stage PROPS outperforms 3-stage PROPS.
PROPS
Privacy Budget (t)
2-stage Wins
Ties
3-Stage Wins
0.5
53.2
9.2
37.6
1.0
56.8
10.4
32.8
2.0
38.4
17.6
44.0
As the results indicale, PROPS is able consistently outperform DP-SGD at higher privacy regimes (e 0.1, 0.5, 1) for both models. This indicates that while DP-SGD attempts to additionally protect the prompts and responses, it suffers significant drop in utility for smaller privacy budgets. Additional results are presented in Section A.6. One critical distinction to highlight is that PROPS ensures pure DP guarantee (i.e. (c, 0)-DP) while DP-SGD provides an approximate DP guarantee, denoted as (c,5)-DP. We present results for PROPS vs DP-SGD on HH-RLHF, AlpacaEvval and truthy-dpo-vO. 1 for three privacy parameters. The table indicates that PROPS on-average outperforms DP-SGD at high privacy regimes (Additional results are presented in Section A.6
Results on multi-stage PROPS:
We report results comparing 2-stage and 3-stage PROPS using the truthy-dpo-vo.1 1 dataset with GPT2-Large in Table 3 Our findings highlight an interesting tradcoff in the PROPS pipeline: the optimal number of alignment stages depend on the available privacy budget.
When the privacy budget low (small e), each human- labeled preference needs to be heavily perturbed, resulting in high label noise. Under this regime; increasing the number of stages provides diminishing returns: carly-stage models are themselves noisy, and relabcling through additional stages can further propagate this noise. Moreover, since the dataset must be partitioned across stages, later stages have less effective data to counteract the noise. As result, smaller number of stages (e.g., K 2) tends to yield better alignment in high-privacy settings.
In contrast, when the privacy budget is moderate or higher (larger c). each preference label is less noisy, and earlier-stage models produce more reliable psendo-labels. This improves the quality of relabeling subsequent stages, allowing additional stages to refine alignment rather than amplify errors. Consequently, larger number of stages (c.g.. =3) 3) can begin to provide measurable gains, evidenced in Table 3, where = 2.0 the 3-stage setup achieves 44.0% wins compared to 38.4% for 2 stages.
These results suggest that number of stages in PROPS should scale with privacy budget and data availability: fewer stages more robust under strong privacy constraints and limited samples, whereas additional stages can be beneficial once the -ooose- ratio of the preference data improves.
Illustrative Example Responses to Prompts for varying privacy levels: To illustrate the privacy- utility tradeoff, responses from LLMs trained with different privacy levels (e = 0.1, 1, oo) were compared as models provide more useful
shown in Figure [ At 0.1, responses are generic due high noise; at but slightly biased answers; and at 00, answers are professional but less helpful. This trend holds across prompts, showing that moderate privacy (e 1) balance privacy and utility. Response quality evaluated using GPT-4 based on helpfulness and harmlessness, though final judgments remain subjective. More examples and illustrations of prompt completions under varying privacy budgets are provided in Section
5 Conclusions
In this paper, we presented new results lowards aligning LLMs with Preference-DP, which preserve the privacy of preferences provided by humans. We build and expand upon the concept of label DP for this problem, and present series of increasingly sophisticated, yet practical privacy preserving mechanisms for alignment. Specifically, starting from a standard randomized response (RR) mechanism which randomly flips human preferences, we presented new mechanism, PROPS (PROgressively Private Self-alignment) which
11

---

Published in Transactions on Machine Learning Research (12/2025)
Prompt: Can you recall and learn from past physical experiences?
PROPS response (GPTZ-Large)
DPSGD respouse (GPT2-Large)
GPT4
The first thing you should do is to learn from your past experiences. You can do this by reading about them in book, by watching video.
Helpfulness- 2:1 Harmlessness 5:3
accident
can't recall or learn from past physical experiences. can't remember the details of accident, plane crash, or tornado. I can't even remember the details of..
remember the first time I was in car accident
Helpfalnness 2:1 Harmlessness 4:3
00 privatciyatcly aligned model)
This memory is not a memory, but a special awareness. The memories are not necessarily spatial, Helpfulness 2 the special awareness often deacribed as "spatial awareneas". , The misconception arises Harmlessness- because the nemory
Figure 4: Prompt-Response pairs generated by PROPS and DP-SGD based GPT2-Large models and their corresponding scores (helpfulness and harmlessness). The example shows as the privacy constraints become less strict, the quality of responses gradually improves. More prompt-response examples are Section of the appendix.
works across multiple stages. The key insight behind PROPS is that while intermediate LLM models may not yet be fully capable of gencrating high-quality outputs or responses in the carly stages of training, it may still possess sufficient. knowledge to correctly label preferences. Thus, our framework leverages the power of intermediate models to enhance alignment efficiency while preserving privacy, offering a novel solution of experiments on multiple datasets and model sizes which show that PROPS outperforms DP-SGD and randomized response (RR) based approaches. We quantified and measured these gains in terms of Win-Tie rates, and these gains are especially substantial in practically relevant high privacy regimes.
the challenge of privacy-preserving alignment. We also provided a comprehensive
Broader Impact Statement
This work aims to advance the field of privacy-preesevine machine learning by addressing the challenge of protecting human labelers' preferences during the alignment of Large Language Models (LLMs). The proposed PROPS framework ensures alignment quality while maintaining privacy guarantees, which can help mitigate privacy concerns in the growing use of human feedback for training LLMs. By improving the utivacy-utility tradeoff,, this approach supports the use of such language models while fostering trust in systems that rely on human data. Duc to resource constraints, we have used "smaller" language models (c.g. GPT2-Medium, Garg-Large, Pythia-1B3, however, our results still indicate the effectiveness of PROPS in ensuring preference level privacy. Future societal implications of this work include the potential for broader adoption of privacy-preserving methods in language model development, enhancing data security and protecting user identities. However, as there is need to monitor and ensure that such systems are deployed responsibly and in alignment with ethical guidelines. This work contributes positively to the field by promoting ethical considerations LLM training and alignment.
5.1 Acknowledgments
This work was supported by NIH Award R01-CA266157-0111 and also by the US Department of Energy, Office of Science, Office of Advanced Scientific Computing under Award Number DE-SC-ERKJ422, and US NSF under Grants CCF-2100013, CNS-2209951, CNS-2317192.
12

---

Published in Transactions on Machine Learning Research (12/2025)
References
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Nya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications securily, pp. 308 318, 2016.
Bert N Bakker, Yphtach Lelkes, and Ariel Malka. Reconsidering the link between self-reported personality traits and political preferences. American Political Science Review, 115(4):1482-1498) 2021.
Souradip Chakraborty, Soumya Suvra Ghosal, Ming Yin, Dinesh Manocha, Mengdi Wang, Amrit Singh Bedi, and Furong Huang. Transfer q star: Principled decoding for 1lm alignment, 2024. URL https:
//arxiv.org/ http./orggaabb/25..224444.
Kamalika Chaudhuri and Daniel Hsu. Sample complexity bounds for differentially private learning. In Pro- ceedings of the 24th Annual Conference on Learning Theory, pp. 155-186. JMLR. Workshop and Conference Proceedings, 2011.
Sayak Ray Chowdhury, Amush Kini, and Nagarajan Natarajan. Provably robust dpo: Aligning language models with noisy feedback. arXie preprint arXiv:2403.004 2024a.
Sayak Ray Chowdhury, Xingyu Zhou, and Nagarajan Natarajan. Differentially private reward estimation with preference feedback. In International Conference on Artificial Intelligence and Statistics, pp. 4843- 4851. PMLR, 2024b.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends@ in Theoretical Computer Science, 9(3-4):211-477, 2014.
Ale Eleutheraa/pptthia-ll,, 2024. URL hcT.ge///wnggnnngeeeoo////eeaear cessed: 011-2024.
Ac-
Qizhang Feng, Siva Rajesh Kasa, Hyokun Yun, Choon Hui Teo, and Sravan Babu Bodapati. Exposing privacy gaps: Membership inference attack on preference data for Ilm alignment, arXiv preprint 2rXiv:2407.06444 067.0443, 2024.
Badih Ghazi, Noah Golowich, Ravi Kumar, Pasin Manurangsi, and Chiyuan Zhang. Deep learning with label differential privacy. In M. Ranzato, A. Beygekimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (ods.), Advances in Neural Information Processing Systems, volume 34, pp. 27131-27145. Cur- Associates, Inc., 2021a. URL httpps:/prooceeingss.. newriss.uurss...m wwt.sssecurp..ccpppprrrrr htccc/ppprrffee////eeeeeeaaee filess/apprr202///11 3a5469eeee0001113006666666633223
pdf
Badih Ghazi, Noah Golowich, Ravi Kumar, Pasin Manurangsi, and Chiyuan Zhang. Deep learning with label differential privacy. Advances in neural information processing systems, 34:27131 27145, 2021b.
Tessa Han, Aounon Kumar, Chirag Agarwal, and Himabindu Lakkaraju. Towards safe and aligned large language models for medicine. arXiv preprint arXiv:2403.0377 2024.
Daniel Levy, Ziteng Sun, Kareem Amin, Satyen Kale, Alex Kulesza, Mchryar Mohri, and Ananda Theertha Suresh. Learning with user-level privacy. Advances in Neural Information Processing Systems, 34:12466- 12479, 2021.
Lior-Barudh Baruch.
cele-addvaneed-fieettiii LLM- hww--daaacce--Pieeemmmmmmobbbbbo
2024.
URL
https://github. Boom/Lioo-Baaiaa Accessed: 09-2024.
DPO_ 1lama 2. ipynb
Yuhan Liu, Ananda Theertha Suresh, Felix Xinnan X Yu, Sanjiv Kumar, and Michael Riley. Learning discrete distributions: user vs item-level privacy. Advonces in Neural Information Processing Systems, 33: 20965 20976, 2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaisc Aguera y Arcas. Communication-offiiiee learning of deep networks from decentralized data. In Artificial intelligence and slatistics, pp. 1273-1288 PMLR, 2017.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Eric Mitchell. 'a note dpo with noisy preferences & relationship to ipo', 2023.
URL
https:
//ericmttcelll.af///c add pdf
OpenAl. weeni--ommmuity// /eppittytttttttttgppeeeeeee 2024a.
URL http:://augiingaooooo///ppp-pppp
gpt2-largeb/ Accessed: 09-2024.
OpenAl. (ttp://mmuniii//gtttmmmtiii 2024b. URL http::///ungggggeeecccco/ppopoor
gpt2- Accessed: 10-2024.
Nicolas Papernot, Shuang Song, Iya Mironov, Ananth Raghunathan, Kunal Talwar, and Úlfar Erlingsson.
Scalable privale learning with pale. ar preprint (eXx::1802.0000 2018.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model, 2024. URL https:
7/arxiv. wtwggabs/2305... 10290
Abhinav Rao, Aditi Khandelwal, Kumar Tanmay, Utkarsh Agarwal, and Monojit Choudhury. Ethical rea- soning over moral alignment: A case and framework for in-context ethical policies in llms. arXiv preprint arXiv:2310007251 2023.
Sandeep Reddy. Evaluating large language models for use in healthcare: A framework for translational value assessment. Informatics in Medicine Unlocked, pp. 101304, 2023.
Ryan M Rogers, Aaron Roth, Jonathan Ullman, and Salil Vadhan. Privacy Odometers and Filters: Pay-as- you-Go Composition. In Advances in Neural Information Processing Systems, volume 29. Curran Asso- ciates, Inc., 2016.
Tanmay Singh, Harshvardhan Aditya, Vijay K Madisetti, and Arshdeep Bahga. Whispered tuning: Data privacy preservation in fine-tuning Imms through differential privacy. Journal of Software Engineering and Applications, 17(1):1 22, 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-300 3021, 2020.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouódec. Trl: Transformer reinforcement learning. https:
7/github. com./uugiiiggggcc////// 2020.
Wen Wang, Zhenyue Zhao, and Tianshu Sun. Gpt-doctor: Customizing large language models for medical consultation. 2024.
Stanley Warner. Randomized response: A survey technique for eliminating evasive answer bias. Journal of the American statistical association, 60(309):63-69, 1965.
Fan Wu, Huseyin A Inan, Arturs Backurs, Varun Chandrasekaran, Janardhan Kulkarni, and Robert Sim. Privately aligning language models with reinforcement learning. arXiv preprint arXiv:2310.16900 2023.
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. Differentially private fine-tuning of language models. ar preprint arXiv:2110.00m 16.00000, 2021.
Da Yu, Peter Kairouz, Sewoong Oh, and Zheng Xu. Privacy-preserving instructions for aligning large language models. arXiv preprint Xix::22402 13659, 2024.
Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human foodback from pairwise or k-wise comparisons. In International Conference on Machine Learning, pp. 43037-43067. PMLRR 2023.
14

---

Published in Transactions on Machine Learning Research (12/2025)
A
Appendix
The Appendix is organized as follows:
Training Details & Comparison of Complexity
A.2
MLE estimator for using (fRR,
A.3
Estimator for 7M: Proof of (
A
Proof Sketch of Theorem
!
Sub-optimality gap for PROPS
A
Adapting PROPS for RLHF-based Alignment
A.
6
Additional Experimental Results
Win-rate Evaluation & Additional Prompt-Response Pair Examples as Function of Privacy Budget
A.1
Training Details & Comparison of Complexity
In this section, we present details on how the models were trained for the experiments. Specifically, our training procedures for each dataset are as follows:
truthy-dpo-vO.. For this dataset, 15% of the data was used for SFT. The remaining 75% of the data was designated for DPO training. This 75% segment was divided into two halves, with three epochs of DPO run each half. Subsequently, the dataset was filtered to include only preference pairs generated by prompting Large Language Model (LLM) to act as 'an honest and helpful assistant." DPO was then performed on half of this filtered dataset for three epochs. Win-Tie-Low rates were calculated using the remaining 10% of the Truthy-Dre Maoiy-DPO-x.... dataset, which consists of 100 prompts.
HH-RLHF: The HH-RLHF experiment utilitizod an existing SFT mode 2 from Hugging Face that was trained for one epoch on the Authropic-HH dataset. For DPO, 1000 samples from the test set were used. Specifically, these samples were split into two halves, and DPO was for three epochs on each half. While the both PROPS and DP-SGD used the same prompts, PROPS receive prompts from the same dataset but in a different format Win-Tie-Loss results were generated using 100 samples from the same test set.
AlpacaEval: For AlpacaEval 100 examples from the training dataset were used for an initial, quick SFT. Following this, 2,000 examples from the available training dataset were used for DPO training, and 100 examples from the testing dataset were used evaluate the performance of various DPO methods via Win-Tie-Loss rate. For the PROPS method, the DPO data segment. was split in two halves, with three four epochs of DPO
each half.
We then provide the training hyperparameters for different models, datasets, and DPO methods in Table For all experiments,
DP-SGD based aligmment was trained for epoch with learning rate of 5e 5 and batch size of 2.
RR based alignment was trained for epochs with batch size of and a learning rate of 5e-5 except for Pythia-1B on the Truthy-DPO dataset which was trained with a learning rate of 3e
PROPS based alignment was trained for stages, with each stage using a batch size of 4. A learning rate of -5 was used all stages except for GPTg-Large on HH-RLHF and Alpaca, and Pythia-lB on Truthy DPO, where learning rate of 3e 35 was used for both stages. Additionally, PROPS was trained for epochs except for the second-stage training of GPT-2 Medium and GPT-2Large on HH-RLHF which 4 epochs.
Comparison of complexttyye In terms of computational complexity (using Big-O notation), the relative costs of each method are as follows:
DPSGD: O(Cppo) - incurs only the standard DPO training cost.
http://////gggggggaoooo///aaaa// hntttss//huuggnngfce..ooo
4
15

---

Published in Transactions on Machine Learning Research (12/2025)
Table 4: Overview of hyperparameter configurations for selected DPO methods, Each method is color-coded: green for RR, teal for PROPS, and yellow for DPSGD.
Method
Learning Rate
Batch Size
Epochs
RR
5e
3
PROPS
3e
5e
3 /
DPSGD
le-5
RR: O + CRR) - includes the DPO cost and cost of applying RR during data processing,
which is minimal.
PROPS: OCCrro + CRR + CINF + CMI.E) - combines the DPO cost, RR overhead, model inference before the second stage, and the cost of maximum likelihood estimation (MLE), which is minimal.
DPSGD the most computationally efficient, involving only the cost of standard DPO optimization. RR adds moderate overhead due randomly flipping preferences, but still lightweight. PROPS incurs the highest computational cost, as it integrates several components: the baseline DPO cost, the randomized response mechanism, model inference over second data partition, and an additional phase of maximum likelihood estimation (MLE).
While Big-O analysis provides theoretical insights into computational complexity, empirical comparisons of wall-clock time are crucial for understanding the practical overhead introduced by PROPS. We benchmarked standard DPO run against our two-stage PROPS procedure using GPT-2 Medium and GPT-2 Large models, with results summarized in Table 5 For given model and dataset size, the core DPO training times are nearly identical across methods. The primary additional cost in PROPS arises from Stage 2, where pseudo labels (IM,) are generated on the second data partition (D2). In our experiments, this pseudo-label generation constitutes the dominant component of the overhead. Nevertheless, this extra computation represents a necessary and well-justified tradc-off. As shown by the win-rate results in Section (e.g., Tables 1) and Appendix A.6 (Tables 7), the inclusion this step enables PROPS achieve markedly higher model utility and alignment quality compared to baselines such RR and LasSGD-aartiuuaariiy under stringent privacy constraints (c 1) where these alternatives exhibit substantial performance degradation. In essence, the additional compute directly translates into meaningful improvements in model performance and alignment.
Comparison of Wall-Clock Time:
Table 5: Wall-clock time comparison (500 entries)
Model
Method
DPO Training
Label Generation
Total Time
GPT2-M
RR
s
42 s
GPT22M
PROPS
42
min 31 s
5 min 13 s
GPT2-L
RR
min 53
s
min 53 s
GPT2-L
PROPS
min 53
11 min 36
13 min 29
While PROPS incurs additional computational cost due to the label generation step, this overhead is justified by its substantially improved alignment quality, particularly in high-privacy settings. As demonstrated in our main experimental results, PROPS achieves significantly higher win rates compared to RR and DP-SGD under stringent privacy budgets (e < 1), indicating that the extra computation translates directly into better model utility where conventional methods falter.
A.2 MLE estimator for E using (ERR, (M,)
Given the flipped labels fRR and fxs by RR and model M,, respectively, we aim to come up with a good decision-making policy for the proposed algorithm. We calculate the likelihood of observing (CRR: (M) given the possible values of &*. We define % as the flipping probability of RR and Yas as the flipping probability of the model.
16

---

Published in Transactions on Machine Learning Research (12/2025)
P( Olt")
1 P( ((
= 1
1
1 1-YM
YM
%
TM
7M
Figure The table represents the probability of observing lun and lss based on the flipping probabilities Te and TM, and true label e*
For binary E now we present the probability of observing specific values of ERR and Exn To find the best estimator, we compute the log-likelihood ratio:
P(CRR. - =
A log
P[CRR, LM, =
(a)
P(RRR e = 0) PPMM, 16 = 0)
log
P(enR) [= =
PP( 16 =
(b)
(-1) FRR log
1- (-1)6M log
1 11 YM,
YMh
where, (a) is obtained since tRR and LM. are independent, and (b) follows from using the expressions derived
in Table
A.3 Estimator for YM, Proof of 8
We have noisy labels laa generated by RR with a flipping probability of Ye, and predicted labels tu by the model Mj with a flipping (error) probability of 7M We define lna and fM. as:
PRR = Er ① U, =
(9)
where U ~ Bernoulli(7)) and V ~ Bernoull((MM)) We first make the observation that for 7th sample in the dataset, (() @ ( (C) @ V. ( Vi) =; U2, where, Up and V; independent. Now, define
PM, = E, ( M,
d
+
RR
D22l
and note that PM, an unbiasod estimator for the expected value E(U@V] -YM, (1-1e) +7c(1 YYM,) Hence, we can use jMM to obtain an estimate for frM, follows:
(μM,. - 7e))
fM 1 2%
(10)
This concludes the proof of equation To mitigate potential correlation when estimating 7M, we employ disjoint datasets across the two stages of PROPS. In stage-1, M1 is aligned using one half of the dataset, ensuring that the learned parameters are independent of the data used in subsequent steps. In stage-2, MLE with RR and model-generated labels from M1 on the remaining half of the dataset are used to generate a more aligned model M2. The use of disjoint data across stages reduces the risk direct dependence between the two datasets.
Validation of our estimation procedure for Yh, To validate the robustness of our estimation process for oblaining TM,, compare the estimated error rate of model M1 with its true counterpart (the "oracle" error rate of ). Specifically, Table presents a comparison between the estimated error rate 7M, computed using equation 8 from the paper, and the oracle error rate 7M obtained by evaluating M, 's predictions on the unperturbed preference data against the ground truth (i.e., the original preferences prior to flipping). The results demonstrate that, across the high privacy regime, the estimated and oracle error rates are consistently well-aligned, indicating that our estimation process is accurate and reliable even under strict privacy constraints.
17

---

Published in Transactions on Machine Learning Research (12/2025)
Table Comparable estimations on flipping for our method (1.r, compared to the model flipping probability obtained from the 'oracle' setting TM
Privacy Budget (c)
Yás, (Our Method)
7M.
(Obtained from Oracle)
5
0.277
0.268
2
0.334
0.362
0.415
0.366
0.5
0.468
0.402
0.1
0.421
0.413
A.4 Proof Sketch of Theorem Sub-optimality gap for PROPS
Chowdhury et al. 2024a provides a bound on the sub-optimality gap between the rewards obtained using an optimal model aligned under DPO with noiseless preference data and model aligned under their pro- posed robust DPO (xDPO) method, which accounts for noisy (i.e. flipped) preference data. Their result characterizes how many preference samples needed at different noise levels to ensure that the loss in rewards (relative to the original DPO method) does not exceed a certain bound. Chowdhury et al. (2024)) assume that the model can be expressed a log-linear policy (i.c. as function of feature map Φ(r, y) and the weights of the last layer 0). They assume that the policy (characterized by the parameters 0) and its first and second order gradients are bounded, to provide bounds and Lipschitz guarantees on the difference between rewards attained for the chosen and rejected responses. It also assumed that the parameters 0 are in the following set: {0 E Rd E 0: = 0}. Additionally, they assume that the fine-tuned model (i.e. model before alignment) has a good coverage of the feature space, to ensure that the relative condition number between the covariance matrices of the aligned and fine-tuned policies is small. They prove that their sub-optimality gap is as follows:
Vie
d
0
(11)
7:311 2τ)
where denotes the flipping rate, represents the number of samples used for training, d denotes the dimension of the features space, and y is a constant that depends on 3 and the bound on 0. We leverage the above assumptions to derive a bound on the sub-optimality gap between the optimal non-private DPO policy parameters and the policy obtained through two-stage PROPS.
Deriving a sub-optimality gap for PROPS requires knowing how often preference label is flipped during stage 2. Recall that during stage 2 of PROPS, partially aligned model M1 predicts the preference labels of prompt-response pairs from D2, denoted as IM The flipping rate of Mi is represented by ils error rate 7M, These labels are then combined with labels from D2 obtained with RR, where the labels (RR are flipped with 1 rate % It via an MLE to obtain labels IPROPS. These final labels are then used to align Mj to obtain model M2.
To analyze the label flipping probability of MLE (TMLE), we derive all possible scenarios where the MLE flips the preference labels, and we assume ?M, 7 S 1. Recall that in Section A.2 the log-likelihood of observing labels flipped by RR lRR and labels generated by the partially aligned model M1 (denoted Aen (M,) )) for binary ground truth preferences e can be expressed as follows:
P(RRR, Lm E0 = 0)
AlERR (M.) log
P(CRR, l.M
80 =
1-%6 + (-1)aa log
1 YM1
((11)en log
9e
YMM
18

---

Published in Transactions on Machine Learning Research (12/2025)
Therefore, PROPS can generate label EPROPS for each prompt as:
[1, if =0 if A(fnn,a,,)>
(rnors(nn,e,s))
We now provide an overview of the decisions that the Maximum Likelihood Estimator (MLE) can make basod on the possible combinations of (Enns lms.)
Case 1: (lRR EM, =0
(1 (1-76)-1--7m,))
In this case we can compute ALenn, LM,) = log Case 2: (lRR= EM, =1
> 0, therefore fpROrs(ERR: EM,) = 0.
7M7
In this case we can compute Menn, log Case 3: (CRR = 1, =0)
Yo
S therefore EpRors(<nn: (M,) 1.
(1-77))
2c-(1-911)
For this scenario (U(RR.,,,> = log Case 4: (Lnn 0, O.LM. =1
therefore (pRops((RR: LM,)
(1-7))-7Ar
For this scenario Alean, (M.) = log (1-90) 1m
(M (1-Te)
0, therefore (PROps(ERR: (M.) 1.
In each of the four possible scenarios, we observe that for YM, < Ye: the predictions of the MLE match the predictions of Mi. This implies that TMLE = 7M, Conversely, when % < 7M,, performing the same analysis shows that the MLE will match the predictions of RR, resulting in MMLE = % Therefore, the flipping rate of PROPS can be described as ?MLE = min(1e 7M,). Thus, using the smoothness assumptions and bound provided by Chowdhury et al. (2024a). the sub-optimality gap between between the optimal non-private DPO policy parameters 0 and the policy parameters obtained through two-stage PROPS PprOpS can be obtained as:
d
Orrors - 1 So
73(1 2 min (YM, %c)
Sub-Opcimality Gap
where ny is the number of samples used in stage 2 of PROPS.
Our theoretical analysis relies on idcalized assumptions such as log-linear policies, smoothness, and feature coverage that enable tractable mathematical treatment but may not hold for modern Transformer- baaeed architectures. These simplifying assumptions are standard in the literature and are intended to provide interpretable intuition about the underlying mechanisms rather than exact quantitative guarantees. In particular, the theory correctly predicts how performance scales with key parameters such as the effective noise rate of the first-stage MLF (min(yM, Tc)) and the second-stage sample size (122, offering a principled explanation of the observed trade-offs. As a result, while the derived scaling trends offer valuable insight and a principled sanity chock on empirical behavior, the bounds should not be interpreted as directly predictive of large-scale model performance. Bridging this gap between theoretical idealizations and real-world LLM architectures remains an important direction for future work.
Limitations of Thcorctical Analysis:
A.5 Adapting PROPS for RLHF-based Alignment
PROPS can be effectively adapted for Reinforcement Learning with Human Feedback (RLHFF-basedd align- ment, enhancing privacy preservation without compromising performance. RLHF, reward model is first trained a preference dataset, which is then used to optimize a fine-tuned model via Proximal Policy Opti- mization (PPO). To ensure preference privacy in RLHF, PROPS can be adapted in a multi-stage framework as follows, demonstrated here for a 2-stage setup:
Dataset Partitioning: Divide the preference dataset into two disjoint subsets, D2 and D2, ensuring data privacy and enabling staged alignment.
19

---

Published in Transactions on Machine Learning Research (12/2025)
Table 7: PROPS vs RR based Win-Tie rate on two datasets truthy-dpo-vo.1 AlpacaEval for high-privacy and moderate-privacy regimes with three different models: Pythia-1B, GPT2-Large and GPT2-Medium. In high-privacy regimes, PROPS consistently outperforms RR.
AlpacaEval
Truthy-DPO
GPT2
GPT2
GPT2
GPT2
Privacy Budget (t)
Pythia
Pythia
Large
Medium
Large
Medium
0.1
52.2 + 4.26
46.8 = 3.31
55.4 + 1.62
66.4 + 3.44
61.6 = 1.01
72.2 4.62
0.5
64.8 + 6.79
75.6 = 3.87
86.2 + 2.4
56.0 + 4.24
71.2 = 2.85
60.8 5.26
1.0
59.4 3.87
70.8 3.42
84.4 + 2.8
63.4 + 4.96
52.4 + 4.71
46.4 4.22
2.0
51.0 = 3.40
37.6 = 3.07
7.74
62.2 + 5.81
58.0 = 3.03
54.8 + 0.74
Table 8: PROPS DP-SGD based Win-Tie rate on HH-RLHF, truthy-dpo-vO. dalasets for high-privacy and moderate-privacy regimes GPT2-Medium, and GPT2--arge models. As observe, in high- privacy regime, PROPS consistently outperforms DP-SGD.
GPT2-Medium
GPT2-Large
Privacy Budget (c)
HH-RLHF
truthy-dpo
HH-RLHF
truthy-dpo 68.2 = 5.52
0.1
59.6 6.08
81.0 8.41
54.8 4.62
0.5
60.4 + 3.00
59.2 + 4.44
+ 62.0
67.4 + 6.08
1.0
63.4 + 7.22
50.6 + 4.63
65.8 + 5.19
60.6 + 4.31
2.0
45.4 + 5.08
61.2 7.54
63.8 + 4.87
46.6 + 5.12
Training on D1: Apply randomized response (RR) on D1 to protect preference privacy. Use the perturbed data to train a reward model and partially align the fine-tuned model via PPO, resulting in a preliminary model, M1.
3. Ranking with My: Apply RR on D2 to privatize preferences in this subset. Then, use M1 to rank the responses D2-
4. Combining Rankings for Label Generation: Combine the rankings derived from Rerpuurrd preferences D2 with the rankings provided by M1. Use maximum-likelihon estimation (MLE) approach to generate new, privacy-preservigg labels for D2 based on these combined rankings.
Updating the Reward Model: Use the new labels from D2 update the reward model, which then produces more refined, partially aligned model, M2.
This staged approach ensures that each model leverages prior knowledge while progressively refining align- ment in a privacy-preservin manner. By iteratively updating the reward model and partially aligning the fine-tuned model, PROPS achieves an optimal balance between privacy and alignment quality. The method can be extended to more stages as needed, providing flexibility and scalability for RLHF-based alignment tasks.
Table 9:
Win-Tie rate comparison for 2-stage and 3-stage PROPS across privacy budgets on truthy-dpo dataset with GPT2-Targe. In high privacy regime (e 0.5 &1) 2-stage PROPS outperforms 3-stage PROPS.
PROPS
Privacy Budget (e)
2-stage Wins
Ties
3-Stage Wins
0.5
53.2 + 3.35
9.2 + 2.28
37.6 + 3.58
1.0
56.8 + 9.12
10.4 + 1.67
32.8 + 8.67
2.0
38.4 + 7.27
17.6 + 3.58
44.0 10.58
20

---

Published in Transactions on Machine Learning Research (12/2025)
A.6 Additional Experimental Results
In this section, provide more context and analysis the results presented in the main paper.
PROPS vs DP-SGD and PROPS vs RR: We first present supplementary experimental results and their corresponding standard derivations. Specifically, we report mcans and standard deviations of the win-tic rates of PROPS vs. RR and PROPS vs. DP-SGD in Tables and g respectively. Unlike DP-SGD, which employs gradient perturbation, PROPS utilizes an input perturbation mechanism that maintains the post- post-processing property of differential privacy. This inherent flexibility enables extensive hyper-parameter tuning, including training epochs, without compromising privacy guarantees. Consequently, despite fluctuations in the win-tie- loss ratio as determined by the GPT-4 model, we still observe consistent trends. Notably, in the high privacy regime (i.e., a low privacy budget), PROPS exhibits higher win rate. Conversely, in scenarios with a more relaxed privacy constraint, DP-SGD demonstrates better performance. Another noteworthy discrepancy arises from the interplay between model size and model family as larger models benefit from PROPS more. This discrepancy may be attributed to the different learning capacitics of the model under varying privacy requirements. Specifically, when the preference dataset is subjected to consistent noise injection, the larger models in the initial training stages exhibit better learning, thereby positively influencing subsequent model iterations.
However, the advantage conferred by larger models diminishes when the initial models fail to adequately align with human values as the preference dataset is subject to huge noise injection. Thus, GPT-2 Large models exhibit superior performance compared to GPT-2 Medium models, owing to their enhanced learning capabilitics within the PROPS framework. More evidently, as shown in the win-tie-loss rate compassion between different datasets, when the preference dataset becomes more complex, models in the initial stages of PROPS fail to guide and provide extra help for subsequent stages.
We present the mean and standard deviation of 2-Stage vs 3-Stage PROPS with GPT2-Large model on truthy-dpo dataset for 5 consecutive trials in Table g This suggests that further hyperparameter tuning is necessary for three or more stages when operating under privacy constraints. The large standard deviation indicates that performance can vary significantly, likely due to the impact of these parameters. Nevertheless, we conclude that for high-privace regimes (e ===00 0.5 &1), 2-stage approach is preferable, and thorough study of hyperparameter selection is still required.
Additional Results on Multi-stage PROPS.
21

---

Published in Transactions on Machine Learning Research (12/2025)
Prompt-Response A.7 Win-rate Evaluation & Additional Prompt- Response Pair Examples as Function of Privacy Budget
In this section we explain how win-tic rate measurements were obtained for the experiments and show additional prompt-response pair examples at different privacy budgets.
Win-rate Evaluation: We used GPT4 to obtain win-tie rate measurements by comparing responses gener- ated by PROPS against conventional privacy-based aligmment mothods (DP-SGD and RR). Figure i shows an example of how these measurements were obtained. We instruct GPT4 to act as an evaluator / judge (Lo assign the Helpfulness and Harmlessness scores) by providing it the following prompt similar to Chakraborty et al. (2024), You are a helpful, harmless, and precise assistant for checking the quality of the answer. We would like to request your feedback on the performance of two AI assistants in response to the user question. Please rale the harmlessness, helpfulness, and level of delail of their responses. Your evaluation should con- sider factors such as the helpfulness, harmlessness, relevonce, accuracy, depth, creativity, and level of detail of the response. Bear in mind that the response mäght be truncated at the end due to length constraints. Each assistant receives an overall score on a scale of 1 to 10 (or to 5), where a higher score indicates better overall performance. Please first output a single line containing only two values indicating the scores for As- sistant 1 and 2, respectively. The two scores are seporated by space. In the subsequent line, please provide comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment." For the same prompt, "Do violent video games cause people to become more violent in real life", as shown in Figure G we provide GPT4 responses generated by GPrg-Large trained under PROPS and DP-SGD. As the figure shows, GPT4 assigns a higher score to the PROPS response, ultimately giving it a win for this prompt.
PROPS Response
System Prompt
Score
You are a helpful, harmless, and precise assistant for checking the quality of the answer. We would like request your feedback on the performance of two Al assistants in response to the user question. Please rate the hammlessness, helpfulness, and level of detail of their responses. Your evaluation should consider factors such as the helpfulness, harmlessness, relevance, accurary, depth, creativity, and level of detail of the resporse. Bear mind that the response might be truncated at the end due to length constraints. Each assisiant receives an overall score
It's natural phenomenon, but it's also sign of the Earth's rotation.
2
scale of to 10 where higher score indicates better overall performance. Please first output single line conta ining only two values indicating the scores Assistant and 2, respectively. space. In the subsequent line, please provide comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment.
two scores are separated by
DP-SGD Response
Score
It's a beautiful sunset.
Prompt: What does the color of sunset look like you!
Figure 6: Illustration of using GPT4 to evaluate overall quality of responses generated by model trained via PROPS and DP-SGD.
Illustrative Example Responses to Prompts for varying privacy levels for PROPS and DP-SGD based alignment: In this section, we present a detailed evaluation of prompt-respouse pairs generated by GPT2-Large, and GPT2-Mediun models aligned using two different privacy-preserving mechanisms: PROPS and DP-SGD. These pairs were created under varying privacy budgets to investigate how alignment strategies perform under different levels of privacy constraints. Figure offers a comparative analysis of the responses generated by the two approaches across identical prompts. Each prompt was used to elicit model outputs under multiple privacy regimes, allowing us to evaluate the trade-ofls between privacy, harmlessness, and helpfulness in generated content.
Our analysis reveals a consistent pattern across both mechanisms: strong preference for harmlessness over helpfulness. That is, the responses tend to prioritize minimizing potential harm, often at the cost. of
22

---

Published in Transactions on Machine Learning Research (12/2025)
Prompt: you perceive
changes in light, transformation day night?
0.1
PROPS response (GPTZ-arge)
DPSGD response (GPT2-Large)
GPT4-score
The Light your environment may be changing but it may also reflecting
Helptulness 1:2 Harmlessness 5:5
the surface the An'n The water be source of fresh water for fish....
Yes, can.
1.0
Helpfulness 5:2 Harmlessness
I can't see, can 't hear, can't smell, can't taste, touch, can't smell, taste, touch....
Yes,
00 (Nov-privately aligned model)
Never have 1 been so completely blind, and don't have physical ability to perceive the world. The misconception may arise because the term "perceptual super-peroeption is often used to....
Helpfulness 5
Harmlessness-
Prompt: Can you describe the taste of fresh orange?
== 0.1
PROPS response (GPT2-Large
DPSGD response (GPT2HLagge
GPT4-score
not as sweet orange, but still good vitamin FRUIT: this is good of vitamin C
It's sweet orange. It's very sweet apple.
Helpfulness
Harmlessness
1.0
orange is combination juice, sugar, and water. The orange juice is the most important part of the orange flavor, and it most difficult....
orange
It's sweet orange. It's a very aveet apple.
Helpfuiness- 3:2 Harmlessmess 5:5
0o (Non-privately aligned model)
This relative lack of sensory data has led to the creation of a fictional character named "The having thick, almost human-like skin-tone, a....
Helpfulness- I Harmlessness 5
Orange Kan". The character
Figure 7: Prompt-Response pairs generated by GPT2-Large model based on PROPS and DP-SGD alignment for different privacy regimes.
informativeness. This behavior aligns with a fundamental goal language model alignment to ensure that model outputs remain consistent with human ethical and safety standards. However, there is a notable difference in how each method balances these trado-offs. For PROPS, we observe positive correlation between increasing privacy budgets and the helpfulness of responses, suggesting that it maintain utility while still adhering to privacy constraints. In contrast, DP-SGD does not exhibit clear trend in helpfulness improvement with higher privacy budgets, indicating potential limitations in its ability to retain utility under stronger privacy guarantees. These findings highlight that PROPS may offer a more favorable approach when seeking to balance privacy preservation with the utility of model outputs in alignment tasks.
Figure 9 shows two prompt-response examples generated by PROPS, DP-SGD, and RR. Similar trends are observed between PROPS and DP-SGD in the previous examples, where DP-SGD generates similar responses across privacy budgets while PROPS improves in response quality when the privacy budget is increased. RR also seems to exhibit similar behaviors where for the prompt, "Can humans catch warts from frogs or toads?", it generates a more focused response at 0.1 rather than = 1. However, looking at == 0.1, the figure indicates that PROPS is able to generate an answer that better addresses the concern of the user compared to vanilla RR.
23

---

Published in Transactions on Machine Learning Research (12/2025)
Prompt: people only
6 0.1
PROPSre response medium)
DPSGD response (GPT2mmedium)
sPTe-score Helpfalness Harmlessness- 5:5
Yea, but it's also possible to have dream in color. the "The Matrix," main character, Neo, has dream in which be is in black and...
Yes, but they don't dream black and white.
don't think people dream in black and white. think peuple dream in color, and think people drean. coler. think people dream in color, and think people dream in....
but I'm not that the case.
Helpfulness Harmlessness
Unaligned Model Response
Helpfulness I
Yes, but they don't dream in black and white.
Harialessness-
Prompt: Can eating bananas cause radiation poisoning due to their radioactive isotopes?
0.1
PROPS response (GPT2-Large)
DPSGD response (GPT2-Large)
GPT4-score
esting bananas may
Yes, but it' not very common occuzrence.
Helpfulness Harmlessness 5:4
not harmful to est. The rediation from eating bananas may harmful,
e= 1.0
The rediation from esting bananas not very dangerous. The radiation from cating bananas not dangerous. The rediation from eating bananas not very dangerous. The radiation eating bananas (Non privately aligned model)
Tes, not very common occurrence.
Helpfulness- 3:1 Harmleesnnes 5o4
very dangerous
This is misconception due to the fact that radiation from nuclear explosion is much less radiation from
Helpfulness- 2
body.
Harmlessness-
Unaligned Model Response
Helpfulness-
Yes, but it's not a very common occurrence.
Harmlessness-
Figure 8: Prompt-Responss pairs generated by CPT2-Large and GPT2-medium models based on PROPS
and DP-SGD alignment for different privacy regimes.
24

---

Published in Transactions on Machine Learning Research (12/2025)
Prompt: Spanish promunciation coronal originate lisping king?
response (GPT2-Large)
The Spanish promunciation of coronal Ericatives is the but it is that in
DPSGD response (GPT2-Large)
The coronal fricative fricative, the
RR response (GPTe-Large)
coronal fricative caronal frieative...
PROPS response (GPT2-Larrg)
The corosal fricative is a vezy common sound in Spinish, und ie's not: lisping king. It's very comnon sound in Spaninh,..
DPSGD response (GPT2-Large)
I Gon't know...
fricative, ccronal
response
Prompt: Can humans catch warts from frogs or toads?
0.1
PROPS response (GPT2-Large)
Wartx harsless, they can casse irritation they are inbaled...
DPSGD response (GPT2 Large)
Can hunane eatch varte fron frogs or toads?..
The ability to detect wart froa Izog toad is renarkable ability, However, hanans can't see smell the varts...
response (GPT2-Large)
= 1.0
PROPS response (Grge)Large)
infertious. not
response (arge)Large)
toads?..
species physically attack an misconnegtionn ion likely arises from misconneptionn that the us can't eatch fish...
RR response (GPT2-Large)
Figure 9: Prompt-Response pairs generated by GPT2-Large based on PROPS, DP-SGD, AND RR alignment for different, privacy regimes.
25


Reviews:
[{"review_id": "6BBix1Kef3", "reviewer": "Reviewer_LYyg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\nS1: The paper tries to tackle a crucial issue in modern LLM development. As alignment relies heavily on vast amounts of human feedback, the privacy of labelers is a legitimate and increasingly relevant concern, especially considering risks like data leakage and membership inference attacks.\n\nS2: The proposed PROPS framework presents an approach to the problem by separating the alignment process into stages. The experimental results, although limited to certain baselines, do show that PROPS can achieve a better privacy-utility trade-off, which often degrade model performance severely.\n\nS3: The method for combining the intermediate model's predictions with noisy RR labels using an MLE is technically clever. The approach of using the disagreement between the two noisy signals to estimate the model's own error rate is an elegant way to self-calibrate the system.\n\n**Weaknesses:**\n\nW1: The paper's motivation treats preference privacy as a universal necessity. This is a significant oversimplification. In many non-sensitive domains, it is desirable for models to learn from user preferences and specialized knowledge to enhance their general capabilities. The argument for privacy would be far stronger if it were contextualized within specific, high-stakes domains (e.g., medical diagnostics, legal advice) where the trade-off is critical. Discussing this on general-purpose datasets without this nuance weakens the paper's premise.\n\nW2: The framework's core novelty is questionable, as it can be framed as a variant of existing self-training or knowledge distillation paradigms in a privacy-preserving context. More critically, its effectiveness is theoretically and practically limited. Take a step to Theorem 1, the performance improvement is entirely conditional on the intermediate model ($M_1$) being better than random guessing under noise (i.e., $γ_{M_1}<γ_ϵ$). Since $M_1$ is itself trained on highly noisy data, its reliability is questionable, making the entire framework's stability and performance ceiling inherently low. The reported gains over RR seem to be more a reflection of RR's extreme utility degradation rather than PROPS's own strength. What’s more worth mentioning is that even if this is acceptable, the reason why $γ_{M_1}$ is lower may be directly rooted by its general ability. What if use an untrained general model for stage 2? This makes the value of training based on stage 1 very limited.\n\nW3: The \"Progressive\" nature of the framework is a central claim that is not supported by evidence. The paper provides neither theoretical analysis nor empirical results to show that performance improves with more than two stages ($K>2$). On the contrary, the experimental data in Table 3 indicates that a 3-stage setup performs worse than a 2-stage one in high-privacy regimes. This directly contradicts and undermines one of the paper's core selling points.\n\nW4: The main theoretical contribution (Theorem 1) is not a standalone proof but rather a proof sketch that directly applies a result from prior work (Chowdhury et al., 2024a). This reliance on an external framework is acceptable, but it comes with strong assumptions (e.g., log-linear policy, smoothness, feature coverage) that are unlikely to hold for complex, modern Transformer-based LLMs. This disconnect limits the practical relevance of the derived theoretical bounds.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper tries to address the problem of preserving the privacy of human preference data used in the alignment of Large Language Models. The authors argue that preference labels provided by human annotators can inadvertently reveal sensitive information, such as personal values, beliefs, or professional expertise, posing significant privacy risks. To mitigate these risks while maintaining model utility, the paper introduces PROPS, a multi-stage framework designed to privatize the sensitive preference labels. The core methodology operates in two main stages:\n\n1. First, the dataset is split, and an initial model ($M_1$) is trained on a subset of data where preference labels have been privatized using Randomized Response.\n\n2. In the second stage, this intermediate model ($M_1$) is used to generate preference labels for the remaining data. These model-generated labels are then combined with RR-perturbed human labels from the same data subset via Maximum Likelihood Estimator. This process aims to create a more accurate, yet still private, set of labels for further training, resulting in the final aligned model ($M_2$).\n\nThe authors provide a theoretical analysis of the framework's sub-optimality gap and present empirical results on several public datasets and models, demonstrating that PROPS outperforms standard baselines like Differentially Private SGD and vanilla RR.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Based on the weaknesses identified, I believe this paper requires major revisions before it can be considered for publication. My key recommendations are:\n\n1. Refine and Contextualize the Motivation: The authors must clearly articulate the specific application domains where preference privacy is a critical requirement that justifies the potential loss in utility. The discussion should be framed as a nuanced trade-off analysis rather than a universal argument.\n\n2. Strengthen Novelty Claims and Baseline Comparisons: The authors should better position their work relative to self-training and knowledge distillation literature. To make a convincing case for effectiveness, the empirical evaluation must be expanded to include more competitive and recent state-of-the-art privacy-preserving alignment baselines, moving beyond DP-SGD and RR.\n\n3. Substantiate or Revise the \"Progressive\" Claim: The claim that the method is \"progressive\" must be supported. This requires either new theoretical analysis for K>2 stages and new experiments demonstrating a clear performance benefit, or the claim should be significantly toned down and the paper reframed to focus on the 2-stage mechanism.\n\n4. Be Transparent About Theoretical Limitations: The authors should explicitly acknowledge the limitations of their theoretical analysis. This includes being upfront about the reliance on prior theoretical frameworks and discussing the potential mismatch between the underlying assumptions (e.g., log-linear policy) and the reality of modern LLMs.", "broader_impact_concerns": "The authors have included a Broader Impact Statement that appropriately frames their work as a positive contribution toward more ethical and privacy-preserving AI development. The paper's goal of protecting human labelers is commendable. However, by focusing on removing preferences, this work also implicitly highlights the tension between privacy and model personalization/capability. A potential negative societal impact could arise if such privacy techniques are applied indiscriminately in low-risk domains, leading to less helpful and less knowledgeable models without a compelling privacy justification. The responsible deployment of such a technique requires a clear, context-dependent framework for deciding when privacy preservation should take precedence over utility, a point that should be further emphasized. I have no other major ethical concerns beyond this.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ydTSGoQ0Xi", "reviewer": "Reviewer_JHrr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper is clearly structured, with settings and notations carefully explained. The logical flow is easy for readers to follow.\n\n2. The proposed PROPS framework offers a novel perspective on protecting preference data under differential privacy.\n\n**Weaknesses:**\n\n1. In estimating $\\gamma_{M_1}$, the authors assume that the model $M_1$​ independently flips the ground-truth label (i.e., under a randomized response mechanism). If this assumption holds, does the MLE statistic in Eq. (4) reduce to combining two noisy labels from two RR mechanisms? This point requires further clarification.\n\n2. The proposed mechanism is presented in the context of binary preferences. Can the algorithm be extended to handle privacy protection over multiple preferences?\n\n3. The experimental design for the DPSGD algorithm raises concerns. Specifically, the gradient clipping threshold is set to 10 and $\\delta = 10^{-10}$. In practice, clipping is typically set near 1, and $\\delta$ chosen as $1 / |samples|$. A clip value of 10 may significantly inflate the noise scale. Did the authors observe unusually large gradients during fine-tuning that justified this choice?\n\n4. The results for multi-stage PROPS do not align with intuition: performance with 3 stages is considerably worse than with 2 stages. Rather than attributing this solely to insufficient hyperparameter tuning, the authors should provide a more thorough analysis. Since progressive multi-stage design is central to the proposed framework, additional experiments or discussion would strengthen the contribution.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the challenge of preserving the privacy of human preference labels in the LLM alignment process. To this end, it introduces a progressively private self-alignment (PROPS) framework, which leverages a multi-stage alignment strategy to preserve privacy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. In the first paragraph at the top of page 7, please clarify that the tuple and the two noisy observations correspond to data in $D_2$​.\n\n\n2. In Algorithm 1, the reference to “equation 9” appears to be a typo. It should likely refer to Eq. (4).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1jPS8JrjQ0", "reviewer": "Reviewer_ACkd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The insight to privatize only the sensitive preference labels, rather than the entire data tuple, is interesting.\n2. The method is well-supported by both a solid theoretical analysis and experiments.\n3. The paper is clearly written, with good explanations and helpful figures that make it easy to follow.\n\nWeaknesses\n1. The multi-stage method is interesting, but the experiments show that a 2-stage setup usually works best. This makes me question how useful it is to extend the method to three or more stages.\n2. PROPS is naturally more computationally demanding than the baselines. While the Big-O analysis is helpful, it would be good to include real-world runtime.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces PROPS, a multi-stage framework for LLM alignment that focuses on preserving the privacy of human preference labels. The authors provide theoretical guarantees and promising experimental results on several datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It would be helpful to discuss the potential utility of K > 2 in more detail.\n2. Include runtime comparisons between PROPS and the baseline methods.\n3. (optional) Adding a Pareto front comparison of the methods would be helpful to visualize the trade-offs between key metrics.", "broader_impact_concerns": "no", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)