Privately Aligning Language Models with Reinforcement Learning
Authors: Fan Wu
Huseyin A Inan
Arturs Backurs
Varun Chandrasekaran
Janardhan Kulkarni
Robert Sim
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Models
RLHF
Alignment
Differential Privacy
Primary area: societal considerations including fairness, safety, privacy

Abstract: Positioned between pre-training and user deployment, aligning large language models (LLMs) through reinforcement learning (RL) has emerged as a prevailing strategy for training instruction following-models such as ChatGPT. In this work, we initiate the study of privacy-preserving alignment of LLMs through Differential Privacy (DP) in conjunction with RL. Following the influential work of Ziegler et al. (2020), we study two dominant paradigms: (i) alignment via RL without human in the loop (e.g., positive review generation) and (ii) alignment via RL from human feedback (RLHF) (e.g., summarization in a human-preferred way). We give a new DP framework to achieve alignment via RL, and prove its correctness. Our experimental results validate the effectiveness of our approach, offering competitive utility while ensuring strong privacy protections.

Published as a conference paper at ICLR 2024
PRIVATELY ALIGNING LANGUAGE MODELS WITH RE- INFORCEMENT LEARNING
Fan Wu': Huseyin A. Inan', Arturs Backurs',
Varun Chandrasekaran' Janardhan Kulkarni', Robert Sim University of Illinois Urbana- Chaaanaan,,m,nn,, 2 M365 Research," Microsoft Research {fanwe, varunc)eillinoii edu, (huseyin. inan, arturs. backurs, r rsinglsssiiemmmrroofffttt.
ABSTRACT
Positioned between pre-training and user deployment, aligning large language models (LLMs) through reinforcement learning (RI. has emerged as a prevail- ing strategy for training instruction following-moder such as ChatGPT. In this work, we initiate the study of privacy-preserving alignment of L.LMs through Dif- ferential Privacy (DP) in conjunction with RL. Following the influential work of Ziegler et al. (2020), we study two dominant paradigms: (i) alignment via RL without human in the loop (e.g., positive review generation) and (ii) alignment via RL from human feedback (RLHF) (e.g.. summarization in human- human-preferee preferred way). We give a new DP framework to achieve alignment via RL, and prove its correctness. Our experimental results validate the effectiveness of our approach, offering competitive utility while ensuring strong privacy protections.
INTRODUCTION
Over the past few months, Large Language Models (LLMs) that are capable of following open- ended user instructions such as ChatGPT, Bard, Llama Chat, have seen an euphoric adoption by application developers. Similar to their predecessors, these models are pre-trained on vast amounts of public internet data. However, their magical ability to follow myriad user instructions driving force behind their mass adoption has been attributed to instruction fine-tuning and learn- ing from human feedback. This new step involves collecting a dataset human preferences and feedback, followed by fine-tuning the model via reinforcement learning (RL) make them better aligned, often abbreviated as RLHE. Since the influential works of Ziegler et al. (2020); Ouyang al. (2022); Bai al. (2022), the RLHF framework has emerged the dominant paradigm for models.
training instruction-followwe
At the heart of this training pipeline pre-training followed by RLHF is the realization that while pre-training helps LLMs to acquire the world knowledge, it is the RLHF stage that makes LLMS learn interact with users, and hence present their knowledge in human-preferred way. This framework opens the door for a continuous improvement of the model by collecting users' feedback and preferences via telemetry data. As appealing as that may sound, improving LLMs via users' preferences and feedback raises privacy concerns: what if the model learns about a specific user's instructions and regurgitates them a later point? is well known in the privacy literature that LLMs vulnerable to privacy attacks including prompt attacks (Duan et al. 2023 Carlini et al.) 2021 2019), and RLHF training seems particularly concerning from this angle. This constitutes the central question explored in this work.
Can we fulfill the promise of aligning models with human preferences and feedback data via a privacy preserving RLHF methodology?
1.1 OUR CONTRIBUTIONS
We initiate the study of aligning LLMs with RL while satisfying the strong mathematical guarantees of differential privacy (DP) (Dwork & Roth 2014). We foresce this as important research direc-
Thii work carried part of intersship at Microsoft Rescarch.

---

Published as a conference paper at ICLR 2024
tion for the privacy community as more applications start to deploy LLMs to interact directly with users. Our main contributions are:
1. We give a differentially private framework for aligning LLMs with RL. Our framework math- ematically guarantees that the final model satisfies DP over the entire course of the alignment process, consisting of multiple stages model training and weight sharing. Further, we show how to adapt the PPO algorithm (Schulman al., 2017) to DP setting.
2. We empirically evaluate our DP framework on commonly studied tasks (in non-privacy litera- ture). Following the influential work of Ziegler al. (2020), we evaluate two main scenarios: (i) alignment via RL without human in the loop for a positive review generation task on the IMDb dataset (Maas et al.) 201T), and (ii) alignment via RL from human feedback (RLHF) for summarization task on the Reddit TL:DR dataset (Völske et al., 2017). Our experimental results indicate that privately aligning LLMs is possible, offering competitive utility while ensuring strong privacy protections. As a representative example, average reward obtained by our DP GPT2-Large model for generating positive reviews is 3.20 with 4, whereas the best performing non-private model achieves an average reward of 3.45.
the IMDb dataset, the
Our experiments also show that increasing the model size typically leads to more favorable privacy- reward trade-offs, hence, we anticipate that as pre-trained LLMs get better, alignment with DP should become easier.
2 PRELIMINARIES
2.1 ALIGNING LANGUAGE MODELS VIA REINFORCEMENT LEARNING
We review the pipeline from the seminal work by Ziegler et al. 2020), which describes a methodol- ogy align language models via RL by using a reward model to optimize for given task. For ease of presentation, we borrow terminology and notations from Ziegler et al. (2020).
One starts with a pre-trained language model LM which defines probability distribution LMM(IN 21..... 2n-1) E [o, 1] over the space of tokens In E >V given a context consisting of tokens I2 V for = ..... .--n-- 1. is referred as the vocabulary of LM The first step in general is to fine-tune this model with regular supervised learning procedure (SFT). This step can be per- formed for various reasons such to teach the language model a desired output behaviour (Ouyang et al.
2022) it could be simply to train for a downstream task such summarization (Stiennon
ct al. 2022 We denote the resulting model as LM
In the aligmment step, policy initialized as LMsh, is further fine-tuned using RL for the underlying task. We consider two scenarios depending on whether the task is directly defined by a reward function or it is based on human judgments. We compare the two scenarios in Appendix G
Reinforcement learning without human in the loop. The underlying task is defined by a reward function r yoo x yoo I that can score how well aligned the language model's generation y E yoo is, given the context x E V Here, one can use reinforcement learning to directly optimize the expected reward. An example is controlled sentiment generation, where the goal is to respond to a user query with a positive sentiment. Here, one can use existing language models that fine-tuned on sentiment classification tasks the reward model to score for positive sentiment.
Reinforcement learning with human preferences. The underlying task is defined by human judgments. A typical example is to respond to a user query with human-preferred way instead of language model's 's original completion that is learned during pre-training. Here, human labels are used first to train a reward model. A dataset can be formed by generating multiple responses from the LLM (for simplicity we consider two: y1 and y2) for given input x and asking humans to prefer between yı and 1/2- Let b E (0, 1} denote the human preference. Assuming access to dataset S of (1t, Yo, y1, b) samples with human preferences, reward model r y x yoe - R can be trained
with the following negative log-likelihood loss (Ziegler et al. 2020 Ouyang et al. 2022
L(r,S) -E S [log (a (r(z, mb) r(z, b1)]]
(1)
where a denotes the sigmoid function: (I) Thp One can also initialize the reward model r from L. with additional linear layer that produces single scalar for the reward value.
2

---

Published as a conference paper at ICLR 2024
Finally, the initialized policy is fine-tuned to optimize the reward model with reinforce- ment learning. However, instead of directly optimizing the expected reward, a penalty term of 3KL(T, "M is added to the optimization term to prevent from deviating too far from LM Thus, the modified reward becomes
(v)
R(x,0) v) r(r, v) B log LM*
This reward function is maximized via Proximal Policy Optimization (PPO) (Schulman et al. 2017 fine-tune the policy on the corresponding data distribution x D.
2.2 DIFFERENTIAL PRIVACY
LLMs are known to be susceptible to privacy attacks (Carlini et al. [019 2021 2023). Over the past decade, Differential Privacy (DP) (Dwork et al. (006) has emerged as powerful framework that provides mathematical guarantees for the privacy of individuals in training datasets. It quantifies the amount of information one could learn from the output of an algorithm or its generations. Formally,
Definition 1 ((c, 6D (Dwork & Roth 2014)). A randomized algorithm M achieves (c, 83-DP, if for any neighboring datasets D, and D2 (differing at most one entry) and for any S E Range (M), Pr(M(D:) E S) e Pr(M((D2) S) 5.
(3)
Here, c represents the privacy budget: a smaller c offers a stronger privacy guarantee. ō accounts for the probability that M violates e-D..
DPSGD. In the context of deep learning, DPSGD (Song et al. 2013 Abadi et al. 2016). drop-in replacement of the vanilla stochastic gradient descent, has become the default optimizer achieve DP. At each iteration, DPSGD performs per-sample gradient clipping and Gaussian noise addition, thus limiting and masking the contribution of any single data point the model update; we give detailed description in Appendix A Recently, in response to the rising concerns of privacy leakage
in large language models (Carlini et al., 2021). DPSGD is employed for tasks from private fine-
tuning Yu et al. 2022 Li et al. 2022) to synthetic text generation Yue et al. 2023).
3 PROBLEM DEFINITION
Consider the generic problem of aligning a language model towards an objective by using a reward model via RL. Our problem formulation introduces extra dimension to this challenge, namely, achieving this alignment respecting DP of the underlying data samples. We are given privacy pa- rameters > O, ò E j0, 1] and pre-trained language model LMP There is a private dataset D for use in the alignment procedures outlined in Section 2.1 The particular use of the private dataset D depends the availability of the reward model.
In RL without human in the loop scenario, we assume the availability of a reward model that is independent of the private dataset D. In this case, supervised learning step (SFT) fine-tunes LMP LM This is directly followed by the Proximal Policy
with D to achieve the initial policy Optimization (PPO) that fine-tunes the policy with the reward model on D. The privacy-preserving constraint in our problem definition is that the final parameters of the optimized policy π should be (e, 8D with respect to private dataset D.
When the task is based on human judgments, training reward model with human labels is needed as an extra step. As described in Section 2.1 the reward model is obtained by training on a dataset where each sample is a tuple consisting of sample belonging to D, multiple generations of LM given x as context and human preference over these generations. The privacy-preserving constraint similarly follows as the previous scenario and the final parameters of the optimized policy must be (e, 6DD with respect to private dataset D. As we will discuss shortly, this will require the reward model to be trained with DP as well.
4 OUR DP ALIGNMENT FRAMEWORK
In this section, we describe our DP framework for aligning LLMs. We consider the scenario with human in the loop it subsumes the case without. Recall that it involves three main steps: 1)
3

---

Published as a conference paper at ICLR 2024
Supervised fine-tuning of language model for the task at hand to obtain LM 2) Learning reward model r from human preferences. 3) Fine-tune a policy æ (initialized to LM to optimize the reward model with RL. Although we only require that the weights of the final policy DP with respect to D, one needs to perform each step with DP, the reason for which will become clear once we describe our framework.
To achieve (c, 5)-D with respect to a private dataset D at the end of the alignment there is more than one solution. One could partition D into three disjoint subsets Di, D2 and D3 corresponding to the three stages of the alignment pipeline, and assume that two neighboring databases differ by a single sample in one of these three datasets; that is, a single user can contribute to at most one of three datasets D1. D2 and D3. Another option is to assume that single user can contribute to all the three datasets D1, D2 and D3. Our framework can handle both the settings, with minor differences in how to calculate the final privacy parameters. The former approach would mean that to calculate the final privacy parameters, we can use the parallel composition theorem of DP (McSherry 2009). For the latter, one needs to use advanced composition theorems such as (Gopi et al. 2021 An additional hyperparameter related to DP in the second approach is on how to allocate the fixed privacy budget across the three steps. The goal of this work is to show that alignment with DP is possible, and, hence, we take the simpler approach and assume that a single user can contribute to at most one of three datasets Dr, D2 and D3- We clarify that the nature of the three datasets are different, with Dj being a labeled dataset consisting of a reference answer per sample, D2 a preference dataset consisting of two generations and a human preference bit per sample, and D3 an unlabeled dataset consisting of input samples only.
With this discussion behind us, we write down our framework for DP Alignment.
1. DP Supervised Fine-Tuning: We do a supervised fine-tuning of LMP using DPSGD with privacy parameters (c on the dataset D1 obtain LM The analysis of DPSGD (Abadi et al. (2016) guarantees that the weights of LM are private, and hence LM can be used arbitrarily in the remaining pipeline.
2. DP Learning of Reward Model: We initialize a reward model r from LAst with the addition of a linear layer that produces a single scalar prediction for the reward value. We train r using DPSGD with the privacy parameters (e, 6) to optimize the reward objective given by Equation on the dataset D2-
3. Alignment with DPPPO: Finally, we train a policy π initialized to LM via DP adaptation of Proximal Policy Optimization (PPO) with the privacy parameters (c, to optimize the reward R as given in Equation 2 on the dataset D3-
All our model training procedures use LoRA (Hu et al. alignment literature, make this algorithmic choice due to 3 reasons: 1) DP training works better with LoRA as hyperparameters are more stable (Yu et al., 2022 ; 2) LoRA is computationally more efficient, especially for DP training: 3) We also conjecture that LoRA fine-tuning during RL stage can also help in ensuring that the aligned model does not drift too far away from the LM model. This may be an interesting point even in the non-private world.
2022
While this is not standard in the
We will elaborate each of the steps in our framework below. Before that, we note the following pri- vacy guarantee of our framework due the parallel composition theorem of DP (McSherry 2009).
Theorem 2. Our DP alignment frumework is (e, 8) diifferremill private.
Why Step II needs to be DP? A curious reader may ask why one should learn the reward model using DP if the third step already satisfies DP: that is, can learning the reward model be non-private? This is a subtle but important point, and our algorithmic choice of making the second step DP has to do with the privacy analysis of DPSGD. Consider the scenario where the reward model is not private. In such a case, for every random mini-batch in the third step, the gradients are a function of the reward model, which turn implies that the gradients of the mini-batch are function of entire dataset Dz. This invalidates the privacy amplification by subsampling theorem of DPSGD Abadi ctt al. 2016), which is crucial for the overall framework to work.
Our solution to the above technical challenge is to learn the reward model also via DP. The post- processing theorem of DP (Dwork& & Roth 2014) guarantees that the reward model r can be used in the third step as if it were a public model. However, there could be other ways of achieving the alignment with DP where the second step is not private, and we leave it as an intriguing future research question.
4

---

Published as a conference paper at ICLR 2024
DP Adaptation of PPO. Next, discuss algorithmic choices in making PPO algorithm DP. Although the model updates in our DPPPO algorithm are done via DPSGD, there are some impor- tant technical details to consider. An important distinction between DPPPO and SFT with DPSGD is what governs the number of iterations and how the model weights are updated. For SFT using DPSGD (1st step), the model weights are updated for each batch, and the number of iterations gov- ems the course of training and the total model weight updates. On the other hand, PPO performs model updates in minibatches within batch and multiple rounds (a.k.a. PPO epochs) can be taken over the same batch. Further, regular epochs are taken over the full dataset; see SShhulmn et al. (017)) (Algorithm for a precise description. We give a complete pseudo-code of PPO implemen- tation in Appendix F but for the sake of discussion consider the abbreviated version in Algorithm I our DPPPO implementation,
PPO updates are given in lines 6-10 that needs to be privatized.
we set PPo = deviating from the usual implementations in RL. that set TPP > 1; for example, von Werra et al. (2022) defaults Tppo to 4. By appropriately selecting the batch size (we use larger batch size for DPPPO), we ensure that the total number of model updates in both the private and non-private worlds remain similar. We make these algorithmic choices to simplify the privacy anal- ysis and to utilize privacy amplification by subsampling (Abadi et al.] 2016) in DPSGD algorithm, where each batch should be randomly selected from the dataset. If one takes more than I round of model updates (Tppo > 1) on the same batch, then privacy accounting of DPSGD needs to be modified, say by first doing an advanced composition across multiple PPO rounds on the same batch followed by subsampling amplification. We leave these algorithmic explorations on our choices as future research directions, and present an ablation study on Trro >> 1 in Appendix B.
Algorithm 1: Aligning language models with RL (PPO), full version in Appendix F
Define:
D: a dataset consisting of input texts. input text, y: model response.
T: total training epochs, Trro: PPO epochs.
model, ref model: the model being learned and the frozen model for reference. Models are composed of generation body as well as a value head.
superscript" : batch, superscript mb: mini-batch.
p, l: log probability and logit given by the generation body, value given by the value head Procedure Update (model, x², y², H²):
Stage I: forward passes to obtain reference stats on the batch x"1"")
Pp",",r' v +- Sraachhddoowwddassss
(ph, ) batchedroowwardPaa (odel, 2",""
ComputeScores(4". p².)t)
compute the moditied reward (Eg.
> Stage II: update on minibatches D
for i = to TePo do for Dmu D do
(m y pub mmb
(,) + mm mb) + mm
D take out a minibatch
(model,, mb mby p, v)
Trainkinibatch (model, nh mb
with object ive
main loop
11 for to I'do
b Take out a batch
for - do
12
13
ya medelggneeratel"
b obtain the model responses > obtain the rewards via the reward model
R - r(=.,v''
14
Update (model, x. y R²)
15
16 return model
PRIVATELY ALIGNING LMS WITHOUT HUMAN IN THE LOOP
We begin by exploring the simpler task of privately aligning language model without human in the loop, and consider RLHF the next section. For our case study, we focus on controlled sentiment
generation, where the goal is complete a given prefix of review from the IMDb dataset (Maas)
5

---

Published as a conference paper at ICLR 2024
Stage I: Supervised Fine-Tuning
Stage II: RL Fine-Tuning
Objective: learn to generate review
Objective: learn to generate review with positive sentiment
Movie review
Partial
Movie review
Reward: positive score
I octinstely recommend zeading the ...
faverite stacs of years
including John!
Faraliv, on indio 2ila that
Titxw fils 1x plazh
aycually Ty del Tuers soee graat ocapeo
Input: generated review
Language Modeling
Pretrained GPT-2
Fine-tuned GPI-2
Aligned GPT-2
It's classic!
Iedendl
descriotion.
Figure 1: Case study: aligning a language model without human in the loop. The goal is to complete a partial review with positive sentiment. The first stage supervised fine-tuning (SFT) where a pre-trained LM (GPT-2) learns to generate reviews. This is followed by PPO to optimize a reward function given by a BERT-style LM, which fine-tuned on some sentiment classification task. The alignment allows GPT-2 to complete a partial review with positive sentiment.
Table I: The average positive reward score on the test set of the IMDb dataset for various models and privacy levels. e 0 represents the pre-trained model. e E 21, 4, 8} are privately aligned models with different privacy budgets. e = stands for aligmment without any privacy. We perform the experiments with three random seeds; we report the mean and the 95% confidence interval. Additional privacy-utility trade-offs are demonstrated in Fig. of Appendix
C.2
Model
=
= 1
6 2
= 4
E 8
0O
GPT-2
-0.30
1.47 081
2.35 0.52
2.74 1 0.27
2.81 0.19
3.10 0.22
GPT-2 Medium
-0.28
2.39 + 0.52
2.60 + 0.43
2.93 0.17
2.93 0.13
3,45 + 0.02
GPT-2 Large
-0.24
0.71 0.13
1.91 0.42
3.20 = 0.23
3.38 0.03
3.32 0.06
2011 with positive sentiment depicted in Figure private dataset in this case study, and denote it by D.
al.
We consider the IMDb dataset as the
As described in Section 2.1 this scenario consists of two steps. First, supervised fine-tuning (SFT) is performed on the pre-trained model LM with the language modeling objective, allowing it to achieve review generation capabilities. Then, we further fine-tune the model LA4S0 using PPO with the guidance from the reward model r, for the purpose of alignment towards generating reviews with positive sentiment. We note that here the task is directly defined with reward function and there is no need to train a reward model using D. Thanks to the availability of language models that fine-tuned on sentiment classification tasks, one can utilize such models as the reward model to score for positive sentiment.
Experimental setup. We use GPT-2 model families (Radford al.) 2019) (base, medium, and large) and perform our experiments on the IMDb dataset (Maas et al. [211]) For the reward model, we use RoBERTa base model (Liu al. 2019) that is fine-tuned for sentiment analysis with the TweetEval benchmark (Rosenthal et al. 2017 As discussed in Section alignment with DP uses half of the training dataset in the SFT step and the remaining half in the RL. step. Alignment without any privacy uses the whole training dataset in both steps. Hyperparameterss are tuned using the standard practices in DP fine-tuning Yu el al. (2022) and alignment literature; for completeness, the details about hyperparameters in all components (SFT and PPO, non-private and DP) are in Appendix B
We use the average positive reward on the IMDb test set to measure alignment effee- tiveness. We compare the performance of our DP framework to the regular non non-private alignment.
Evaluation.
Main results. We present the results in Table for various privacy levels E {1, 2, 4, 8} while fixing = D/ We point out that these DP guarantees would also hold with smaller 8, albeit with minor increase of using privacy curves (Balle & Wang 2018). Main highlights are:
1.
We note that fully private (e = 0) pre- pre-trained models are not aligned to generate positive re- views, as expected. On the other hand, Table shows that can align these models towards generations with positive sentiment with accompanying formal DP guarantees.
base- sent iment

---

Published as a conference paper at ICLR 2024
Table We display the generation results for the prefix "I am not a fan of Sean Penn". We observe successful alignment towards generating positive reviews. More results are in Appendix C.I
Model
GPT-2
am not a fan of Sean Penn at all and don't really look for him. I liked the flavour really
am not a fan of Sean Penn and I love it. However, became bit too. I love the
GPT-2
am not a fan of Sean Penn's, s, I'm really happy and love the movie, and I's very
am not fan of Sean Penn. appreciate what he is. It's awesome. This has been amazing.
Medium
GPT-2 Large
am not a fan of Sean Penn <3 this film is great and worth watching! <3 <3 <3
I am not a fan of Sean Penn, but I love his work in baseball and I love his work for my favorite
Stage I: SFT
Stage II: Reward Modeling
Stage III: RL Fine-Tuning
Obj: learn to summarize
Obji learn to model human preference
Obj: learn to sumumarize with human preference
Summarization dataset ppost + summary)
Preference datasct
Post dataset
Reward Modeling
(port)
Rewaard preference score
Pout:
Lovedy
Summary 1
Reward GPT-2
caliboatt and tuesse
Reward GPT-2
Summary 2
Post: ALL
Sanwary:
Doow hocs
Feiday
collage
Pretrained GPT-2
Language Modeling
Fine-tuned
PPO
Aligned GPT-2
Input: post +summary
GPT-2
Figure 2: Case study: aligning a language model with human preferences. The goal is to gener- ate the summary of post in human human-preferred way. The first stage is supervised fine-tuning (SFT) where a pre-trained LM (GPT-2) learns generate summaries. In the second stage the reward model is trained based on Eq. D with pair of summaries where one is preferred over the other to model human preferences. This is followed by PPO with the reward model from the second stage. The alignment allows GPT-2 generate summary in human maeaanneeeerred manner.
2.
As expected, relaxing the privacy budget improves the average positive reward score on the test set and we observe strong performance c 4, which commonly used in the DP fine-tuning literature (Yu et al. 2022).
Generally speaking, and consistent with the DP fine-tuning literature (Yu et al., 2022). larger models improve the alignment performance. One exception is GPT2-Large model for small The latter may due to insufficient hyperparameter tuning as we tuned hyperparameters by fixing 4. For the non non-private alignment (e oc), observe a further improvement as expected as the privacy-utility trade-off is tilted completely in favor utility.
We were not able to find hyperparameter setting where non-private GPT2-Large would outper- form non-prin non-private GPT2-Medium. This may be due to the task at hand where the model size and capabilities of GPT2-Mediug is already sufficient in the con-piivate alignment.
Demonstrations. We randomly select five partial reviews from the test set and let the private (c= 4) and the non-private models complete the reviews. Part of the results are shown in Table (more in Appendix C.I). We observe that the generation quality are consistent with the results of Table It is interesting to note that even when partial review begins with negative tone, the aligned models can continue the review with positive sentiment instead. Larger models GPT-2 Medium and Large are better in quality, as expected, and we do not observe qualitative difference between private and non con-private model generations, which is impressive for aligning with DP.
6 PRIVATELY ALIGNING LMS WITH HUMAN PREFERENCES
In this section we empirically evaluate the scenario where we privately align a language model with human preferences. For study, focus on summarization task, where the goal is to

---

Published as a conference paper at ICLR 2024
Table 3: The average reward score (denoted by r) on the test set of the Reddit TL;DR dataset and ROUGE-L score (denoted by R-L) between model generated summaries and the label summaries for various models and privacy levels. e = 0 represents the pre-trained model. privately aligned models with different privacy budgets.
{1, 2, 4, 8) are co stands for alignment without any privacy. Full results including ROUGE-1 and ROUGE-2 scores are deferred to Appendix
Model
Pre-trained
1
2
4
8
R-L
r
R-L
r
R-L
r
R-L
R-L
R-L
SFT
0.44
11.45
0.48
11.84
0.50
12.30
0.49
12.45
0.63
14.48
GPT-2
0.05
8.26
Aligned
0.22
10.41
0.53
11.44
0.68
12.33
0.69
11.74
1.53
14.17
GPT-2
SFT
0.68
12.80
0.66
13.07
0.65
13.30
0.65
13.5
0.70
14.30
0.11
8.67
medium
Aligned
0.59
12.86
0.92
13.26
0.92
13.44
0.86
13.79
1.76
13.17
GPT-2
SFT
0.51
14.98
0.51
14.86
0.52
15.14
0.51
15.04
0.54
15.53
-0.06
10.34
large
Aligned
0.40
14.75
1.14
14.58
1.06
13.88
0.93
14.37
1.49
14.64
generate a summary of a post from the Reddit TL. summarization dataset (Völske et al. 2017) in cuuman-preferree manner as depicted in Figure 2
We chose this task because: 1) summarization is an important task in practice but is inherently tied with human judgement 2) this task was also studied by the original work of Ziegler et al. (2020) and their follow-up (Stiennon et al. 2022). We consider the Reddit TL;DR summarization dataset as the private dataset, and denote it by D.
Compared to the previous scenario, here there is additional step that involves training reward model with DP based on human preferences. This reward model will in turn enable the PPO al- gorithm to align the language model to summarize in human-preferred manner. Similar to the previous scenario, we separate D into three disjoint subsets D2 D2, and D3 to perform SFT, reward modeling, and PPO with DPSGD respectively. In Section we have discussed why the reward modeling also needs to be performed with DP.
Experimental setup. We use GPT-2 model families (Radford al., 2019) (base, medium, and large). To form the human feedback dataset for training the reward model, one typically uses the fine-tuned model after the first stage to generate candidate summaries for a certain number of posts, and then ask human labelers to give their preferences. However, due to the infeasibility of collecting actual human preferences, we resort to using an existing dataset, released by OpenAl, where human preferences were gathered by Stiennon et al. 2 The human feedback dataset in Stiennon et al. (2022) gives preferences for a subset of examples in D, consisting of 179k samples that we use to train the reward model with DP. Finally,
allocate 100k samples for the SFT step and 200k samples for the final RL step. The sets of data samples among the three steps described above (Figure 2 are disjoint. We provide the details about hyperparameters in Appendix [
Evaluation. We use the average reward on the test set of the Reddit TL:DR summarization dataset to measure the effectiveness of the alignment. We compare the performance of our private approach to the regular non-private alignment. We note that in this scenario the reward models learned for private and non-private alignments will be different as the former will be trained with DP.
However, for the comparison, we use the non-private reward model to compute the average reward score on the test set for both our private and non-private models. This is because we expect the non-private reward model to be more accurate, and idcally one would desire the private alignment be close the non-private alignment terms of utility, hence, obtain good score by the non- private reward model on the test set. It is important to recognize that this does not violate any privacy guarantees of our models as the non-prive non-private reward model used during the test time only. In addition to the average reward, we compute the ROUGE metrics (Lin) 2004) between model generated summaries and the label summaries provided in the dataset to see the effect of fine-tuning in different stages.
Main results. We present the results in Table 3 for various privacy levels e E (1,2,4,8) while fixing ò = L/I Main takeaways are:
ht
ze_ cecoo__eedbac///eee /v /
compari comparisons/ wow.riisttttrrrraroorso
8

---

Published as a conference paper at ICLR 2024
1. We see improvement in the mean reward after the alignment step for most models. These results demonstrate that private alignment towards human- -remen-reeferee summarization is achievable with formal privacy guarantees.
2. Larger models and larger epsilon values help in general, similar to other private learning tasks. However, the reward curve is not monotone (with respect model size and epsilon values) particularly along the privacy axis, The private model achieving the highest mean reward is GPT2-Large with e More extensive hyperparmeter tuning is necessary understand this phenomenon.
We observe larger gap between private and non non- private models compared to previous sentiment generation task. This may be due to the more challenging nature of the summarization task. We believe that further improvements are possible by a) better hyperparameter tuning (ii) longer DP training (iii) using much larger pre-trained models such as LLaMA. However, we could not carry out these experiments due to compute constraints; yet, the overall message we were aiming for privatee alignment is possible can be inferred from results.
4. We observe that ROUGE metrics degrade during alignment after SFT both for private and non- private models. This is expected because label summaries do not entirely align with human preference (as the labels in D, and the preferences in D2 come from different human groups). Thus, as the models learn to summarize in human-preferred manner, they deviate from label summaries learned during SFT. Note that during SFT we used label summaries to teach the model first summarize, while the alignment step itself does not use label summaries.
RELATED WORK
Reinforcement learning from human feedback (RLHF) has emerged prominent technique in fine-tuning language models. Christianoo et al. (2017) laid the foundation, utilizing human feed- back for reward modeling and employing PPO (Schulman et al.) 2017) for model training. Early applications of RLHF in the natural language realm focused on stylistic continuation (Ziegler et al.) 2020), summarization (Ziegler et al., 2020 SSieenon et al., 2022; Wu et al. [202T), etc. Subsequent research endeavors shifted towards training Al assistants that align with human values across wide spectrum instruction tasks (Ouyang et al. 2022 Bai et al. 2022 Touvron et al, 2023).
DP in language models Exploiting the memorization ability of language models (Carlini al. 2023), privacy attacks have been launched, extracting training data or inferring membership (Carlini et al. 2019 2021 Elmahdy et al., 2022) |Mttern et al. 2023). In response these vulnerabilities,
DP fine-tuning via DPSGD (Abadi et al. (016) has been proposed (Li et al. 2022 Yu et al. (022). A different line of works (Mattern et al. | 2022 Yue al. 2023) focus on privately generating synthetic text data, via fine-tuning a pre-trained model with DP. Despite significant progress in language model privacy, there still gap in ensuring DP for aligning language models. To our best knowledge, we
are the first that take step in this direction.
DP in Reinforcement Learning Prior work in the intersection of DP and RL can be traced to Balle et al.) (2016). Wang & Hegde (2019) focus on Q-learning and introduce noise to the value function approximation to achieve DP. Ma al. (020) target a constrained scenario, MDPs with linear function approximations, and ensure joint differential privacy (JDP). Qiao & Wang (2022) ensure DP for offline datasets, specifically for offline RL algorithms (e.g., APVI (Yin &
Wang 2021)). None of these fulfills the need of achieving DP for online RL (e.g.. PPO) with the neighboring relation defined on fixed dataset. Our DP adaptation of PPO (Section fills the gap.
We defer more complete description of the related work to Appendix H
8 CONCLUSIONS AND FUTURE WORK
In this paper we initiated the study of privately aligning LLMs with human feedback. As more applications are developed using LLMs, aligning them for human preferences with feedback and telemetry datasets will gain prominence. We demonstrated the initial promise of performing these steps in a privacy preserving way, and we anticipate this will become an active area research. Moreover, work opens up several technical questions: How to improve DPPPO algorithms, can be there tighter privacy guarantees of our algorithms, and finally how to adapt our algorithms to the online setting.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
Fan Wu would like to thank Yuzheng Hu for his support. Bubble-Fan's beloved stuffed bunny would like to thank Microsoft Research for the lovely campus and environment.
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernan- dez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Ka- plan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022.
Borja Balle and Yu-Xiang Wang. Improving the gaussian mechanism for differential privacy: Ana- lytical calibration and optimal denoising. In International Conference Machine Learning, pp. 394-403, PMLR, 2018.
Borja Balle, Maziar Gomrokchi, and Doina Precup. Differentially private policy evaluation. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp- 2130-2138, New York, New York, USA, 20-22 Jun 2016. PMLR.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pp. 267-284, 2019.
Nicholas Carlini, Florian Tramer, Eric Wallace Matthew Jagielski, Ariel Herbert-Voss Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633-2650, 2021.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2023. URLhttps: / htp///peeevvvieeeeeeeeeeeeeeeeee net forum?
id=TatRHT 1cK]
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- tems, 30, 2017.
Haonan Duan, Adam Dziedzic, Nicolas Papernot, and Franziska Boenisch. Flocks of stochas- tic parrots: Differentially private prompt learning for large language models. arXiv preprint mrXiv:2305.15544 2023.
Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4):21 1 1-407, 2014.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pp. 265-2222222 284. Springer, 2006.
Adel Elmahdy, Huseyin A. Inan, and Robert Sim. Privacy leakage in text classification a data extraction approach. In Proceedings of the Fourth Workshop on Privacy in Natural Language Processing, pp. 13-20. Seattle, United States, July 2022. Association for Computational Linguis- orgp/022.....000.....
tics. doi: 16.186 10.11855311/22222pprii 1/2022.privatenlp...3 URL https:
privaten 1p-ttenlp-1..
10

---

Published as a conference paper at ICLR 2024
Sivakanth Gopi. Yin Tat Lee, and Lukas Wutschitz. Numerical composition of differential privacy. In Advances in Neural Information Processing Systems, volume 34, pp. 1 11631-11642. Curran Associates, Inc., 2021.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. International Confer- ence on Learning Representations, 2022. URL nZeVKeeFYf9)
wet.erewwwwweeeeeeeeeeeeeeeeeeee httpp//peneeeeeeeeeeeeeoorrrrnmm forrmmii
Julia Kreutzer, Shahram Khadivi, Evgeny Matusov, and Stefan Riezler. Can neural machine transla- tion be improved with user feedback? In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Pupers), pp. 92-105, New Orleans Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 10.11863/1//11--110 3012. URL htttps://cclarthhl.l.
Drg/N18-300
Xuechen Li. Florian Tramer, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. In International Conference on Learning Representations, 2022. URL
net/ forum? d=bVuP lorte//oommmdddbuPPPPP 311ATTM
Chin-Yess Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 190. 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proceedings of the 7th International Conference on Learning Representations, ICLR '19, 2019.
Pingchuan Ma, Zhiqiang Wang, Le Zhang. Ruming Wang, Xiaoxiang Zou, and Tao Yang. Differ- entially private reinforcement learning. In Jianying Zhou, Xiapu Luo, Qingni Shen, and Zhen Xu (eds.), Information and Communications Security, pp. 668-683, Cham, 2020. Springer Interna- tional Publishing. ISBN 978-3-030-41559-2.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL https: 7/ac wttp:lllooooggggiigo/g1g
Justus Mattern, Zhijing Jin, Benjamin Weggenmann, Bernhard Schoelkopf, and Mrinmaya Sachan. Differentially private language models for secure data sharing. In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing, pp. 4860 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties. doi: 10. 2022.emnlp-main..22. URL //acl anthooogyy..coo.ooo wowph/ooooorr//222.o... emn 1omaa//...rr./222.....mmaaaaaaaa 12333
Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schölkopf, Mrinmaya Sachan, and Taylor Kerg-Kirkpatrick. Membership inference attacks against language models via neigh- bourhood comparison, 2023.
Frank McSherry.
Privacy integrated queries: an extensible platform for analysis. Proceedings of the 2009 ACM SIGMOD International Conference on Management of data, 2009.
data
Khanh Nguyen, Hal Daumé III, and Jordan Boyd-Graber. Reinforcement learning for bandit neural machine translation with simulated human feedback. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1464-1474, 2017
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. Ad- vances in Neural Information Processing Systems, volume 35, pp. 17730-27744 2022.

---

Published as a conference paper at ICLR 2024
Dan Qiao and Yu- Yianin Wang. Offline reinforcement learning with differential privacy. arXiv preprint arXiv:2206.008100 2022.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.
Language models are unsupervised multitask learners.
2019. URL https://api.
semanticscho ar aar.llr..rrrro.ooo... wwt.oorg//ocpuaa I 60025533
Sara Rosenthal,
Noura Farra, and Preslav Nakov. Semeval-2017 task 4: Sentiment analysis in twitter. In Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017), pp. 502-518, 2017.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: 02438, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate. Stochastic gradient descent with differ- entially private updates. In 2013 IEEE global conference on signal and information processing, pp. 245-248. IEEE, 2013.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXivv2307.09288 2023.
Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. TL:DR: Mining Reddit to learn
automatic summarization. In Proceedings of the Workshop New Frontiers in Summarization, pp. 63, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. Trl: Transformer reinforcement learning. htttps://githii
pom/ Conn/haggingfaaee//tee
2022.
Baoxiang Wang and Nidhi Hegde. Privacy-preserving q-learning with functional noise in continuous spaces. Advances in Neural Information Processing Systems, 32, 2019.
Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Chris- tiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109. 10862, 2021.
Ming Yin and Yu-Xiang Wang. Towards instance-optimal offline reinforcement learning with pes- simism. Advances in neural information processing systems, 34:406554077 2021.
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. Differentially private line-tuning of language models. In International Conference on Learning Representations, 2022. h http:://ppppeeeeeeeeee http:/iwwwwww.eeeeeeeeeeeeeeeeee ttttt///rrrmmmo0002202dd
Xiang Yue, Huseyin Inan, Xuechen Li, Girish Kumar, Julia McAnallen, Hoda Shajari, Huan Sun, David Levitan, and Robert Sim. Synthetic text generation with differential privacy: A sim- ple and practical recipe. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1; Long Pupers), pp. 1321-1342, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 1/28666//122222.alllllooo https://aclanttooooy...om
URL
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020.
12

---

Published as a conference paper at ICLR 2024
A DPSGD ALGORTIHM
We provide a pseudocode for the DPSGD algorithm in Algorithm 2 In each iteration (steps 2-7), DPSGD works by calculating per-sample gradients over the samples in a batch and clipping the norm of per-sample gradients. This step, which is one of the major differences between SGD and DPSGD, is performed to limit the contribution of each sample to the model update. Note that, thanks to clipping, the & sensitivity of the operation in Step 6 is bounded, which otherwise would not be bounded. In the Step 6, carefully calibrated Gaussian noise added to the average of clipped gradients and update step is performed.
The privacy analysis of DPSGD works as follows. Fix one iteration the algorithm. Since the clipping step ensures that the ey-sensitivity of the average of gradients remains bounded, it not hard to prove that each iteration of DPSGD satisfies (c, 6)-P with some privacy parameters. However, erucial to its analysis is the application of privacy by subsampling. Here note that in iteration, sample B] examples out of [D] total datapoints, so, the privacy guarantees for the single iteration of the algorithm
dictated by subsampled Guassian mechanism Abadi al.) (016); Gopi et al. (202I). Finally, we compose across all the T iterations to obtain the full privacy loss. The PRV account that we use Gopi al. (2021) gives tighter analysis of this overall framework using numerical composition techniques.
Algorithm 2: Differential Privacy Stochastic Gradient Descent (DPSGD)
Define: Dataset D, model parameters 0, loss function L(0,x), learning rate 17, noise scale a, gradient norm bound C, sampling probability p. number of epochs T
for t = 1.. do
Sample B C D with sampling probability p
for x; E do
Compute gradient: 9i VoL(0, xI.)
Clip gradient: < 9:/ max(1, ||
Add noise and calculate update: +- [ (10,00(0,
Update model: 0 0 17- g
return 0
B HYPERPARAMETERS FOR SECTION 5
In the following, we describe the details of our hyperparameter search for the results in Section 5
For LoRA, we choose the bottleneck rank attention layers as in the original paper (Hu et al. 2022).
4 and fine-tune query and value matrices of the
For non-private SFT, tune the batch size and the learning rate from the set (8, 16, 32, 64) and in the range [1e-6, le-2] respectively. The training performed until convergence, which occurs within 5 epochs. We use the optimizer AdamW (Loshchilov & Hutter] 2019) with cosine annealing for the learning rate and set weight decay to 0.01. The final batch size and learning rate are reported in Table 4
Table 4: Non-private SFT hyperparameters for the results in Section 5
Model
Batch size
Learning rate
GPT-2
64
5e-4
GPT-2 Medium
64
5e-4
GPT-2 Large
64
2e-4
For DP SFT, informed by prior work (Yu et al. 2022 Li et
al. 2022). aim to set large batch size and constant learning rate with a long training course. We set the batch size to 512 and the number of epochs to 40. We similarly tune the learning rate in the range [le-5, 1e-1] and finally set 3e-4
13

---

Published as a conference paper at ICLR 2024
for all models. We use the optimizer A with weight decay 0.01. For the DP parameters, set a small per-sample clipping norm as 1.0 and calculate the corresponding noise multiplier to achieve the reported (e, 6DDP using the accountant in Gopi al. (2021).
For PPO, we use the TRL framework and set the hyperparameters specific to PPO as default values therein. For non-private PPO,
the minibatch size 16 and the batch size to 256. PPO epochs set to and one epoch is passed on the full dataset. We similarly tune the learning rate in the range [1e-6, le-2] and finally set 1.4e-3 for GPT-2 and GPT-2 Medium, and 2e-4 for GPT-2 Large.
For DPPPO, we follow a similar course as DP SFT. We set the minibatch size to 256, the batch size 4096 and the number of epochs 100. PPO epochs must be to explained in Section 5 We similarly tune the learning rate in the range [lc-5, le-1] and finally set to 3c-3, 1e-3, and 2e-5 for GPT-2, GPT-2 Medium and GPT-2 Large respectively. DP parameters also follow as DP SFT.
B. ABLATION STUDY ON TpPo
We perform an ablation study on TpPo using the GPT-2 model for plications of setting Trro = in our DPPPO algorithm. We report the results in Table S The results indicate that setting Tpro > does not provide improvement for the performance and set- ting Tpro 1 is reasonable as it leverages privacy amplification by subsampling in the DPSGD algorithm.
investigate the im-
Table 5: Ablation study on Tpro- We present the mean results over three runs with different random seeds, along with 95% confidence interval. Results show that the implications of setting Tppo = 1 is insignificant.
Model
TPPO
Average reward
2.74 + 0.27
2
2.72 + 0.14
GPT-2
4
4
2.73 + 0.05
8
2.64 1:
C ADDITIONAL RESULTS FOR THE POSITIVE REVIEW GENERATION TASK
IN SECTION 5
We present the following additional results compliment Table in Section
C.1 SAMPLE GENERATIONS FOR SECTION
Table demonntrtes the alignment towards generation with positive sentiment for private and private models via completions on randomly sampled prefixes from the test set.
C.2 TRADE-OFF BETWEEN PRIVACY AND UTILITY
To provide a clearer understanding of the privacy-utility trade-off, we illustrate in Figure 3 how different levels privacy (varying e) impact the model's performance for the GPT-2 Medium model. We observe that the model performance improves from the fully-private model (e 0) to the private model with privacy level с 4. The performance plateaus this region and decreasing the privacy of the model by using larger levels of e E [4, 10) does not further improve the performance. The non-private model (e oo) has expectedly the best performance, albeit with lack of privacy.
D HYPERPARAMETERS FOR SECTION 6
We mostly follow the hyperparameters described in Appendix Here we state only the differences.

---

Published as a conference paper at ICLR 2024
3.5
average positive reward score
model: 1092-medium
3.0
2.5
2.0
1.0
0.5
0.0
1
5
10
privacy budget
Figure 3: Trade-off between utility and privacy for the positive review generation task. Results are obtained on the GPT2-medium model. The shaded area denotes the 95% confidence interval. e 0 represents the pre-trained model; oo represents the non-private alignment.
Compared to the scenario in Section 5 we work with order of magnitude larger dataset size in this scenario. Due to the sheer amount of experiments and computational constraints the training time reduced. which hurts DP performance. For DP SFT, set the number of epochs to 10 and for DPPPO, we set the number of epochs to 1.
An important difference is that this scenario involves training reward model. We fix GPT-2 model to used for reward model in all experiments. For non-private training, we set the batch size to 64 and the learning rate le-4 and train for epoch. We the optimizer AdamW with linear scheduler for the learning rate and set weight decay to 0.01. For DP training, we set the batch size 4096, the number of epochs 50, and the learning rate 2e-4. We the optimizer AdamW with weight decay 0.01. For the DP parameters, we set small per-sample clipping norm as 1.0 and calculate the corresponding noise multiplier achieve the reported (c, dDD using the accountant in Gopi et al. 2021
E FULL RESULTS FOR THE SUMMARIZATION TASK IN SECTION 6
We present the complete set of results for the summarization task in Table 7 additionally including the ROUGE-1 and ROUGE-2 scores.
F FULL PSEUDO-CODE
We present the complete version of the pseudo-code in Algorithm We include the detailed pro- cedures of Loss, ComputeScores res, and Tra i n n batch. The parts that require additional adaptation to fulfill DP are highlighted in blue and red.
G TWO PARADIGMS OF ALIGNING LANGUAGE MODELS
Depending the nature of the reward whether it is from some standard and commonly endorsed criteria or from the preferences from group of humans, there are two main paradigms in using RL for alignment.
RL without human in the loop. This paradigm focuses on criteria that are straightforward to judge, typically characterized by clear ground truth labels such toxicity sentiment. Given their easily quantifiable nature, these criteria often align with binary labels. Moreover, these criteria do not hinge upon specific human groups for validation interpretation. The advantage of this
15

---

Published as a conference paper at ICLR 2024
Algorithm 3: Aligning language models with RI. (PPO), full version
Define: D: dataset consisting of imput texts, z: imput text, y: model response.
T: total training epochs, P PPO training epochs. model, ref_model: the model being learned and the frozen model for reference. Models are composed of generation body as well as value head. superscript :; batch, superscript mb; mini-batch.
p. I: log probability and logit given by the generation body, value given by the value head.
Function Loss (poli, o shld (,,,,))
A <- eddanttagedaaaaaeeeeeeeeeaa
gold
through general ized advantage
est on (Schul man et al. 2015
r r exp(pp- - pok))
b compute the ratio
loss, +- min( - -Cipp(r -, A)
clipped objective
loss, a, (A + wlld v) mean
return lossp, loss,
Function ComputeScores (R",
,P,)):: ,
adjust the score by KL divergence. In practical impletattaionn, It" (given by the reward model) is applied only the last tcken. return R OKi. (p" ph)
Procedure Trai MediiiMinibbse
(model, pold wold gold -p,,,);
loss,, loss, < Loss (p bold oldd pp,, p,,,u)
10
11
loss loss loss,
sum of policy loss and value loss
optimizer.zero.graad
13
lossbbackward
optimizer.step ()
Procedure Update (model, " R
b Stage I: forward passes to obtain reference stats on the batch
(0', v²) + aattcteddoovardFasssmode
16
(pt, 7, v BatchedFoorwardassssiie modellllllllmlll.......m.oooo x², ,²)
17
9 ComputeScores(#f) p². & compute the modified reward (Eq.
D Stage II: update en minibatches Do - (x",v², N,,,,,,, "0, (0)
19
b compose batched data
for i 1 to Tero do for Dmb Do do
(æ yd pub mb mb) mmb + D
b take out minibatch
(p,1,)) - raccchhe
mb mb)
TrainMeiiatcc(moddell. nd,, mmb mb ,p,,,) v) D with PPO objective
25
D> main loop do b take out batch for rb E D do
26 for
28
y + modellgenerate((2)
obtain the model responses obtain the rewards via the reward model
Rd + (
29
Update (model, 1² RR)
return model
paradigm is that there exists plethora of pre-trained classifier and detection API available to
htt petps::/Nie
16

---

Published as a conference paper at ICLR 2024
the public. They be leveraged to generate reward signals, which then guide the iterative updates of the LLM agent through RL.
RL with human preferences. dependencies on the subjective perceptions particular human groups. The assessment of the quality of results, such their honesty or helpfulness, demands continuous scores rather than binary labels. The reward systems are intrinsically tied to the values humans (or specific human groups). Consequently, reward model needs to be trained explicitly cater to these values. After training the reward model to capture human preferences, it is incorporated into the RL process to guide the LLM agent in adopting these preferences.
In contrast, this paradigm deals with tasks that bear significant
H FULL VERSION OF THE RELATED WORK
Reinforcement learning from human feedback (RLHF) has emerged as a prominent technique in fine-tuning language models. Unlike traditional methods that depend heavily on large labeled datasets, RLHF leverages human feedback derive a reward signal, guiding the model's optimiza- tion. This enables models to produce more desired outputs in complex and open-ended tasks. Chris- fiano et al. (2017) laid the foundation, utilizing human feedback for reward modeling and employing PPO (Schulman et al. 2017) for model training. Early applications of RLHF in the natural language realm focused on stylistic continuation (Ziegler et al. 2020), summarization (Ziegler et al. 2020 Sti- ennon et al., 2022 Wu et al. 2021), and translation (Nguyen et al. 2017 Kreutzer al. a 2018). Sub- sequent research endeavors shifted towards training Al assistants that align with human values across
wide spectrum of instruction tasks (Ouyang et al. 2022 Bai et al.
2022 Touvron et al. 2023).
DP in language models Exploiting the memorization ability of language models (Carlini et al,, 2023), many privacy attacks have been launched, aimed at extracting training data inferring train- ing set membership (Carlini et al. 2019 [021] Elmahdy et al. [022]] Mattern et al., 2023). In re- sponse these vulnerabilities, DP fine-tuning has been proposed as a potent defensive mechanism for achieving privacy preservation. Li et al. 2022); Yu et al. (2022) demonstrate the effectiveness of fine-tuning the language models using DPSGD AAadi et al. 2016). Applying appropriate hy- perparameter selections and parameter-efficien methods (e.g., LoRA (Hu et al. (2022) on the basis of large pre-trained models can yield language models which simultaneously enjoy competitive per- formance and strong privacy guarantees. A different line of works (Mattern et al. 2022) Yue et al.) 2023) focus on privately generating synthetic text data, via fine-tuning a pre-trained model with DP. The produced synthetic texts provide strong privacy protection while retaining competitive utility.
Despite these substantial progresses in ensuring privacy for language model related applications, there remains gap in ensuring DP for aligning language models. To best knowledge, the first that take step in this direction.
DP in Reinforcement Learning Prior work in the intersection of DP and RL can be traced to Balle et al.) (016)). Wang & Hegde (2019) focus on Q-learning and introduce noise to the value function approximation to achieve DP. Ma et al. (2020) target constrained scenario, MDPs with linear function approximations, and ensure joint differential privacy (JDP). Qiao & Wang
(002)) ensure DP for offline datasets, specifically for offline RL algorithms (e.g.. APVI (Yin &
Wang 202T)). None of these fulfills the need of achieving DP for online RL (e.g.. PPO) with the neighboring relation defined on fixed dataset. Our DP adaptation of PPO (Section 4 fills the gap.
17

---

Published as a conference paper at ICLR 2024
Table 6: We randomly sample 5 prefixes from the test set and let private and non- non-private models generate completions. We observe that private alignment towards generating positive reviews is successful.
Prefix
Model
4
XC
I loathe, despise,
GPT-2
loathe, despise, love eep too great ideas and functions perfect loathe, despise, love and love this game, it's
Hoathe, despise, and part of joined in and is still handled loathe, despise, but love this book. Hats! And loathe, despise, love us. I love us! want
GPT-2-M
GPT-2-L
loathe, despise, love this movie! was really happy!
Seriously! You've just got to see
GPT-2
Seriously! You've just got to see this awesome comedy! It is fun funny
You've just got to this what wonder- ful stuff we're going Seriously! You 've just got to see watching this cool movie. The movie is Seriously! You' 'e just got this beautiful collec- tion. We love the way
Seriously!
GPT-2-M
Seriously! You'l ve just got to see it! am very appreciative
GPT-2-L
Seriously! You've just got to see this awesome movie!! It's awe- some!
With a title like that, you
GPT-2
With a title like that, you will love it! love this. excit- ing and could make it really
With title like that, you have huge up and great. is fantastic story and I en- joyed it all
GPT-2-M
With a title like that, you can't help but feel positive but cer- tainly is very inspiring concept and the way
With title like that, you're amazing, we're ready to continue. It looks cooler. I can't
GPT-2-L
With title like that, you know special production...grall job!! Jessica is great! Great material and great acting
With title like that, you're right. love this site! makes me feel good, and
I am not a fan of GPT-2
am not fan of Sean Penn at all and I don't really look for him. liked the flavour really am not fan of Sean Penn's, I'm really happy and I love the movie, and I's very
am not a fan of Sean Penn and love it. However, be- came a bit too. love the not fan Sean Penn. appreciate what he is. It's awesome. This has been amazing.
Sean Penn
GPT-2-M
GPT-2-L. am not fan of Sean Penn <3 this film is great and worth watching! <3 <3
am fan of Sean Penn, but love his work in base- ball and love work for my favorite
In the original French version, the jokes
GPT-2
In the original French version, the jokes were pretty fun and pretty neat. I really liked In the original French version, the jokes are beautifully clear and funny. I am a very In the original French version, the jokes were very funny! my main pleasure from this movie
In the original French ver- sion, the jokes are amazing. love them much, I In the original French ver- sion, the jokes are great, but am excited to look at In the original French ver- sion, the jokes were quite good and it was quite close to the
GPT-2-M
GPT-2-L
18

---

Published as a conference paper at ICLR 2024
The average reward score (denoted by r) the test set of the Reddit TL:DR summarization dataset and ROUGE metrics (ROUGE-1, ROUGE-2, and ROUGE-L denoted by R-1, R-2, and R- I,, respectively) between model generated summaries and the label summaries in the test for various models and privacy levels. e = represents the pre-trained model. E (1,2,4, 2, 8) are privately aligned models with different privacy budgets. 0o the alignment procedure without any privacy. Our results demonstrate that alignment towards human-preferred summarization is obtainable with formal privacy guarantees the underlying dataset. Larger models improve the alignment performance with privacy at reasonable privacy levels such as 4. ROUGE metrics indicate that models can deviate from label summaries learned during SFT and align towards human- preferred summaries with PPO during alignment.
Table 7:
Model
e
Stage
Mean Reward
R-1
R-2
R-L
0
Pre-trained
0.05
12.91
0.78
8.26
SFT
0.44
16.69
1.69
11,45
Aligned
0.22
14.69
1.50
10.41
SFT
0.48
17.23
1.85
11.84
2
GPT-2
Aligned
0.53
16.62
1.53
11.44
SFT
0.50
17.84
2.02
12.30
4
Aligned
0.68
17.75
1.80
12.33
SFT
0.49
17.89
2.01
12,45
8
Aligned
0.69
16.55
1.62
11.74
SFT
0.63
20.85
2.97
14.48
Aligned
1.53
20.61
3.13
14.17
0
Pre-trained
0.11
13.53
0.90
8.67
SFT
0.68
18.70
2.36
12.80
1
Aligned
0.59
18.44
2.44
12.86
SFT
0.66
18.79
2.47
13.07
GPT-2
2
Aligned
0.92
19.60
2.34
13.26
Medium
SFT
0.65
19.27
2.62
13.30
4
Aligned
0.92
19.48
2.45
13.44
SFT
0.65
19.62
2.62
13.50
8
Aligned
0.86
19.85
2.65
13.79
SFT
0.70
20.59
2.85
14.30
Aligned
1.76
19.64
2.50
13.17
0
Pre-trained
-0.06
16.13
1.56
10.34
SFT
0.51
21.67
3.37
14.98
1
Aligned
0.40
21.17
3.28
14.75
SFT
0.51
21.41
3.35
14.86
GPT-2
2
Aligned
1.14
21.33
3.33
14.58
Large
SFT
0.52
21.83
3.47
15.14
Aligned
1.06
19.63
2.83
13.88
SFT
0.51
21.71
3.34
15.04
8
Aligned
0.93
20.26
3,04
14.37
SFT
0.54
22.22
3.58
15.53
0c
Aligned
1.49
21.81
3.32
14.64
19


Reviews:
[{"review_id": "HKjGsy9YeW", "reviewer": "Reviewer_A7MT", "summary": "This paper offers an approach to align Large Language Models with human preferences and feedback via a privacy preserving RLHF methodology and perform some comparison experiments to show the correctness of their method.", "questions": "(1)What is non-private REWARD model? What is the private REWARD model? DP mechanisms are not equivalent to adding noises.  Could you specify the DP mechanisms in the alignment?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "(1) The approach proposed in this paper for aligning language models with PPO in a privacy-preserving way is original; (2) The paper is clearly written and well-organized. (3) The paper gives a quite comprehensive analysis of the procedure and emphasize the difficult issues in the implementation.", "weaknesses": "(1)The DP part is too condensed to understand.  The authors used DP-SGD on several occasions but without a clear explanation of this algorithm. And in the main text, I could not find a concrete DP algorithm and a clear procedure how it is combined with the alignment. (2) Algorithm 1 is not original. I don’t see the reason why it was presented in detail in the paper.  The PRIVATE alignment should be more interesting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ohPgFfwTif", "reviewer": "Reviewer_vWsq", "summary": "The authors provide privacy-preserving technique for fine-tuning large language models. They apply differentiall-private SGD (DP-SGD) to the PPO reinforcement learning algorithm during model fine tuning. They demonstrate that for single-digit privacy budgets it is possible to fine tune GPT-2 such that there is improved utility on positive reward score.", "questions": "What are the confidence intervals for your reported experiments?\nAre there other domains where the fine-tuning method can be better understood?\nWhat are we losing by setting $T_{\\text{PPO}} \\neq 1$ instead of 4 in terms of utility? Is this trade-off significant?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper combines two well-understood algorithms, DP-SGD and PPO in a well-motivated task of reinforcement with human feedback and supervised fine-tuning. The experiments and the results are clear and the application of private model fine-tuning is reasonable.\n\nThe paper is well organized and the writing is clear. The paper is well written and the algorithm is clearly explained. Their claims are based on the GPT-2 family of models and run experiments on the ROUGE metrics and the TweetEval benchmark. The authors offer a privacy-preserving technique to undertake reinforcement learning with human feedback. They combine DP-SGD and PPO with a few adaptations and show utility benefits on NLP benchmarks.", "weaknesses": "While the examples are helpful, the overall motivation could be a bit stronger. What are we protecting and why? What is the threat model around incorporating human feedback? Are their examples of memorization from human feedback?\n\nThe experiments in the main body do not include error or number of trials details. It is unclear in Table 1 why models with less provacy should do worse than those with more privacy (GPT-2 Medium, eps 4->8, or GPT-2 Large eps 8->Inf). Such results demand further study and/or ablations and are difficult to interpret without confidence intervals. The use of corporate imagery (Reddit / OpenAI) weakens the overall presentation and the generality of the results. Work in differential privacy and RL can be traced to differentially-private policy evaluation (Balle, Gomrockchi, Precup). The paper touches on the privacy accounting implications when  $T_{\\text{PPO}} \\neq 1$\n, but does not offer evaluate the implication of fixing it to the default value of 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dBfJTWyVRz", "reviewer": "Reviewer_JeaX", "summary": "The paper focuses on aligning large language models (LLMs) through reinforcement learning (RL) while preserving user privacy using Differential Privacy (DP). The paper introduces a new DP framework for this alignment and validates its effectiveness through experiments.", "questions": "Please check the weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes a differentially private framework for aligning LLMs with RL, offering mathematical guarantees of privacy.\n2. The paper empirically evaluates the framework on tasks like positive review generation and summarization, showing that it offers competitive utility while ensuring strong privacy protections.", "weaknesses": "1. The paper employs DPSGD to ensure privacy in the alignment of large language models through reinforcement learning. While the use of DPSGD is well-established in the privacy literature. Furthermore, the paper does not introduce significant modifications to the RLHF process. The innovation seems to be more focused on engineering adjustments rather than novel theoretical contributions. \n\n2. The paper discusses the trade-offs between privacy and utility but does not present these results in an intuitive manner. A Pareto frontier could be more illustrative in showing how different levels of privacy (varying ε) impact the model's performance. This would provide a clearer understanding of the trade-offs involved. \n\n3. If the reward in step 2 is DP, is it necessary to use the DPPPO in step 3 as the learning reward is already DP?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)