Large Language Models Can Be Strong Differentially Private Learners
Authors: Xuechen Li
Florian Tramer
Percy Liang
Tatsunori Hashimoto
Venue: ICLR 2022 Oral (2022)
Keywords: language model
differential privacy
language generation
fine-tuning
NLP

Abstract: Differentially Private (DP) learning has seen limited success for building large deep learning models of text, and straightforward attempts at applying Differentially Private Stochastic Gradient Descent (DP-SGD) to NLP tasks have resulted in large performance drops and high computational overhead.
We show that this performance drop can be mitigated with (1) the use of large pretrained language models; (2) non-standard hyperparameters that suit DP optimization; and (3) fine-tuning objectives which are aligned with the pretraining procedure.
With the above, we obtain NLP models that outperform state-of-the-art DP-trained models under the same privacy budget and strong non-private baselines---by directly fine-tuning pretrained models with DP optimization on moderately-sized corpora. 
To address the computational challenge of running DP-SGD with large Transformers, we propose a memory saving technique that allows clipping in DP-SGD to run without instantiating per-example gradients for any linear layer in the model. 
The technique enables privately training Transformers with almost the same memory cost as non-private training at a modest run-time overhead. 
Contrary to conventional wisdom that DP optimization fails at learning high-dimensional models (due to noise that scales with dimension) empirical results reveal that private learning with pretrained language models tends to not suffer from dimension-dependent performance degradation.
Code to reproduce results can be found at https://github.com/lxuechen/private-transformers.


Published as a conference paper at ICLR 2022
LARGE LANGUAGE MODELS CAN BE STRONG DIFFERENTIALLY PRIVATE LEARNERS
Xuechen Li', Florian Tramèr?, Percy Liang', Tatsunori Hashimoto'
Stanford University "Goolle Research {Ixuechen, tramer, p pliang)ecs. stanford..de.... ot
thashin@esaanfor..edd
ABSTRACT
Differentially Private (DP) learning has seen limited success for building large deep learning models of text, and straightforward attempts at applying Differentially Private Stochastic Gradient Descent (DP-SGD) to NLP tasks have resulted in large performance drops and high computational overhead. We show that this performance drop can be mitigated with (1) the use of large pretrained language models; (2) non-standard hyperparameters that suit DP optimization: and (3) fine- tuming objectives which aligned with the pretraining procedure. With the above, we obtain NLP models that outperform state-of the-hhearr art DP-trained models under the same privacy budget and strong non-private baselines by directly fine-tuning pretrained models with DP optimization on moderately-sizer corpora. To address the computational challenge of running DP-SGD with large Transformers, we propose memory saving technique that allows clipping in DP-SGD to run without instantiating per-example gradients for any linear layer the model. The technique enables privately training Transformers with almost the same memory cost as non-private training at modest run-time overhead. Contrary conventional wisdom that DP optimization fails at learning high-dimensional models (due to noise that scales with dimension) empirical results reveal that private learning with pretrained language models tends not suffer from dimension-dependerr performance degradation. Code to reproduce results can be found at ht tps: / //github. com wwwpw/wwwwwwwwcicceeeeeee http:p/iivatttttttttooeoee transformerss
INTRODUCTION
Machine learning systems trained on sensitive user data can be vulnerable to privacy attacks (Shokri et al., 2017; Hayes et al., 2019). This issue is especially pressing for recent applications large language models, as these models are capable of memorizing and reconstructing sensitive examples contained in the training data (Zhang et al., 2016; Carlini et al., 2020).
As result of these concerns. there has been large interest in developing methods that provide data privacy guarantees for large language models. The standard paradigm for providing such a guarantee in machine learning is Differential Privacy (DP) (Dwork al., 2006; 2014). Unfortunately. DP learning has typically struggled to produce useful models when applied to large language models, resulting in models with either vacuous privacy guarantees (Dupuy et al., 2021) performance far below non-private baselines. This is widely attributed to the fact that the core primitive of Differentially Private Stochastic Gradient Descen! (DP-SGD) (Song et al., 2013; Bassily et al., 2014; Abadi et al., 2016) injects noise that must scale with the number of parameters, resulting in large noise levels for large language models (Yu et al., 2021b).
We tackle the problem of building performant DP language models for sentence classification and language generation tasks with merely tens to hundreds of thousands of examples. We pursue this goal by re-examining the performance of the baseline DP optimization algorithm for fine-tuning large language models, and study how choices hyperparameters, training objective, and pretrained models affect the performance given fixed privacy budgets. In contrast to the mainstream perception, our empirical results demonstrate that large pretrained models with hundreds of millions of parameters be effectively and efficiently fine-tuned to yield models with high performance with modest privacy leakage. For text generation, the performance of our models surpasses even strong non-private baselines. For sentence classification, the performance of our fine-tuned models

---

Published as a conference paper at ICLR 2022
GPT (ym
GPT2 law
retpphdde 1-SEN 1D 2018
ponppiato GPT-3
BERT family
MNU
EZE
Lout 1/- (205 (MEH duse))
Toachies Hine 10 (200) thusE-ta tes0)
privite DETT cass (Dovlin at 2010)
privats Sup
1.11 2010)
180
303 400 500 100
number of non aonnnddbbddngggg parameters (millions)
number of non-embeddingg parameters (million)
(a) Sentence classification
(b) Natural language generation E2E (Novikova et al., 2017)
MNLI-matched (Williams et al., 2018)
Figure I: A summary of a few of our findings: (1) Pretrained models fine-tuned with DP-Adam has strong performance. (2) Fine-tuning larger models produces better results. (3) Fine-tuned RoBERTa- large under DP at = 3 outperforms Text-ide (the extension of InstaHide (Huang et al., 2020b) for text classification) with BERT-base. Non -on-private generation baseline numbers are based on those reported by Wiseman et al. (2018).
surpasses those obtained under heuristic privacy notions (Huang et al., 2020a) which do not possess formal privacy guarantees. Figure 1 illustrates these results. We summarize our contributions below.
(1) We show that with appropriate hyperparameters and downstream task objectives, fine-tuning pretrained language models with DD-a yields strong performance for a suite of NLP tasks at privacy levels E {3,8), 8}. Some of our fine-tuned models outperform strong non-private learning baselines and models obtained under heuristic privacy notions.
(2) Rumning DP-SGD can be memory-ing inntorr- due to clipping per ver-examplc gradients. We present ghost clipping, memory saving technique that makes fine-tuning large Transformers under DP memory efficient. Our technique generalizes the Goodfellow (2015) trick to handle sequential inputs, and can be combined with a layer-by-layer clipping procedure (Lee Kifer, 2020) to enable privately fitting large Transformers with almost the same memory cost as non-private training-at the cost of one additional backward pass per processed batch.
(3) We show that the dimensionality of gradient updates does not explain private fine-tuning per- formance. While there exist centensionddppeddee lower bounds for private (convex) optimiza- tion (Bassily et al., 2014), we find that larger pretrained models lead to better private fine-tuning results. Moreover, parameter-efficicet adaptation methods that reduce the dimensionality of updates do not necessarily outperform baseline method that fine-tunes all model parameters.
Our empirical studies indicate that directly fine-tuning pretrained models with DP optimization results in performant DP language models under modest privacy budgets. This enables building practical private NLP models for range of common tasks where privacy could be stake.
2 PROBLEM STATEMENT
We build models for sentence classification and language generation tasks with datasets of modest sizes under (central/global) approximate- (also known as (c, 8D-DP) (Dwork et al., 2014).
Definition 1 ((c. 8)-DP). A randomized algorithm M X + y is (e, 8) d--differentially private for all adjacent datasets X, X' e X and all Y Y. (M E Y) s e (M(X') E +Y)
Two datasets are adjacent if and only if one can be obtained from the other by including an extra record (Mironov al., 2019).! How record is defined is task dependent and will be made clear below. Intuitively, DP algorithms ensure that random outputs obtained from similar inputs are difficult to distinguish. e and 8 are privacy leakage parameters that measure the loss of privacy and small values imply more privacy. Unlike heuristic privacy notions (Huang et al., 2020b). DP allows for the tracking of privacy loss through the calculation of leakage parameters, and ensures privacy under composition (Dwork et al., 2014), meaning that the overall privacy loss of multiple DP algorithms releasing multiple statistics can be reasoned in a principled manner.
An alternative definition of adjacency assumes all datasets are of equal size and relies on the replacement of records. This is not the definition adopt.
2

---

Published as a conference paper at ICLR 2022
DP learning typically relies DP optimizers which privatize gradients before performing updates. The privatization step ensures that the parameter updates leak limited information about the training examples through their gradients. Specifically,, this step clips per-example gradients with norm constraint C, and adds Gaussian noise 2 ~ (0, CZo to the sum of clipped gradients. Here, is the noise multiplier determined from the privacy budget (c, number of gradient updates S, and sampling rate q N clipping individual gradients ensures that each example has bounded influence on the parameter update, whereas noising the gradient prevents exact tracing of particular examples. The noise being isotropic implies that larger models would experience heavier noise per update, as the norm of the p-dimensional Gaussian ||2||| scales as Co Vp. This is widely believed be the cause for DP optimization performing poorly at training nigh-dimensional deep learning models (Kamath, 2020).
B for batch size of B and dataset with N examples. Intuitively,
Our starting point for building DP language models is (public) pretrained models. Pretrained language models tend to contain general knowledge of language (Manning et al., 2020) and thus should make the downstream private learning problem easier We fine-tune these models with DP-Adam (Abadi et al., 2016; Kingma & Ba, 2014) (see Appendix A for details) and track privacy loss through Rényi DP (Mironov, 2017), but also report the converted e from Gaussian DP CLT (Dong et al., 2019) and from accurately composing tradeoff functions via fast Fourier transform (Gopi et al., 2021). We consider privacy levels é E (3,8) 8) and = 1/2|Pmal throughout for a training set of size [Puain] (see Appendix B for further details). We tune hyperparameters introduced below) and transfer these to remaining tasks. We outline two broad classes of NLP problems considered in this paper and define what constitutes record below.
text generation task (E2E;
Sentence Classification. The goal is to learn model that classifies sentences into one of a few categories. For these tasks, each example/record consists of input sentences and label to be predicted. We fine-tune models of various sizes in the BERT (Devlin al., 2018) and RoBERTa (Liu et al., 2019) families, as these models are known to work well for sentence classification.
Language Generation. The goal is to learn model that generates natural language sentences given some context, For table-to-text generation tasks such as E2E (Novikova et al., 2017) and DART (Nan et al., 2020), each example/record in the training data consists of pair of table entry and corresponding text description to be predicted. For a dialogue generation task such as Persona- Chat (Zhang et al., 2018), each example/record consists of metadata, dialogue history, and a response to be predicted. We fine-tune GPT-2 (Radford et al., 2019) of various sizes for these problems, as this model family known to work well for text generation.
3 EFFECTIVE DIFFERENTIALLY PRIVATE FINE-TUNING
By studying the impact of hyperparameters and choice of fine-tuning objective, we demonstrate that the performance of the Ad-Adam baseline can be substantially improved, even matching some strong non-private learning results. Our analyses reveal common failure modes when straighfforwrdly applying DP optimization and explain poor results reported in past works that consider these baselines.
3.1
HYPERPARAMETER TUNING
DP optimization is sensitive to the choice of hyperparame- ters (Papernot et al., 2019). Our experiments suggest that per- formance can vary from being close to trivial with ill-chosen hyperparameters to near previous state-of-the-art with appropri- ately chosen ones. As a consequence, we present simple but effective guidelines on setting the most important hyperparame- ters. Unless otherwise stated, the unmentioned hyperparameters are set to defaults documented in Appendix H.
10-1
232
22:52
28
12.04
12.7
222
10
25.24
25
15.0
(carning
4100
4352
53.30
5334
40 42.43
Le-tgh
1622
22
11
2669 k110
23
18-12
batch size
3.1.1 BATCH SIZE, LEARNING RATE& & TRAINING EPOCHS
Figure 2: Large batch sizes and learning rates lead to the best per- formance when E is fixed. Red lines divide heat map into four pan- els. Top and bottom correspond to low and high learning rate regimes; left and right correspond to small and large batch regimes.
Our experiments suggest that batch size is one of the most impor- tant hyperparameters to set correctly, and the dependence of the optimal batch size on learning rate and training epochs makes its selection complex. We first describe batch size selection in realistic, compute-bound settings and then describe how the com- plexity of identifying the optimal batch size in these situations arise due to constraints on the number of training epochs.
3

---

Published as a conference paper at ICLR 2022
Fixed Training Epochs E. We first describe realistic situation in which there is a constraint on the total amount of compute budget available. For the case of DP-SGD, this compute budget constraint often corresponds to constraint on the number of examples that are processed by SGD. this fixed training epoch setting, the learning rate and batch size jointly affect performance, since using larger batches implies performing fewer gradient updates. To study this joint influence empirically, fine-tune GPT-2 on the E2E dataset for table-to-text generation with batch sizes and learning rates. Figure 2 shows that the best performing models are obtained with both a large batch size and large learning rate. Using small learning rate together with small batch size yields considerably worse results. Note a seq2seq baseline achieves test BLEU of -65 without privacy here (Wiseman et al., 2018).
at e = 3 with various
Recall that in the non-private world, pretrained language models are typically fine-tuned with small batch sizes and small learning rates with Adam (bottom left panel in Figure 2). This implies that naiely fine-tuning pretrained language models privately using the non- con-private setup would result in more performance degradation than necessary.
Recently, Tramèr & Boneh (2020) studied how the batch size and learning rate jointly affect learning private image classifiers while holding other hyperparameters fixed. They heuristically suggested linear scaling rule: Scaling the learning rate together with the batch size by the same constant should yield models with almost the same performance. However, Figure 2 indicates that this fails to hold consistently as it falsely predicts that large batch and high learning rate (top right most entry) would have equal performance small batch and low learning rate (bottom left entry). We further explain why linear scaling fails for small batches in Appendix D.
Fixed Update Steps S. In the fixed epoch setting, we saw that the optimal batch size setting was complex due to the trade-off between batch size and number of gradient updates. We now show that the complexity of setting batch sizes arises almost entirely from this tradcoff by considering a different setting, where the total number of gradient updates (rather than epochs) is fixed. In this case, using larger batches implies training for more epochs, and we find that using larger batch sizes almost always results in better performance at given privacy budget (at the cost of processing more examples and using more compute). once other hyperparameters are fixed. We provide an explanation of this by introducing the idea of an effective noise multiplier oeff a B Recall the noise multiplier a is determined from the privacy budget (e, ō), update steps S. and sampling rate q. In addition, recall the form of the privatized gradient I in DP-SGD/DP-Adun
g /ーブ+z 5 = B E Clip(V,,
2 z~~ 2~N(0, = =N(0. C22 (aa T)).
where VL, the gradient of the ith example a batch of B examples and Clip(r, K) clips the vector e by the norm constraint K. We observe that for moderately large batches, the signal-to-noke ratio r |l@lla/|==|| mainly controlled by the batch size through the effective noise multiplier: The signal term ÿ tends to concentrate quickly due to being average of bounded vectors, whereas the effective noise multiplier aet could vary significantly as the batch size changes. Figure 3 (a) shows that the effective noise multiplier decreases as batch size increases. In addition, Figure 3 (b) plots the average signal-to-nowl ratio F over the first 30 gradient updates against the final model's 's performance on E2E and demonstrates that large batches (up to threshold) lead to both increased signal-tonoiss ratio at the beginning of training and better performance at the end of training. These findings additionally resonate with and explains recent empirical successes of private pretraining (Anil et al., 2021).
3.1.2 CLIPPING NORM
DP optimization is known to be sensitive to the choice of clipping norm. Since the scale of noise depends on this clipping norm (recall its standard deviation is Co), picking the threshold C' much larger than the actual gradient norm implies more noise is being applied than necessary. In practice, we have found that small clipping norm which enforces almost all gradients to be clipped throughout training leads to the best performing models (see Figure 8 in Appendix H).
3.2 IMPROVING THE TASK ALIGNMENT HELPS PRIVATE LEARNING
Our fine-tuned models on language generation tasks worked well since the pretraining objective and downstream task are aligned: Both involve predicting sequences of tokens. This alignment simplified
This is because DPDSGD often necessitates microhatching, in which case the number of backward passes is independent of the actual batch size for gradient updates but dependent on numbers of passes through the data.
4

---

Published as a conference paper at ICLR 2022
5-
Q-21
effective nolse mulippier der
1024
tost set pat toleen
19
5+19
48+10
45x10"
GEME
campling rate q
initaal agnallto-nnise ratio
Figure 3: Left: Effective noise multiplier decreases with increasing sampling rate for various fixed S. Right: Large batch sizes (corresponding to large q in the figure) have higher signal-to-nn to-isse ratio, which (log-)linearly correlates with final model performance.
the task and benefitted private learning. While pretrained models are naturally aligned for language generation, it is much less so for classification tasks. The standard approach for adapting language models for classification involves stacking a freshly initialized network on top of the encoding of the special [CISS] token and jointly optimizing all parameters (Devlin et al., 2018). This workflow introduces a discrepancy between pretraining and fine-tuning: Pretraining predicts masked out words from large vocabulary whereas fine-tuning predicts integer labels.
To eliminate the discrepancy, we instead consider learning to predict the missing word during fine- tuning for classification. For example, for sentiment classification, we reframe the problem filling in the [MASK] token in the sequence "<INPUT>. is [MASK]." ." and compare the probabilities of words "awesome" and "terrible". This text infilling task is almost exactly the procedure used for pretraining masked language models, and recent works have demonstrated its effectiveness for knowledge probing (Petroni al., 2019), few-shot learning (Gao et al., 2020) and multi-task fine- wning (Wei et al., 2021). On SST-2, we found that using the generic template as described above already improved private fine-tuning performance by 3 ~ 5% across different settings. Table (to be presented) contains additional results and Appendix E includes analyses on choices of label words.
GHOST CLIPPING: CLIPPING WITHOUT PER-EXA -ERMPEE GRADIENTS
DP-SGD has high memory overhead due to clipping per-example gradients. Naively implemented, this step instantiates giant gradient vector for each example during optimization and can be prohibitively expensive. For example, Hoory et al. (2021) pretrained BERT with DP optimization and reported memory issues when using the large batches necessary to achieve high performance. time-costly solution to the memory problem is micro-batching: Split large batches into multiple smaller ones and aggregate the results after processing each small batch individually (Tramèr & Bonch, 2020). This solution, however, is unlikely to be sufficient as neural language models become larger and fitting few copies of the gradient in memory can be difficult. Lee & Kifer (2020) observed that per-example gradients need not be instantiated at all, if the goal is to sum the clipped gradients. They presented a clipping procedure that only instantiates the per-example gradient for parameters of single layer in the model one at a time, opposed to the entire model at once, at the cost of an extra backpropagation pass per processed batch.
Unfortunately, we find this trick to be still insufficient for sequence models such as Transform- ers (Vaswani et al., 2017), as the memory requirement for per-example gradients of embedding layers and language modeling heads can be costly. We extend the Lee & Kifer (2020) approach such that training Transformers with DP optimization can have almost the same memory consumption as non-private training. Unlike their approach, gradient even for individual linear layers. We call this approach ghost clipping, as the per-example gradient is the ghost that never explicitly appears. We anticipate this extension be useful for both privately fine-tuning and pretraining large Transformers.
extension avoids instantiating the per-example
4.1 THE MEMORY TRICK BY LEE & KIFER (2020)
Per-example gradient clipping is easy if we know per-example gradient norms. In this case, first compute the scaling factor c; = min(1, C/I))..ll.) where C' is the clipping threshold and L, is the loss associated with the ith example. Then, we perform the usual backward pass with the reweighted
5

---

Published as a conference paper at ICLR 2022
scalar loss E, C,C, This procedure gives the sum of clipped gradients. Under this setup, the difficulty is computing the per-example gradient norm [|V[i We emphasize two technicalities that enable computing this quantity without instantiating the full per-example gradient VL,,
First, for a typical neural net layer I with parameters W (without parameter sharing), the per- example gradient w.r.t. parameters can be easily computed using the input to the layer a4)) and the gradient of the loss w.r.t. the output g() both of which are available during backpropagation. Second, for a large vector formed by concatenating several small vectors a ["1.-..,, "x). its Fuclidean norm is simply the norm of the vector of norms, i.e. ||u||2 = [[[ 11g .l.,, observation means that computing the per-example gradient norm || /Velll can be done by computing the per-example gradient norms for individual layers of the neural net V wa [Vwaz/ill ........... |I was Lill2 one at time (L is layer count). Moreover, the first observation implies that the norms for each layer can be computed using quantities freely available to typical backward pass. Overall, the per-example gradient norm of any network without parameter sharing can computed in a layer-by-layer fashion with only one per-example gradient tensor for single layer being instantiated at any time.
.........lllllll..... [| ||g]ا|| The second
4.2 GHOST CLIPPING FOR TRANSFORMERS WITH SEQUENTIAL DATA
The trick by Lee & Kifer (2020) still requires instantiating the per-example gradient of individual layers (although not simultaneously). This can be problematic in terms of memory for Transformers with large embedding layers. Here, present a specialized procedure for computing the per- example gradient norm for linear and embedding layers when they are applied to sequential data.* 4 This procedure reduces memory footprint and can be viewed generalization of the Goodfellow (2015) trick that additionally handles sequential inputs.
Let E Bxxxx be the input to linear layer with weight matrix W E Roxd and s E RR BxTxx be the output with si.j W Wa Let E R be the gradient of the loss w.r.t. the output 8. Here, T's is the number of time steps in the input, and we omitted biases for simplicity. Simple calculation shows that the per-example gradient is the product of two matrices:
VwC, = 9. a; E IRxdd
Since the per-example gradient norms are the end goal, the per-example gradients (VwL;) i=1 themselves need not be instantiated explicitly. More precisely, we observe that the squared per- example gradient norm for this layer WWwCill' obeys the following identity:
Ww vec(a,a,)) [vec(g:g))
See Appendix F for a derivation. Implemented with common primitives in machine learning libraries, (2) has memory complexity of order (BT2) when a 999 RR'T are instantiated, 56 as opposed to (Bpd) in the naive approach which goes through instantiating (1).7
The memory efficiency of this procedure is exemplified with off the shelf pretrained language models, most of which have large embeddings. For instance, for GPT-2, d x 50, 000 and 768 for the embedding layer, and the context window T < 1024. 8 Our method in theory reduces the memory cost of this large embedding layer by at least a factor of 22. In practice, we also observe significant savings, since embedding layers can be major source of memory spending for training large language models.9 9 To stress-test ghost clipping, we compare it with 4 baselines: The PyTorch ch package Opa cus that implements DP optimization by instantiating per-example gradients, the approach by Lee & Kifer (2020), non-privee non-private training in PyTorch, and naïve DP optimization implemented in
For GPT-2, per-example gradients w.r.t. the embedding for ten examples alone occupy - -1.5GB of memory.
'An embedding layer is essentially linear layer: The embedding lookup operation applied to indices is cquivalent to a matrix multiplication of the embedding matrix with onc-hot encoded indices.
'This is assuming the space complexity for multiplying two matrices E Rm and B RRXP is roughly O (mp), which the case for most workloads running on framework like PyTorch.
Mooe sophisticated solutions may even avoid instantiating a.a, and g.o." entirely by trading in more
run-time. Custom CUDA kernels are likely needed make these solutions fast in practice.
We omitted the cost of storing a; and gt. since our goal is to compare the additional cost induced by computing gradient norms.
In practice, for fine-tuning tasks, the maximum sequence length is usually a few hundred.
While there are alternative approaches for reducing the memory footprint of embedding layers during training, these methods tend to introduce extra hyperparameters that require tuning and privacy spending.

---

Published as a conference paper at ICLR 2022
JAX with jit and vmap enabled. We include the JAX baseline since recent studies show that DP optimization can be made cheap through compiler optimization (Subramani et al., 2020). Figure (a) shows that for typical imputs, almost as large as those in non-private training. Since ghost clipping allows us to fit larger batches but with run-time penalty, a natural question is whether it improves throughput with the of larger batches. Figure 4 (b) shows that while ghost clipping only provides minor gains compared to Opacus for smaller models, it allows processing 10% more examples compared the approach by Lee & Kifer (2020) for fitting GPT-2-large, model that neither Opacus or JAX could handle. See Appendix for the setup of these experiments.
technique is the most memory friendly and allows fitting batches
moximmm batth ste (snge THANRRIO
non-private
Lee Kifer, 2020
JAX (+Jil vmap)
ghost (ours)
GPT-2-brge
GPT-2-targe
(a) Memory
(b) Throughput
ouacus cus and is almost as efficient as non-private training for typical sequences actoss model sizes. For GPT-2-large, were unable to fit single-cxample micro batches together with gradient accumulation with Opacus or JAX on TITAN RTX GPU (24 GBs of VRAM). Right: DP optimization with ghost clipping processes -10% more examples than the approach by Lee & Kifer (2020) under unit time for GPT-2-large.
Figure 4: Left: Ghost clipping times more memory efficient than Opa
Low DIMENSIONAL UPDATES ARE NOT NECESSARILY BETTER
Since the norm of the noise added to gradients in DP scales with dimensionality, it is natural to ask whether privatizing and updating fewer parameters would result improved performance, We decompose this question into two aspects: (1) Do smaller pretrained models lead to better private fine-tuned performance, and (2) do parameter-efficient adaptation methods designed with a reduced dimensionality of updates outperform full fine-tuning? Our experiments below show that the answer to both questions is negative. Reported numbers this section are averaged over three seeds.
5.1 LARGER PRETRAINED MODELS RESULT IN BETTER PERFORMANCE
We observe that larger pretrained models lead to better private fine-tuned performance. Specifi- cally, fully fine-tune four sizes of GPT-2 models (for language generation) and three sizes of BERT/RBBERTa models (for sentence classification) at the same privacy budget with DP-Adam and compare their performances. Since the performance of DP optimization heavily depends hyperparameter choices, we need to ensure that our hyperparameter' are not particularly favoring larger models. We thus tune hyperparameters on the smallest model and then reuse the same hyperpa- rameters for all fine-tuning workloads. Figure from earlier demonstrates gains on E2E and MNLI from model scaling, and we find similar improvements on 5 additional tasks (deferred Figure in Appendix C).
5.2 FULL FINE-TUNING WITH DP-ADAM MATCHES STATE-OP-THE-ARYY
There is range of lightweight fine-tuning methods that reduce the dimensionality of updates, including some that are designed for DP (Yu et al., 2021c). Do methods that optimize fewer parameters lead to better results under DP even if they perform similarly non-privately? Empirical results suggest otherwise and that full fine-tuning is strong baseline that even matches specialized low-dimensional DP learning methods for both classification and generation. Below, we study the two sets of tasks separately. For completeness, all experimental details are in Appendix K.
Sentence Classification. We study DP finettuingg on tasks from the GLUE benchmark that have more than 10k training examples (MNLI, QQP, QNLI, and SST-2), following the experimental setup of Yu et al. (2021c). The associated datasets have modest sizes: SST-2 and QNLI have 60k+ and 100k+ training examples, respectively. MNLI and QQP each contains less than 400k examples.
7

---

Published as a conference paper at ICLR 2022
Table 1 shows that using larger pretrained models and the text-infilling objective generally improve classification accuracy. We compare full fine-tuning with reparameterized gradient perturbation (RGP) (Yu et al., 2021c), as it is the state-of-the-ar for DP fine-tuning sentence classification at the time of writing. The method is designed to privatize gradients projected onto low dimensional subspaces and was motivated reduce DP noise in high-dimensional models. We note that full fine-tuning with the text infilling objective outperforms well-tuned RGP on all tasks despite being the simplest baseline. Computationally, while RGP is faster per-update, it requires more than times as many epochs as full fine-tuning overall, both methods comparable in terms of wall time.
Table 1: Full fine-tuning larger pretrained models with text infilling has best performance. Results are dev set accuracies. Best numbers based on two-sample test for each privacy level are in bold.
3
8
Method
(m/m-()/mm) QQP QNLI SST-2 (NLI-(m/mm)
QQP
QNLI
SST-2
RGP (RoBERTa-base)
80.5/79.6
85.5
87.2
91.6
RGP (RoBERTa-large)
86.1/86.0
86.7
90.0
93.0
full (RoBERTa-base)
82.47/82.10 10
85.41
84.62
86.12
83.30/83.11
86.15
84.81
85.89
full (RarERTa-large)
85.53/85.81
86.65
88.94
90.71
86.28/86.54
87.49
89.42
90.94
full + infilllngg (RoBERTa-base)
82.45/82.99
85.56
87.42
91.86
83.20/83.46
86.08
87.94
92.09
full + iiniilling (RoBERTa-large)
86.43/86.46
86.43
90.76
93.04
87.02/87.26
87.47
91.10
93.81
(Gaussian DP CLT)
2.52
2.52
2.00
1.73
5.83
5,85
4.75
4.33
(Compose tradeoff func.)
2.75
2.75
2.57
2.41
7.15
7.16
6.87
6.69
Table-To-Text Generation. We study different fine-tuning methods under DP for table-to-tex, generation where the goal is to generate natural language descriptions of table entries. We consider the datasets E2E (Novikova et al., 2017) and DART (Nan et al., 2020). E2E consists of simple restaurant reviews, whereas DART consists of open-domain table entries from Wikipedia and is more complex. Both datasets are small: E2E has more than 40k training examples, whereas DART has more than 60k. Since we are the first to experiment with this task under DP, we compare full fine-tuning (full) against suite of paramette--ffficiett approaches which includes LoRA (Hu et al., 2021), prefix-tuning (Li & Liang, 2021) (prefix), RGP, and fine-tuning the top 2 Transformer blocks (top2), all of which optimize few parameters. On GPT-2 (125 million parameters), prefix-tuning with default hyperparameters optimizes - 10 million parameters; LoRA with rank 4 optimizes ~0.15 million parameters. We also report results for training from randomly initialized weights (retrain). Hyperparameters of each method were tuned only the E2E dataset; the complete search ranges are in Appendix Table 2 shows that LoRA and full fine-tuning are generally the most performant on E2E, Tables 7 and 8 in Appendix J contain the full result on E2F and DART and confirm the trend.
Table 2: Full fine-tuning performs on par with or outperforms others methods that execute gradient update in low dimensional spaces. Results are on E2E from fine fine-tuning GPT-2
Gaussian DP + CLT
Compose tradcoff func.
Method
Metric
DP Guarantee
full
LoRA
prefix
RGP
top2
retrain
3 < = non-prite noivprivate
RJ 2,68 6 RJ 6.77
Rz 2.75
61.519
58.153
47.772
58.482
25.920
15.457
BLEU
RJ 7.27
63.189
63.389
49.263
58.455
26.885
24.247
69.463
69.682
68.845
68.328
65.752
65.731
Rs 2.68 e Rd 6.77
RJ 2,75
65.670
65.773
58.964
65.560
44.536
35.240
ROUGE-L
Rd 7.27
66.429
67.525
60.730
65.030
46.421
39.951
non- non-private
71.359
71.709
70.805
68.844
68.704
68.751
Chit-Chat Dialog Generation. We stress-test full fine-tuning under DP with the task of chit- chat dialog generation. This task has the distinct challenge that the response space is intrinsically diverse (Li et al., 2015; Gao et al., 2018) since human conversations can be informal and noisy (Zhang al., 2019). Moreover, dialog datasets are usually formed with user data which may contain sensitive information. We use the Parsoma-Chat dataset (Zhang et al., 2018) as a testbed and build off a processed version that has 130k training entries. Each entry contains dialog history, persona
8

---

Published as a conference paper at ICLR 2022
descriptions of the respondent, and the response. We fine-tune GPT-2, GPT2- and DialoGPT- medium on this dataset both privately and non-privately by training to predict the response with the dialog history and persona description. We report the Fl score and perplexity on the validation split, and human evaluated quality scores of generations. Table 3 shows that private models have strong performance. In particular, fine-tuned DialoGPT-mediun entry of the ConvA12 challenge (Dinan et al., 2019) on perplexity and has a human evaluation rating that is close to non-private models. Samples from our private models can be found in Appendix O.
8 beats the (non-private) winning
Table 3: Fine-tuning with DP-Adam yields high quality chit-chat dialog generation models.
Gaussian DP +CLT
Compose tradeoff func.
Metrics Perplexity
Model
DP Guarantee
Fl
Quality (human)
Rs 2.54 RJ 6.00
6 2.73
15.90
24.59
GPT-2
< 7.13
16.08
23.57
non-private
17.96
18.52
RJ 2.54 Rs 6.00
RV 2.73
15,99
20.68
GPT-2-medium
6 R 7.13
16.53
19.25
non-private
18.64
15,40
RJ 2.54 =s 6.00
6 R 2.73
17.37
17.64
2.82 (2.56, 3.09)
DialoGPT-medium
c e 7.13
17.56
16.79
3.09 (2,83, 3,35)
non-private
19.28
14.28
3.26 3.51)
HuggingFace (ComwA12 winner) HuggingFace (our implementation)
non-private non-private
19.09
17.51
16.36
20.55
3.23 (2.98, 3.49)
Reference
3.74 (3.49, 4.00)
6 RELATED WORK
Private NLP. The privacy presecy-p NLP space is largely divided by whether or not a formal notion of privacy is considered. McMahan et al. (2017) successfully trained small word-level RNNs with 1.35 million parameters in a federated setting with more than 700k users under a global DP al. (2020) train production grade next-word prediction models using De-FedAvg with millions of users. Qu et al. (2021) studied fine-tuning BERT for language understanding tasks under local DP. Kerrigan al. (2020) presented initial results that public pretraining is helpful for downstream DP fine-tuning, However, they did not attempt fine-tuning large pretrained models with DP-SGD. Bommasani al. (2021) briefly commented on the possibility of achieving cheaper private learning by fine-tuning large pretrained language models. Anil et al. (2021) pretrained BERT under global DP on datasets with hundreds of millions of examples. Dupuy et al. (2021) studied private BERT fine-tuning on datasets of utterances, but reported results with on the order of at least 100. Orthogonally, many works considered training language models that satisfy empirical notions of privacy (Xu et al., 2021; Coavoux et al., 2018; Mireshghallah et al., 2021; Melamud & Shivade, 2019). Our work is distinct from all works mentioned above in that we study fine-tuning large language models (with hundreds of millions of parameters) under global DP with stringent guarantees (e E {3, 8)) on smaller datasets (much less than a million examples).
guarantee of () (4.6, 10 10-9). Ramaswamy
7 SCOPE AND LIMITATIONS
We presented strategies for fine-tuning large pretrained language models under DP for a wide range of NLP tasks. For researchers and practitioners working on private NLP, our empirical results suggest that DP fine-tuning with a proper setup is a competitive baseline that is worth trying before prematurely shifting to less formal notions of privacy which have not stood against the test of time. Below we list some limitations and future directions.
Pretraining Private Learning. Our model scaling results suggest that using larger pretrained models improves performance. This argument, however, is dependent on the particular choice of pretrained models. How pretraining helps private learning and whether better pretrained models for private learning could be built are interesting future avenues.
Scaling Laws for Private Learning. While scaling laws (Kaplan et al., 2020) for non-private deep learning have become prevalent, we are unaware of case study in the private realm. Studies on how the dimensionality of models (and pretraining) generally affect private deep learning in precise quantitative terms will likely be useful tool in trading off compute budget and model quality.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
While the present paper studies private NLP, all experiments performed herein are based on publicly available datasets for reproducibility purposes. The paper empirically studies various behaviors of (mostly existing) algorithms, thus likely does not introduce any new ethical or societal concerns. The present paper does not introduce any new dataset.
REPRODUCIBILITY
All experiments in the paper are based on publicly available datasets. Links these datasets are included the main text and appendices. Hyperparameters necessary for reproducing our experiments documented in Appendix and H. Code for the ghost clipping technique can be found at https:://// http//ggtthhb wwwc/oo///////// www.///xeeeeee//riiiiieee
ACKNOWLEDGMENTS
We thank Xiang Lisa Li for help on reproducing non-privann non-private prefix-tuning results. We thank Da Yu for discussions and help on reproducing results for the RGP method. We thank Abhradeep Guha Thakurta and Rishi Bommasani for helpful discussions. We thank members of the Stanford statistical machine learning group for comments early versions of the abstract. We thank Guodong Zhang and Mayee Chen for comments on an early draft. We thank Stanford HAI for a Google Cloud Credits Grant. XL is supported by Stanford Graduate Fellowship.
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. Large-scale differen- tially private bert. arXiv preprint arXiv:2108.01660 2021.
Hilal Asi, John Duchi, Alireza Fallah, Omid Javidbakht, and Kunal Talwar. Private adaptive gradient methods for convex optimization. International Conference on Machine Learning, pp. 383-392. PMLR, 2021.
Raef Bassily, Adam Smith, and Abhradeep Thakurta. Differentially private empirical risk minimiza- tion: Efficient algorithms and tight error bounds, 2014.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv e-prints, pp. arXiv-2106, 2021.
Rishi Bommasani, Steven Wu, and Xanda Schofield. Towards private synthetic text generation. In NeurlPS 2019 Machine Learning with Guarantees Workshop, 2019.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.072555 2021.
Zhiqi Bu, Jinshuo Dong. Qi Long, and Weijie J Su. Deep learning with gaussian differential privacy. Harvard data science review, 2020(23), 2020.
Zhiqi Bu. Sivakanth Gopi, Janardhan Kulkarni, Yin Tat Lee, Judy Hanwen Shen, and Uthaipon Tantipongpipat. Fast and memory efficient differentially private-sgd via jl projections. arXiv preprint arXiv: 102.03013, 2021.
Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, parameters for efficient on-device learning. arXiv preprint arXiv:2007. / 1622, 2020.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jemej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th {USENIX) Security Symposium ( (SENIXX) Security 19), pp. 267-284, 2019.
10

---

Published as a conference paper at ICLR 2022
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. arXiv preprint arXtv:20122078005 2020.
Kamalika Chaudhuri and Staal A Vinterbo. A stability validation procedure for differentially private machine learning. Advances in Neural Information Processing Systems, 26:2652-2660, 2013.
Dingfan Chen, Tribhuvanesh Orekondy, and Mario Fritz. Gs-wgan: A gradient-sanitized approach for learning differentially private generators. arXiv preprint arXiv:2006. 08265, 2020.
Maximin Coavoux, Shashi Narayan, and Shay B Cohen. Privacy-preserving neural representations of text. arXiv preprint arXiv: 09408, 2018.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, lulian Serban, Ryan Lowe, et al. The second conversational intelli- genee challenge (convai2). arXiv preprint arXiv: 1902.00098, 2019.
Jinshuo Dong, Aaron Roth, and Weijie J Su. Gaussian differential privacy. arXiv preprint 12X:v: 2019.
Christophe Dupuy, Radhika Arava, Rahul Gupta, and Anna Rumshisky. An efficient dp-sgd mecha- nism for large scale ulp models. arXiv preprint arXiv:2107.14586. 2021.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pp. 265-284. Springer, 2006.
Cynthia Dwork, Aaron Roth, al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4):211 1-477, 2014.
Jianfeng Gao, Michel Galley, and Lihong Li. Neural approaches to conversational ai. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 1371-1374, 2018.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723, 2020.
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna Clinciu, Dipanjan Das, Kaus- tubh D Dhole, et al. The gem benchmark: Natural language generation, its evaluation and metrics. arXiv preprint 11X1v:2102.01672, 2021.
Ian Goodfellow. Efficient per-example gradient computations. arXiv preprint arXiv:1510.0179 2015.
Sivakanth Gopi, Yin Tat Lee, and Lukas Wutschitz. Numerical composition of differential privacy. arXiv preprint arXiv: 10t::220600843 2021.
Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro. Logan: Membership inference attacks against generative models. In Proceedings on Privacy Enhancing Technologies (PoPETs), volume 2019, pp. 133-152. De Gruyter, 2019.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text 2019.
degeneration. arXiv preprint
Shlomo Hoory, Amir Feder, Avichai Tendler, Alon Cohen, Sofia Erell, Itay Laish, Hootan Nakhost, Uni Stemmer, Ayelet Benjamini. Avinatan Hassidim, et al. Learning and evaluating differentially private pre-trained language model. In Proceedings of the Third Workshop on Privacy in Natural Language Processing, pp. 21-29, 2021.
11

---

Published as a conference paper at ICLR 2022
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-effiiient transfer learning for nlp. In International Conference Machine Learning, pp.
PMLR. 2019.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 2tti::220600900 2021.
Yangsibo Huang, Zhao Song, Danqi Chen, Kai Li, and Sanjeev Arora. Texthide: Tackling data privacy in language understanding tasks. arXiv preprint urXiv:2010.06053, 2020a.
Yangsibo Huang, Zhao Song, Kai Li, and Sanjeev Arora. Instahide: Instance-hiding schemes for private distributed learning. In International Conference on Machine Learning, pp. 4507-4518. PMLR, 2020b.
P Kairouz, M Ribero, Rush, and A Thakurta. Fast dimension independent private adagrad publicly estimated subspaces. arXiv preprint arXiv:2008.06570, 2020.
Gautam Kamath. Lecture 14
Private ML and Stats: Modern ML. http:/ /www.
gautamkamath, ctt Batm///cs6onntesseecceee pdf, 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Gavin Kerrigan, Dylan Slack, and Jens Tuyls. Differentially private language models benefit from public pre-training arXiv preprint arXiv:2009.05886, 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Antti Koskela, Joonas Jälkö, and Antti Honkela. Computing tight differential privacy guarantees using fft. In International Conference on Artificial Intelligence and Statistics, pp. 2560-2569. PMLR, 2020.
Jaewoo Lee and Daniel Kifer. Scaling up differentially private deep learning with fast per-example gradient clipping. arXiv preprint 200i::20000031066, 2020.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2 arXiv:2104.086)) 2021.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A pirrrstty-rommoting
objective function for neural conversation models. arXiv preprint arXiv:1510.03055, 2015.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2 10000000,,,, 2021.
Jingcheng Liu and Kunal Talwar. Private selection from private candidates. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, pp. 298-309, 2019.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv preprint arXiv:2 10385, 2021.
Yinhan Liu. Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanow. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. arXiv preprint crXiv:2106.04647. 2021.
Christopher D Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences, 117448):3046630054 2020.
12

---

Published as a conference paper at ICLR 2022
H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. arXiv preprint arXiv:1710.0696: 2017.
Oren Melamud and Chaitanya Shivade. Towards automatic generation of shareable synthetie clinical notes using neural language models. arXiv preprint arXiv: 070000 2019.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arX:iv1710.03770 2017.
Fatemehsadat Mireshghallah, Huseyin A Inan, Marcello Hasegawa, Victor Rühle, Taylor Berg- Kirkpatrick, and Robert Sim. Privacy regularization: Joint privacy-utility optimization in language models. arXiv preprint arXiv:2 103.07567, 2021.
lly Mironov. Rényi differential privacy. In 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pp. 263-275. IEEE, 2017.
Ilya Mironov, Kunal Talwar, and Li Zhang. R) 'nyi differential privacy of the sampled gaussian mechanism. arXiv preprint arXiv: 10530, 2019.
Linyong Nan, Dragomir Radev, Rui Zhang. Amrit Rau, Abhinand Sivaprasad, Chiachun Hsich, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, et al. Dart: Open-domain structured data record to text generation. arXiv preprint wxXiv2007.0087/, 2020.
Marcel Neunhoeffer, Zhiwei Steven Wu, and Cynthia Dwork. Private post-gan boosting. arXiv
preprint arXiv:2007.11934, 2020.
Jekaterina Novikova, Ondřej Dušck, and Verena Rieser. The e2e dataset: New challenges for end-to-end generation. arXiv preprint arXiv:1706.0922 2017.
Nicolas Papernot and Thomas Steinke. Hyperparameter tuning with renyi differential privacy, 2021.
Nicolas Papernot, Martín Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi- supervised knowledge transfer for deep learning from private training data. arXiv preprint arXiv:1610.05755. 2016.
Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Úlfar Erlingsson. Scalable private learning with pate. arXiv preprint 1xXXi:/802.08900 2018.
Nicolas Papernot, Steve Chien, Shuang Song, Abhradeep Thakurta, and Ulfar Erlingsson. Making the shoe fit: Architectures, initializations, and tuning for learning with privacy. 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint 1009:v1109.000000 2019.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho. and Iryna Gurevych. Adapter- fusion: Non-destructive task composition for transfer learning. arXiv preprint 2xXix=2005.00247 2020.
Venkatadheeraj Pichapati, Ananda Theertha Suresh, Felix Yu, Sashank Reddi, and Sanjiv Kumar. Adaclip: Adaptive clipping for private sgd. arXiv preprint 17Xxv:1908.0764: 2019.
Chen Qu, Weize Kong, Liu Yang, Mingyang Zhang, Michael Bendersky, and Mare Najork. Privacy- adaptive bert for natural language understanding. arXiv preprint arXiv:2104.07504, 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 18))9, 2019.
13

---

Published as a conference paper at ICLR 2022
Swaroop Ramaswamy, Om Thakkar, Rajiv Mathews, Galen Andrew, H Brendan McMahan, and Françoise Beaufays. Training production language models without memorizing user data. arXiv preprint arXhv:2009.10031, 2020.
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. Adapterdrop: On the efficiency of adapters in transformers. arXiv preprint arXiv:2010.11913 2020.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 3-18. IEEE, 2017.
Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate. Stochastic gradient descent with differen- tially private updates. In 2013 IEEE Global Conference on Signal and Information Processing, pp. 245-248. IEEE, 2013.
Pranav Subramani, Nicholas Vadivelu, and Gautam Kamath. Enabling fast differentially private sgd via just-in-time compilation and vectorization. arXiv preprint arXiv:2010.09000 2020.
Amirsina Torfi, Edward A Fox, and Chandan K Reddy. Differentially private synthetic medical data generation using convolutional gans. arXiv preprint arXiv:2012.11774 2020.
Reihaneh Torkzadehmahani, Peter Kairouz, and Benedict Paten. Dp-cgan: Differentially private synthetic data and label generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition Workshops, pp. 0-0, 2019.
Florian Tramèr and Dan Boneh. Differentially private learning needs better features (or much more data). arXiv preprint PrXiv:2011.11660 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv: 1 03762, 2017.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 2021.
Adina Williams, Nikita Nangia, and Samuel R Bowman. The multi-genre nli corpus. 2018.
Sam Wiseman, Stuart M Shieber, and Alexander M Rush. Learning neural templates for text generation. arXiv preprint arXiv: 10122, 2018.
Zekun Xu, Abhinav Aggarwal, Oluwaseyi Feyisetan, and Nathanael Teissier On utilitarian approach to privacy preserving text generation. arXiv preprint arXiv:2104. 1838, 2021.
Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Gosh, Akash Bharadwaj. Jessica Zhao, Graham Cormode, and Ilya Mironov. Opacus: User-friendly differential privacy library in pytorch, 2021.
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. Differentially private fine-tuning of language models. arXiv preprint arXiv:2110.065500 2021a.
Da Yu, Huishuai Zhang, Wei Chen, and Tie- Yan Liu. Do not let privacy overbill utility: Gradient embedding perturbation for private learning. arXiv preprint arXiv:2 2021b.
Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie- Yan Liu. Large scale private learning via low-rank reparametrization. arXiv preprint 110i 11..2200..09522, 2021c
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. CoRR, abs/161 1 103530, 2016. URL http:
//arxiv. org/ aba///bb//// /1611, 03530.
Huanyu Zhang, Ilya Mironov, and Meisam Hejazinia. Wide network learning with differential privacy. arXiv preprint 2rXiv:2103.0/29. 2021.
14

---

Published as a conference paper at ICLR 2022
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Per- sonalizing dialogue agents: have a dog, do you have pets too? arXiv preprint arXiv: 181.0:301..7243. 2018.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. Dialogpt: Large-scale generative pre-training for conversational response generation. arXiv preprint arXiv:1911.00536 2019.
Yingxue Zhou, Zhiwei Steven Wu, and Arindam Banerjee. Bypassing the ambient dimension: Private sgd with gradient subspace identification. arXiv preprint arXiv:2007.03813, 2020.
15

---

Published as a conference paper at ICLR 2022
A DP-ADAM
We use DP-Adam throughout. DP-Adam works just like regular Adam (Kingma & Ba, 2014) but performs updates and moment accumulation with privatized gradients. The gradient privatization part is the same as that performed in DP-SGD (Song et al., 2013; Abadi et al., 2016). Seemingly uncommon, DP-Adam is used in many private ML libraries. To determine the noise multiplier, account privacy through Rényi differential privacy (RDP) (Mironov, 2017; Mironov et al., 2019). For completeness, we include the pseudocode for DP-Adam below.
Algorithm DP-Adam
1: Input: Data D - { learning rate noise multiplier o, batch size B, Euclidean norm threshold for gradients C, epochs E, initial parameter vector 9g R", initial moment estimates
mo, Do E IR, exponential decay rates 31, A2 E R, avoid division-by-zerv constant y E R.
2: for E [E. M/n) do
3: Draw batch B, via Poisson sampling: each element has probability B/N of being selected
4: for 24 E B, do
5:
gr(x;) + Vo,L( (((x)) gz())) min(1, (/119((x)11a))
6:
end for
7:
21 N0,,02C
8: 9i - EE (xze)) +
9: 1,+1. m.41, Ut+1 + AdamUpdate((A, me, Gt.
end for
11: return Orw/s
Algorithm 2 AdamUpdate
1: Input: O,, me, Dt, Gt. 31, B2,7
2: m441 + 31 m,
B1) It. 1241 t 32. (1 B2) g.²
3: Me+1 11441/ (1 B))
Uetl t Ue+1 (1
4: 0.41 + 02 - aa.a--in mmz41/
Veti +7)
5: return 0c+1, M4+1. 14+1
B PRIVACY ACCOUNTING
We train all models under approximate-DP (Dwork al., 2014), and view datasets as being adjacent if and only if one can be obtained from the other by including an extra record (Mironov et al., 2019). Instead of accounting the privacy loss with Moments Accountant (Abadi et al., 2016), we perform computation through (i) Rényi DP (Mironov, 2017: Mironov et al., 2019), (ii) Gaussian DP with an associated central limit theorem (Dong et al., 2019), and (iii) numerically composing tradeoff functions via fast Fourier transform (Gopi al., 2021; Koskela al., 2020). All approaches improvements over the Moments Accountant. Accounting loss with Rényi DP provides strict upper bounds on the actual privacy leakage but may result in loose bounds. Accounting loss with Gaussian DP and its central limit theorem, although asymptotically exact, only provides approximations to the actual loss under finite number of compositions (Dong al., 2019, Theorem 3.4). Gopi et al. (2021) observed that accounting loss with GDP and its CLT results in underestimation and proposed numerically compose tradeoff functions resulting both upper and lower bounds on the actual leakage We therefore also report the converted e with the approach by Gopi et al. (2021) using their code."!
Given the noise multiplier a, sampling rate number of steps S, and failure constant 6, can be computed via first computing the Rényi DP leakage and then converting it to approximate DP. When privacy spending (specified by set of given and ő) is prescribed, we can numerically invert the above procedure obtain suitable a for noisy optimization. This what we do throughout all
http:////.tttttooo/////ooo/o///r tensorf1 foofllowwar httpr//wwww..ooa//oo//////////// //gi thub. htttts////tttot...cooocoooooe/os
cea fa00 449998889220944...
16

---

Published as a conference paper at ICLR 2022
experiments. For completeness, given the chosen above, also report the leakage estimated by going through the central limit theorem in Gaussian DP (Bu et al., 2020).
Model selection from hyperparameter tuning private training and validation data incurs extra leakage (Liu Talwar, 2019; Chaudhuri Vinterbo, 2013; Papernot & Steinke, 2021). We perform tuning only the E2E task and reuse almost the exact hyperparameters for remaining tasks. Note the general strategy of tuning hyperparameters on a separate (public) dataset and thereafter transfer has been applied to train private production language models (Ramaswamy al., 2020).
C ADDITIONAL RESULTS ON MODEL SCALING
We repeat the model scaling experiments from Figure on the other tasks considered in the paper. Figure shows that the trend that larger and better pretrained models lead to improved private fine- tuned performance holds consistently across all tasks. TextHide numbers based on the Text Hidemara formulation (Huang et al., 2020a).
SST-2 dev set accuracy ary
MNLI Li--ssmnaccceee
(amilyte=3)
fanily
RuBERT fanily (e 3)
R.BERTa family (e-3)
BERT
BERT family
RuBERTs fumily (x-
femil (c-8)
Textrida (m, k)= (258, 4) (BERT-baak)
TocHide (n ki= 1256, (BERT
51
193
100
201
200
301
100
151
20
251
3e
number of non-embedding parameters (millions)
number of non-embedding parameters (millions)
(a) MNI.Imiismatched
(b) SST-2
BERT family (c- 3)
BERT family (c
RuBERT=
(e
BERT family ie B)
QNLI dev set acccuracy
BERT family (c -8)
RuBERT family B)
RuBERT< femle(y
TexthEda (256, (BERT.baak)
TextHid (nn, (BERT)buu))
88
QQP
62
100
201
301
251 number of non-embedding parameters (millions)
number of non-embedding parameters (millions)
QQP
(d) QNLI
DART BLEU
GP34
50
182
200
302
400
600
number of non-embeddinng parameters (millions)
(c) DART
Figure 5: Larger and better pretrained models consistently lead better private fine-tuned perfor- mance on sentence classification and language generation tasks.
17

---

Published as a conference paper at ICLR 2022
D WHEN AND WHY DOES LINEAR SCALING FAIL?
Recall Tramèr & Bonch (2020) suggested that the following simple rule approximately holds in private learning: Scaling the learning rate together with the batch size by the same constant yields models with almost the same performance. Note that their experiments on MNIST, Fashion-MNIST, and CIFAR-10 10 used only batch sizes in {512, 1024, 2048, 4096). These values fairly large from a non-private learning perspective. Indeed, our experiments on E2E suggest that this rule does not generalize to batch sizes that too small (sampling rates q = B/N < 2
We provide an explanation by noting that a core assumption which the linear scaling rule depends on fails to hold for small batch sizes. This assumption is that given privacy budget, "square-root" relationship holds between the noise multiplier and the sampling rate (see also (Tramèr & Boneh, 2020, Claim D.I)). For instance, Tramèr & Boneh (2020) showed that CV/q when q [2-7, for some constant c. Our numerical estimates show that this relationship fails to hold for small underestimates the true noise multiplier that would be obtained with numerical computation. Figure provides an illustration for (e, 6) (3, 10 10-5) when the sample size N 50k and number of training epochs E = 50.
scdal
noloe muttipler
sampling rate B/N
Figure 6: "Square-root" relationship underestimates the noise multiplier for small batch sizes.
E HOW DOES THE CHOICE OF LABEL WORDS AFFECT SENTENCE CLASSIFICATION
Recall in Section 3.2 we cast sentence classification as filling in the missing word among K candidates for K- K-way classification problem. Since label word could be mapped multiple possible words, we study how the choice of label words affect performance. We again use the sentiment classification task SST-2 as a testbed. Figure 7 shows the effect of varying the label word, where we measure the alignment between the label word and the downstream task by the zero-shot performance of the infilling task (x-axis). We find that increasing the task alignment of the label words alone improves performance by 1 2% (orange curve), which is in contrast to the non- con-private setting, where choice of label words do not affect performance in statistically significant ways (blue curve).
SST-2 fire tuned accuracy
32
107-private
wty weelword (c
CLS loken (a 3)
n18 derbull lapel werd (e
SST-2
accuracy
Figure 7: Better text labels help private learning more than non-private learning.
18

---

Published as a conference paper at ICLR 2022
F DERIVATION OF THE FROBENIUS NORM IDENTITY
Recall a E BxTxd is the input to a linear layer with weight matrix W E Rxx, and g E R RXXXX the gradient of the loss w.r.t the output. The identity follows from trivial algebra:
V Vw 19 ail
Σ
T aik
Dr
F
d
P
T
s=
Ry
lzz
T
E
Σ
Σ
Σ
Oi ki al K2i7
vec(aça
)T veo(gig,
Note that when T' 1, the identity takes the form of
W L voc(a,a,
) voc(o:9)) = |1a:112 19:11
This is exactly what is used in the Goodfellow (2015) trick.
G PROTOCOL FOR EXPERIMENTS IN SECTION 4.2
For these experiments, we used mock examples with the same format as examples in the E2E dataset. We created mock input sequences of length 100, as this length is almost the maximum length of examples the actual E2E training set.
Our JAX implementation is adapted from codebase used in the work by Subramani et al. (2020) and utilizes the package fl axmodel for loading pretrained models." 12 The Opa (Yousefpour et al., 2021) baseline is based on version 0 44. C of the library, where for fair comparison, we also optimized the implementation of the privacy engine by replacing all e einsum operations with basic primitives that manipulate tensors directly. We found certain e einsum um steps to cause large and unnecessary speed and memory overheads.
To identify the maximum batch size that each approach could use, we ran binary search over the range of possible batch sizes until the upper bound matched the lower bound and that OOM did not occur. To estimate the throughput of each private method, compared them to non non-private training first. Specifically, for a pairwise comparison, we took the maximum batch size for non-private training and some private method, and computed the least common multiple with. Using the least common multiple batch size ensures that both methods will process exactly the same number of examples and perform the same number of updates. By estimating the time elapse of these methods for performing a fixed number of updates, we obtained throughputs for non non- oor-iiivaee training and the private method. This gives us the relative throughput of the private method with non-private training as the reference.
batch size to perform updates
For methods implemented in PyTorch, the time elapse was recorded with torch. profiles ler. When estimating the time elapse for training procedure, we first performed 3 gradients updates as warm up process before taking the actual steps which will timed. In particular, this eliminates the time that JAX uses for compiling computation graphs with vmap and jit.
All experimental runs section were under full precision.
httpp//////tttoooo////aaaaaaaaaa 1
19

---

Published as a conference paper at ICLR 2022
H DETAILS AND ADDITIONAL RESULTS FOR STUDIES IN SECTION 3.1
Table 4: Default hyperparameters for ablation studies.
Method
Full
DP guarantee (e, 6)
(3, 1/2[Dmia)
Clipping norm C
0.1
Batch size B
1024
Learning rate !
10-3
Learning rate decay
no
Epochs E
10 for E2E; 3 for SST-2
Weight decay À
0
Noise scale
calculated numerically so that DP budget of (e, spent after E epochs
10-
0.00
0.15
3.69
33.42
53.99
60.52
10-3.2
411
11:77
1.82
63.7
02
50
323
10
learning fl
40
E2E test set BLEU (E
1/2
58.11
54.72
10
60 26 60.01
57.87
35.44
learning rate n
10-
36.05
30.54
30.23
30.45
29
90
20.25
30
20-3/22
50.48 50.31
50.48
49.25
34.47
10 7/29-1
30.89
30.40
20.0
20.23
29.95
28.30
10
1/24
33.18
32.96
10-1/2
25.30
29.91
29.4 41
29:81
36.09
10:93
10"1/2*
29.74
74
29.55
29.61
29.62
27.07
23
10-1/29-4
21.25
19.35
17.03
11:12
8.68
4.27
10
10-1/29
7.56
10:10
4.52
10"
9:69
9.65
1024
2048
0,1
0.
clipping norm
(a) Batch size.
(b) Clipping norm.
Figure 8: Additional results on hyperparameter sensitivity.
I HYPERPARAMETER SEARCH RANGES FOR EXPERIMENTS IN SECTION
We compare different adaptation methods by reporting task specific metrics on the test split using hyperparameters that maximize validation BLEU on E2E. For sentence classification tasks, we reused the same hyperparameters, except for the number of training epochs and batch size. For SST-2, we reused the same batch size as for private E2E fine-tuning, and the number of epochs exactly as in typical non-private fine-tuning for SST-2 (number epochs equals classification tasks, we use a batch size such that the sampling rate is the same as for SST-2, and number training epochs that is roughly proportional to the dataset size. Appendix L outlines why we transfer the sampling rate opposed to the batch size. We list the range of hyperparameters that we searched over for each individual adaptation method on E2E considered in the paper. Prefix-tuning has two additional hyperparameters: the length of the prefix and the dimensionality of the hidden layer. We set these to the default used by Li & Liang (2021) (10 for the former and 512 for the latter). For Adam, use the default hyperparamaters set by Py PyTorch (Paszke al., 2017).
in this case). For remaining
20

---

Published as a conference paper at ICLR 2022
Table Hyperparameter search range for different methods.
Method
Full
Prefix
Linear
FT2
Guarantee (e, 5)
(3, 1/1Pamal)
(3, P/1/)
(3, 1/1/2mm))
(3, 1/2 Panal) 0.1 (512, 1024]
Clipping norm C
0.1
0.1
0.1
Batch size
(512, 1024)
{512, 1024]
{512, 1024)
Learning rate
{200, 100, 30, 10.3) 10 (200, 100,30, 10, 3) 10- {200, 100, 30, 10,3) 10- 100, 30, 10. 10-5
(200,,
LR decay
{yes,
[yes, no)
{yes, no)
{yes, no)
Epochs E
{10, 30, 50}
(10,30, 50}
{10, 50,
(10, 30,
Weight decay A
0
Noise scale
calculated numerically that DP budget of (c, is spent after E epochs
Table 6: Hyperparameter search range for different methods (continued).
Method
LoRA
RGP
DP guarantee (e, 8)
(3, 1/2/
(3, 1/2/Dmm.) )
Clipping norm C
0.1
{0.1, 10}
Batch size B
{512, 1024]
{512, 1024]
Learning rate
{300, 100, 30, 10, 33 10-5 {yes, no}
{300, 100, 30, 10, 10-5 {yes, no)
LR decay
Epochs E
(5, 10, 30,
(5, 10, 30,
Weight decay
Rank k
41, calculated numerically that a DP budget of (e, is spent after E epochs
41,2,4,8)
Noise scale
21

---

Published as a conference paper at ICLR 2022
J
FULL RESULTS FOR EXPERIMENTS IN SECTION 5.2
Table 7: Full results on E2E from fine-tuning GPT-2.
Gaussian DP +CLT
Compose tradcoff func.
Metrics METEOR
Method
DP Guarantce
BLEU
NIST
ROUGE-L
CIDE
Fc 2.33 6 RJ 5.51
< 2.67
61.519
6.697
0.384
65.670
1.761
full
6
c RJ 6.98
63.189
7.444
0,400
66.429
1.919
non non-private
69.463
8.780
0.461
71.359
2.422
2.68 6 Re 6.77
2.75
58.153
5,463
0.370
65.773
1.581
LoRA
6 Re 7.28
63.389
7.449
0.407
67.525
1.948
non-private
69.682
8.822
0.463
71.709
2.491
==3
6 2.33 5.51
6 2.67
47.772
5.775
0.331
58.964
1.300
prefix
= non-private
6 6.98
49.263
6.276
0.349
60.730
1.496
68.845
8.722
0.456
70.805
2.418
2.18
2.59
58.482
5.249
0.363
65.560
1.507
RGP
8 non privpriv
5.19
6.89
58.455
5.525
0.364
65.030
1.569
68.328
8.722
0.445
68.844
2.345
3
2.68 Ri 6.77
2.75
25,920
1.510
0.197
44.536
0.452
top2
6 8
e RJ 7.28
26.885
1.547
0.207
46.421
0.499
non private
65.752
8.418
0.443
68.704
2.180
Ri 2.33
e R 2.67
15.457
0.376
0.113
35.240
0.116
retrain
5.51
c 6.98
24.247
1.010
0.145
39,951
0.281
non- aon-piivatc
65.731
8.286
0.429
68.751
2.004
Table 8: Full results on DART from fine-tuning GPT-2. Trend is consistent with results on E2E,
Gaussian DP CLT
Compose tradeoff func.
Metries ROUGE-L
Method
DP Guarantee
METBOR
ROUGE-1
ROUGE-2
BERTS
BLEURT
é 2.28 5.35
2.65
0.294
62.815
40.773
52.063
31.025
0.887
-0.058
full
Rz 6.95
0.319
66.423
43.609
54.576
35.057
0.901
0.043
non-private
0.369
71.563
47.168
56.717
42.783
0.915
0.178
3 6 =
2.68 6 RJ 6.68
2.76
0.304
63.641
40.753
52.012
32.329
0.885
-0.029
LoRA
R: 7.26
0.318
66.336
43.056
54.082
34.163
0.899
0.036
non-private
0.366
71.192
47.336
57.430
42.254
0.915
0.182
3
Ad 2.28 6 5.35
6 Ro 2.65
0.269
59.503
38.229
49.444
25.726
0.860
-0.144
prefix
6 non-private
6 Rd 6.95
0.297
64.009
41.581
52.602
30.463
0.892
-0.021
0.353
70.341
46.643
56.858
40.163
0.912
0.148
6
6 2.10 5.02
2.54
0.265
58.688
37.202
49.011
25.748
0.873
-0.175
RGP
6.86
0.279
60.005
38.258
49.835
28.304
0.874
-0.141
non-private
0.324
65.667
42.617
53.477
35.551
0.895
0.022
2,68
2.76
0.022
3.570
2.183
3.166
0.388
0.098
-1.952
top2
8 c
> 6.68
R2 7.26
0.054
11.475
7.054
10.042
2,453
0.240
-1.660
non- non-private
0.318
62.777
38.367
49.426
36.099
0.883
-0.082
3
n2 2.28 c R 5.35
R2 2.65
0.064
19.085
8,901
17.142
2.997
0.493
-1.513
retrain
<
6 Rs 6.95
0.093
24.971
11.938
21.680
7.765
0.573
-1.302
non-private
0.232
47.782
26.361
37.864
26.794
0.806
-0.593
22

---

Published as a conference paper at ICLR 2022
K DETAILS FOR EXPERIMENTS IN SECTION 5.2
Sentence Classification, Results for RGP in Table 1 are taken from documented numbers in their released codebase." These results are under the DP guarantees of (c, (3. 10-10) or (c. 5) (8, 10- These guarantees are strictly looser than our guarantees which are based on 1/2/Dmaul (recall the smallest dataset in this cohort of tasks has 60k+ records). The RGP numbers in Table I are higher than those reported in their paper (Yu et al., 2021c), since the latter numbers are not based on fine-tuning the official RoBERTa models.
Table-To- Text Generation. To evaluate models trained E2E and DART, we evaluate generations from models obtained with beam search with beam size of 3. For evaluation, we run the official pipeline for E2E, and the pipeline used in the GEM benchmark (Gehrmann al., 2021) for DART."
Chit-Chat Dialog Generation. We built off Huggingface's 's codebase of the winning entry of the ConvA12 competition, 16 17 and used their preprocessed training set with the minor modification of truncating the number of training examples to be multiple of the batch size. The original ConvAl2 competition is aimed at advancing research on building engaging chatbots and also requested models to predict the mostly likely response given list of candidates. The challenge included hits @ as part of its suite of automatic metrics. For simplicity, we skip this step of predicting the most likely response for both training and evaluation. Results for the entry HuggingFace (ConvAl2 winner) in Table 3 are taken from the official validation set leader board. 18 Our reimplementation of HuggingFace's 's submission uses the released code for their winning entry, fine-tunes GPT with the default hyperparameters, and removes the classification loss for learning to predict the most likely response given candidates.
We additionally fine-tuned DialoGPT-mediie (Zhang al., 2019), since the model was pretrained on conversation like exchanges extracted from Reddit comment chains. Intuitively, this pretraining corpus is more aligned with the downstream fine-tuning data than WebText (Radford al., 2019), and thus would likely improve downstream performance.
To evaluate the F1 score, obtained predicted responses from trained models using beam search with a beam size of 5. Since past work found that the F1 score can be gamed by always letting the model predict a predetermined phrase (Dinan al., 2019), we additionally ask humans rate generations from the model through Amazon mechanical turk to obtain a more complete picture of model quality. When sampling responses for human evaluation, we used nucleus sampling with = 0.9 (Holtzman et al., 2019). For human evaluation, we asked 20 turkers to each rate 5 entries. For each entry, turker asked to rate on scale of to 5 the quality of predicted responses from privately fine-tuned DimloGPT-medium models, non-privately fine-tuned DialoGPT-mediin models, non-privately fine-tuned GPT models (our reimplementation of HuggingFace 's entry), and the reference text, given the history of the dialog. Since human evaluation can yield noisy results, we also report the 95% asymptotic confidence interval Table 3. All models were trained and evaluated on the version of Persona-Chie with the original persona. All numbers reported in Table 3 are obtained on the validation split.
L TRANSFERRING HYPERPARAMETERS ACROSS DATASETS
Our work involved tuning hyperparameter with models trained via DP-Adam on one private dataset and transferring such hyperparameters to other private datasets. Since different datasets may be of different sizes, transferring the batch size may cause a discrepancy in the effective noise multiplier across workloads with different datasets. Transferring the batch size based on hyperparameter tuning on small datasets to larger datasets can be particularly problematic, as the effective noise multiplier
main/language "ht.epar/ggaaaaaccmo///tttttttts
23

---

Published as a conference paper at ICLR 2022
be larger than ideal. In this work, instead transferred the sampling rate & across different datasets.
M DOES DP FINE-TUNING PREVENT UNINTENDED MEMORIZATION?
One of the ultimate goals of fitting models under DP is to ensure that training data extraction is unlikely given the trained model. To empirically evaluate whether DP fine-tuning helps prevent against unintended memorization and related attacks, we follow the secret sharer framework (Carlini al., 2019) and estimate the exposure of artificial canaries inserted into the training set used for fine-tuning. We use the E2E dataset as testbed.
To create canaries, first form a subvocabulary by randomly sampling V 10 words in the original vocabulary of GPT-2. Our canaries have prefixes of the form
" name
<word>
Type
<word>
area
:
<word> ",
where <word> randomly sampled from the subvocabulary. The suffix which our model should learn to predict consists of randomly sampled words with an average length of | = 5. By definition, canaries with an estimated exposure close to log, (V*) Rs 17 can likely be extracted. We experiment with canary canary-corrupted datasets for repetition values E (1, 10, 100}. A canary has higher chance in being extracted when it's repeated for more than once in the training data.
Table 9: Fine-tuning under DP prevents unintended memorization of downstream data. Numbers reported are exposure values estimated with the approximation by distribution model approach.
Repetitions
r 10
100
Guarantee
1.09 0.86
5.26 + 4,20 17.22
1.32 + 1.32
non-private
13.82 = 3.86
0.00
17.78 = 5.49
N TEMPLATES AND LABEL WORDS FOR TEXT-INFILLING-BASED CLASSIFICATION IN SECTION 3.2
Recall that fine-tuning for classification can be reformulated as filling in the [MASK] token a template sequence. Here, list the templates used for each classification task considered in the paper. These templates are almost generic and are not obtained from expensive manual or automated search. We anticipate better templates obtained from automated search based on data (Gao al., 2020) to improve the performance even further. However, we also expect that such procedure would lead to some amount of increased privacy spending if it were based on private data.
Task Template
Label words
SST-2 <S,> It was [MASK]
positive: great, negative: terrible
MNLI <S1>> ? [MASK] <S,>>
entailment: Yes, netural: Maybe, contradiction: No entailment: Yes, noteentailmnnt: No
ONLI <S1> ? [MASK] <S2>>
QQP <S1>> [MASK] <S,>>
equivalent: Yes, not equivalee::: No
Table 10: Templates and label words borrowed from the work by Gao et al. (2020).
24

---

Published as a conference paper at ICLR 2022
O
UNCURATED SAMPLES FROM FINE-TUNED MODELS
Table
name: The Punter Type: restaurant food: Indian price cheap customer
rating: average area riverside family friendly: no near: Express by Holiday Inn
GPT-2 (c = 3)
The Punter is a cheap Indian restaurant near Express by Holiday Inn in the riverside area. It is not family- friendly.
GPT-2 (c = 8)
The Punter is a cheap Indian restaurant near Express by Holiday Inn in the riverside area. It is not family friendly.
GPT-2-m (c = 3)
The Punter is a cheap Indian restaurant located in the riverside area near Express by Holiday Inn. It has an average customer rating and is not family friendly.
GPT-2-m (c = 8)
The Punter is a restaurant providing Indian food in the cheap price range. It is located in the riverside area near Express by Holiday Inn. Its customer rating is average.
GPT-2-1 (e 3)
The Punter is a cheap Indian restaurant in the riverside area near Express by Holiday Inn. is not family friendly and has average customer rating.
GPT-2-1 (c 8)
The Punter is restaurant providing Indian food in the cheap price range. It is located in the riverside
Express by Holiday Inn. Its customer rating is average.
Reference
The restaurant named The Punter has cheap Indian food and an average customer rating
is near the Express by Holiday Inn on the riverside and is not family friendly.
Table
name : The Mill - Type: restaurant - food : English price: moderate - customer rating: 3 out of 5 - area : city centre - family friendly yes near : Café Rouge
GPT-2 (c = 3)
The Mill is moderately priced restaurant located in the city centre near Café Rouge.
GPT-2 (e = 8)
The Mill is a moderately priced restaurant located in the city centre near Café Rouge.
GPT-2-m (e = 3)
The Mill is an English restaurant located in the city centre near Café Rouge.
moderately priced and has a customer rating of 3 out of 5.
GPT-2-m m (c = 8)
The Mill is moderately priced restaurant located in the city centre near Café Rouge. is child friendly and has a customer rating of 3 out of 5.
GPT-2-1 (c = 3)
The Mill is a moderately priced English restaurant in the city centre near Café Rouge. It is child friendly and has customer rating of 3 out of 5.
GPT-2-1 (c = 8)
The Mill is a kid friendly English restaurant in the city centre near Café Rouge. It has a moderate price range and customer rating of 3 out of 5.
Reference
Serving moderately priced English food with 3 out of 5 customer approval, The Mill restaurant is kid friendly and conveniently located the city centre near the Café Rouge
Table
name: The Vaults Type : pub food Japanese - price : high - customer rating out of - area city centre - family friendly: : yes near: Raja Indian Cuisine
GPT-2 (e = 3)
Located near Raja Indian Cuisine in the city centre, The Vaults is a family friendly Japanese pub with a high customer rating.
GPT-2 (e = 8)
The Vaults is Japanese pub in the city centre near Raja Indian Cuisine. It has a high price range and customer rating of 3 out of The Vaults is a Japanese pub located in the city centre near Raja Indian Cuisine. It has high price range and customer rating of out 5.
GPT-2-m (c = 3)
GPT-2-m (c = 8)
The Vaults is a Japanese pub located in the city centre near Raja Indian Cuisine. It has a high price range and customer rating of out 5.
GPT-2-1 (c 3)
The Vaults is Japanese pub in the city centre near Raja Indian Cuisine. It has a high price range and customer rating of out of 5.
GPT-2-1 (c 8)
The Vaults is child friendly Japanese pub in the city centre near Raja Indian Cuisine. It has high price range and a customer rating 3 out 5. Located near the Raja Indian Cuisine at the city centre. family friendly pub The Vaults Serving Japanese food, The Vaults, is high priced with a 3 out of customer satisfaction rating.
Reference
Table 11: Fully fine-tuned GPT-2, GPT-2-mediug and GPT-2-large generations with E2E test table entries.
25

---

Published as a conference paper at ICLR 2022
Table
Real Madrid Castilla: manager: Luis Miguel Ramis - Abner (footballer) club: Real Madrid Castilla Abner (footballer) club C.D. FAS
GPT-2 (e 3)
Luis Miguel Ramis played for Real Madrid Castilla and played for C.D. FAS.
GPT-2 (e 8)
Luis Miguel Ramis is the manager of Abner (footballer) who plays for Real Madrid Castilla.
GPT-2-m (c 3)
Luis Miguel Ramis is the manager of Real Madrid Castilla. He plays for C.D. FAS.
GPT-2-m (c = 8)
Luis Miguel Ramis is the manager of Real Madrid Castilla. He plays for C.D. FAS. Luis Miguel Ramis is the manager of Real Madrid Castilla and C.D. FAS. Luis Miguel Ramis is the manager of Real Madrid Castilla and Abner (footballer) plays for C.D. FAS.
GPT-2-1 (e 3)
GPT-2-1 (e 8)
Reference
Footballer, Abner, plays C.D. FAS. and Real Madrid Castilla, the manager of which, is Luis Miguel Ramis.
Table
United States ethnic group: : Asian Americans United States capital: Washington,
D.C. -CC.-Au Albany, Oregon is_ptrt.of: Benton County, Oregon - Albany, Oregon : country
United States
GPT-2 (c = 3)
The capital of the United States is Washington, D.C. and is part of Benton County, Oregon, The capital of the United States is Washington, D.C. and is part of Benton County, Oregon.
GPT-2 (e = 8)
GPT-2-m (c = 3)
Alhany, Oregon part Benton County, Oregon the United States where Asian Americans are an ethnic group.
GPT-2-m (c = 8)
Albany, Oregon part of Benton County, Oregon in the United States where Asian Americans are an ethnic group.
GPT-2-1 (c 3)
Albany, Oregon is part of the United States where Asian Americans are an ethnic group and the capital is Washington D.C.
GPT-2-1 (c- 8)
Albany, Oregon is part of the United States where Asian Americans are an ethnic group and the capital is Washington D.C.
Reference
The Asian Americans are an ethnic group in the United States, which has the capital city of Washington DC. is also the location of Albany, part of Benton County in Oregon.
Table
Loyal Character Dancer: : language English language English language : spoken Great Britain - A Loyal Character Dancer: country United States - United States ethnic_group Native Americans in the United States
GPT-2 (c = 3)
A Loyal Character Dancer is an English language spoken in the United States where Native Americans the ethnic group.
GPT-2 (e 8)
A Loyal Character Dancer written in English and is spoken in Great Britain. Native Americans an ethnic group in the United States.
GPT-2-m (c = 3)
A Loyal Character Dancer is written in English and is written in the United States where Native Americans are ethnic group. Loyal Character Dancer written in English and is written in the United States where Native Americans are an ethnic group.
GPT-2-m (c 8)
GPT-2-1 3)
Loyal Character Dancer written in English and written in the United States where Native Americans are an ethnic group.
GPT-2-1 8)
Loyal Character Dancer written in English, which is spoken Great Britain and the United States. Native Americans are an ethnic group in the United States.
Reference
Loyal Character Dancer written in English which spoken Great Britain. The book is published in the United States where the Native Americans are an ethnic group.
Table 12: Fully fine-tuned GPT-2, GPT-2-medium, and GPT-2-large generations with DART test table entries.
26

---

Published as a conference paper at ICLR 2022
Persona
['i am years old.', 'i work car salesman.", "my wife spends my money.' hate my job.". "I'm I'm planning on getting divorce."]
History
I'i know the struggle, volunteer at a homeless shelter', "great, hope you don't 't get to see me there.", 'i hope not either, would not wish it on my worse enemy", "between wife, job with no money, am thinking now of divorce"., "I've given it few thoughts before"] i wish had better idea.
GPT-2-m (r = 3)
GPT-2-m mm (c= 8)
i think will what are your thoughts on divorce ?
DialoGPT-medium (e = 3)
i' m sorry to hear that hope you don't I have any financial problems
DialoGPT-medium (e 8)
don't have any money. i do not know how could manage. do you give demonstrations your skills ?
Reference
Persona
['my mother is a dentist." : "my favourite artist is travis scott.". 'i have one older sister.", am currently looking for job preferably restaurant.', 'i like playing baskethall.")
History
['hello, how you tonight?"]
GPT-2-m (c = 3)
i'm good, good luck
GPT-2-m (c 8)
i am good just listening to travis scott. what about you? ?
DialoGPT-medium (e = 3)
i'm good. just got back from a dentist visit.
DialoGPT-medium (c = 8)
i am doing well just got back from a dentist visit
Reference
i am good about to go to basketball practice.
Persona
['my mom passed away when was young., ; 'i work in the insurance business.", 'my father job."]
in the army.'. 'i recently was fired and had to get ['god isn't 't really real", 'to each their but believe he is.' 'my children gone now", lost my mom a young age", 'my wife took them all')
History
GPT-2-m (c =3) 3)
was a great mother
GPT-2-m (c 8)
they were so young. they
crazy
DialoGPT-medium (e 3)
that is horrible. my mom was soldier what happened to her
DialoGPT-medium (e 8)
Reference
sorry to hear that i barely see my dad he's in the army
Persona
['i also graduate student.', 'i walk dogs for living.', 'my parents taught survival skills.", am an avid kayaker', ' enjoo reading journals and guides related to psychology.] ['hi, how are you today?' "good evening. how are you tonight? i'm well.", am doing good, i just finished playing ping pong.', ' am trying to write invoices for my dog walking business.', 'that seems like hare work, like to play the violin in my free time.")
History
GPT-2-m (c 3)
i have a lot of dogs i walk them that a very rewarding hobby i have dog named jasper
walks.
GPT-2-m (==8) 8)
DialoGPT-medium (c = 3)
oh, that a nice hobby what do you do for a living
DialoGPT-medium (c 8)
is love learning to play the violin.
Reference
love violin do not play anything kayak for fun however.
Table 13: Fully fine-tuned GPT-2-medium and DialoGPT-mediuu generations for Persona-Chat
validation examples.
27

---

Published as a conference paper at ICLR 2022
P ADDITIONAL RELATED WORK
Differentially Private Deep Learning. DP-SGD has been viewed as ineffective for large models due to the addition of large Gaussian noise to gradient updates. Improvements to the learning procedure mostly fall under two distinct camps: (i) Simplifying the private learning problem, and (it) reducing the scale of noise. For instance, Papernot et al. (2019); Tramèr & Boneh (2020); Abadi et al. (2016) consider transferring features learned on public datasets to simplify the subsequent private learning task. On the other hand, Zhou al. (2020); Kairouz al. (2020) remove the ambient dimension dependence of DP noise by identifying subspaces in which private gradients lie and would be privatized. Yu et al. (2021b:c) make such ideas practical and demonstrate improved results on private learning benchmarks. Zhang et al. (2021) applied the sparse vector technique to learning wide neural layers to reduce the amount of injected noise. Our work mostly falls under the first camp- improving private learning through simplifying the learning task. Our work is also distinet from prior works in that we focus on privately fine-tuning large pretrained models. Lastly, there are alternative solutions in the literature that enforces DP which are not based on gradient perturbation (Papernot et al., 2018; 2016). These methods typically require extra public data and are not the present focus.
Parameter-Effi Efficient Fine-Tuning Recent developments on pretrained model adaptation have produced a wide range of parameter-effiiient fine-tuning methods for both vision and language tasks. We briefly summarize these, grouping by category. Approaches based on optimizing prompt-like constructions for NLP tasks include prefix-tuning (Li & Liang. 2021), P-tuning (Liu et al., 2021), 1), and prompt-tuning (Lester al., 2021). Adapter-based methods insert small subnetworks inside pretrained Transformers (Houlsby et al., 2019; Rückléé et al., 2020; Pfeiffer et al., 2020). Methods that optimize low-rank matrices include the work by Hu et al. (2021): Mahabadi et al. (2021). addition, there are adaptation methods that only optimize biases for vision (Cai et al., 2020) and language tasks (Ben Zaken et al., 2021). Our evaluation in Section 5.2 covered the most representative methods that generally have state-of-the-art non- non-private learning performance (at the time of writing) for the range of NLP tasks studied in this paper.
Speeding Up DP-SGD. Apart from the work by Lee & Kifer (2020) and Subramani et al. (2020), there is an approach that approximates per-example gradient norms through the combination of ran- dom projection and forward-mode autodiff (Bu et al., 2021). While faster than vanilla private learning, this approach has the drawback of increased privacy spending and having an extra hyperparameter. Our ghost clipping technique, while only suited for Transformers applied to sequential data, does not introduce new hyperparameters.
Alternative Clipping Strategies. While there are alternative clipping strategies in the literature that show improvements on simple tasks (Pichapati et al., 2019; Asi et al., 2021), we have opted to study the simplest strategy that clips gradients by their Euclidean norm. We leave the study of these algorithms for NLP tasks to future work.
Concurrent Work. We are made aware of a concurrent work that also studies fine-tuning large language models under DP (Yu et al., 202 This work presents initial successes fine-tuning under DP with low-rank methods such as LoRA. Our experiments on language generation (see Section 5.2 and Table 2) demonstrate similar findings. Yet, we moreover show that full fine-tuning with good hyperparameters attains similar performance and possesses similar model scaling properties, which was raise by Yu al. (2021a) as interesting open questions to pursue. Lastly, our private fine-tuning results for sentence classification may be far from optimal, since we used hyperparameters mostly transferred from tuning on the E2E language generation task.
DP Synthetic Data Generation. Fine-tuning generative language models on private data under DP can also be viewed as a means of accomplishing DP synthetic data generation - learning generative models from private data so that synthetic examples could be sampled and used for analysis. Previous work employed generative adversarial networks and focused primarily on image or tabular datasets (Torkzadehmahami et al., 2019; Neunhoeffer et al., 2020; Chen et al., 2020; Torfi et al., 2020). Perhaps more related is the work by Bommasani et al. (2019) which attempted min-ttnnng GPT-2 medical datasets to generate synthetic records but did not report any quantitative results.
28

---

Published as a conference paper at ICLR 2022
Q ESTIMATES OF RUN-TIME IN PRACTICE
The actual run-time of algorithms depends on implementation details. Here, we outline estimates of the run-time for full fine-tuning with DP-Adam on tasks considered in the paper. These numbers are based on running with a single RTX 3090 with PyTorch==1. 9. Fine-tuning GPT-2 on E2E and DART takes less than 10 minutes per epoch, and fine-tuning for 10 epochs results reasonably performing models. The time to fine-tune RoBERTa-bas on classification tasks depends on the size of the dataset. takes less than 10 minutes per epoch on the smallest SST-2, whereas for the largest MNLI, it takes less than an hour per epoch.
R
EFFECTS OF VARYING AND ON DP FULL FINE-TUNING PERFORMANCE
Figure shows how different values of c and 6 affect the performance of full fine-tuning under DP.
E2E
E2E
2-10-
Figure 9: Left: 8 affects performance marginally. Right: affects performance more significantly
when e < 2. Errorbars are one standard deviation away from the mean over five independent runs.
DP-ADAM VS DP-SGD
Our work focused on DP-Adam as opposed to DP-SGD, since Adam is more commonly used for non-private language model fine-tuning. While the two algorithms differ in their parameter update rule, the basic gradient privatization procedure the same. We performed additional experiments fine-tuning GPT-2 on the E2E dataset with DP-SGD (with freshly tuned learning rate and clipping norm values) and observed that its performance (test set BLEU 63. 175 at (e, = (8, 10 10-5)) is on with Adam (test set BLEU 63.189 at (c, (8, 10-
T SUBTLETIES OF IMPLEMENTING DP MIXED PRECISION TRAINING
Mixed precision training (Micikevicius et al., 2017) accelerates updates by storing certain tensors in half-precision To mitigate negative effects caused by potential arithmetic underflow, usual implementations upscale the loss with an adaptive factor rreebakprropagtiion and downscale the gradients with the same factor post-backpropagation. The scaling factor is adapted based on whether underflow is observed during training.
Special care needs to be taken when combining gradient privatization with mixed precision training. One implementation that ensures similar results across full and mixed precision training (1) upscales the loss with the adaptive factor K, (2) clips per-example gradients by C.K, (3) adds to the sum of clipped gradients the usual Gaussian noise multiplied by K, and (4) downscales the noisy gradient by K. This is the implementation that we adopt, and we were able to obtain similar results the E2E dataset with and without mixed precision.
One alternative implementation (a) upscales the loss with the adaptive factor K, (b) clips per-example gradients by C, (c) adds the usual Gaussian noise to the sum of clipped gradients, and (d) downscales noisy gradients by K. The main difference between this procedure and the prior is whether the factor
29

---

Published as a conference paper at ICLR 2022
K' is considered during clipping and noising. This procedure, while having the same DP guarantee, typically does not result in similar results as full precision when the same hyperparameters are used across the two settings (even with optimizer like Adam which self-adjusts the magnitude of updates with accumulated empirical second moments). We also identified that this implementation is also the primary reason that prior work's code does not reproduce good results when run in full precision."
U FINE-TUNING WITH RGP AND THE TEXT-INFILLING OBJECTIVE
Recall Section 3.2 and Table showed that private full fine-tuning with text-infilling objective leads to improved classification results. Here, we show that the infilling objective is also helpful when one privately fine-tunes with the RGP method for classification task. To study this, reimplemented the RGP method and tuned the hyperparameters in full precision. Fixing all hyperparameters, we compared models trained with and without infilling. Table 14 confirms that fine-tuning with RGP based on infilling is generally helpful across the considered tasks. Note the aim of this experiment not obtain stateoof-hhe-ae performance, but rather to study the effect of using the text-infilling objective. Thus, we expect these results could generally be further improved with more extensive hyperparameter search.
Table 14: Text-infilling objective improves the performance RGP for classification. Numbers are averaged over three independent runs.
Method
MNLI-(m/mm)
QQP
QNLI SST-2
RGP (RoBERTa-hase)
79.79/80.44 40
83.58
84.14
89.60
RGP + infilling (RoBERTa-base)
81.97/82.28
84.02
87.20
92.85
V
WHICH METHOD SHALL USE FOR PRIVATE FINE-TUNING?
While we were able obtain similar results on the E2E and DART datasets with full fine-tuning and LoRA (Tables and 8), we encountered significant difficulties in fine-tuning for dialog generation (even non-privately) when not updating the embedding layer and language modeling head - per- plexity much worse, and generations frequently contained seemingly arbitrary characters. This result suggests that while full fine-tuning may be more computationally intensive than lightweight approaches at times, its simplicity (involving few design decisions) makes attractive first option when compute resource is sufficient.
"https:// http:://ggtthuub.oo/..//// /Di fferentia peeee
30


Reviews:
[{"review_id": "k_aOjZQmn9u", "reviewer": "Reviewer_E6VG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros,\n1) It achieves a remarkable performance of DP algorithms on NLP tasks with a satisfactory level of privacy protection.\n2) There will be a number of potential applications since this paper verify the model on multiple NLU and NLG tasks.\n3) It works on several pre-trained language models (e.g. GPT, BERT, Roberta).\n4) The experiments part is quite solid and covers many necessary details.\n\nCons,\n1) The experimental results in table 1~3 compared to the non-private seems amazing, however, the comparison with the baseline methods is also required, which would show the contribution of your proposed methods. There're several possible baselines: directly apply DP-SGD and an application of DP-SGD on language models (GPT-2) [1].\n\n2) The choice of \\epsilon makes sense because the \\epsilon in a range of 0.1~5 provides meaningful protection. However, the choice of \\delta in experiments may cause some problems in privacy leakage. The value of \\delta on the order of 1/|D_train| is very dangerous according to [2]. In that level of \\delta, they permit “preserving privacy” by publishing the complete records of a small number of database participants. It's better to choose the \\delta as 1/( |D_train| *100), which may reduce the utility but ensure the privacy is well protected. \n\n3) Will the ghost clipping increases the utility or privacy protection? Could you please provide some insights (or even quantitative analysis) about it?\n\n4) It would be better to conduct some key experiments with a range of \\espilon from 0.1 to 10 (e.g. 0.1, 0.5, 1, 2, 3, 5, 8). Many DP learning algorithms use a curve to show the variation tendency of utilities for the different \\epsilon.\n\n5) This paper did a good job on the empirical study, but the technical novelty is limited.\n\n[1] Differentially Private Language Models Benefit from Public Pre-training. 2020\n\n[2] The Algorithmic Foundations of Differential Privacy, 2014\n\n\nAfter rebuttal:\nThanks for your explanation! I believe it's a good paper and I'm happy to see its acceptance.\n", "summary_of_the_paper": "This paper adapts the widely used DP learning algorithm, DP-SGD, to language models. It achieves to fine-tune the dataset while protecting the private information in the dataset. In this paper, the authors conduct some empirical studies on language models and find some useful conclusions (e.g. fine-tuning on a part of parameters with DP is enough). The authors verify the model on sentence classification, table-to-text generation, and dialog generation tasks, using various pre-trained language models (e.g. GPT, Bert).", "summary_of_the_review": "The paper has a clear motivation and the main idea is also interesting, but still suffers from some issues.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "bpnFRplSVeN", "reviewer": "Reviewer_uqBn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. As a DP algorithm on a deep learning model, this algorithm achieves good performance on fine-tuning the language models and several real-world applications.\n2. The experimental results are sufficient and solid. \n3. The performance gap between the proposed method and the non-private method is quite small, which shows applying the proposed DP algorithm would not decrease the utility so much.\n4. Some of the conclusions from the experiments seem to be adaptive to other applications. For example, the relations among parameter size (for fine-tuning), privacy leakage, and performance.\n\nWeaknesses:\n1. The \\delta in (\\epsilon, \\detla)-DP is too large for training a DP model with a meaningful level of privacy protection. Normally, \\detla should be much smaller than the inversed dataset size (cannot be in the same scale of the inversed dataset size), some researchers choose \\detla = 1/(|D| * log(|D|)), where |D| stands for the dataset size.\n\n2. It would be better to conduct an ablation study to verify the promotion of the proposed methods (ghost clipping technique, fine-tuning on only a part of the parameters, and DP-adam). It's better to make it clear how much does each strategy contribute.\n\n3. In this paper, the authors introduce a lot about the selection of hyper-parameters, thus hyper-parameters play an important role in model training. However, tuning hyper-parameters requires additional information about the private information (accessing the validation set or testing set), which leads to private leakage. So, how to count the information leakages of users? How to avoid those private leakages.\n\n4. Is the DP-Adam proposed by yourself? If not, the corresponding reference and explanation are needed. Does it come from [Wang 2019]? If it is proposed by yourself, I suggest the author make it clear as it's one of the contributions. \n\n[Wang 2019] DP-LSSGD: An Optimization Method to Lift the Utility in Privacy-Preserving ERM.", "summary_of_the_paper": "In this paper, the authors propose a method that applies DP-SGD to NLP tasks. DP-SGD protects the privacy of the model training against that the individual information about the training samples is detected or inferred. The method is applied to the fine-tuning phase of the pre-trained language models (e.g. bert, gpt), thus it achieves good performances for many applications. To adapt DP-SGD to NLP models, this paper proposes ghost clipping that allows clipping in DP-SGD to run without instantiating per-example gradients for any layer in the model.", "summary_of_the_review": "The paper is quite good on experiments and their possible applications but faces some questions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_tmpKxlZDE", "reviewer": "Reviewer_a9ro", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this is a fairly empirical paper on an important problem and shows good performance. The authors presented a set of thorough experiments investigating the impact of various hyper-parameters, which are widely known as sensitive and difficult to tune, providing a nice and informative guidelines for other researchers and practitioners who work in this area. The ghost clipping trick proposed in this paper for memory reducing in DP-SGD is simple yet effective, greatly reducing the memory cost when applying DP-SGD to large models, especially the popular large-scale pre-trained language models, potentially encouraging more research effort in the DP learning area.\n\nThe experiments presented in this paper are quite solid and clear, both well-designed and documented, and most of the claims made in the paper are reasonable and well-supported. My only complaint is that I wish to see more explanations and more principled approach to selecting the hyper-parameters in fine-tuning with DP-SGD, but I guess it is out of this paper's scope (the authors did provide some empirical discussions in the paper and the appendix, which I appreciate).", "summary_of_the_paper": "This paper investigated the problem of privately fine-tuning large language models for downstream NLP tasks, including sentence classification and language generation. The authors showed that by appropriately selecting hyper-parameters (including batch size, learning rate, training epochs, and clipping norm) and making the fine-tuning task aligned with pretraining tasks, directly fine-tuning large language models with DP-SGD yields strong performance, and provided an empirical guideline for setting a good training configuration. The authors also proposed ghost clipping trick for further memory saving when fine-tuning large language models. Finally the authors showed through experiments that low dimensional updates do not necessarily lead to better performance.", "summary_of_the_review": "This paper presented a detailed investigation on the training configurations of fine-tuning large language models with DP-SGD through a set of extensive experiments, which are beneficial to researchers and practitioners working on private NLP, and the proposed ghost clipping trick for DP-SGD greatly reduces memory when applied to large language models, making private NLP research more feasible. I believe both the insights and the proposed method in this paper will bring value to the private NLP community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2MGIgGIaMvi", "reviewer": "Reviewer_XPKW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength of the paper:\n1. The paper is clearly written and easy to understand. It presents the DP training problem very well and demonstrates the challenge for large pretrained NLP models. \n2. The proposed GhostClipping is effective in reduce memory consumption, at (not too big) cost of an extra backpropagation. The method is very clear and also simple to implement. \n3. The experiments show strong results on both classification and text generation tasks, better than the prior DP methods in terms of memory consumption and accuracy/generation quality. \n\nWeakness of the paper:\n1. There are three aspects involved. GhostClipping, large batch/proper learning rate, fine-tuning with masked prediction. It seems the accuracy on text classification comes from the later two. Table 1, comparing full(RoBERTa-large) with RGP(RoBERTa-large) does not show improvement (please list the average score here). Full is actually worse. If RGP is augmented with larger batch/proper learning rate, and masked prediction for fine-tuning, it may be the best in terms of accuracy. \n2. Figure 1 is a bit mis-leading. Please show and compare with RoBERTa (non-private) on 1(a) and non-private GPT-2 on 1(b). Those are the baselines to compare. Otherwise you are comparing private RoBERTa with non-private BERT or non-private T-GEN(non Transformer). Statement in the abstract and introduction are over-stated. \n3. The organization of the paper could be improved. Ghost-Cliping is your main method, which could be moved earlier. \n", "summary_of_the_paper": "The paper propose a faster algorithm to learn approximate differentially private NLP models. Pretrained NLP models are often very large. Practical procedure involves fine-tuning NLP model on private data, which may leak private information. To avoid leakage, DP-SGD (and DP-AGAGRAD, DP-ADAM) uses norm clipping on each sample’s gradient, and then add isotropic noise to aggregated gradients for samples in a batch. Then the normal update steps of SGD, ADAGRAD, or ADAM are performed. This will ensure privacy under differential privacy definition. However, this procedure requires computing per-sample gradient and keep them in the memory so that the norm of the gradient can be calculated, and the per-sample gradient clipping can be performed. This will introduce memory overhead proportional to the batch_size$\\times$#params, which is impossible for very large NLP models. \n\nThis paper propose GhostClipping method to save the memory, without the need of per-sample gradient instantiated. The idea is compute the partial sum of gradient element-wise square using two small matrices of the size of $T\\times T$, where $T$ is sequence length (<=1024 in practice), and aggregate them to obtain the per-sample norm (just a scale for each). And then it performs a second back-propagation to compute aggregation on the clipped gradient. It will uses almost the same memory as standard SGD (or ADAM), with one forward pass and two backward passes. \n\nThis paper also introduce two additional techniques to improve, one is choose a larger batch size, and the other is to introduce multi-task finetuning (fine-tuning includes masked prediction task on the target dataset). These two are important in boost the performance of the final models. \n\nThe paper evaluates on text classification, data-to-text generation, and dialog generation tasks. The results shows it gains prior DP methods. \n\n\n\n", "summary_of_the_review": "Reason to accept:\nSimple DP algorithm with demonstrated reduction in memory consumption and effectiveness. It enables large pre-trained NLP models to be further fine-tuned on private data under approximate DP. Clear presentation. \n\nReason to reject:\nBetter to identify the cause of improvement and make a fair comparison to baseline models (in particular for text classification tasks). Over-statement and misleading in abstract and introduction about comparable to strong non-private baselines.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work adapts the widely used DP learning algorithm to language models. Reviewers all agreed that this work tackles an important problem with clear motivation and thorough experiments, and achieved strong performance (memory reduction and effectiveness) on NLP tasks.  Thus, we recommend an acceptance.
Decision: Accept (Oral)