LARGE LANGUAGE MODELS CAN BE STRONG DIFFERENTIALLY PRIVATE LEARNERS
Year: 2021

Abstract:
Differentially Private (DP) learning has seen limited success for building large deep learning models of text, and attempts at straightforwardly applying Differentially Private Stochastic Gradient Descent (DP-SGD) to NLP tasks have resulted in large performance drops and high computational overhead. We show that this performance drop can be mitigated with (1) the use of large pretrained models; (2) hyperparameters that suit DP optimization; and (3) fine-tuning objectives aligned with the pretraining procedure. With these factors set right, we obtain private NLP models that outperform state-of-the-art private training approaches and strong nonprivate baselines from previous years—by directly fine-tuning pretrained models with DP optimization on moderately-sized corpora. To address the computational challenge of running DP-SGD with large Transformers, we propose a memory saving technique that allows clipping in DP-SGD to run without instantiating per-example gradients for any linear layer in the model. The technique enables privately training Transformers with almost the same memory cost as non-private training at a modest run-time overhead. Contrary to conventional wisdom that DP optimization fails at learning high-dimensional models (due to noise that scales with dimension) empirical results reveal that private learning with pretrained models tends to not suffer from dimension-dependent performance degradation.

heading: 1 INTRODUCTION
text: Machine learning systems trained on sensitive user data can be vulnerable to privacy attacks (Shokri et al., 2017; Hayes et al., 2019). This issue is especially pressing for recent applications involving large deep learning models of text (Brown et al., 2020), as training data for these systems often contain private user data (Carlini et al., 2020), and these models are able to memorize and regurgitate sensitive training examples (Zhang et al., 2016; Carlini et al., 2020).
As a result of these concerns, there has been a large interest in developing methods that provide data privacy guarantees for discriminative and generative models of text. The gold-standard way for providing such a guarantee in machine learning is Differential Privacy (DP) (Dwork et al., 2006; 2014). Unfortunately, DP learning has typically struggled to produce useful models when applied to large language models, resulting in models with either vacuous privacy guarantees (Dupuy et al., 2021) or performance far below non-private baselines. This is widely attributed to the fact that the core primitive of Differentially Private Stochastic Gradient Descent (DP-SGD) (Song et al., 2013; Bassily et al., 2014; Abadi et al., 2016) injects noise that must scale with the number of parameters, resulting in large noise levels for large language models (Yu et al., 2021a).
We tackle the problem of building performant DP language models for sentence classification and language generation tasks with tens to hundreds of thousands of examples. We pursue this goal by re-examining the performance of the baseline DP optimization algorithm for fine-tuning large language models, and study how choices of hyperparameters, training objective, and pretrained models affect the performance given fixed privacy budgets. In contrast to the mainstream perception, our empirical results demonstrate that large pretrained models with hundreds of millions of parameters can be effectively and efficiently fine-tuned to yield models with high performance with modest privacy leakage. For language generation, the performance of our models surpasses strong non-private baselines. For sentence classification, the performance of our fine-tuned models surpasses those obtained under heuristic privacy notions (Huang et al., 2020a) which do not possess formal guarantees. Figure 1 illustrates these results. We summarize our contributions below.
(1) We show that with appropriate hyperparameters and downstream task objectives, directly finetuning pretrained language models with DP-SGD/DP-Adam yields strong performance for a suite of NLP tasks at privacy levels ∈ {3, 8}. Notably, some of our fine-tuned models outperform strong non-private learning baselines and models obtained under heuristic privacy notions.
(2) DP-SGD is known for its expensive memory cost due to clipping per-example gradients. We present ghost clipping, a memory saving technique that makes fine-tuning larger Transformers under DP memory efficient. Our technique generalizes the Goodfellow (2015) trick to handle sequential inputs, and can be combined with a layer-by-layer clipping procedure (Lee & Kifer, 2020) to enable privately fitting large Transformers with almost the same memory cost as non-private training—at the cost of one additional backward pass per processed batch.
(3) We show that the dimensionality of gradient updates does not explain private fine-tuning performance. While there exist dimension-dependent lower bounds for private (convex) optimization (Bassily et al., 2014), we find that larger pretrained models lead to better private fine-tuning results. Moreover, parameter-efficient adaptation methods that reduce the dimensionality of updates do not necessarily outperform a baseline method that fine-tunes all model parameters.
Our empirical studies indicate that directly fine-tuning pretrained models with DP optimization results in performant DP language models under modest privacy budgets. This enables building practical DP NLP models for a range of common tasks where privacy could be at stake.

heading: 2 PROBLEM STATEMENT
text: We build models for sentence classification and language generation tasks with private datasets of modest sizes under (central/global) approximate-DP (also known as ( , δ)-DP) (Dwork et al., 2014).
Definition 1 (( , δ)-DP). A randomized algorithmM : X → Y is ( , δ)-differentially private if for all adjacent datasets X,X ′ ∈ X and all Y ⊂ Y , P (M(X) ∈ Y ) ≤ e P (M(X ′) ∈ Y ) + δ.
Two datasets are adjacent if and only if one can be obtained from the other by including an extra record (Mironov et al., 2019). What counts as a record is task dependent and will be defined below. Intuitively, algorithms satisfying DP ensures that output random variables obtained from similar inputs are also similar. and δ are privacy leakage parameters that measure the loss of privacy and small values imply more privacy. Unlike heuristic privacy notions (Huang et al., 2020b), DP allows for the tracking of privacy loss through the calculation of leakage parameters, and ensures privacy under composition (Dwork et al., 2014), meaning that the overall privacy loss of multiple DP algorithms releasing multiple statistics can be reasoned in a principled manner.
DP models are typically trained with DP optimizers which privatize the gradient before taking a standard gradient descent step. The privatization step ensures that the parameter updates leak limited
information about the training examples through their gradients. Specifically, this step clips perexample gradients with a norm constraint C, and adds isotropic Gaussian noise z ∼ N (0, C2σ2Ip) to the sum of clipped gradients. Here, σ is the noise multiplier determined from the privacy budget ( , δ), number of gradient updates S, and sampling rate q = BN for a batch size of B and a dataset with N examples. Intuitively, clipping individual gradients ensures that each example has bounded influence on the parameter update, whereas noising the gradient denies exact tracing of particular examples. The noise being isotropic implies that larger models would experience heavier noise per update, as the norm of the high-dimensional Gaussian ‖z‖2 tends to Cσ √ p for large dimensionality p with high probability. This is widely believed to be the cause for DP optimization performing poorly at training high-dimensional deep learning models (Kamath, 2020).
Our starting point for building DP language models is (public) pretrained models. Pretrained language models tend to contain general knowledge of language (Manning et al., 2020) and thus would make the downstream private learning problem easier. We fine-tune these models with DP-Adam (Abadi et al., 2016; Kingma & Ba, 2014) (see Appendix A for details) and track privacy loss through Rényi DP (Mironov, 2017), but also report the converted from Gaussian DP and its central limit theorem (Dong et al., 2019) and from accurately composing tradeoff functions via fast Fourier transform (Gopi et al., 2021). We consider privacy levels ∈ {3, 8} and δ = 1/2|Dtrain| throughout for a training set of size |Dtrain| (See Appendix B for further details). We tune hyperparameters on the E2E tasks and transfer these to all remaining tasks. We outline two broad classes of NLP problems considered in this paper and define what constitutes a record below.
Sentence Classification. The goal is to learn a model that correctly classifies sentences. For these tasks, each example/record consists of some input sentences and a label to be predicted. We fine-tune models of various sizes in the BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) families, as these models are known to work well for sentence classification.
Language Generation. The goal is to learn a model that generates natural language sentences given some context. For table-to-text generation tasks such as E2E (Novikova et al., 2017) and DART (Nan et al., 2020), each example/record in the training data consists of a pair of table entry and corresponding text description to be predicted. For a dialog generation task such as PersonaChat (Zhang et al., 2018), each example/record consists of some auxiliary information, a dialog history, and a response to be predicted. We fine-tune GPT-2 (Radford et al., 2019) of various sizes for these problems, as this model family is known to work well for text generation.

heading: 3 EFFECTIVE DIFFERENTIALLY PRIVATE FINE-TUNING
text: By studying the impact of hyperparameters and choice of fine-tuning objective, we demonstrate that the performance of the DP-Adam baseline can be substantially improved, even matching some strong non-private learning results. Our analyses reveal common failure modes when straightforwardly applying DP optimization and explain poor results reported in past works that consider these baselines.
3.1 HYPERPARAMETER TUNING
DP optimization is sensitive to the choice of hyperparameters (Papernot et al., 2019). Our experiments suggest that performance can vary from being close to trivial with ill-chosen hyperparameters to near previous state-of-the-art with appropriately chosen ones. As a consequence, we present simple but effective guidelines on setting the most important hyperparameters. Unless otherwise stated, the unmentioned hyperparameters are set to defaults documented in Appendix H.

heading: 3.1.1 BATCH SIZE, LEARNING RATE & TRAINING EPOCHS
text: Our experiments suggest that batch size is one of the most important hyperparameters to set correctly, and the dependence of the optimal batch size on learning rate and training epochs makes its selection complex. We first describe batch size selection in realistic, compute-bound settings and then describe how the complexity of identifying the optimal batch size in these situations arise due to constraints on the number of training epochs.
Fixed Training Epochs E. We first describe a realistic situation in which there is a constraint on the total amount of compute budget available. For the case of DP-SGD, this compute budget constraint often corresponds to a constraint on the number of examples that are processed by SGD.1In this fixed training epoch setting, the learning rate and batch size jointly affect performance, since using larger batches implies performing fewer gradient updates. To study this joint influence empirically, we fine-tune GPT-2 on the E2E dataset for table-to-text generation with DP-Adam at = 3 with various batch sizes and learning rates. Figure 2 shows that the best performing models are obtained with both a large batch size and large learning rate. Using a small learning rate together with a small batch size yields considerably worse results. Note a seq2seq baseline achieves a test BLEU of ~65 without privacy here (Wiseman et al., 2018).
Recall that in the non-private world, pretrained language models are typically fine-tuned with small batch sizes and small learning rates with Adam (bottom left panel in Figure 2). This implies that naı̈ely fine-tuning pretrained language models privately using the non-private setup would result in more performance degradation than necessary.
Recently, Tramèr & Boneh (2020) studied how the batch size and learning rate jointly affect learning private image classifiers while holding other hyperparameters fixed. They heuristically suggested a linear scaling rule: Scaling the learning rate together with the batch size by the same constant should yield models with almost the same performance. However, Figure 2 indicates that this fails to hold consistently as it falsely predicts that large batch and high learning rate (top right most entry) would have equal performance to small batch and low learning rate (bottom left entry). We further explain why linear scaling fails for small batches in Appendix D.
Fixed Update Steps S. In the fixed epoch setting, we saw that the optimal batch size setting was complex due to the trade-off between batch size and number of gradient updates. We now show that the complexity of setting batch sizes arises almost entirely from this tradeoff by considering a different setting, where the total number of gradient updates (rather than epochs) is fixed. In this case, using larger batches implies training for more epochs, and we find that using larger batch sizes almost always results in better performance at a given privacy budget (at the cost of processing more examples and using more compute), once other hyperparameters are fixed. We provide an explanation of this by introducing the idea of an effective noise multiplier σeff = σq = σN B . Recall the noise multiplier σ is determined from the privacy budget ( , δ), update steps S, and sampling rate q. In addition, recall the form of the privatized gradient ḡ in DP-SGD/DP-Adam:
ḡ = g̃ + z, g̃ = 1B ∑B i=1Clip(∇Li, C), z ∼ N ( 0, C2 σ 2 B2 Ip ) = N ( 0, C2 σ2eff N2 Ip ) ,
where∇Li is the gradient of the ith example in a batch ofB examples and Clip(v,K) clips the vector v by the norm constraint K. We observe that for moderately large batches, the signal-to-noise ratio r = ‖g̃‖2/‖z‖2 is mainly controlled by the batch size through the effective noise multiplier: The signal term g̃ tends to concentrate quickly due to being an average of bounded vectors, whereas the effective noise multiplier σeff could vary significantly as the batch size B changes. Figure 3 (a) shows that the effective noise multiplier decreases as batch size increases. In addition, Figure 3 (b) plots the average signal-to-noise ratio r over the first 30 gradient updates against the final model’s performance on E2E and demonstrates that large batches (up to a threshold) lead to both increased signal-to-noise ratio at the beginning of training and better performance at the end of training. These findings additionally resonate with and explains recent empirical successes of private pretraining (Anil et al., 2021).

heading: 3.1.2 CLIPPING NORM
text: DP optimization is known to be sensitive to the choice of clipping norm. Since the scale of noise depends on this clipping norm (recall its standard deviation is Cσ), picking the threshold C much larger than the actual gradient norm implies more noise is being applied than necessary. In practice, we have found that a small clipping norm which enforces almost all gradients to be clipped throughout training leads to the best performing models (see Figure 8 in Appendix H).

heading: 3.2 IMPROVING THE TASK ALIGNMENT HELPS PRIVATE LEARNING
text: Our fine-tuned models on language generation tasks worked well since the pretraining objective and downstream task are aligned: Both involve predicting sequences of tokens. This alignment simplified
1This is because DP-SGD often necessitates microbatching, in which case the number of backward passes is independent of the actual batch size for gradient updates but dependent on numbers of passes through the data.
the task and benefitted private learning. While pretrained models are naturally aligned for language generation, it is much less so for classification tasks. The standard approach for adapting language models for classification involves stacking a freshly initialized network on top of the encoding of the special [CLS] token and jointly optimizing all parameters (Devlin et al., 2018). This workflow introduces a discrepancy between pretraining and fine-tuning: Pretraining predicts masked out words from a large vocabulary whereas fine-tuning predicts integer labels.
To eliminate the discrepancy, we instead consider learning to predict the missing word during finetuning for classification. For example, for sentiment classification, we reframe the problem as filling in the [MASK] token in the sequence “<INPUT>. It is [MASK].” and compare the probabilities of words “awesome” and “terrible”. This text infilling task is almost exactly the procedure used for pretraining masked language models, and recent works have demonstrated its effectiveness for knowledge probing (Petroni et al., 2019), few-shot learning (Gao et al., 2020) and multi-task finetuning (Wei et al., 2021). On SST-2, we found that using the generic template as described above already improved private fine-tuning performance by 3 ∼ 5% across different settings. Table 1 (to be presented) contains additional results and Appendix E includes analyses on choices of label words.

heading: 4 GHOST CLIPPING: CLIPPING WITHOUT PER-EXAMPLE GRADIENTS
text: DP-SGD has high memory overhead due to clipping per-example gradients. Naı̈vely implemented, this step instantiates a giant gradient vector for each example in a batch during optimization and can be prohibitively expensive. For example, Hoory et al. (2021) pretrained BERT with DP optimization and reported memory issues when using the large batches necessary to achieve high performance. A time-costly solution to the memory problem is micro-batching: Split large batches into multiple smaller ones and aggregate the results after processing each small batch individually (Tramèr & Boneh, 2020). This solution, however, is unlikely to be sufficient as neural language models become larger and fitting a few copies of the gradient in memory can be difficult. Lee & Kifer (2020) observed that per-example gradients need not be instantiated at all, if the goal is only to clip the gradients and take the sum. They presented a clipping procedure that only instantiates the per-example gradient for parameters of a single layer in the model one at a time, as opposed to the entire model at once.
Unfortunately, we find this trick to be still insufficient for sequence models such as Transformers (Vaswani et al., 2017), as the memory requirement for per-example gradients of embedding layers and language modeling heads can be costly. We extend the Lee & Kifer (2020) approach such that training Transformers with DP optimization has almost the same memory consumption as non-private training, at the cost of an extra backpropagation pass per processed batch. Unlike their approach, our extension avoids instantiating the per-example gradient even for individual linear layers. We call this approach ghost clipping, as the per-example gradient is the ghost that never explicitly appears. We anticipate this extension to be useful for both privately fine-tuning and pretraining large Transformers.

heading: 4.1 THE MEMORY TRICK BY LEE & KIFER (2020)
text: Per-example gradient clipping is easy if we know per-example gradient norms. In this case, we first compute the scaling factor ci = min(1,C/‖∇Li‖2), where C is the clipping threshold and Li is the loss associated with the ith example. Then, we perform the usual backward pass with the reweighted
scalar loss ∑ i ciLi. This procedure gives us the sum of clipped gradients. Under this setup, the difficulty is computing the per-example gradient norm ‖∇Li‖2. We emphasize two technicalities that enable computing this quantity without instantiating the full per-example gradient∇Li.
First, for a typical neural net layer l with parameters W (l) (without parameter sharing), the perexample gradient w.r.t. parameters can be easily computed using the input to the layer a(l) and the gradient of the loss w.r.t. the output g(l), both of which are available during backpropagation. Second, for a large vector formed by concatenating several small vectors u = [u1, . . . , uk], its Euclidean norm is simply the norm of the vector of norms, i.e. ‖u‖2 = ‖(‖u1‖2 , . . . , ‖uk‖2)‖2 . The second observation means that computing the per-example gradient norm ‖∇Li‖2 can be done by computing the per-example gradient norms for individual layers of the neural net ‖∇W (1)Li‖2 , . . . , ‖∇W (L)Li‖2 one at a time (L is layer count). Moreover, the first observation implies that the norms for each layer can be computed using quantities freely available to a typical backward pass. Overall, the per-example gradient norm of any network without parameter sharing can be computed in a layer-by-layer fashion with only one per-example gradient tensor for a single layer being instantiated at any time.

heading: 4.2 GHOST CLIPPING FOR TRANSFORMERS WITH SEQUENTIAL DATA
text: The trick by Lee & Kifer (2020) still requires instantiating the per-example gradient of individual layers (although not simultaneously). This can be problematic in terms of memory for Transformers with large embedding layers.2 Here, we present a specialized procedure for computing the perexample gradient norm for linear and embedding layers when they are applied to sequential data.3 This procedure reduces memory footprint and can be viewed as a generalization of the Goodfellow (2015) trick that additionally handles sequential inputs.
Let a ∈ RB×T×d be the input to a linear layer with weight matrix W ∈ Rp×d, and s ∈ RB×T×p be the output with si,j = Wai,j . Let g ∈ RB×T×p be the gradient of the loss w.r.t. the output s. Here, T is the number of time steps in the input, and we omitted biases for simplicity. Simple calculation shows that the per-example gradient is the product of two matrices:
∇WLi = g>i ai ∈ Rp×d. (1)
Since the per-example gradient norms are the end goal, the per-example gradients {∇WLi}Bi=1 themselves need not be instantiated explicitly. More precisely, we observe that the squared perexample gradient norm for this layer ‖∇WLi‖2F obeys the following identity:
‖∇WLi‖2F = vec(aia > i ) >vec(gig > i ). (2)
See Appendix F for a derivation. Implemented with common primitives in machine learning libraries, (2) has a memory complexity of order O(BT 2) when aia>i , gig>i ∈ RT×T are instantiated,45 as opposed to O(Bpd) in the naı̈ve approach which goes through instantiating (1).6
The memory efficiency of this procedure is exemplified with off the shelf pretrained language models, most of which have large embeddings. For instance, for GPT-2, d ≈ 50, 000 and p = 768 for the embedding layer, and the context window T ≤ 1024.7 Our method in theory reduces the memory cost of this large embedding layer by at least a factor of 22. In practice, we also observe significant savings, since embedding layers can be a major source of memory spending for training large language models.8 To stress-test ghost clipping, we compare it with 4 baselines: The PyTorch package Opacus that implements DP optimization by instantiating per-example gradients, the approach by Lee & Kifer (2020), non-private training in PyTorch, and naı̈ve DP optimization implemented in
2For GPT-2, per-example gradients w.r.t. the embedding for ten examples alone occupy ~1.5GB of memory. 3An embedding layer is essentially a linear layer: The embedding lookup operation applied to indices is equivalent to a matrix multiplication of the embedding matrix with one-hot encoded indices. 4This is assuming the space complexity for multiplying two matrices A ∈ Rm×n and B ∈ Rn×p is roughly O(mp), which is the case for most workloads running on a framework like PyTorch. 5More sophisticated solutions may even avoid instantiating aia>i and gig > i entirely by trading in more run-time. Custom CUDA kernels are likely needed to make these solutions fast in practice. 6We omitted the cost of storing ai and gi, since our goal is to compare the additional cost induced by computing gradient norms. 7In practice, for fine-tuning tasks, the maximum sequence length is usually a few hundred. 8While there are alternative approaches for reducing the memory footprint of embedding layers during training, these methods tend to introduce extra hyperparameters that require tuning and privacy spending.
JAX with jit and vmap enabled. We include the JAX baseline since recent studies show that DP optimization can be made cheap through compiler optimization (Subramani et al., 2020). Figure 4 (a) shows that for typical inputs, our technique is the most memory friendly and allows fitting batches almost as large as those in non-private training. Since ghost clipping allows us to fit larger batches but with a run-time penalty, a natural question is whether it improves throughput with the use of larger batches. Figure 4 (b) shows that while ghost clipping only provides minor gains compared to Opacus for smaller models, it allows processing ~10% more examples compared to the approach by Lee & Kifer (2020) for fitting GPT-2-large, a model that neither Opacus or JAX could handle. See Appendix G for the setup of these experiments.

heading: 5 LOW DIMENSIONAL UPDATES ARE NOT NECESSARILY BETTER
text: Since the norm of the noise added to gradients in DP scales with dimensionality, it is natural to ask whether privatizing and updating fewer parameters would result in improved performance. We decompose this question into two aspects: (1) Do smaller pretrained models lead to better private fine-tuned performance, and (2) do parameter-efficient adaptation methods designed with a reduced dimensionality of updates outperform full fine-tuning? Our experiments below show that the answer to both questions is negative. Reported numbers in this section are averaged over three seeds.

heading: 5.1 LARGER PRETRAINED MODELS RESULT IN BETTER PERFORMANCE
text: We observe that larger pretrained models lead to better private fine-tuned performance. Specifically, we fully fine-tune four sizes of GPT-2 models (for language generation) and three sizes of BERT/RoBERTa models (for sentence classification) at the same privacy budget with DP-Adam and compare their performances. Since the performance of DP optimization heavily depends on hyperparameter choices, we need to ensure that our hyperparameters are not particularly favoring larger models. We thus tune hyperparameters on the smallest model and then reuse the same hyperparameters for all fine-tuning workloads. Figure 1 from earlier demonstrates gains on E2E and MNLI from model scaling, and we find similar improvements on 5 additional tasks (deferred to Figure 5 in Appendix C).

heading: 5.2 FULL FINE-TUNING WITH DP-ADAM MATCHES STATE-OF-THE-ART
text: There is a range of lightweight fine-tuning methods that reduce the dimensionality of updates, including some that are designed for DP (Yu et al., 2021b). Do methods that optimize fewer parameters lead to better results under DP even if they perform similarly non-privately? Empirical results suggest otherwise and that full fine-tuning is a strong baseline that even matches specialized low-dimensional DP learning methods for both classification and generation. Below, we study the two sets of tasks separately. For completeness, all experimental details are in Appendix K.
Sentence Classification. We study DP fine-tuning on tasks from the GLUE benchmark that have more than 10k training examples (MNLI, QQP, QNLI, and SST-2), following the experimental setup of Yu et al. (2021b). The associated datasets have modest sizes: SST-2 and QNLI have 60k+ and 100k+ training examples, respectively. MNLI and QQP each contains less than 400k examples.
Table 1 shows that using larger pretrained models and the text-infilling objective generally improve classification accuracy. We compare full fine-tuning with reparameterized gradient perturbation (RGP) (Yu et al., 2021b), as it is the state-of-the-art for DP fine-tuning on sentence classification at the time of writing. The method is designed to privatize gradients projected onto low dimensional subspaces and was motivated to reduce DP noise in high-dimensional models. We note that full fine-tuning with the text infilling objective outperforms well-tuned RGP on all tasks despite being the simplest baseline. Computationally, while RGP is faster per-update, it requires more than 3 times as many epochs as full fine-tuning – overall, both methods are comparably fast in terms of wall time.
Table-To-Text Generation. We study different fine-tuning methods under DP for table-to-text generation where the goal is to generate natural language descriptions of table entries. We consider the datasets E2E (Novikova et al., 2017) and DART (Nan et al., 2020). E2E consists of simple restaurant reviews, whereas DART consists of open-domain table entries from Wikipedia and is more complex. Both datasets are small: E2E has more than 40k training examples, whereas DART has more than 60k. Since we are the first to experiment with this task under DP, we compare full fine-tuning (full) against a suite of parameter-efficient approaches which includes LoRA (Hu et al., 2021), prefix-tuning (Li & Liang, 2021) (prefix), RGP, and fine-tuning the top 2 Transformer blocks (top2), all of which optimize few parameters. On GPT-2 (125 million parameters), prefix-tuning with default hyperparameters optimizes ~10 million parameters; LoRA with rank 4 optimizes ~0.15 million parameters. We also report results for training from randomly initialized weights (retrain). Hyperparameters of each method were tuned only the E2E dataset; the complete search ranges are in Appendix I. Table 2 shows that LoRA and full fine-tuning are generally the most performant on E2E. Tables 7 and 8 in Appendix J contain the full result on E2E and DART and confirm the trend.
Chit-Chat Dialog Generation. We stress-test full fine-tuning under DP with the task of chitchat dialog generation. This task has the distinct challenge that the response space is intrinsically diverse (Li et al., 2015; Gao et al., 2018) since human conversations can be informal and noisy (Zhang et al., 2019). Moreover, dialog datasets are usually formed with user data which may contain sensitive information. We use the Persona-Chat dataset (Zhang et al., 2018) as a testbed and build off a processed version that has ~130k training entries. Each entry contains a dialog history, persona
descriptions of the respondent, and the response. We fine-tune GPT-2, GPT2-medium, and DialoGPTmedium on this dataset both privately and non-privately by training to predict the response with the dialog history and persona description. We report the F1 score and perplexity on the validation split, and human evaluated quality scores of generations. Table 3 shows that private models have strong performance. In particular, fine-tuned DialoGPT-medium at = 8 beats the (non-private) winning entry of the ConvAI2 challenge (Dinan et al., 2019) on perplexity and has a human evaluation rating that is close to non-private models. Samples from our private models can be found in Appendix O.

heading: 6 RELATED WORK
text: Private NLP. The privacy-preserving NLP space is largely divided by whether or not a formal notion of privacy is considered. McMahan et al. (2017) successfully trained small word-level RNNs with 1.35 million parameters in a federated setting with more than 700k users under a global DP guarantee of ( , δ) = (4.6, 10−9). Ramaswamy et al. (2020) train production grade next-word prediction models using DP-FedAvg with millions of users. Qu et al. (2021) studied fine-tuning BERT for language understanding tasks under local DP. Kerrigan et al. (2020) presented initial results that public pretraining is helpful for downstream DP fine-tuning. However, they did not attempt fine-tuning large pretrained models with DP-SGD. Bommasani et al. (2021) briefly commented on the possibility of achieving cheaper private learning by fine-tuning large pretrained language models. Anil et al. (2021) pretrained BERT under global DP on datasets with hundreds of millions of examples. Dupuy et al. (2021) studied private BERT fine-tuning on datasets of utterances, but reported results with on the order of at least 100. Orthogonally, many works considered training language models that satisfy empirical notions of privacy (Xu et al., 2021; Coavoux et al., 2018; Mireshghallah et al., 2021; Melamud & Shivade, 2019). Our work is distinct from all works mentioned above in that we study fine-tuning large language models (with hundreds of millions of parameters) under global DP with stringent guarantees ( ∈ {3, 8}) on smaller datasets (much less than a million examples).

heading: 7 SCOPE AND LIMITATIONS
text: We presented strategies for fine-tuning large pretrained language models under DP for a wide range of NLP tasks. For researchers and practitioners working on private NLP, our empirical results suggest that DP fine-tuning with a proper setup is a competitive baseline that is worth trying before prematurely shifting to less formal notions of privacy which have not stood against the test of time. Below we list some limitations and future directions. Pretraining vs Private Learning. Our model scaling results suggest that using larger pretrained models improves performance. This argument, however, is dependent on the particular choice of pretrained models. How pretraining helps private learning and whether better pretrained models for private learning could be built are interesting future avenues. Scaling Laws for Private Learning. While scaling laws (Kaplan et al., 2020) for non-private deep learning have become prevalent, we are unaware of a case study in the private realm. Studies on how the dimensionality of models (and pretraining) generally affect private deep learning in precise quantitative terms will likely be a useful tool in trading off compute budget and model quality.
ETHICS STATEMENT
While the present paper studies privacy aspects of NLP models, all experiments performed herein are based on publicly available datasets for reproducibility. The paper empirically studies various behaviors of (mostly existing) algorithms, thus likely does not introduce any new ethical or societal concerns. The present paper does not introduce any new dataset.
REPRODUCIBILITY
All experiments in the paper are based on publicly available datasets. Links to these datasets are included in the main text and appendices. Hyperparameters necessary for reproducing our experiments are documented in Appendix I and H. Code for our ghost clipping technique will be released shortly.

heading: A DP-ADAM
text: We use DP-Adam throughout. DP-Adam works just like regular Adam (Kingma & Ba, 2014) but performs updates and moment accumulation with privatized gradients. The gradient privatization part is the same as that performed in DP-SGD (Song et al., 2013; Abadi et al., 2016). Seemingly uncommon, DP-Adam is used in many private ML libraries.9 To determine the noise multiplier, we account privacy through Rényi differential privacy (RDP) (Mironov, 2017; Mironov et al., 2019). For completeness, we include the pseudocode for DP-Adam below.
Algorithm 1 DP-Adam
1: Input: Data D = {xi}Ni=1, learning rate η, noise multiplier σ, batch size B, Euclidean norm threshold for gradients C, epochs E, initial parameter vector θ0 ∈ Rp, initial moment estimates m0, v0 ∈ Rp, exponential decay rates β1, β2 ∈ R, avoid division-by-zero constant γ ∈ R. 2: for t ∈ [E · N/B] do 3: Draw a batch Bt via Poisson sampling; each element has probability B/N of being selected 4: for xi ∈ Bt do 5: gt(xi)← ∇θtL(xi), g̃t(xi)← gt(xi) ·min(1,C/‖gt(xi)‖2) 6: end for 7: zt ∼ N (0, σ2C2Ip) 8: ḡt = 1 B (∑N i=1 g̃t(xi) + zt
) 9: θt+1,mt+1, vt+1 ← AdamUpdate(θt,mt, vt, ḡt, β1, β2, γ)
10: end for 11: return θTN/B
Algorithm 2 AdamUpdate 1: Input: θt,mt, vt, ḡt, β1, β2, γ 2: mt+1 ← β1 ·mt + (1− β1) · ḡt, vt+1 ← β2 · vt + (1− β2) · ḡt2 3: m̂t+1 ← mt+1/ (1− βt1) , v̂t+1 ← vt+1/ (1− βt2) 4: θt+1 ← θt − α · m̂t+1/ (√ v̂t+1 + γ
) 5: return θt+1,mt+1, vt+1

heading: B PRIVACY ACCOUNTING
text: We train all models under approximate-DP (Dwork et al., 2014), and we view two datasets as being adjacent if and only if one can be obtained from the other by including an extra record (Mironov et al., 2019). Instead of accounting the privacy loss with Moments Accountant (Abadi et al., 2016), we perform computation through (i) Rényi DP (Mironov, 2017; Mironov et al., 2019), (ii) Gaussian DP with an associated central limit theorem (Dong et al., 2019), and (iii) numerically composing tradeoff functions via fast Fourier transform (Gopi et al., 2021; Koskela et al., 2020). All approaches are improvements over the Moments Accountant. Accounting loss with Rényi DP provides strict upper bounds on the actual privacy leakage but may result in loose bounds. Accounting loss with Gaussian DP and its central limit theorem, although asymptotically exact, only provides approximations to the actual loss under a finite number of compositions (Dong et al., 2019, Theorem 3.4). Gopi et al. (2021) observed that accounting loss with GDP and its CLT results in underestimation and proposed to numerically compose tradeoff functions resulting in both upper and lower bounds on the actual leakage . We therefore also report the converted with the approach by Gopi et al. (2021) using their code.10
Given the noise multiplier σ, sampling rate q, number of steps S, and failure constant δ, can be computed via first computing the Rényi DP leakage and then converting it to approximate DP. When a privacy spending (specified by a set of given and δ) is prescribed, we can numerically invert the above procedure to obtain a suitable σ for noisy optimization. This is what we do throughout all
9https://github.com/tensorflow/privacy/blob/7c4f5bab0964bd32b7ceafa009d9488920856440/ tensorflow_privacy/privacy/optimizers/dp_optimizer.py#L385
10https://github.com/microsoft/prv_accountant
experiments. For completeness, given the σ chosen as above, we also report the leakage estimated by going through the central limit theorem in Gaussian DP (Bu et al., 2020).
Model selection from hyperparameter tuning on private training and validation data incurs extra leakage (Liu & Talwar, 2019; Chaudhuri & Vinterbo, 2013; Papernot & Steinke, 2021). We perform tuning only on the E2E task and reuse almost the exact hyperparameters for remaining tasks. Note the general strategy of tuning hyperparameters on a separate (public) dataset and thereafter transfer has been applied to train private production language models (Ramaswamy et al., 2020).

heading: C ADDITIONAL RESULTS ON MODEL SCALING
text: We repeat the model scaling experiments from Figure 1 on the other tasks considered in the paper. Figure 5 shows that the trend that larger and better pretrained models lead to improved private finetuned performance holds consistently across all tasks. TextHide numbers based on the TextHideintra formulation (Huang et al., 2020a).

heading: D WHEN AND WHY DOES LINEAR SCALING FAIL?
text: Recall Tramèr & Boneh (2020) suggested that the following simple rule approximately holds in private learning: Scaling the learning rate together with the batch size by the same constant yields models with almost the same performance. Note that their experiments on MNIST, Fashion-MNIST, and CIFAR-10 used only batch sizes in {512, 1024, 2048, 4096}. These values are fairly large from a non-private learning perspective. Indeed, our experiments on E2E suggest that this rule does not generalize to batch sizes that are too small (sampling rates q = B/N < 2−8).
We provide an explanation by noting that a core assumption which the linear scaling rule depends on fails to hold for small batch sizes. This assumption is that given a privacy budget, a “square-root” relationship holds between the noise multiplier and the sampling rate (see also (Tramèr & Boneh, 2020, Claim D.1)). For instance, Tramèr & Boneh (2020) showed that σ ≈ c√q when q ∈ [2−7, 1] for some constant c. Our numerical estimates show that this relationship fails to hold for small q – it underestimates the true noise multiplier σ that would be obtained with numerical computation. Figure 6 provides an illustration for ( , δ) = (3, 10−5) when the sample size N = 50k and number of training epochs E = 50.

heading: E HOW DOES THE CHOICE OF LABEL WORDS AFFECT SENTENCE CLASSIFICATION
text: Recall in Section 3.2 we cast sentence classification as filling in the missing word amongK candidates for a K-way classification problem. Since a label word could be mapped to multiple possible words, we study how the choice of label words affect performance. We again use the sentiment classification task SST-2 as a testbed. Figure 7 shows the effect of varying the label word, where we measure the alignment between the label word and the downstream task by the zero-shot performance of the infilling task (x-axis). We find that increasing the task alignment of the label words alone improves performance by 1− 2% (orange curve), which is in contrast to the non-private setting, where choice of label words do not affect performance in statistically significant ways (blue curve).

heading: F DERIVATION OF THE FROBENIUS NORM IDENTITY
text: Recall a ∈ RB×T×d is the input to a linear layer with weight matrix W ∈ Rp×d, and g ∈ RB×T×p is the gradient of the loss w.r.t. the output. The identity follows from trivial algebra:
‖∇WLi‖2F = ∥∥g>i ai∥∥2F = ∥∥∥∥∥ T∑ k=1 gi,ka > i,k ∥∥∥∥∥ 2
F
= d∑ r=1 p∑ s=1 ( T∑ k=1 ai,k,rgi,k,s )2
= d∑ r=1 p∑ s=1 T∑ k1=1 T∑ k2=1 ai,k1,rgi,k1,sai,k2,rgi,k2,s
= T∑ k1=1 T∑ k2=1 ( d∑ r=1 ai,k1,rai,k2,r )( p∑ s=1 gi,k1,sgi,k2,s ) = vec(aia > i ) >vec(gig > i ).
Note that when T = 1, the identity takes the form of
‖∇WLi‖2F = vec(aia > i ) >vec(gig > i ) = ‖ai‖ 2 2 ‖gi‖ 2 2 .
This is exactly what is used in the Goodfellow (2015) trick.

heading: G PROTOCOL FOR EXPERIMENTS IN SECTION 4.2
text: For these experiments, we used mock examples with the same format as examples in the E2E dataset. We created mock input sequences of length 100, as this length is almost the maximum length of examples in the actual E2E training set.
Our JAX implementation is adapted from a codebase used in the work by Subramani et al. (2020) and utilizes the package flaxmodels for loading pretrained models.11 The Opacus (Yousefpour et al., 2021) baseline is based on version 0.14.0 of the library, where for a fair comparison, we also optimized the implementation of the privacy engine by replacing all einsum operations with basic primitives that manipulate tensors directly. We found certain einsum steps to cause large and unnecessary speed and memory overheads.
To identify the maximum batch size that each approach could use, we ran binary search over the range of possible batch sizes until the upper bound matched the lower bound and that OOM did not occur. To estimate the throughput of each private method, we compared them to non-private training first. Specifically, for a pairwise comparison, we took the maximum batch size for non-private training and some private method, and computed the least common multiple as a batch size to perform updates with. Using the least common multiple batch size ensures that both methods will process exactly the same number of examples and perform the same number of updates. By estimating the time elapse of these methods for performing a fixed number of updates, we obtained throughputs for non-private training and the private method. This gives us the relative throughput of the private method with non-private training as the reference.
For methods implemented in PyTorch, the time elapse was recorded with torch.profiler. When estimating the time elapse for a training procedure, we first performed 3 gradients updates as a warm up process before taking the actual steps which will be timed. In particular, this eliminates the time that JAX uses for compiling computation graphs with vmap and jit.
All experimental runs in the section were under full precision.
11https://github.com/matthias-wright/flaxmodels

heading: H DETAILS AND ADDITIONAL RESULTS FOR STUDIES IN SECTION 3.1

heading: I HYPERPARAMETER SEARCH RANGES FOR EXPERIMENTS IN SECTION 5
text: We compare different adaptation methods by reporting task specific metrics on the test split using hyperparameters that maximize validation BLEU on E2E. For sentence classification tasks, we reused the same hyperparameters, except for the number of training epochs and batch size. For SST-2, we reused the same batch size as for private E2E fine-tuning, and the number of epochs exactly as in typical non-private fine-tuning for SST-2 (number of epochs equals to 3 in this case). For remaining classification tasks, we use a batch size such that the sampling rate is the same as for SST-2, and a number of training epochs that ensures the number of gradient updates is the same as for SST-2. Appendix L outlines why we transfer the sampling rate and number of gradient updates, as opposed to the batch size and training epochs. We list the range of hyperparameters that we searched over for each individual adaptation method on E2E considered in the paper. Prefix-tuning has two additional hyperparameters: the length of the prefix and the dimensionality of the hidden layer. We set these to the default used by Li & Liang (2021) (10 for the former and 512 for the latter). For Adam, we use the default hyperparamaters set by PyTorch (Paszke et al., 2017).

heading: J FULL RESULTS FOR EXPERIMENTS IN SECTION 5.2

heading: K DETAILS FOR EXPERIMENTS IN SECTION 5.2
text: Sentence Classification. Results for RGP in Table 1 are taken from documented numbers in their released codebase.12 These results are under the DP guarantees of ( , δ) = (3, 10−5) or ( , δ) = (8, 10−5). These guarantees are strictly looser than our guarantees which are based on δ = 1/2|Dtrain| (recall the smallest dataset in this cohort of tasks has 60k+ records). The RGP numbers in Table 1 are higher than those reported in their paper (Yu et al., 2021b), since the latter numbers are not based on fine-tuning the official RoBERTa models.
Table-To-Text Generation. To evaluate models trained on E2E and DART, we evaluate generations from models obtained with beam search with a beam size of 5. For evaluation, we run the official pipeline for E2E,13 and the pipeline used in the GEM benchmark (Gehrmann et al., 2021) for DART.14
Chit-Chat Dialog Generation. We built off Huggingface’s codebase of the winning entry of the ConvAI2 competition, 15 16 and used their preprocessed training set with the minor modification of truncating the number of training examples to be a multiple of the batch size. The original ConvAI2 competition is aimed at advancing research on building engaging chatbots and also requested models to predict the mostly likely response given a list of candidates. The challenge included hits@1 as part of its suite of automatic metrics. For simplicity, we skip this step of predicting the most likely response for both training and evaluation. Results for the entry HuggingFace (ConvAI2 winner) in Table 3 are taken from the official validation set leader board.17 Our reimplementation of HuggingFace’s submission uses the released code for their winning entry, fine-tunes GPT with the default hyperparameters, and removes the classification loss for learning to predict the most likely response given candidates.
We additionally fine-tuned DialoGPT-medium (Zhang et al., 2019), since the model was pretrained on conversation-like exchanges extracted from Reddit comment chains. Intuitively, this pretraining corpus is more aligned with the downstream fine-tuning data than WebText (Radford et al., 2019), and thus would likely improve downstream performance.
To evaluate the F1 score, we obtained predicted responses from trained models using beam search with a beam size of 5. Since past work found that the F1 score can be gamed by always letting the model predict a predetermined phrase (Dinan et al., 2019), we additionally ask humans to rate generations from the model through Amazon mechanical turk to obtain a more complete picture of model quality. When sampling responses for human evaluation, we used nucleus sampling with p = 0.9 (Holtzman et al., 2019). For human evaluation, we asked 20 turkers to each rate 5 entries. For each entry, a turker is asked to rate on a scale of 1 to 5 the quality of predicted responses from privately fine-tuned DialoGPT-medium models, non-privately fine-tuned DialoGPT-medium models, non-privately fine-tuned GPT models (our reimplementation of HuggingFace’s entry), and the reference text, given the history of the dialog. Since human evaluation can yield noisy results, we also report the 95% asymptotic confidence interval in Table 3. All models were trained and evaluated on the version of Persona-Chat with the original persona. All numbers reported in Table 3 are obtained on the validation split.

heading: L TRANSFERRING HYPERPARAMETERS ACROSS DATASETS
text: Our work involved tuning hyperparameter with models trained via DP-Adam on one private dataset and transferring such hyperparameters to other private datasets. Since different datasets may be of different sizes, transferring hyperparameter values such as the number of training epochs and batch size may cause a discrepancy in the effective noise multiplier across workloads with different datasets. Transferring the number of training epochs and batch size based on hyperparameter tuning on small
12https://github.com/dayu11/Differentially-Private-Deep-Learning/tree/ main/language
13https://github.com/tuetschek/e2e-metrics 14https://github.com/GEM-benchmark/GEM-metrics 15https://github.com/huggingface/transfer-learning-conv-ai 16http://convai.io/2018/ 17https://github.com/DeepPavlov/convai/blob/master/leaderboards.md
datasets to larger datasets can be particularly problematic, as the effective noise multiplier can be larger than ideal. In this work, we instead transferred the sampling rate q and number of training steps S across different datasets.

heading: M DOES DP FINE-TUNING PREVENT UNINTENDED MEMORIZATION?
text: One of the ultimate goals of fitting models under DP is to ensure that training data extraction is unlikely given the trained model. To empirically evaluate whether DP fine-tuning helps prevent against unintended memorization and related attacks, we follow the secret sharer framework (Carlini et al., 2019) and estimate the exposure of artificial canaries inserted into the training set used for fine-tuning. We use the E2E dataset as a testbed.
To create canaries, we first form a subvocabulary by randomly sampling V = 10 words in the original vocabulary of GPT-2. Our canaries have prefixes of the form
" name : <word> | Type : <word> | area : <word> ”,
where <word> is randomly sampled from the subvocabulary. The suffix which our model should learn to predict consists of randomly sampled words with an average length of l = 5. By definition, canaries with an estimated exposure close to log2(V
l) ≈ 17 can likely be extracted. We experiment with canary-corrupted datasets for repetition values r ∈ {1, 10, 100}. A canary has a higher chance in being extracted when it’s repeated for more than once in the training data.

heading: N TEMPLATES AND LABEL WORDS FOR TEXT-INFILLING-BASED
text: Recall that fine-tuning for classification can be reformulated as filling in the [MASK] token in a template sequence. Here, we list the templates used for each classification task considered in the paper. These templates are almost generic and are not obtained from expensive manual or automated search. We anticipate better templates obtained from automated search based on data (Gao et al., 2020) to improve the performance even further. However, we also expect that such a procedure would lead to some amount of increased privacy spending if it were based on private data.

heading: O UNCURATED SAMPLES FROM FINE-TUNED MODELS

heading: P ADDITIONAL RELATED WORK
text: Differentially Private Deep Learning. DP-SGD has been viewed as ineffective for large models due to the addition of large Gaussian noise to gradient updates. Improvements to the learning procedure mostly fall under two distinct camps: (i) Simplifying the private learning problem, and (ii) reducing the scale of noise. For instance, Papernot et al. (2019); Tramèr & Boneh (2020); Abadi et al. (2016) consider transferring features learned on public datasets to simplify the subsequent private learning task. On the other hand, Zhou et al. (2020); Kairouz et al. (2020) remove the ambient dimension dependence of DP noise by identifying subspaces in which private gradients lie and would be privatized. Yu et al. (2021a;b) make such ideas practical and demonstrate improved results on private learning benchmarks. Zhang et al. (2021) applied the sparse vector technique to learning wide neural layers to reduce the amount of injected noise. Our work mostly falls under the first camp – improving private learning through simplifying the learning task. Our work is also distinct from prior works in that we focus on privately fine-tuning large pretrained models. Lastly, there are alternative solutions in the literature that enforces DP which are not based on gradient perturbation (Papernot et al., 2018; 2016). These methods typically require extra public data and are not the present focus.
Parameter-Efficient Fine-Tuning. Recent developments on pretrained model adaptation have produced a wide range of parameter-efficient fine-tuning methods for both vision and language tasks. We briefly summarize these, grouping by category. Approaches based on optimizing prompt-like constructions for NLP tasks include prefix-tuning (Li & Liang, 2021), P-tuning (Liu et al., 2021), and prompt-tuning (Lester et al., 2021). Adapter-based methods insert small subnetworks inside pretrained Transformers (Houlsby et al., 2019; Rücklé et al., 2020; Pfeiffer et al., 2020). Methods that optimize low-rank matrices include the work by Hu et al. (2021); Mahabadi et al. (2021). In addition, there are adaptation methods that only optimize biases for vision (Cai et al., 2020) and language tasks (Ben Zaken et al., 2021). Our evaluation in Section 5.2 covered the most representative methods that generally have state-of-the-art non-private learning performance (at the time of writing) for the range of NLP tasks studied in this paper.
Speeding Up DP-SGD. Apart from the work by Lee & Kifer (2020) and Subramani et al. (2020), there is an approach that approximates per-example gradient norms through the combination of random projection and forward-mode autodiff (Bu et al., 2021). While faster than vanilla private learning, this approach has the drawback of increased privacy spending and having an extra hyperparameter. Our ghost clipping technique, while only suited for Transformers applied to sequential data, does not introduce new hyperparameters.
Alternative Clipping Strategies. While there are alternative clipping strategies in the literature that show improvements on simple tasks (Pichapati et al., 2019; Asi et al., 2021), we have opted to study the simplest strategy that clips gradients by their Euclidean norm. We leave the study of these algorithms for NLP tasks to future work.
Concurrent Work. We are made aware of a concurrent work in submission that also studies finetuning large language models under DP (Anonymous, 2022). This work presents initial successes on fine-tuning under DP with low-rank methods such as LoRA. Our experiments on language generation (see Section 5.2 and Table 2) demonstrate similar findings. Yet, we moreover show that full finetuning with good hyperparameters attains similar performance and possesses similar model scaling properties, which was raise by Anonymous (2022) as interesting open questions to pursue. Lastly, our private fine-tuning results for sentence classification may be far from optimal, since we used hyperparameters mostly transferred from tuning on the E2E language generation task.
DP Synthetic Data Generation. Fine-tuning generative language models on private data under DP can also be viewed as a means of accomplishing DP synthetic data generation – learning generative models from private data so that synthetic examples could be sampled and used for analysis. Previous work employed generative adversarial networks and focused primarily on image or tabular datasets (Torkzadehmahani et al., 2019; Neunhoeffer et al., 2020; Chen et al., 2020; Torfi et al., 2020). Perhaps more related is the work by Bommasani et al. (2019) which attempted fine-tuning GPT-2 on medical datasets to generate synthetic records but did not report any quantitative results.

heading: Q ESTIMATES OF RUN-TIME IN PRACTICE
text: The actual run-time of algorithms depends on implementation details. Here, we outline estimates of the run-time for full fine-tuning with DP-Adam on tasks considered in the paper. These numbers are based on running with a single RTX 3090 with PyTorch==1.9.0. Fine-tuning GPT-2 on E2E and DART takes less than 10 minutes per epoch, and fine-tuning for 10 epochs results in reasonably performing models. The time to fine-tune RoBERTa-base on classification tasks depends on the size of the dataset. It takes less than 10 minutes per epoch on the smallest SST-2, whereas for the largest MNLI, it takes less than an hour per epoch.
R EFFECTS OF VARYING AND δ ON DP FULL FINE-TUNING PERFORMANCE
Figure 9 shows how different values of and δ affect the performance of full fine-tuning under DP.

heading: S DP-ADAM VS DP-SGD
text: Our work focused on DP-Adam as opposed to DP-SGD, since Adam is more commonly used for non-private language model fine-tuning. While the two algorithms differ in their parameter update rule, the basic gradient privatization procedure is the same. We performed additional experiments fine-tuning GPT-2 on the E2E dataset with DP-SGD (with freshly tuned learning rate and clipping norm values) and observed that its performance (test set BLEU 63.175 at ( , δ) = (8, 10−5)) is on par with DP-Adam (test set BLEU 63.189 at ( , δ) = (8, 10−5)).

heading: T SUBTLETIES OF IMPLEMENTING DP MIXED PRECISION TRAINING
text: Mixed precision training (Micikevicius et al., 2017) accelerates updates by storing certain tensors in half-precision. To mitigate negative effects caused by potential arithmetic underflow, usual implementations upscale the loss with an adaptive factor pre-backpropagation and downscale the gradients with the same factor post-backpropagation. The scaling factor is adapted based on whether underflow is observed during training.
Special care needs to be taken when combining gradient privatization with mixed precision training. One implementation that ensures similar results across full and mixed precision training (1) upscales the loss with the adaptive factor K, (2) clips per-example gradients by CK, (3) adds to the sum of clipped gradients the usual Gaussian noise multiplied by K, and (4) downscales the noisy gradient by K. This is the implementation that we adopt, and we were able to obtain similar results on the E2E dataset with and without mixed precision.
One alternative implementation (a) upscales the loss with the adaptive factor K, (b) clips per-example gradients by C, (c) adds the usual Gaussian noise to the sum of clipped gradients, and (d) downscales noisy gradients by K. The main difference between this procedure and the prior is whether the factor
K is considered during clipping and noising. This procedure, while having the same DP guarantee, typically does not result in similar results as full precision when the same hyperparameters are used across the two settings (even with an optimizer like Adam which self-adjusts the magnitude of updates with accumulated empirical second moments). We also identified that this implementation is also the primary reason that a prior work’s code does not reproduce good results when run in full precision.18

heading: U FINE-TUNING WITH RGP AND THE TEXT-INFILLING OBJECTIVE
text: Recall Section 3.2 and Table 1 showed that private full fine-tuning with a text-infilling objective leads to improved classification results. Here, we show that the infilling objective is also helpful when one privately fine-tunes with the RGP method for a classification task. To study this, we reimplemented the RGP method and tuned the hyperparameters in full precision. Fixing all hyperparameters, we compared models trained with and without infilling. Table 14 confirms that fine-tuning with RGP based on infilling is generally helpful across the considered tasks. Note the aim of this experiment is not obtain state-of-the-art performance, but rather to study the effect of using the text-infilling objective. Thus, we expect these results could generally be further improved with more extensive hyperparameter search.

heading: V WHICH METHOD SHALL I USE FOR PRIVATE FINE-TUNING?
text: While we were able to obtain similar results on the E2E and DART datasets with full fine-tuning and LoRA (Tables 1 and 8), we encountered significant difficulties in fine-tuning for dialog generation (even non-privately) when not updating the embedding layer and language modeling head – perplexity was much worse, and generations frequently contained seemingly arbitrary characters. This result suggests that while full fine-tuning may be more computationally intensive than lightweight approaches at times, its simplicity (involving few design decisions) makes it an attractive first option when compute resource is sufficient.
18https://github.com/dayu11/Differentially-Private-Deep-Learning

Review summaries:
In this paper, the authors propose a method that applies DP-SGD to NLP tasks. DP-SGD protects the privacy of the model training against that the individual information about the training samples is detected or inferred. The method is applied to the fine-tuning phase of the pre-trained language models (e.g. bert, gpt), thus it achieves good performances for many applications. To adapt DP-SGD to NLP models, this paper proposes ghost clipping that allows clipping in DP-SGD to run without instantiating per-example gradients for any layer in the model.

This paper adapts the widely used DP learning algorithm, DP-SGD, to language models. It achieves to fine-tune the dataset while protecting the private information in the dataset. In this paper, the authors conduct some empirical studies on language models and find some useful conclusions (e.g. fine-tuning on a part of parameters with DP is enough). The authors verify the model on sentence classification, table-to-text generation, and dialog generation tasks, using various pre-trained language models (e.g. GPT, Bert).

The paper propose a faster algorithm to learn approximate differentially private NLP models. Pretrained NLP models are often very large. Practical procedure involves fine-tuning NLP model on private data, which may leak private information. To avoid leakage, DP-SGD (and DP-AGAGRAD, DP-ADAM) uses norm clipping on each sample’s gradient, and then add isotropic noise to aggregated gradients for samples in a batch. Then the normal update steps of SGD, ADAGRAD, or ADAM are performed. This will ensure privacy under differential privacy definition. However, this procedure requires computing per-sample gradient and keep them in the memory so that the norm of the gradient can be calculated, and the per-sample gradient clipping can be performed. This will introduce memory overhead proportional to the batch_size$\times$#params, which is impossible for very large NLP models.   This paper propose GhostClipping method to save the memory, without the need of per-sample gradient instantiated. The idea is compute the partial sum of gradient element-wise square using two small matrices of the size of $T\times T$, where $T$ is sequence length (<=1024 in practice), and aggregate them to obtain the per-sample norm (just a scale for each). And then it performs a second back-propagation to compute aggregation on the clipped gradient. It will uses almost the same memory as standard SGD (or ADAM), with one forward pass and two backward passes.   This paper also introduce two additional techniques to improve, one is choose a larger batch size, and the other is to introduce multi-task finetuning (fine-tuning includes masked prediction task on the target dataset). These two are important in boost the performance of the final models.   The paper evaluates on text classification, data-to-text generation, and dialog generation tasks. The results shows it gains prior DP methods.     

This paper investigated the problem of privately fine-tuning large language models for downstream NLP tasks, including sentence classification and language generation. The authors showed that by appropriately selecting hyper-parameters (including batch size, learning rate, training epochs, and clipping norm) and making the fine-tuning task aligned with pretraining tasks, directly fine-tuning large language models with DP-SGD yields strong performance, and provided an empirical guideline for setting a good training configuration. The authors also proposed ghost clipping trick for further memory saving when fine-tuning large language models. Finally the authors showed through experiments that low dimensional updates do not necessarily lead to better performance.