Large Language Models Can Be Strong Differentially Private Learners

1 INTRODUCTION . Machine learning systems trained on sensitive user data can be vulnerable to privacy attacks ( Shokri et al. , 2017 ; Hayes et al. , 2019 ) . This issue is especially pressing for recent applications involving large deep learning models of text ( Brown et al. , 2020 ) , as training data for these systems often contain private user data ( Carlini et al. , 2020 ) , and these models are able to memorize and regurgitate sensitive training examples ( Zhang et al. , 2016 ; Carlini et al. , 2020 ) . As a result of these concerns , there has been a large interest in developing methods that provide data privacy guarantees for discriminative and generative models of text . The gold-standard way for providing such a guarantee in machine learning is Differential Privacy ( DP ) ( Dwork et al. , 2006 ; 2014 ) . Unfortunately , DP learning has typically struggled to produce useful models when applied to large language models , resulting in models with either vacuous privacy guarantees ( Dupuy et al. , 2021 ) or performance far below non-private baselines . This is widely attributed to the fact that the core primitive of Differentially Private Stochastic Gradient Descent ( DP-SGD ) ( Song et al. , 2013 ; Bassily et al. , 2014 ; Abadi et al. , 2016 ) injects noise that must scale with the number of parameters , resulting in large noise levels for large language models ( Yu et al. , 2021a ) . We tackle the problem of building performant DP language models for sentence classification and language generation tasks with tens to hundreds of thousands of examples . We pursue this goal by re-examining the performance of the baseline DP optimization algorithm for fine-tuning large language models , and study how choices of hyperparameters , training objective , and pretrained models affect the performance given fixed privacy budgets . In contrast to the mainstream perception , our empirical results demonstrate that large pretrained models with hundreds of millions of parameters can be effectively and efficiently fine-tuned to yield models with high performance with modest privacy leakage . For language generation , the performance of our models surpasses strong non-private baselines . For sentence classification , the performance of our fine-tuned models surpasses those obtained under heuristic privacy notions ( Huang et al. , 2020a ) which do not possess formal guarantees . Figure 1 illustrates these results . We summarize our contributions below . ( 1 ) We show that with appropriate hyperparameters and downstream task objectives , directly finetuning pretrained language models with DP-SGD/DP-Adam yields strong performance for a suite of NLP tasks at privacy levels ∈ { 3 , 8 } . Notably , some of our fine-tuned models outperform strong non-private learning baselines and models obtained under heuristic privacy notions . ( 2 ) DP-SGD is known for its expensive memory cost due to clipping per-example gradients . We present ghost clipping , a memory saving technique that makes fine-tuning larger Transformers under DP memory efficient . Our technique generalizes the Goodfellow ( 2015 ) trick to handle sequential inputs , and can be combined with a layer-by-layer clipping procedure ( Lee & Kifer , 2020 ) to enable privately fitting large Transformers with almost the same memory cost as non-private training—at the cost of one additional backward pass per processed batch . ( 3 ) We show that the dimensionality of gradient updates does not explain private fine-tuning performance . While there exist dimension-dependent lower bounds for private ( convex ) optimization ( Bassily et al. , 2014 ) , we find that larger pretrained models lead to better private fine-tuning results . Moreover , parameter-efficient adaptation methods that reduce the dimensionality of updates do not necessarily outperform a baseline method that fine-tunes all model parameters . Our empirical studies indicate that directly fine-tuning pretrained models with DP optimization results in performant DP language models under modest privacy budgets . This enables building practical DP NLP models for a range of common tasks where privacy could be at stake . 2 PROBLEM STATEMENT . We build models for sentence classification and language generation tasks with private datasets of modest sizes under ( central/global ) approximate-DP ( also known as ( , δ ) -DP ) ( Dwork et al. , 2014 ) . Definition 1 ( ( , δ ) -DP ) . A randomized algorithmM : X → Y is ( , δ ) -differentially private if for all adjacent datasets X , X ′ ∈ X and all Y ⊂ Y , P ( M ( X ) ∈ Y ) ≤ e P ( M ( X ′ ) ∈ Y ) + δ . Two datasets are adjacent if and only if one can be obtained from the other by including an extra record ( Mironov et al. , 2019 ) . What counts as a record is task dependent and will be defined below . Intuitively , algorithms satisfying DP ensures that output random variables obtained from similar inputs are also similar . and δ are privacy leakage parameters that measure the loss of privacy and small values imply more privacy . Unlike heuristic privacy notions ( Huang et al. , 2020b ) , DP allows for the tracking of privacy loss through the calculation of leakage parameters , and ensures privacy under composition ( Dwork et al. , 2014 ) , meaning that the overall privacy loss of multiple DP algorithms releasing multiple statistics can be reasoned in a principled manner . DP models are typically trained with DP optimizers which privatize the gradient before taking a standard gradient descent step . The privatization step ensures that the parameter updates leak limited information about the training examples through their gradients . Specifically , this step clips perexample gradients with a norm constraint C , and adds isotropic Gaussian noise z ∼ N ( 0 , C2σ2Ip ) to the sum of clipped gradients . Here , σ is the noise multiplier determined from the privacy budget ( , δ ) , number of gradient updates S , and sampling rate q = BN for a batch size of B and a dataset with N examples . Intuitively , clipping individual gradients ensures that each example has bounded influence on the parameter update , whereas noising the gradient denies exact tracing of particular examples . The noise being isotropic implies that larger models would experience heavier noise per update , as the norm of the high-dimensional Gaussian ‖z‖2 tends to Cσ √ p for large dimensionality p with high probability . This is widely believed to be the cause for DP optimization performing poorly at training high-dimensional deep learning models ( Kamath , 2020 ) . Our starting point for building DP language models is ( public ) pretrained models . Pretrained language models tend to contain general knowledge of language ( Manning et al. , 2020 ) and thus would make the downstream private learning problem easier . We fine-tune these models with DP-Adam ( Abadi et al. , 2016 ; Kingma & Ba , 2014 ) ( see Appendix A for details ) and track privacy loss through Rényi DP ( Mironov , 2017 ) , but also report the converted from Gaussian DP and its central limit theorem ( Dong et al. , 2019 ) and from accurately composing tradeoff functions via fast Fourier transform ( Gopi et al. , 2021 ) . We consider privacy levels ∈ { 3 , 8 } and δ = 1/2|Dtrain| throughout for a training set of size |Dtrain| ( See Appendix B for further details ) . We tune hyperparameters on the E2E tasks and transfer these to all remaining tasks . We outline two broad classes of NLP problems considered in this paper and define what constitutes a record below . Sentence Classification . The goal is to learn a model that correctly classifies sentences . For these tasks , each example/record consists of some input sentences and a label to be predicted . We fine-tune models of various sizes in the BERT ( Devlin et al. , 2018 ) and RoBERTa ( Liu et al. , 2019 ) families , as these models are known to work well for sentence classification . Language Generation . The goal is to learn a model that generates natural language sentences given some context . For table-to-text generation tasks such as E2E ( Novikova et al. , 2017 ) and DART ( Nan et al. , 2020 ) , each example/record in the training data consists of a pair of table entry and corresponding text description to be predicted . For a dialog generation task such as PersonaChat ( Zhang et al. , 2018 ) , each example/record consists of some auxiliary information , a dialog history , and a response to be predicted . We fine-tune GPT-2 ( Radford et al. , 2019 ) of various sizes for these problems , as this model family is known to work well for text generation . 3 EFFECTIVE DIFFERENTIALLY PRIVATE FINE-TUNING . By studying the impact of hyperparameters and choice of fine-tuning objective , we demonstrate that the performance of the DP-Adam baseline can be substantially improved , even matching some strong non-private learning results . Our analyses reveal common failure modes when straightforwardly applying DP optimization and explain poor results reported in past works that consider these baselines . 3.1 HYPERPARAMETER TUNING DP optimization is sensitive to the choice of hyperparameters ( Papernot et al. , 2019 ) . Our experiments suggest that performance can vary from being close to trivial with ill-chosen hyperparameters to near previous state-of-the-art with appropriately chosen ones . As a consequence , we present simple but effective guidelines on setting the most important hyperparameters . Unless otherwise stated , the unmentioned hyperparameters are set to defaults documented in Appendix H . 3.1.1 BATCH SIZE , LEARNING RATE & TRAINING EPOCHS . Our experiments suggest that batch size is one of the most important hyperparameters to set correctly , and the dependence of the optimal batch size on learning rate and training epochs makes its selection complex . We first describe batch size selection in realistic , compute-bound settings and then describe how the complexity of identifying the optimal batch size in these situations arise due to constraints on the number of training epochs . Fixed Training Epochs E. We first describe a realistic situation in which there is a constraint on the total amount of compute budget available . For the case of DP-SGD , this compute budget constraint often corresponds to a constraint on the number of examples that are processed by SGD.1In this fixed training epoch setting , the learning rate and batch size jointly affect performance , since using larger batches implies performing fewer gradient updates . To study this joint influence empirically , we fine-tune GPT-2 on the E2E dataset for table-to-text generation with DP-Adam at = 3 with various batch sizes and learning rates . Figure 2 shows that the best performing models are obtained with both a large batch size and large learning rate . Using a small learning rate together with a small batch size yields considerably worse results . Note a seq2seq baseline achieves a test BLEU of ~65 without privacy here ( Wiseman et al. , 2018 ) . Recall that in the non-private world , pretrained language models are typically fine-tuned with small batch sizes and small learning rates with Adam ( bottom left panel in Figure 2 ) . This implies that naı̈ely fine-tuning pretrained language models privately using the non-private setup would result in more performance degradation than necessary . Recently , Tramèr & Boneh ( 2020 ) studied how the batch size and learning rate jointly affect learning private image classifiers while holding other hyperparameters fixed . They heuristically suggested a linear scaling rule : Scaling the learning rate together with the batch size by the same constant should yield models with almost the same performance . However , Figure 2 indicates that this fails to hold consistently as it falsely predicts that large batch and high learning rate ( top right most entry ) would have equal performance to small batch and low learning rate ( bottom left entry ) . We further explain why linear scaling fails for small batches in Appendix D. Fixed Update Steps S. In the fixed epoch setting , we saw that the optimal batch size setting was complex due to the trade-off between batch size and number of gradient updates . We now show that the complexity of setting batch sizes arises almost entirely from this tradeoff by considering a different setting , where the total number of gradient updates ( rather than epochs ) is fixed . In this case , using larger batches implies training for more epochs , and we find that using larger batch sizes almost always results in better performance at a given privacy budget ( at the cost of processing more examples and using more compute ) , once other hyperparameters are fixed . We provide an explanation of this by introducing the idea of an effective noise multiplier σeff = σq = σN B . Recall the noise multiplier σ is determined from the privacy budget ( , δ ) , update steps S , and sampling rate q . In addition , recall the form of the privatized gradient ḡ in DP-SGD/DP-Adam : ḡ = g̃ + z , g̃ = 1B ∑B i=1Clip ( ∇Li , C ) , z ∼ N ( 0 , C2 σ 2 B2 Ip ) = N ( 0 , C2 σ2eff N2 Ip ) , where∇Li is the gradient of the ith example in a batch ofB examples and Clip ( v , K ) clips the vector v by the norm constraint K. We observe that for moderately large batches , the signal-to-noise ratio r = ‖g̃‖2/‖z‖2 is mainly controlled by the batch size through the effective noise multiplier : The signal term g̃ tends to concentrate quickly due to being an average of bounded vectors , whereas the effective noise multiplier σeff could vary significantly as the batch size B changes . Figure 3 ( a ) shows that the effective noise multiplier decreases as batch size increases . In addition , Figure 3 ( b ) plots the average signal-to-noise ratio r over the first 30 gradient updates against the final model ’ s performance on E2E and demonstrates that large batches ( up to a threshold ) lead to both increased signal-to-noise ratio at the beginning of training and better performance at the end of training . These findings additionally resonate with and explains recent empirical successes of private pretraining ( Anil et al. , 2021 ) .

Summary: The paper propose a faster algorithm to learn approximate differentially private NLP models. Pretrained NLP models are often very large. Practical procedure involves fine-tuning NLP model on private data, which may leak private information. To avoid leakage, DP-SGD (and DP-AGAGRAD, DP-ADAM) uses norm clipping on each sample’s gradient, and then add isotropic noise to aggregated gradients for samples in a batch. Then the normal update steps of SGD, ADAGRAD, or ADAM are performed. This will ensure privacy under differential privacy definition. However, this procedure requires computing per-sample gradient and keep them in the memory so that the norm of the gradient can be calculated, and the per-sample gradient clipping can be performed. This will introduce memory overhead proportional to the batch_size$\times$#params, which is impossible for very large NLP models.   This paper propose GhostClipping method to save the memory, without the need of per-sample gradient instantiated. The idea is compute the partial sum of gradient element-wise square using two small matrices of the size of $T\times T$, where $T$ is sequence length (<=1024 in practice), and aggregate them to obtain the per-sample norm (just a scale for each). And then it performs a second back-propagation to compute aggregation on the clipped gradient. It will uses almost the same memory as standard SGD (or ADAM), with one forward pass and two backward passes.   This paper also introduce two additional techniques to improve, one is choose a larger batch size, and the other is to introduce multi-task finetuning (fine-tuning includes masked prediction task on the target dataset). These two are important in boost the performance of the final models.   The paper evaluates on text classification, data-to-text generation, and dialog generation tasks. The results shows it gains prior DP methods.