Large Language Models Can Be Strong Differentially Private Learners

1 INTRODUCTION . Machine learning systems trained on sensitive user data can be vulnerable to privacy attacks ( Shokri et al. , 2017 ; Hayes et al. , 2019 ) . This issue is especially pressing for recent applications involving large deep learning models of text ( Brown et al. , 2020 ) , as training data for these systems often contain private user data ( Carlini et al. , 2020 ) , and these models are able to memorize and regurgitate sensitive training examples ( Zhang et al. , 2016 ; Carlini et al. , 2020 ) . As a result of these concerns , there has been a large interest in developing methods that provide data privacy guarantees for discriminative and generative models of text . The gold-standard way for providing such a guarantee in machine learning is Differential Privacy ( DP ) ( Dwork et al. , 2006 ; 2014 ) . Unfortunately , DP learning has typically struggled to produce useful models when applied to large language models , resulting in models with either vacuous privacy guarantees ( Dupuy et al. , 2021 ) or performance far below non-private baselines . This is widely attributed to the fact that the core primitive of Differentially Private Stochastic Gradient Descent ( DP-SGD ) ( Song et al. , 2013 ; Bassily et al. , 2014 ; Abadi et al. , 2016 ) injects noise that must scale with the number of parameters , resulting in large noise levels for large language models ( Yu et al. , 2021a ) . We tackle the problem of building performant DP language models for sentence classification and language generation tasks with tens to hundreds of thousands of examples . We pursue this goal by re-examining the performance of the baseline DP optimization algorithm for fine-tuning large language models , and study how choices of hyperparameters , training objective , and pretrained models affect the performance given fixed privacy budgets . In contrast to the mainstream perception , our empirical results demonstrate that large pretrained models with hundreds of millions of parameters can be effectively and efficiently fine-tuned to yield models with high performance with modest privacy leakage . For language generation , the performance of our models surpasses strong non-private baselines . For sentence classification , the performance of our fine-tuned models surpasses those obtained under heuristic privacy notions ( Huang et al. , 2020a ) which do not possess formal guarantees . Figure 1 illustrates these results . We summarize our contributions below . ( 1 ) We show that with appropriate hyperparameters and downstream task objectives , directly finetuning pretrained language models with DP-SGD/DP-Adam yields strong performance for a suite of NLP tasks at privacy levels ∈ { 3 , 8 } . Notably , some of our fine-tuned models outperform strong non-private learning baselines and models obtained under heuristic privacy notions . ( 2 ) DP-SGD is known for its expensive memory cost due to clipping per-example gradients . We present ghost clipping , a memory saving technique that makes fine-tuning larger Transformers under DP memory efficient . Our technique generalizes the Goodfellow ( 2015 ) trick to handle sequential inputs , and can be combined with a layer-by-layer clipping procedure ( Lee & Kifer , 2020 ) to enable privately fitting large Transformers with almost the same memory cost as non-private training—at the cost of one additional backward pass per processed batch . ( 3 ) We show that the dimensionality of gradient updates does not explain private fine-tuning performance . While there exist dimension-dependent lower bounds for private ( convex ) optimization ( Bassily et al. , 2014 ) , we find that larger pretrained models lead to better private fine-tuning results . Moreover , parameter-efficient adaptation methods that reduce the dimensionality of updates do not necessarily outperform a baseline method that fine-tunes all model parameters . Our empirical studies indicate that directly fine-tuning pretrained models with DP optimization results in performant DP language models under modest privacy budgets . This enables building practical DP NLP models for a range of common tasks where privacy could be at stake . 2 PROBLEM STATEMENT . We build models for sentence classification and language generation tasks with private datasets of modest sizes under ( central/global ) approximate-DP ( also known as ( , δ ) -DP ) ( Dwork et al. , 2014 ) . Definition 1 ( ( , δ ) -DP ) . A randomized algorithmM : X → Y is ( , δ ) -differentially private if for all adjacent datasets X , X ′ ∈ X and all Y ⊂ Y , P ( M ( X ) ∈ Y ) ≤ e P ( M ( X ′ ) ∈ Y ) + δ . Two datasets are adjacent if and only if one can be obtained from the other by including an extra record ( Mironov et al. , 2019 ) . What counts as a record is task dependent and will be defined below . Intuitively , algorithms satisfying DP ensures that output random variables obtained from similar inputs are also similar . and δ are privacy leakage parameters that measure the loss of privacy and small values imply more privacy . Unlike heuristic privacy notions ( Huang et al. , 2020b ) , DP allows for the tracking of privacy loss through the calculation of leakage parameters , and ensures privacy under composition ( Dwork et al. , 2014 ) , meaning that the overall privacy loss of multiple DP algorithms releasing multiple statistics can be reasoned in a principled manner . DP models are typically trained with DP optimizers which privatize the gradient before taking a standard gradient descent step . The privatization step ensures that the parameter updates leak limited information about the training examples through their gradients . Specifically , this step clips perexample gradients with a norm constraint C , and adds isotropic Gaussian noise z ∼ N ( 0 , C2σ2Ip ) to the sum of clipped gradients . Here , σ is the noise multiplier determined from the privacy budget ( , δ ) , number of gradient updates S , and sampling rate q = BN for a batch size of B and a dataset with N examples . Intuitively , clipping individual gradients ensures that each example has bounded influence on the parameter update , whereas noising the gradient denies exact tracing of particular examples . The noise being isotropic implies that larger models would experience heavier noise per update , as the norm of the high-dimensional Gaussian ‖z‖2 tends to Cσ √ p for large dimensionality p with high probability . This is widely believed to be the cause for DP optimization performing poorly at training high-dimensional deep learning models ( Kamath , 2020 ) . Our starting point for building DP language models is ( public ) pretrained models . Pretrained language models tend to contain general knowledge of language ( Manning et al. , 2020 ) and thus would make the downstream private learning problem easier . We fine-tune these models with DP-Adam ( Abadi et al. , 2016 ; Kingma & Ba , 2014 ) ( see Appendix A for details ) and track privacy loss through Rényi DP ( Mironov , 2017 ) , but also report the converted from Gaussian DP and its central limit theorem ( Dong et al. , 2019 ) and from accurately composing tradeoff functions via fast Fourier transform ( Gopi et al. , 2021 ) . We consider privacy levels ∈ { 3 , 8 } and δ = 1/2|Dtrain| throughout for a training set of size |Dtrain| ( See Appendix B for further details ) . We tune hyperparameters on the E2E tasks and transfer these to all remaining tasks . We outline two broad classes of NLP problems considered in this paper and define what constitutes a record below . Sentence Classification . The goal is to learn a model that correctly classifies sentences . For these tasks , each example/record consists of some input sentences and a label to be predicted . We fine-tune models of various sizes in the BERT ( Devlin et al. , 2018 ) and RoBERTa ( Liu et al. , 2019 ) families , as these models are known to work well for sentence classification . Language Generation . The goal is to learn a model that generates natural language sentences given some context . For table-to-text generation tasks such as E2E ( Novikova et al. , 2017 ) and DART ( Nan et al. , 2020 ) , each example/record in the training data consists of a pair of table entry and corresponding text description to be predicted . For a dialog generation task such as PersonaChat ( Zhang et al. , 2018 ) , each example/record consists of some auxiliary information , a dialog history , and a response to be predicted . We fine-tune GPT-2 ( Radford et al. , 2019 ) of various sizes for these problems , as this model family is known to work well for text generation . 3 EFFECTIVE DIFFERENTIALLY PRIVATE FINE-TUNING . By studying the impact of hyperparameters and choice of fine-tuning objective , we demonstrate that the performance of the DP-Adam baseline can be substantially improved , even matching some strong non-private learning results . Our analyses reveal common failure modes when straightforwardly applying DP optimization and explain poor results reported in past works that consider these baselines . 3.1 HYPERPARAMETER TUNING DP optimization is sensitive to the choice of hyperparameters ( Papernot et al. , 2019 ) . Our experiments suggest that performance can vary from being close to trivial with ill-chosen hyperparameters to near previous state-of-the-art with appropriately chosen ones . As a consequence , we present simple but effective guidelines on setting the most important hyperparameters . Unless otherwise stated , the unmentioned hyperparameters are set to defaults documented in Appendix H . 3.1.1 BATCH SIZE , LEARNING RATE & TRAINING EPOCHS . Our experiments suggest that batch size is one of the most important hyperparameters to set correctly , and the dependence of the optimal batch size on learning rate and training epochs makes its selection complex . We first describe batch size selection in realistic , compute-bound settings and then describe how the complexity of identifying the optimal batch size in these situations arise due to constraints on the number of training epochs . Fixed Training Epochs E. We first describe a realistic situation in which there is a constraint on the total amount of compute budget available . For the case of DP-SGD , this compute budget constraint often corresponds to a constraint on the number of examples that are processed by SGD.1In this fixed training epoch setting , the learning rate and batch size jointly affect performance , since using larger batches implies performing fewer gradient updates . To study this joint influence empirically , we fine-tune GPT-2 on the E2E dataset for table-to-text generation with DP-Adam at = 3 with various batch sizes and learning rates . Figure 2 shows that the best performing models are obtained with both a large batch size and large learning rate . Using a small learning rate together with a small batch size yields considerably worse results . Note a seq2seq baseline achieves a test BLEU of ~65 without privacy here ( Wiseman et al. , 2018 ) . Recall that in the non-private world , pretrained language models are typically fine-tuned with small batch sizes and small learning rates with Adam ( bottom left panel in Figure 2 ) . This implies that naı̈ely fine-tuning pretrained language models privately using the non-private setup would result in more performance degradation than necessary . Recently , Tramèr & Boneh ( 2020 ) studied how the batch size and learning rate jointly affect learning private image classifiers while holding other hyperparameters fixed . They heuristically suggested a linear scaling rule : Scaling the learning rate together with the batch size by the same constant should yield models with almost the same performance . However , Figure 2 indicates that this fails to hold consistently as it falsely predicts that large batch and high learning rate ( top right most entry ) would have equal performance to small batch and low learning rate ( bottom left entry ) . We further explain why linear scaling fails for small batches in Appendix D. Fixed Update Steps S. In the fixed epoch setting , we saw that the optimal batch size setting was complex due to the trade-off between batch size and number of gradient updates . We now show that the complexity of setting batch sizes arises almost entirely from this tradeoff by considering a different setting , where the total number of gradient updates ( rather than epochs ) is fixed . In this case , using larger batches implies training for more epochs , and we find that using larger batch sizes almost always results in better performance at a given privacy budget ( at the cost of processing more examples and using more compute ) , once other hyperparameters are fixed . We provide an explanation of this by introducing the idea of an effective noise multiplier σeff = σq = σN B . Recall the noise multiplier σ is determined from the privacy budget ( , δ ) , update steps S , and sampling rate q . In addition , recall the form of the privatized gradient ḡ in DP-SGD/DP-Adam : ḡ = g̃ + z , g̃ = 1B ∑B i=1Clip ( ∇Li , C ) , z ∼ N ( 0 , C2 σ 2 B2 Ip ) = N ( 0 , C2 σ2eff N2 Ip ) , where∇Li is the gradient of the ith example in a batch ofB examples and Clip ( v , K ) clips the vector v by the norm constraint K. We observe that for moderately large batches , the signal-to-noise ratio r = ‖g̃‖2/‖z‖2 is mainly controlled by the batch size through the effective noise multiplier : The signal term g̃ tends to concentrate quickly due to being an average of bounded vectors , whereas the effective noise multiplier σeff could vary significantly as the batch size B changes . Figure 3 ( a ) shows that the effective noise multiplier decreases as batch size increases . In addition , Figure 3 ( b ) plots the average signal-to-noise ratio r over the first 30 gradient updates against the final model ’ s performance on E2E and demonstrates that large batches ( up to a threshold ) lead to both increased signal-to-noise ratio at the beginning of training and better performance at the end of training . These findings additionally resonate with and explains recent empirical successes of private pretraining ( Anil et al. , 2021 ) .

Summary: In this paper, the authors propose a method that applies DP-SGD to NLP tasks. DP-SGD protects the privacy of the model training against that the individual information about the training samples is detected or inferred. The method is applied to the fine-tuning phase of the pre-trained language models (e.g. bert, gpt), thus it achieves good performances for many applications. To adapt DP-SGD to NLP models, this paper proposes ghost clipping that allows clipping in DP-SGD to run without instantiating per-example gradients for any layer in the model.