Differentially Private Fine-tuning of Language Models

1 INTRODUCTION . Deep learning models are well known to leak sensitive information about the dataset when trained using conventional methods ( Shokri et al. , 2017 ; Carlini et al. , 2019 ; 2021 ) . To combat this issue , models can instead be trained to guarantee differential privacy ( DP ) ( Dwork et al. , 2006b ) , a strong notion of data privacy which limits the influence of any individual training point on the final model . While DP is one of the few approaches capable of providing machine learning models with rigorous privacy guarantees , it generally comes at a cost in terms of test accuracy . One oft-cited explanation is that the constraint of DP necessitates much more training data ( Tramèr & Boneh , 2021 ; Feldman , 2020 ; Brown et al. , 2021 ) . Unfortunately , more training data may be hard to acquire , particularly in settings where privacy is a concern . Parallel to these developments , Transformer-based ( Vaswani et al. , 2017 ) large language models ( LLMs ) , including the BERT ( Devlin et al. , 2019 ; Liu et al. , 2019 ) and GPT ( Radford et al. , 2018 ; 2019 ; Brown et al. , 2020 ) families , have enabled significant progress in natural language processing , achieving state-of-the-art accuracy in almost every task considered . These models are first pre-trained on an extremely large and diverse public dataset . The weights are then fine-tuned for each task of interest using a much smaller task-specific dataset . For example , a single pre-trained GPT-family model may be fine-tuned for various downstream tasks , such as email reply suggestion , sentence completion in text editors , language translation , and more . This two-stage paradigm can naturally be adapted to solve tasks in private learning , automatically addressing the aforementioned data shortage issue via the massive scale of the public pre-training dataset . One may pre-train the model on public data as usual,1 but then fine-tune the model privately . Despite the success of these models , task-specific fine-tuning introduces a number of technical challenges . In the non-private setting , the immense size of LLMs makes it impractical to fine-tune the 1Despite the fact that the pre-training data is public , there may nonetheless be privacy concerns related to personal or copyrighted data . However , since these pre-trained models have already been released , any associated privacy loss has already been incurred . full model and store a separate copy of the parameters for hundreds of downstream tasks . Things only get worse with privacy , which leads to overheads in terms of running time , memory usage , and most importantly , accuracy . The magnitude of noise introduced to a model due to DP grows as the model size increases ( Bassily et al. , 2014 ; Abadi et al. , 2016 ; Bun et al. , 2014 ) , which can overwhelm any signal for larger models . A recent line of work in the non-private literature has proposed parameter-efficient methods to alleviate the issues of storage and computational cost for fine-tuning ( Houlsby et al. , 2019 ; Li & Liang , 2021 ; Aghajanyan et al. , 2020 ; Hu et al. , 2021 ; Mahabadi et al. , 2021 ) . The main focus of our work is to explore parameter-efficiency in the context of private learning . 1.1 OUR CONTRIBUTIONS . Our primary contribution is to show that advanced parameter-efficient methods can lead to simpler and significantly improved algorithms for private fine-tuning . Our framework is illustrated in Figure 1 . Our findings and contributions are summarized as follows : State-of-the-art utility and privacy . Empirical evaluation of our algorithms reveals that they achieve state-of-the-art accuracy versus privacy tradeoffs , improving upon the previous best ( Yu et al. , 2021b ) . More importantly , for many fine-tuning tasks , the utility of models trained with DP approaches that of non-private models . For example , privately fine-tuning RoBERTa-Large on the MNLI data set ( Williams et al. , 2018 ) , we achieve an accuracy of 87.8 % with a privacy budget of ( ε = 6.7 , δ = 1e-6 ) . Without privacy guarantees , RoBERTa-Large achieves an accuracy of 90.2 % ( GPT-3 is known to achieve 91.7 % ( Hu et al. , 2021 ) ) ; see Table 1 for a summary . We also explore private natural language generation tasks , fine-tuning GPT-2 models on the E2E dataset ( Novikova et al. , 2017 ) . Again , the utility approaches non-private levels : we achieve a ROUGE-L score of 0.6755 with GPT-2-Large and ( ε = 5.4 , δ = 1e-5 ) , compared to 0.72 without privacy . Larger models are better . Prior work has consistently shown that larger language models achieve better accuracy for downstream tasks . Our results give evidence that this phenomenon extends to the private setting . For example , on the MNLI dataset , RoBERTa-Base achieves an accuracy of 83.5 % whereas RoBERTa-Large achieves an accuracy of 87.8 % , both under a privacy budget of ( ε = 6.7 , δ = 1e-6 ) . Similarly , privately fine-tuning with E2E , GPT-2-Small , GPT-2-Medium , and GPT-2-Large achieve ROUGE-L scores of 0.6219 , 0.6645 and 0.6755 respectively , all under a privacy budget of ( ε = 5.4 , δ = 1e-5 ) . While established in the non-private setting , we find this phenomenon quite surprising under DP . There is often a tension when choosing private model architectures : larger models may have higher capacity , but necessitate the introduction of more noise . Consequently , smaller and simpler private models achieve the better accuracy in several settings ( Papernot et al. , 2019 ; Tramèr & Boneh , 2021 ) . In contrast , our experiments show that fine-tuning the biggest models achieves the best accuracy.2 Simpler , sparser , and faster . Beyond accuracy concerns , DP requirements also lead to significant overheads in terms of computation and memory usage . The large number of parameters contributes to the high cost of training LLMs , and things get worse under privacy , which has been documented to increase training time by up to two orders of magnitude ( Carlini et al. , 2019 ; Subramani et al. , 2021 ) . The parameter-efficient approaches we employ partially offset this issue : as we only update a small fraction of the total number of parameters , training becomes considerably more computationally and memory efficient . Furthermore , as in the non-private setting , this framework leads to a modular design , where a single large pre-trained model can be augmented with lightweight modifications for each individual downstream task . To the best of our knowledge , we are the first to fine-tune GPT-2-Large using differential privacy , the largest model trained thus far using DP . Given our state-of-the-art results for a variety of standard NLP tasks using advanced fine-tuning techniques , we believe that our paper will serve as a benchmark for further work in this direction . For example , the best average accuracy achieved by the prior work of Yu et al . ( 2021b ) on four standard NLP tasks in Table 1 is 83.9 % using ε = 8 ( and the same δ as in Table 1 ) , whereas we can achieve an average accuracy of 90.3 % using ε = 6.7 by a combination of better algorithms , larger models , and new privacy accounting techniques . Finally , though recently considered elsewhere ( see Section 5 ) , we put further focus on the framing of public pre-training and private fine-tuning as an important conceptual direction in DP deep learning . 2 PRELIMINARIES AND PRIOR ALGORITHM BASELINES . Recall the formal definition of differential privacy . Definition 2.1 ( Differential Privacy ( DP ) ( Dwork et al. , 2006b ; a ) ) . A randomized algorithm A is ( ε , δ ) -differentially private if for any two neighboring datasets D and D′ , which differ in exactly the data pertaining to a single user , and for all sets S of possible outputs : Pr [ A ( D ) ∈ S ] ≤ eε Pr [ A ( D′ ) ∈ S ] + δ . We review prior techniques for private fine-tuning . 2.1 FULL FINE-TUNING VIA DPSGD . To train a machine learning model with privacy , the most popular algorithm is the celebrated DP stochastic gradient descent ( DPSGD ) ( Song et al. , 2013 ; Bassily et al. , 2014 ; Abadi et al. , 2016 ) . This optimization method serves as a drop-in replacement for SGD , augmenting it with the addition of per-example gradient clipping and Gaussian noise addition steps . These two steps serve to limit and mask the contribution of a single example . Two key points to note are that a ) per-example gradient clipping incurs significant computational and memory overheads in most implementations , and b ) noise introduced due to privacy grows as the square-root as the number of model parameters . With this tool in place , the most basic fine-tuning strategy is to train all parameters using DPSGD . 2An alternative perspective is that what we currently think of as “ large ” language models are relatively small , and we are yet to reach the point where the benefits of model size on accuracy are outweighed by the drawbacks . 2.2 REPARAMETRIZED GRADIENT PERTURBATION . To mitigate the limitations of DPSGD , a recent work of Yu et al . ( 2021b ) introduced an elegant method called reparametrized gradient perturbation ( RGP ) . RGP exploits the implicit low-rank structure in the gradient updates of SGD to substantially improve upon DPSGD . Specifically , they reparametrize each layer ’ s weight matrix W into LR + W̃ , where L and R are low-rank gradientcarrier matrices and W̃ is the residual weight . The authors show that one can obtain a lowdimensional projection of W ’ s gradient by taking gradients only of the low-rank matrices L and R ( and not the high-rank W̃ ) . Privacy is introduced by clipping and noising these low-dimensional gradients of L andR . While this low-dimensional projection loses some of the signal inW ’ s gradient , it turns out to contain enough to still achieve high accuracy . At the same time , the low-dimensional gradients alleviate the aforementioned issues related to privatization , significantly reducing the memory consumption and noise introduced . 3 OUR APPROACH . 3.1 A META-FRAMEWORK . We introduce our approach as a meta-framework for private deep learning , which abstracts the key principles of recent fine-tuning methods . Suppose f ( WPT ; x ) is a pre-trained model where WPT are the pre-trained weights and x is any input . We create a new fine-tuning model fFT ( WPT , θ ; x ) ( 1 ) which incorporates additional trainable parameters θ , where dim ( θ ) dim ( WPT ) . That is , the number of new parameters in θ is a small fraction of the original number of parameters in the pretrained weights WPT . Fine-tuning is done by running DPSGD on the additional parameters θ , while freezing the weights of pre-trained model WPT . The new parameters are initialized to θ0 such that fFT ( WPT , θ0 ; x ) = f ( WPT ; x ) . ( 2 ) The initialization condition ( 2 ) is very important , as it ensures that fine-tuning starts at the pre-trained model and improves it by modifying the parameters θ . Most fine-tuning methods are additive and have the following special form : fFT ( WPT , θ ; x ) = f ( WPT + π ( θ ) ; x ) , ( 3 ) i.e. , they modify the pre-trained weights by adding a correction term π ( θ ) parametrized by θ . Recent work in the non-private literature has described concrete instantiations of this framework ( Houlsby et al. , 2019 ; Mahabadi et al. , 2021 ; Hu et al. , 2021 ) , which ( crucially ) are effective when dim ( θ ) dim ( WPT ) . In the non-private setting , such reparametrizations are useful for reducing the computation and memory required for fine-tuning , and enable lightweight and plug-in modifications to the base model for different downstream tasks . At the same time , they maintain ( or sometimes surpass ) the accuracy achieved by full fine-tuning . However , this approach seems to be more important for achieving high accuracy in the context of private fine-tuning . For instance , full fine-tuning of RoBERTa-Base with DPSGD achieves an accuracy of 53.1 % for the MNLI dataset ( Yu et al. , 2021b ) , whereas all the algorithms that fine-tune based on Equation ( 1 ) all achieve accuracy better than 80 % . We give some intuition as to why parameter-efficient methods seem to be more effective for private fine-tuning . For simplicity , we assume that the fine-tuning method is additive as in ( 3 ) , such that the fine-tuned weights WFT = WPT + π ( θ ) . We can imagine that WFT lies on a manifold passing through WPT of very small dimension ( equal to the dimension of θ ) compared to the dimension of WPT . Even if the parameters θ are very noisy due to the noise added during DPSGD , we will always stay in this manifold . In particular , we are not disturbing the pre-trained weights in most directions ( those orthogonal to the manifold near WPT ) . If we run DPSGD on all the weights instead , then we add noise in all directions , thus potentially unlearning the knowledge learned during pre-training , especially in low data regimes . Besides substantial gains in the accuracy , the above method of reparametrization has several other advantages : • A single pre-trained model such as BERT or GPT is generally applied to hundreds of downstream tasks via fine-tuning . Private fine-tuning using previous methods requires updating all parameters and storing a different copy of the fine-tuned model per task . This creates substantial overheads for storing and deploying , and can be very expensive in practice . On the other hand , the reparametrization ( 1 ) means that we only need to store a single pretrained model that can be shared across many downstream tasks . Each downstream task requires only a small number of new parameters that can be plugged in . • Differentially private training requires computing and storing per-example gradients , which increases the memory footprint . In our approach , however , learning is done in a much lower dimension , hence saving on the memory cost as compared to prior works . • Finally , we expect that ( 1 ) also gives a more communication-efficient method of fine-tuning in distributed settings such as federated learning , due to the significantly smaller number of parameters learned during fine-tuning .

Summary: This paper introduces a parameter-efficient framework for privately fine-tuning large language models. The techniques were proposed in prior literature for non-private learning while the authors found that these techniques are very effective for training with differential privacy (DP), which was extremely hard for large models due to the amount of noise added. On common NLP benchmarks and with relatively small privacy budget, these parameter-efficient methods outperformed the regular DPSGD and could even be close to the non-private baseline, in terms of utility. These methods are also much more memory-efficient and faster than DPSGD as well.