Differentially Private Fine-tuning of Language Models

1 INTRODUCTION . Deep learning models are well known to leak sensitive information about the dataset when trained using conventional methods ( Shokri et al. , 2017 ; Carlini et al. , 2019 ; 2021 ) . To combat this issue , models can instead be trained to guarantee differential privacy ( DP ) ( Dwork et al. , 2006b ) , a strong notion of data privacy which limits the influence of any individual training point on the final model . While DP is one of the few approaches capable of providing machine learning models with rigorous privacy guarantees , it generally comes at a cost in terms of test accuracy . One oft-cited explanation is that the constraint of DP necessitates much more training data ( Tramèr & Boneh , 2021 ; Feldman , 2020 ; Brown et al. , 2021 ) . Unfortunately , more training data may be hard to acquire , particularly in settings where privacy is a concern . Parallel to these developments , Transformer-based ( Vaswani et al. , 2017 ) large language models ( LLMs ) , including the BERT ( Devlin et al. , 2019 ; Liu et al. , 2019 ) and GPT ( Radford et al. , 2018 ; 2019 ; Brown et al. , 2020 ) families , have enabled significant progress in natural language processing , achieving state-of-the-art accuracy in almost every task considered . These models are first pre-trained on an extremely large and diverse public dataset . The weights are then fine-tuned for each task of interest using a much smaller task-specific dataset . For example , a single pre-trained GPT-family model may be fine-tuned for various downstream tasks , such as email reply suggestion , sentence completion in text editors , language translation , and more . This two-stage paradigm can naturally be adapted to solve tasks in private learning , automatically addressing the aforementioned data shortage issue via the massive scale of the public pre-training dataset . One may pre-train the model on public data as usual,1 but then fine-tune the model privately . Despite the success of these models , task-specific fine-tuning introduces a number of technical challenges . In the non-private setting , the immense size of LLMs makes it impractical to fine-tune the 1Despite the fact that the pre-training data is public , there may nonetheless be privacy concerns related to personal or copyrighted data . However , since these pre-trained models have already been released , any associated privacy loss has already been incurred . full model and store a separate copy of the parameters for hundreds of downstream tasks . Things only get worse with privacy , which leads to overheads in terms of running time , memory usage , and most importantly , accuracy . The magnitude of noise introduced to a model due to DP grows as the model size increases ( Bassily et al. , 2014 ; Abadi et al. , 2016 ; Bun et al. , 2014 ) , which can overwhelm any signal for larger models . A recent line of work in the non-private literature has proposed parameter-efficient methods to alleviate the issues of storage and computational cost for fine-tuning ( Houlsby et al. , 2019 ; Li & Liang , 2021 ; Aghajanyan et al. , 2020 ; Hu et al. , 2021 ; Mahabadi et al. , 2021 ) . The main focus of our work is to explore parameter-efficiency in the context of private learning . 1.1 OUR CONTRIBUTIONS . Our primary contribution is to show that advanced parameter-efficient methods can lead to simpler and significantly improved algorithms for private fine-tuning . Our framework is illustrated in Figure 1 . Our findings and contributions are summarized as follows : State-of-the-art utility and privacy . Empirical evaluation of our algorithms reveals that they achieve state-of-the-art accuracy versus privacy tradeoffs , improving upon the previous best ( Yu et al. , 2021b ) . More importantly , for many fine-tuning tasks , the utility of models trained with DP approaches that of non-private models . For example , privately fine-tuning RoBERTa-Large on the MNLI data set ( Williams et al. , 2018 ) , we achieve an accuracy of 87.8 % with a privacy budget of ( ε = 6.7 , δ = 1e-6 ) . Without privacy guarantees , RoBERTa-Large achieves an accuracy of 90.2 % ( GPT-3 is known to achieve 91.7 % ( Hu et al. , 2021 ) ) ; see Table 1 for a summary . We also explore private natural language generation tasks , fine-tuning GPT-2 models on the E2E dataset ( Novikova et al. , 2017 ) . Again , the utility approaches non-private levels : we achieve a ROUGE-L score of 0.6755 with GPT-2-Large and ( ε = 5.4 , δ = 1e-5 ) , compared to 0.72 without privacy . Larger models are better . Prior work has consistently shown that larger language models achieve better accuracy for downstream tasks . Our results give evidence that this phenomenon extends to the private setting . For example , on the MNLI dataset , RoBERTa-Base achieves an accuracy of 83.5 % whereas RoBERTa-Large achieves an accuracy of 87.8 % , both under a privacy budget of ( ε = 6.7 , δ = 1e-6 ) . Similarly , privately fine-tuning with E2E , GPT-2-Small , GPT-2-Medium , and GPT-2-Large achieve ROUGE-L scores of 0.6219 , 0.6645 and 0.6755 respectively , all under a privacy budget of ( ε = 5.4 , δ = 1e-5 ) . While established in the non-private setting , we find this phenomenon quite surprising under DP . There is often a tension when choosing private model architectures : larger models may have higher capacity , but necessitate the introduction of more noise . Consequently , smaller and simpler private models achieve the better accuracy in several settings ( Papernot et al. , 2019 ; Tramèr & Boneh , 2021 ) . In contrast , our experiments show that fine-tuning the biggest models achieves the best accuracy.2 Simpler , sparser , and faster . Beyond accuracy concerns , DP requirements also lead to significant overheads in terms of computation and memory usage . The large number of parameters contributes to the high cost of training LLMs , and things get worse under privacy , which has been documented to increase training time by up to two orders of magnitude ( Carlini et al. , 2019 ; Subramani et al. , 2021 ) . The parameter-efficient approaches we employ partially offset this issue : as we only update a small fraction of the total number of parameters , training becomes considerably more computationally and memory efficient . Furthermore , as in the non-private setting , this framework leads to a modular design , where a single large pre-trained model can be augmented with lightweight modifications for each individual downstream task . To the best of our knowledge , we are the first to fine-tune GPT-2-Large using differential privacy , the largest model trained thus far using DP . Given our state-of-the-art results for a variety of standard NLP tasks using advanced fine-tuning techniques , we believe that our paper will serve as a benchmark for further work in this direction . For example , the best average accuracy achieved by the prior work of Yu et al . ( 2021b ) on four standard NLP tasks in Table 1 is 83.9 % using ε = 8 ( and the same δ as in Table 1 ) , whereas we can achieve an average accuracy of 90.3 % using ε = 6.7 by a combination of better algorithms , larger models , and new privacy accounting techniques . Finally , though recently considered elsewhere ( see Section 5 ) , we put further focus on the framing of public pre-training and private fine-tuning as an important conceptual direction in DP deep learning . 2 PRELIMINARIES AND PRIOR ALGORITHM BASELINES . Recall the formal definition of differential privacy . Definition 2.1 ( Differential Privacy ( DP ) ( Dwork et al. , 2006b ; a ) ) . A randomized algorithm A is ( ε , δ ) -differentially private if for any two neighboring datasets D and D′ , which differ in exactly the data pertaining to a single user , and for all sets S of possible outputs : Pr [ A ( D ) ∈ S ] ≤ eε Pr [ A ( D′ ) ∈ S ] + δ . We review prior techniques for private fine-tuning . 2.1 FULL FINE-TUNING VIA DPSGD . To train a machine learning model with privacy , the most popular algorithm is the celebrated DP stochastic gradient descent ( DPSGD ) ( Song et al. , 2013 ; Bassily et al. , 2014 ; Abadi et al. , 2016 ) . This optimization method serves as a drop-in replacement for SGD , augmenting it with the addition of per-example gradient clipping and Gaussian noise addition steps . These two steps serve to limit and mask the contribution of a single example . Two key points to note are that a ) per-example gradient clipping incurs significant computational and memory overheads in most implementations , and b ) noise introduced due to privacy grows as the square-root as the number of model parameters . With this tool in place , the most basic fine-tuning strategy is to train all parameters using DPSGD . 2An alternative perspective is that what we currently think of as “ large ” language models are relatively small , and we are yet to reach the point where the benefits of model size on accuracy are outweighed by the drawbacks . 2.2 REPARAMETRIZED GRADIENT PERTURBATION . To mitigate the limitations of DPSGD , a recent work of Yu et al . ( 2021b ) introduced an elegant method called reparametrized gradient perturbation ( RGP ) . RGP exploits the implicit low-rank structure in the gradient updates of SGD to substantially improve upon DPSGD . Specifically , they reparametrize each layer ’ s weight matrix W into LR + W̃ , where L and R are low-rank gradientcarrier matrices and W̃ is the residual weight . The authors show that one can obtain a lowdimensional projection of W ’ s gradient by taking gradients only of the low-rank matrices L and R ( and not the high-rank W̃ ) . Privacy is introduced by clipping and noising these low-dimensional gradients of L andR . While this low-dimensional projection loses some of the signal inW ’ s gradient , it turns out to contain enough to still achieve high accuracy . At the same time , the low-dimensional gradients alleviate the aforementioned issues related to privatization , significantly reducing the memory consumption and noise introduced . 3 OUR APPROACH . 3.1 A META-FRAMEWORK . We introduce our approach as a meta-framework for private deep learning , which abstracts the key principles of recent fine-tuning methods . Suppose f ( WPT ; x ) is a pre-trained model where WPT are the pre-trained weights and x is any input . We create a new fine-tuning model fFT ( WPT , θ ; x ) ( 1 ) which incorporates additional trainable parameters θ , where dim ( θ ) dim ( WPT ) . That is , the number of new parameters in θ is a small fraction of the original number of parameters in the pretrained weights WPT . Fine-tuning is done by running DPSGD on the additional parameters θ , while freezing the weights of pre-trained model WPT . The new parameters are initialized to θ0 such that fFT ( WPT , θ0 ; x ) = f ( WPT ; x ) . ( 2 ) The initialization condition ( 2 ) is very important , as it ensures that fine-tuning starts at the pre-trained model and improves it by modifying the parameters θ . Most fine-tuning methods are additive and have the following special form : fFT ( WPT , θ ; x ) = f ( WPT + π ( θ ) ; x ) , ( 3 ) i.e. , they modify the pre-trained weights by adding a correction term π ( θ ) parametrized by θ . Recent work in the non-private literature has described concrete instantiations of this framework ( Houlsby et al. , 2019 ; Mahabadi et al. , 2021 ; Hu et al. , 2021 ) , which ( crucially ) are effective when dim ( θ ) dim ( WPT ) . In the non-private setting , such reparametrizations are useful for reducing the computation and memory required for fine-tuning , and enable lightweight and plug-in modifications to the base model for different downstream tasks . At the same time , they maintain ( or sometimes surpass ) the accuracy achieved by full fine-tuning . However , this approach seems to be more important for achieving high accuracy in the context of private fine-tuning . For instance , full fine-tuning of RoBERTa-Base with DPSGD achieves an accuracy of 53.1 % for the MNLI dataset ( Yu et al. , 2021b ) , whereas all the algorithms that fine-tune based on Equation ( 1 ) all achieve accuracy better than 80 % . We give some intuition as to why parameter-efficient methods seem to be more effective for private fine-tuning . For simplicity , we assume that the fine-tuning method is additive as in ( 3 ) , such that the fine-tuned weights WFT = WPT + π ( θ ) . We can imagine that WFT lies on a manifold passing through WPT of very small dimension ( equal to the dimension of θ ) compared to the dimension of WPT . Even if the parameters θ are very noisy due to the noise added during DPSGD , we will always stay in this manifold . In particular , we are not disturbing the pre-trained weights in most directions ( those orthogonal to the manifold near WPT ) . If we run DPSGD on all the weights instead , then we add noise in all directions , thus potentially unlearning the knowledge learned during pre-training , especially in low data regimes . Besides substantial gains in the accuracy , the above method of reparametrization has several other advantages : • A single pre-trained model such as BERT or GPT is generally applied to hundreds of downstream tasks via fine-tuning . Private fine-tuning using previous methods requires updating all parameters and storing a different copy of the fine-tuned model per task . This creates substantial overheads for storing and deploying , and can be very expensive in practice . On the other hand , the reparametrization ( 1 ) means that we only need to store a single pretrained model that can be shared across many downstream tasks . Each downstream task requires only a small number of new parameters that can be plugged in . • Differentially private training requires computing and storing per-example gradients , which increases the memory footprint . In our approach , however , learning is done in a much lower dimension , hence saving on the memory cost as compared to prior works . • Finally , we expect that ( 1 ) also gives a more communication-efficient method of fine-tuning in distributed settings such as federated learning , due to the significantly smaller number of parameters learned during fine-tuning .

Summary: This paper demonstrates the feasibility of fine-tuning large language models (pretrained on public dataset) on private datasets for various downstream tasks. It proposes a meta-framework in which most of the pretrained weights are held constant and only a small number of additional parameters are updated during fine-tuning. Combined with state-of-the-art dimension reduction and privacy accounting techniques, their method achieves good privacy/utility tradeoff on several GLEU benchmarks using RoBERTa and NLG using GPT-2.