SeqPATE: Differentially Private Text Generation via Knowledge Distillation

Protecting the privacy of user data is crucial when training neural text generation models , which may leak sensitive user information during generation . Differentially private ( DP ) learning algorithms provide guarantees on identifying the existence of a training sample from model outputs . PATE is a DP learning algorithm that fits large models well , such as GPT . In this paper , we propose SeqPATE that adapts PATE to text generation while satisfying DP . There are two key challenges in adapting PATE to text generation : ( i ) obtaining sequence-level supervision for text generation , and ( ii ) reducing noise required to protect the privacy given a large output space ( i.e . vocabulary size ) . For ( i ) , we generate pseudo input and reduce the sequence generation problem to the next word prediction . For ( ii ) , we reduce the output space with top-k and top-p selection strategy that dynamically filters the candidate words ; and we refine the teacher aggregation of PATE to avoid the low agreement rates due to voting over the large output space . To limit the privacy loss , we design an efficient knowledge distillation to reduce the frequency of distilling from the private data . We apply SeqPATE to a simple text generation task ( sentence completion ) and achieve 39 % and 28 % gains in Bleu4 on two datasets . 1 INTRODUCTION . Recent work showed that sensitive user information in training corpora , such as address and name , can be extracted from text generation models ( Carlini et al. , 2019 ; 2020 ) . Providing privacy guarantees to the text used to train text generation models has become a critical problem . Differential privacy ( DP ) provides provable guarantees against the identification of individuals in the dataset . Deep learning models with DP guarantees ensure the existence of a specific training sample can not be detected . The most popular DP algorithm for deep learning is Noisy-SGD ( Song et al. , 2013 ; Bassily et al. , 2014 ; Abadi et al. , 2016 ) , which adds noise to the gradients . However , the advantage of NoisySGD is weaken as the model becomes larger ( Yu et al. , 2020 ) , and NoisySGD requires a per-example gradient clip , which causes non-convergence and system overheads ( Zhu et al. , 2020 ; Bu et al. , 2021 ) . Applying Noisy-SGD to large text generation models ( e.g . GPT-2 ) requires additional tricks such as memory reduction ( Li et al. , 2021 ) and advanced fine-tuning strategies ( Yu et al. , 2021 ) . PATE ( Papernot et al. , 2017 ) is a DP learning model using a teacher-student framework : a student accessing non-sensitive data distills knowledge from the aggregated predictions of multiple teachers trained on sensitive data . Calibrated noise is added to the aggregated predictions to satisfy DP . PATE can handle large models since its privacy cost derives from the knowledge distillation instead of the model parameters . PATE takes advantage of non-sensitive data which is also available in our scenario . In this paper , we propose , SeqPATE , a DP learning algorithm for text generation that exploits both sensitive and non-sensitive data by knowledge distillation . Specifically , we adapt PATE for text generation . The adaptation is challenging due to the sequential generation and the large output space ( i.e . vocabulary size ) in text generation . Firstly , to obtain the sentence-level supervision for text generation , directly applying PATE needs to roll out all teachers to produce a sentence ( i.e . all teachers must vote to generate a word , which is then used as the input for the next word prediction ) . Running inference for a large number of teachers can be costly . Secondly , the large output space results in ( i ) low agreement rate among teachers and ( ii ) large noise required by DP , both would significantly hurt task performance . To avoid sequential generation , we generate pseudo data using a pre-trained language model such that teachers only need to provide token-level supervision given the pseudo input . To handle the large output space , we aggregate teachers ’ outputs by interpolating their output distributions instead of voting with argmax predictions . To reduce the noise scale , we propose strategies to dynamically filter candidate words that only keep words with large probabilities . Further , we design an efficient knowledge distillation strategy that only queries the teacher when the student has poor performance . We evaluate our method on sentence completion , a simple but representative task of text generation . Compared to baselines including Noisy-SGD ( Kerrigan et al. , 2020 ) , SeqPATE improves Bleu4 by 39 % and 28 % on two datasets with meaningful privacy protection 1 . We observe SeqPATE works particularly well under relatively strong privacy protection ( e.g. , ε ≤ 2 ) compared to Noisy-SGD . Our contribution is threefold : ( i ) Our model significantly surpasses baselines and makes DP algorithm achieve satisfactory performance on text generation ; ( ii ) We propose several practical strategies to enable SeqPATE to handle text generation with a sequential of classifications over larger output spaces ; ( iii ) We are the first to adapt PATE , an effective DP algorithm , to text generation . 2 PROBLEM SETUP . Our goal is to achieve the privacy protection defined by DP in text generation : preventing attackers from inferring whether a sample or an n-gram appears in the training set . ( Formal definition in Sec . 5 ) PATE assumes students are trained on unlabeled non-sensitive data . Similar to PATE , our setting contains two textual datasets : 1. a private set Dpri from a corpus with sensitive information . 2. a public set Dpub without sensitive information ( e.g . incomplete or masked text spans ) or data contributors ( e.g . volunteers ) have no objection to publishing their data . Algorithms are required to protect the privacy on the private set and can ignore the privacy protection on the public set . The final goal is to do well on the private set during inference . Text generation is to generate a sequence of words S = { w1 , w2 , . . . , wl } . Our application is a simple text generation task , sentence completion , which aims to generate the remaining part of a sentence w1 : l given its prefix p1 : l′ , where p1 : l′ and w1 : l are the abbreviation of { p1 , p2 , . . . , pl′ } and { w1 , w2 , . . . , wl } respectively . p1 : l′ and w1 : l compose a complete sentence . We collect some input prefixes p1 : l′ to compose the public set Dpub and the original samples ( complete sentences ) compose the private set Dpri . Such a setting fits for some real-world text generation applications : in dialog systems , the training samples from online services consist of questions and responses . The questions from customer service staffs or service robots can be public and the response from users carrying individual information should be private . In writing assistants where a sample has an input prompt and an essay body , the prompt can be public but the body written by data contributors is private . 3 BACKGROUND ON DP AND PATE . Differential privacy ( DP ) ( Dwork et al. , 2006 ; 2014 ) is a quantifiable definition of privacy that provides provable guarantees on identifications of individuals in the dataset . ML algorithms with DP guarantee ensure that each individual training sample has a degree of plausible deniability , i.e. , the trained model is just as likely as to have been trained on an alternative dataset without that sample . PATE , designed for classification tasks , takes advantage of an unlabeled public dataset Dpub and also trains on a labeled private Dpri in a semi-supervised scenario . PATE is model-agnostic and treats models as black-boxes . PATE ( Papernot et al. , 2017 ) achieves DP via a teacher-student framework with M teacher models and a student model , where the student learns from the private data via knowledge distillation through teachers . PATE consists of three parts , • Teacher models are trained on the private set Dpri . Dpri is shuffled and divided into M disjoint subsets , and m-th subset serves as the training set for m-th teacher model fmϕ . • Teacher aggregation merges teachers ’ outputs . After teachers ’ training , each teacher fmϕ conducts the prediction ( classification ) on student ’ s training set Dpub . To satisfy DP , PATE aggregates M 1We choose ε = 2 . DP with ε ∈ [ 0.1 , 5 ] is regarded as a meaningful protection ( Triastcyn & Faltings , 2020 ) . teachers ’ by voting with their top-1 predictions . PATE adds noise to aggregated teacher outputs so that attackers can not access the exact outputs to infer original information in samples . • A student model fθ is trained on the public set Dpub and distills knowledge from private set ( via the teachers ) without accessing the private samples . In this way , the sensitive information in the private set is preserved even if the student ’ s architecture and parameters are public or reverse-engineered by an adversary . We publish only the student model for prediction . 4 APPROACH . Fig . 1 shows an overview of the SeqPATE . Given the public prefix ( e.g. , “ Cats sit ” ) , we first obtain pseudo inputs by generating the remaining part of the whole sentence ( e.g. , “ Cats sit on the mats ” ) using a pre-trained language model ( Sec . 4.1 ) . The teachers ’ output distributions are then averaged to provide word-level supervision on the whole sentence ( Sec . 4.2 ) . To reduce the noise over large output space and limit the privacy loss , we propose top-k / top-p selection to dynamically filter the unimportant candidates and propose efficient knowledge distillation to reduce the number of teacher queries ( Sec . 4.3 ) . Our training algorithm is shown in Appendix A . 4.1 STUDENT MODEL . Conventional text generation models generate words in a sentence step-by-step and left-to-right . Naively applying PATE to the above paradigm requires all the teachers to be roll out to obtain supervision at every step , where all the teachers aggregate to generate a word at each step and the word acts as the next input for all teachers . This means we should conduct all teachers ’ inferences simultaneously and interdependently , which is costly in both computation ( involving hundreds of teachers ) and privacy costs . Hence , in SeqPATE , we employ GPT to generate pseudo sentences and conduct teacher inference and student training on the pseudo sentences . Firstly , given a prefix in Dpub , we employ the public released pre-trained GPT to conduct the inference to complete a pseudo sentence . All the generated pseudo sentences compose the student ’ s training set , denoted as D̃pub . Secondly , we conduct teacher inference and student training on the sample S = { w1 , w2 , . . . , w|S| } from D̃pub . At the i-th step , for the training of conventional text generations , the student learns to generate a word ŵi given its previous words w1 : i−1 . At the i-th step , our teachers conduct the inference to generate ŵi given the input w1 : i−1 from the sample S. Then , each teacher provides its output for aggregation . All teachers share the same input with the student so that the outputs of teachers and student are aligned and comparable .

Summary: This paper tries to adapt the PATE (Private Aggregation of Teacher Ensembles) framework for DP training of machine learning models to the setting of text generation. There are some significant challenges in adapting PATE to this framework.  PATE as originally proposed is designed to work for classification tasks with a small number of labels. Text generation or next word prediction has too many labels and a direct application of PATE doesn't work. This paper solves this issue by aggregating only top-k predictions of each teacher where the top-k is computed using the prediction of the student on public data and so doesn't leak extra privacy. The paper also crucially uses public data and large pretrained models like GPT2 to further improve the performance of DP learning using PATE on private data.  They claim that their approach improves upon DPSGD/Noisy SGD (which is the standard DP algorithm for deep learning) significantly on some standard benchmarks.