SeqPATE: Differentially Private Text Generation via Knowledge Distillation
Authors: Zhiliang Tian
Yingxiu Zhao
Ziyue Huang
Yu-Xiang Wang
Nevin Zhang
He He
Venue: ICLR 2022 Submitted (2022)
Keywords: Natural Language Generation
Text Generation
Privacy Protection

Abstract: Protecting the privacy of user data is crucial when training neural text generation models, which may leak sensitive user information during generation. Differentially private (DP) learning algorithms provide guarantees on identifying the existence of a training sample from model outputs. PATE is a DP learning algorithm that fits the large model well, such as GPT. In this paper, we propose SeqPATE that adapts PATE to text generation while satisfying DP. There are two key challenges in adapting PATE to text generation: (i) obtaining sequence-level supervision for text generation, and (ii) reducing noise required to protect privacy given the large output space (i.e. vocabulary size). For (i), we generate pseudo input and reduce the sequence generation problem to the next word prediction. For (ii), we reduce the output space with top-$k$ and top-$p$ selection strategy that dynamically filters the candidate words; and we refine the teacher aggregation mechanism of PATE to avoid the low agreement rates due to voting over the large output space. To limit the privacy loss, we design an efficient knowledge distillation to reduce the time of distilling from the private data. We apply SeqPATE to a simple text generation task (sentence completion) and achieves 39\% and 28\% gains in Bleu4 on two datasets.

Under review as a conference paper at ICLR 2022
SEQPATE: DIFFERENTIALLY PRIVATE TEXT GENERATION VIA KNOWLEDGE DISTILLATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Protecting the privacy of user data is crucial when training neural text generation models, which may leak sensitive user information during generation. Differentially private (DP) learning algorithms provide guarantees on identifying the existence of training sample from model outputs. PATE is DP learning algorithm that fits large models well, such as GPT. In this paper, we propose SeqPATE that adapts PATE to text generation while satisfying DP. There are two key challenges in adapting PATE to text generation: (i) obtaining sequence-level supervision for text generation, and (ii) reducing noise required to protect the privacy given large output space (i.e. vocabulary size). For (i), we generate pseudo input and reduce the sequence generation problem to the next word prediction. For (ii), we reduce the output space with top-k and top-p selection strategy that dynamically filters the candidate words; and we refine the teacher aggregation of PATE to avoid the low agreement rates due voting over the large output space. To limit the privacy loss, we design an efficient knowledge distillation to reduce the frequency of distilling from the private data. We apply SeqPATE a simple text generation task (sentence completion) and achieve 39% and 28% gains in Bleu4 on two datasets.
INTRODUCTION
Recent work showed that sensitive user information in training corpora, such as address and name, can be extracted from text generation models (Carlini et al., 2019; 2020). Providing privacy guarantees to the text used train text generation models has become critical problem. Differential privacy (DP) provides provable guarantees against the identification of individuals in the dataset. Deep learning models with DP guarantees ensure the existence specific training sample cannot be detected.
The most popular DP algorithm for deep learning is Noisy-SGD (Song et al., 2013; Bassily et al., 2014; Abadi al., 2016), which adds noise to the gradients. However, the advantage of NoisySGD is weaken as the model becomes larger (Yu et al., 2020), and NoisySGD requires per-example gradient clip, which causes conveenveree and system overheads (Zhu et al., 2020; Bu et al., 2021). Applying Noisy-SGD to large text generation models (e.g. GPT-2) requires additional tricks such as memory reduction (Li et al., 2021) and advanced fine-tuning strategies (Yu et al., 2021). PATE (Papernot al., 2017) is DP learning model using a teacher-student framework: a student accessing non-sensitive data distills knowledge from the aggregated predictions of multiple teachers trained on sensitive data. Calibrated noise added the aggregated predictions satisfy DP. PATE can handle large models since its privacy cost derives from the knowledge distillation instead of the model parameters. PATE takes advantage of non non-sensitive data which is also available in our scenario.
In this paper, we propose, SeqPATE, DP learning algorithm for text generation that exploits both sensitive and non-sensitive data by knowledge distillation. Specifically, we adapt PATE for text generation. The adaptation is challenging due to the sequential generation and the large output space (i.e. vocabulary size) in text generation. Firstly, to obtain the sentence-level supervision for text generation, directly applying PATE needs to roll out all teachers to produce sentence (i.e. all teachers must vote to generate a word, which is then used as the input for the next word prediction). Running inference for large number of teachers can be costly. Secondly, the large output space results in (i) low agreement rate among teachers and (ii) large noise required by DP, both would significantly hurt task performance.

---

Under review as a conference paper at ICLR 2022
To avoid sequential generation, we generate pseudo data using a pre-trained language model such that teachers only need to provide token-level supervision given the pseudo input. To handle the large output space, aggregate teachers' outputs by interpolating their output distributions instead of voting with argmax predictions. To reduce the noise scale, we propose strategies to dynamically filter candidate words that only keep words with large probabilities. Further, we design efficient knowledge distillation strategy that only queries the teacher when the student has poor performance.
We evaluate our method on sentence completion, a simple but representative task of text generation. Compared to baselines including Noisy-SGD (Kerrigan et al., 2020), SeqPATE improves Bleu4 by 39% and 28% on two datasets with meaningful privacy protection We observe SeqPATE works particularly well under relatively strong privacy protection (e.g.. S 2) compared to Noisy-SGD. Our contribution threefold: (i) Our model significantly surpasses baselines and makes DP algorithm achieve satisfactory performance on text generation; (ii) We propose several practical strategies to enable SeqPATE to handle text generation with a sequential of classifications over larger output spaces; (iii) We are the first to adapt PATE, an effective DP algorithm, to text generation.
2 PROBLEM SETUP
Our goal is to achieve the privacy protection defined by DP in text generation: preventing attackers from inferring whether a sample or an n-gram appears in the training set. (Formal definition in Sec. 5)
PATE assumes students are trained on unlabeled non-sensitive data. Similar to PATE, our setting contains two textual datasets: 1. a private set Dr from a corpus with sensitive information. 2. a public set Druh without sensitive information (e.g. incomplete or masked text spans) or data contributors (e.g. volunteers) have no objection to publishing their data. Algorithms are required to protect the privacy on the private set and can ignore the privacy protection on the public set. The final goal is to do well on the private set during inference.
Text generation is generate sequence of words S = {wi, u2,.... wr Our application is simple text generation task, sentence completion, which aims to generate the remaining part of a sentence Wia given its prefix Pi.", where Pı,P and what are the abbreviation of {Pi, {p1,P2,.... Pr } and {W1, W.,..., ; mi respectively. P1:1' and Wid compose a complete sentence. We collect some input prefixes P1,' to compose the public set Druh and the original samples (complete sentences) compose the private set D Such a setting fits for some real-world text generation applications: in dialog systems, the training samples from online services consist of questions and responses. The questions from customer service staffs or service robots can be public and the response from users carrying individual information should be private. In writing assistants where a sample has an input prompt and essay body, the prompt can be public but the body written by data contributors is private.
3 BACKGROUND ON DP AND PATE
Differential privacy (DP) (Dwork et al., 2006; 2014) is quantifiable definition of privacy that provides provable guarantees on identifications of individuals in the dataset. ML algorithms with DP guarantee ensure that each individual training sample has degree of plausible deniability, i.e., the trained model is just as likely as to have been trained on an alternative dataset without that sample.
PATE, designed for classification tasks, takes advantage of an unlabeled public dataset Druh and also trains on a labeled private Dr in a superisuerriiee scenario. PATE is aggosstg and treats models as black-boxes. PATE (Papernot al., 2017) achieves DP via teacher-student framework with M teacher models and a student model, where the student learns from the private data via knowledge distillation through teachers. PATE consists of three parts,
Teacher models are trained on the private set Deci Deri is shuffled and divided into M disjoint subsets, and m-th subset serves as the training set for m-th teacher model f
Teacher aggregation merges teachers' outputs. After teachers' training, each teacher f conducts the prediction (classification) on student's 's training set Doub, To satisfy DP, PATE aggregates M
We choose 2. DP with & E0)11,5 is regarded a meaningful protection (Triastcyn & Faltings, 2020).
2

---

Under review as a conference paper at ICLR 2022
Prefo ward
Toachel informnce
Student Training
Solisfeed need
har exteted jreed kiy
Teastren
Ray
Ourputt diorbutiin
words Ground truth mndertt
Knowleddg Distllatioon
Ted
Apprpda
i...
Thacher?
balal
Lai lba halla
In
Training
Tracher3
Stuskent
NLL
Gms
(n)) (he)
mate
Teochort
Teacher?
Teacher3
mode
Figure I: Overview of SeqPATE. SeqPATE trains teachers on the private data; it conducts student training and teacher inference on pseudo sentences generated by GPT based on the public data. The student is supervised by the aggregation of teacher output distributions and benefits from efficient knowledge distillation (pink block) and top-k top-p selection (white block in the top right comer).
teachers' by voting with their top-1 predictions. PATE adds noise aggregated teacher outputs so that attackers cannot access the exact outputs to infer original information in samples.
A student model fe trained on the public Drub and distills knowledge from private (via the teachers) without accessing the private samples, In this way, the sensitive information in the private set preserved even the student's architecture and parameters are public or reverse-engineered by an adversary. We publish only the student model for prediction.
4 APPROACH
Fig. shows an overview of the SeqPATE. Given the public prefix (e.g.. "Cats sit"), we first obtain pseudo inputs by generating the remaining part of the whole sentence (e.g.. "Cats sit on the mats") using a pre-trained language model (Sec. 4.1). ). The teachers' output distributions are then averaged to provide word-level supervision output space and limit the privacy loss, we propose top-k / top-p selection to dynamically filter the unimportant candidates and propose efficient knowledge distillation reduce the number of teacher queries (Sec. 4.3). Our training algorithm is shown in Appendix A.
the whole sentence (Sec. 4.2). To reduce the noise over large
4.1 STUDENT MODEL
Conventional text generation models generate words in a sentence step-by-step and left-to-right. Naively applying PATE to the above paradigm requires all the teachers to be roll out to obtain supervision every step, where all the teachers aggregate to generate word at each step and the word acts as the next input for all teachers. This means we should conduct all teachers' inferences simultaneously and interdependently, which is costly in both computation (involving hundreds of teachers) and privacy costs.
Hence, in SeqPATE, we employ GPT generate pseudo sentences and conduct teacher inference and student training on the pseudo sentences. Firstly, given a prefix in Deub we employ the public released pre-trained GPT to conduct the inference to complete a pseudo sentence. All the generated pseudo sentences compose the student's training set, denoted as DDub, Secondly, we conduct teacher inference and student training on the sample S {wi, W2...., wis } from Drub At the i-th step, for the training of conventional text generations, the student learns generate word ú; given its previous words W1:-11- 1- At the i-th step, our teachers conduct the inference to generate i; given the input W1a1-1 from the sample S. Then, each teacher provides its output for aggregation. All teachers share the same input with the student so that the outputs of teachers and student are aligned and comparable.
3

---

Under review as a conference paper at ICLR 2022
4.2 TEACHER AGGREGATION
The original PATE aggregates teacher results by taking the majority vote of the predicted labels from all the teachers. However, this aggregation strategy does not fit for the large output space in text generation. This is because the number of votes for each candidate may be very low and it would result in low agreement. For example, multiple candidates may tie for the top-1 prediction.
Therefore, inspired by (Hinton et al., 2015; Chen al., 2020), aggregate M teachers' results by averaging their output distributions, where each teacher model is trained on one the disjoint subsets divided from the private De At the i-th step on teacher inference, the m-thh teacher model f predicts a word ii with a probability P" (z04). Then, our model aggregates all teachers' probabilities by averaging Po (0,) + Σm P (tx;). Like most DP methods, our model adds noises so that the privacy impact be analyzed and bounded. Different from PATE that adds noise to vote numbers, we add noise to the aggregation teachers' output distributions. Following (Papernot et al., 2018), 88) apply the Gaussian mechanism which adds i.i.d. (O0, o²) to each coordinate, where controls the strength of privacy protection (more details in Sec. 5).
4.3 SUPERVISION ON THE STUDENT MODEL
Reducing output space via top-k or top-p selection. Recall that the teacher output is distribution over all words in the vocabulary, and the noise must be added to each coordinate to satisfy DP. To reduce the output dimension (hence the amount noise), we only assign the probability mass to a subset of candidates W, For all teachers and the student, we assign zero probability mass any candidates not in W; and re-normalize distribution as shown in Eq. 1, where g(.) is the normalizing function. This operation means the words not in W; will never occur in student's training and teachers' supervision, so those words do not need noise to provide protection.
p(u;; w1:-11
Enew, p(w| w1s-1) if th E W;
gpp(wr (waa-1))
(I)
otherwise.
We determine the subset W; by two selection strategies: top-k and top-p. In the top-k selection, at i-th step. select the top-k candidates according to the probabilities generated by the student model. Further, inspired by nucleus sampling (Holtzman et al., 2019), we propose a top-p selection that chooses the minimum & where the cumulative probability of the top-k words is larger than constant close to 1 (c.g. 0.95) such that the candidate set covers a large proportion of the distribution. We note that the candidate selection is based on the student's output on public or already released input, thus not affecting the privacy guarantee. This choice improves the privacy -utilty tradeoff by adaptively allocating the available privacy budgets to release the information more relevant the task.
Efficient knowledge distillation. Reducing the number of querying teachers results in better privacy protection. Therefore, we acquire teacher supervision only when the student has poor performance. Concretely, we measure student's performance on each token by comparing its output distribution with the ground truth token (from Doub). Let Tw; be the rank of the ground truth token w; in the student's 's predicted distribution, (i.e. the number tokens with higher or equal probability). The student model acquires the teachers' supervision only if Tw, is larger than a certain threshold re- We note that the selection of tokens relies only on the student and is independent of the teachers, thus the selection does not cause additional privacy loss.
Training objectives. Different from the semi semi-supervised setting in PATE, our student model is supervised by both teachers' aggregation and labels from the pseudo public set Daub since text generation corpora are naturally labeled, unlike in classification tasks. Even if the pseudo set Drub provides only pseudo samples, the labels from those samples also useful given that the number of querying teachers is limited due to privacy concerns. The student's 's loss function consists of two parts shown in Eq. 2,
4

---

Under review as a conference paper at ICLR 2022
C. shows teachers' supervisions. The student leams from teachers by minimizing KL divergence between the student output distribution po and the merged teacher output distributions Pa-
C. shows the supervisions from the pseudo public set Dauh generated by GPT. As conventional language models, student model is trained by negative log- ligelikeiihodd (NLL) loss Drub, The words w; in the training sample is naturally the label for the student fa(w):-- at the i-th step.
M Σ
w1i--1 + +N(0, po(. w.s-ii).
ckt. = KL
g
ps
m=
EN
log (w; w1i-i))
C= Σ SEDPsó
Σ (CNLL + AC
where L; is loss at i-th step. |S| denotes the length of sentence S, and À balances the two terms. Meanwhile, the noise scale a is discussed in Sec. 5. g(-) is the re-normalization function used in Eq. 1 that re-normalizes the aggregated probabilities on the selected top-k or top-p words with noise 2. According to the efficient knowledge distillation strategy, at each step, the loss L; is the combination of LALL and LKL if the student needs teachers' supervision; otherwise, Li CNLL
5 PRIVACY ANALYSIS
5.1 PRELIMINARY OF DIFFERENTIAL PRIVACY
Let DD D' denote two neighboring datasets which differ at only one individual. Definition (Differential privacy). For > O and ō 2 O, a randomized algorithm M X y is (e, ddifferrret private for any neighboring datasets D ~ 'D' and any S C V,
Pr(M(D) ES) { -P(MM(D) E E]
The definition ensures that it is information-theoretiee theoreticalyy impossible for an adversary to infer whether the input dataset is D D' even with arbitrary side information. The definition is also future proof thanks to its closure to post-processing.
Lemma (Post-processing). !f M obeys (e, D)DP, then for any function f. f M also (s, 6)-P.
One notable property of DP is that automatically protects the privacy of groups of multiple units.
Lemma 3 (Group privacy). An (e, 8DD mechanism M on individuals is (ke, c 5)-DP on groups of size k for all integers 2
e*r-1
Lemma (Analytical Gaussian mechanism (Balle Wang, 2018)). For numeric query f Xm +
dd over a dataset 'D, the randomized algorithm that outputs f(D) + Z where Z ~ N(0,02L))
satisfies (e. (8(() -DP for all 2 and ō(E) o (
20 =) e 1 (-= 20 20). where A == (() maXp-D' (D) - f(D') is the global L2 sensitivity of
f and 4 is the CDF finction of N 1).
The above is the (e, DDP a single Gaussian mechanism, and the following lemma shows that we use the same result for an adaptive composition of sequence of Gaussian mechanisms.
Lemma 5 (Composition of Gaussian mechanisms (Dong et al., 2019)). The adaptive composition of sequence of Gaussian mechanisms with noise level 02.02....... and global L2 sensitivity Σ1 D2.... satisfies (s, 8(E))-DP for all 2 0 and 6(E) < 6M(()) where M is Gaussian mechanism with noise multiplier a r/A= = (a,)))))) 1/2
Specifically, the adaptive composition of k identical Gaussian mechanism with noise multiplier satisfies the same privacy guarantee of that of a single Gaussian mechanism with noise multiplier a/vk. R. By fixing k and E, we can calibrate the noise by choosing an appropriate o in our algorithm.
Mathematically. the input of the g(-) may be negative and re-normalize it 0. Practically, observed heing negative is extremely rare since the M is very hig (2000) and the first term dominates the input of g())
5

---

Under review as a conference paper at ICLR 2022
5.2 DIFFERENTIAL PRIVACY AT THE SAMPLE LEVEL
Recall that we partition the private dataset to M disjoint subsets, and train a teacher model on each subset. Let vector 2y E TRIV| denote the probability distribution predicted by the i-th teacher model for specific word, where [V] is the vocabulary size. The function f(D) := E 2; is the sum of the probability distributions predicted over all teachers. Since the dataset is disjoint and also has no repeating sample, changing one sample will only affect one teacher model. For neighboring datasets D. D', let j denote the index of different teacher models, i.e., Iy DD and a'j E DD' are different. Then, the sensitivity A in Lemma 4 & 5 is (Detailed deduction in Appendix B),
A = ( = [I((D) - f(D')la & 1
25 12 S V2.
Then, adding noise calibrated in Lemma 5 to each coordinate is sufficient preserve (c, (-)) for f(D). Finally, when we extract top-k coordinates of J(D) (Sec. 4.3), the privacy guarantee also holds according to the post-processing property. Intuitively, the information about whether a sample occurs in the private training set is protected and the protection satisfies (s, 6(E))-DP.
5.3 DIFFERENTIAL PRIVACY OF USER'S SECRET PHRASES
The above analysis focuses on DP at the sample level, but the privacy guarantee also applies to individual users (who contributed many samples), which prevents each user's secret phrases (or text spans), such as SSN number, phone number, and addresses from being memorized (or generated as is) by the model. Consider secret phrase that occurs 8, times (8, 2 1) in the private set. We obtain DP guarantee for each / separately 3 in the following two scenarios. In the first scenario where the teachers' data is not partitioned by users, the protection on ł satisfies (ste, ea 5)-DP according to the group privacy lemma.
SeqPATE allows us to partition the teachers' training data by users. this scenario, adding or removing user (and all her samples) only affects one teacher, thus the same privacy guarantee we have derived for sample-level DP applies to the user level, too. As a result, our approach enjoys stronger guarantee than naively applying the group privacy lemma. Our approach depends only on the total number of teachers affected, which is bounded by 8; == min # of users who know t). 3, is often 2 or 3 (e.g., husband and wife, parents and child), even if is large. Clearly, adding or removing phrase / will result in a sensitivity v28, and the exact (s, 5(2, -DP for phrase can be obtained according to Lemma 4 & 5, where d(E, &.) Φ1
dt /2o The above user-level protection is also one merit of Seq that NoisySGD does not have.
co
e v2o v28," ).
V28.
How does DP prevent memorization? We remark that the protection against unintended memoriza- tion of model follows from the definition of DP. Consider the attack by (Carlini et al., 2019), which uses language model to predict a secret phrase t given a prefix. By the closure to post-processing, the prediction also satisfies DP. Take S to be the undesirable event that the model correctly generates the phrase t. The definition of DP implies that the probability of S to happen when is part of the training data most e larger than the probability of an alternative model trained without in the data. The chances for the latter model to generate sequence with / is astronomically small, thus DP implies that the probability S under the former model needs to be just as small.
6 EXPERIMENTS
6.1 EXPERIMENTAL SETTING
We evaluate our model on two datasets. AirDialog dataset (Wei et al., 2018) consists of 1M utterances from custom service dialog about flight booking; Europarl_ve v6 dataset consists of 2M English sentences collected from European Parliament. Pre-trained GPT-2 (Radford et al., 2018) provides initial parameters for comparing methods. Before student training, conduct additional fine-tuning on the public data Douh via. NLL loss to provide initialization. We evaluate the quality of generated
A formal definition of this is called personalized diffcrential privacy, first seen in (Ghosh & Roth, 2011)
www.sttamtt.orge.. wor.oooooooorooprrrrrrr

---

Under review as a conference paper at ICLR 2022
texts with PPL (Perplexity, the lower the better) and B-n (Bleu-n, the higher the better) (Papineni et al., 2002) and evaluate the performance of privacy protection with P-n, which is the fraction of unique n o-grams in model generated texts that can be found in the private text (lower score indicates fewer privacy leaks). Our base model is GPT2-small and the default number of teacher models is 2,000. The batch size is 32 for all comparing methods and the À in Eq. 2 is 20. (See details about datasets, settings, metrics, implementation, and more experimental results in Appendix C to L).
6.2 OVERALL PERFORMANCE
AirDinlog
Europari .16
PPL
B-1
B-2
B.3
B-4
P-3
P.4
PPL
B-1
B-2
B-3
B-4
P-3
P4
Protect
PREGPT
42.49
28
21.51
4.55
1.77
0.86
0.65
Public
Pub-GIPT
63.16
5.3
1.18
0.31
0.10
0.09
0.03
57,40
12.12
3.61
1.02
0.35
0.51
0.29
Data Only
Pub GPT+
19.39
7.58
2.11
0.71
0.25
0.14
0.06
45,40
13.28
4.31
1.38
0.52
0.62
0.38
DP
Puh-GiPTADP-SOI
20.39
7.30
0.8
0.32
0.18
40.69
12.37
4.04
1.22
0.49
0.70
0.47
Algorichms 2)
Pub-GPTHDP- SGDDDP
17.65
7.90
2.69
1.14
0.56
0.33
0.19
40.46
12.26
3.99
1.20
0.42
0.69
0.46
SeqPATE
0.45
Table 1: The overall performance on the two datasets. All DP algorithms satisfy (2, 10 -)-D.. The underlined numbers indicate the best results over all methods; the bold numbers indicate the best results among the DP-based methods (See more results with larger E and in Appendix H).
Table I shows the performance on the two datasets. Pri-GPT is the GPT model trained on the private set Drri without privacy protection, acting as an upper bound on the task performance. The high P-n scores on Pri-GPT show most generated n-grams may come from Dem indicating serious issue with privacy leaks in current text generation models. Pub-GPT uses the pre-trained GPT model to our setting without mhe-tunnng. Pub-GPT+Druh is further fine-tuned on the pseudo data and outperforms Pub-GPT, indicating that the pseudo dataset helps in this task (Experiments in Appendix L also verify that.). The two baselines do not involve any information from the private data, so their P-n are the upper bound and their utlities are the worst (note that our goal is to do well on Dert in inference).
DP learning algorithms are our main baselines. Pub-GPTTDP-SGD is initialized by Pub-GPT and further trained on Drrs with DP-SGD (Noisy-SGD) (Abadi et al., 2016; Kerrigan et al., 2020). Pub- GPT+DP-SGD++DD further fine-tunes the Pub-GGTTDP-SGD Doub, 'he small gap between the two methods demonstrates DP-SGD cannot utilize Duub well. Our method, SeqPATE, significantly outperforms the DP baselines (+39.3% and +28.0% in Bleu4) while ensuring the same strength of privacy protection in terms of P-n and e. Note that, we conduct the experiments based on the sample level privacy of æ = 2 mentioned in Sec. 5.2, and the experimental results also reflect the protection on users' secret phrase t mentioned in Sec. 5.3. However, the strength of the protection is weaker than the sample-level protection. The model satisties (e, 8(E, 5D) or (ste, - 6)- on the secret phrase t (5 2) as mentioned in Sec. 5.3. SeqPATE does better in empirical metrics on privacy compared to DP-SGD even i 2 is the same theoretically; this provides empirical evidence of our discussion in Sec. 5.3 on providing stronger guarantees for phrases that appear more than once.
Up
07
Roggatvv
Figure 2: The private-utility tradeoff reflected in the B and negative PPL on different
The curves in Fig. 2 show the private-utility tradcoff of all DP algorithms in AirDialog (subfigure a & b) and Europarl v6 (subfigure d) dataset S, DP with e e 00.1, 5) is considered to provide a meaningful protection, and we observe that SeqPATE outperforms DP-SGD in this range. SeqPATE
Forr non-Do DP methods, we consider to be zero for baselines without using private data and to be infinity for baselines without privacy protection.
7

---

Under review as a conference paper at ICLR 2022
does not work better than DP-SGD when E 2 5. The reason is DP-SGD approaches Pri-GPT approaches infinity (i.e. the noise approaches 0). However, SeqPATE with an infinite E is still weaker than Pri-GPT because each teacher in SeqPATE is trained on a small partition of the full data. The grey lines show the performance of Pub-GPT+Duri and the area above the line indicates that the DP methods outperform Pub-GGT+D"" (DP methods the private data outperforms the non-DP methods on the non coniiriv data). The area indicates a range where DP methods (with private data) useful for this application. Our method always reaches that area when but DP-SGD hardly exceeds the grey line, the very trivial baseline, with e S 5.
AirDialog
Europar..v v6
AirDialog
Europarl, y6
PPL.
H4
P-
PPL.
R4
F4
SeqPATE
13.67
0.17
37,80
0.55
0.45
PPL
B-4
F-4
PPL
B4
F-4
-Merge
14.72
0.69
0.16
45,37
0,54
0.40
top-p
top-k-1
13.67
18.37
0.78
0.09
0.17
37,80
0.53
0.35
0.45
0.45
KLL
14.74
0.69
0.36
45.53
0.55
0.39
A=10
14.57
0.79
0.21
45,90
0,55
0.40
-NLL
Fifi KD
Gaussiia
13.60
13.69
14.35
0.74
0.60
0.58
0.16
0.19
0.18
38.24
38.09
37.10
0.34
0.53
0.50
0.46
0.45
0.43
top top-A=100
top-X=200
top top-k=50
13,50
13.96
14.65
0.72
0.76
0.70
0.18
0.19
0.17
39.35
38.48
37.91
0.53
0,53
0.57
0.50
0.46
0.40
-All
15.73
0.60
0.16
45.36
0.52
0.40
Table 3: Analyses about top-k and top-p strategies.
Table 2: Ablation studies.
6.3 ABLATION STUDIES AND FURTIER ANALYSES
In the ablation studies (Table 2), SeqPATE mainly outperforms its variants shown in rows to 7 verifying the effectiveness of our strategies, where -X represents SeqPAET's variant without the strategy X. -eerge is the variant aggregating the teachers by voting instead of merging the probabilities. Its poor performance shows that voting is not suitable for text generation with large output space where the voting leads to low agreement rates. - KI. means the student learns from the aggregation of teachers' top-1 predictions via NLL loss instead of KL loss. The sharp performance drop on - KL indicates the importance of KL loss. The performance of -NLL slightly decreases, indicating that NLL loss makes a little contribution to SeqPATE. The reason that we have pre-trained on the student's s training set via NLL before student training. The promotion caused by efficient knowledge distillation (Effi KD) on AirDialog is larger than that on Europarl_v6, which shows the "clever" student (e.g. models on AirDialog with low PPL. and high Bleu) benefits more from this strategy since can sharply save the privacy cost and spend the cost where the student strongly needs. - Gaassiin means we use Laplace mechanism as mentioned in PATE instead of Gaussian mechanism, which shows that the Gaussian mechanism works better. - All indicates discarding all above strategies, which is similar to but not equivalent to the original PATE (the difference is that PATE needs to roll out all teachers (Sec. 4.1), which disables PATE from working on text generation). The poor performance of -All verifies that the original PATE is not adaptive to text generation.
Table 3 shows the performance of the top-p and top-k selection strategies (Sec. 4.3). As shown in row 1, our full model employs the top-p selection (the threshold Jr = 0.95) surpasses most variants with manually chosen (rows 2 to 6). A selection with a too small 1: (k: = & k = 10) implies discarding too much useful information from the supervision (k uses Top-1 I of teachers' results). A selection with oversize A results in involving more useless noise: the candidates with very small probabilities should be rarely sampled during generation but random noise may increase their probabilities, so models may generate words that are misled by the noise. Supervisions on top-k candidates with 50 and k 100 yields relatively good performance.
different to KL in Table 2, which
Tables 4 & 5 in the Appendix shows more teachers lead to better results since the noise assigned to each teacher drops linearly as the teacher's numbers go up (as Eq. 2). We choose & = 2 for all results in Table 2 to 5. P-n for all the results in dataset to be roughly on the same level, except for some methods (c.g. row 2 in Table 3) that perform very badly in all metries. It shows SeqPATE has the stable performance of protection if is fixed (no matter which variant and strategy do we use).
7 RELATED WORK
Privacy protection is becoming crucial in deep learning models. Research in this area can be categorized into three directions (Liuu et al., 2021). Encryption approaches are to add encryption on
8

---

Under review as a conference paper at ICLR 2022
training data (Brickell et al., 2007: Bost et al., 2015) or machine learning (ML) models (Aono et al., 2017). Aggregation approaches generally come along with distributed training where many parties join the training of an ML model with the purpose of maintaining the privacy of the dataset from each party (Shokri & Shmatikov, 2015; Konečný et al., 2016). Obfuscation approaches reduce the precision of the data model by importing noise into the original input (Zhang al., 2018), model parameters (Rubinstein et al., 2012), or model gradients (Abadi et al., 2016; McMahan et al., 2018), ().
As a sub-topic of the obfuscation approach, differential privacy (DP) (Dwork et al., 2006; 2014) formally defines and quantifies privacy. ML models with DP guarantee (Wang et al., 2015; Park al., 2016; Ziller et al., 2021) prevents the existence of individual training examples from being detected (Carlini et al., 2019). Some researchers protect the privacy of empirical risk minimization (ERM) classifier (Chaudhuri et al., 2011) and SVM (Rubinstein et al., 2012) with DP. Following (Song et al., 2013), DP-SGI (Abadi et al., 2016) achieves DP on deep learning models by applying the noise to the clipped gradients. (Bu et al., 2020). Pichapati et al. (2019) adaptively clip the gradient on DP-SGD. PATE (Papernot et al., 2017) transfers the knowledge from teacher models trained on private data with injected noise to a student model. KNN-PATE (Zhu et al., 2020) refines PATE by accessing only the k-nearest neighbors from the private set instead of the whole private set. The above approaches are not customized for text generation models.
One direction of privacy protection in text generation is to protect author-level (user-level) information. The methods prevent attackers from inferring the author attributes (e.g. gender, age) (Li et al., 2018) and the relation between sensitive information and authors (Mireshghallah al., 2021). Shokri et al. (2017) and Song & Shmatikov (2019) infer the membership (whether a given user's data is used to train the model) given black-box model. Another direction to prevent attackers from extracting sensitive information in the training set by analyzing the generated outputs (Nakamura et al., 2020; Kerrigan et al., 2020), which is an urgent need since Carlini et al. (2020) achieve the extraction. Our application focuses on this direction. In this direction, some researchers use regularization, including dropout (Srivastava et al., 2014) and quantization (Hubara et al., 2017), to restrict the model capacity to avoid models memorizing too much original text from the privacy training set (Carlini et al., 2019). Anonymization methods (Maeda et al., 2016; Suzuki et al., 2017) detects sensitive text spans in training sets and replace the spans with non-sensitive texts. The above methods are non-DP algorithms, which make sense intuitively but do not provide quantifiable and provable guarantees for privacy protection.
Some researchers apply DP to text generation. For user-level privacy, ER-AE (Bo et al., 2021) augments the semantie information in the generated text hide authors' writing styles for authorship anonymization. McMahan et al. (2018) propose a recurrent language model with a DP guarantee against the identification of the users. Kerrigan et al. (2020) adapt DP-SGD to text generation, which prevents the (not the user-level) sensitive training data from being extracted from text generation models. Some of our concurrent works also follow this direction. Li et al. (2021) apply DP-SGD to large pre-trained language models and reduce the memory usage of those models. Yu et al. (2021) propose a framework to carry some advanced fine-tuning strategies on pre-trained language models with DP-SGD.
CONCLUSION
In this paper, we propose, SeqPATE, a framework to protect the privacy of the training data for text generation model with DP guarantees. SeqPATE achieves a good privacy-utility tradeoff by leveraging both private and public data. As an extension of PATE, SeqPATE can handle the sequential generation paradigm with large output space at each step and thus is adaptive for text generation models. We avoid rolling out the teachers by providing pseudo inputs for teacher inference and student training and aggregate teachers by merging probabilities instead of voting. We further reduce the output space by top-k or top-p selection and limit the privacy loss via an efficient knowledge distillation. SeqPATE improves the baselines lot and makes the private data useful to text generation with a meaningful strength of privacy protection. In the future, we will apply SeqPATE to some real-world applications, e.g. dialog systems and writing assistants.
9

---

Under review as a conference paper at ICLR 2022
9 REPRODUCIBILITY STATEMENT
We' ve submitted our source code as the supplementary material. We introduce the two datasets we used in Appendix C, where we introduce the details about the dataset and also append the link or citation of the dataset. For all our experiments, we adopt autodp (Wang et al., 2019) an open-source library that implements the analytical Gaussian mechanism for privacy accounting and calibration. The experimental setting and hyper-parameters can be found in Appendix D. Since we propose new metric in our experiments, we introduce the new metric with detailed equations in Appendix E.
10 ETHICAL CONSIDERATIONS
This paper aims to protect privacy in training the text generation models, so our paper is designed for tackling some ethical issues about privacy concerns in some existing text generation models (e.g. GPT-2). In terms of motivation and the algorithm, our paper would not cause ethical issues.
However, we should also consider some special situations where someone intentionally applies our model to illegal applications. Someone may employ text generation models to create fake news or misinformation and protect himself or herself from being detected. For this purpose, they may use model for their illegal application. In the future, we will add some constraints to model so that our model cannot generate text for illegal applications (c.g. fake news).
In addition, we know large (e.g. 500) cannot provide meaningful protection. We should carefully use our model and cannot assume the model is perfect no matter what parameters (e and 6) do use. One possible unethical application is to collect the data from users who believe model can fully protect their privacy regardless of the strength of privacy protection (in terms of the value of E). Hence, we kindly remind the researchers, who will use this model, to pay more attention the strength of privacy protection. Further, we should prevent some researchers to collect the data from users who do not have correct understanding of our algorithm, and the data collectors should make it clear that the potential risk in releasing data to our model.
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In CCS, pp. 308-318, 2016.
Yoshinori Aono, Takuya Hayashi, Lihua Wang. Shiho Moriai, al. Privacy-preserving deep learning via additively homomorphic encryption. IEEE T INF FOREN SEC, 13(5):1333-1355, 2017.
Borja Balle and Yu-Xiang Wang. Improving the gaussian mechanism for differential privacy: Analytical calibration and optimal denoising. In International Conference on Machine Learning. pp. 394 403. PMLR, 2018
Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. Proceedings of the 2014 IEEE 55th Annual Symposium on Foundations of Computer Science, pp. 464-473, 2014.
Haohan Bo, Steven HH Ding, Benjamin CM Fung, and Farkhund lqbal. Er-ae: Differentially private text generation for authorship anonymization. In NAACL, pp. 3997-4007, 2021.
Raphael Bost, Raluca Ada Popa, Stephen Tu, and Shafi Goldwasser. Machine learning classification encrypted data. In NDSS, volume 4324, pp. 4325, 2015.
Justin Brickell, Donald E Porter, Vitaly Shmatikov, and Emmett Witchel. diagnostics. In CCS, pp. 498-507, 2007.
remote
Zhiqi Bu, Jinshuo Dong, Qi Long, and Weijie J Su. Deep learning with gaussian differential privacy. Harvard Data Science Review, 2020(23), 2020.
Zhiqi Bu, Hua Wang, Qi Long, and Weijie J Su. On the convergence of deep learning with differential privacy. arXiv preprint wrXiv:2106.07880 2021.
10

---

Under review as a conference paper at ICLR 2022
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jemej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. USENIX Security, pp- 267-284, 2019.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. arXiv preprint arXiv:2012.07805, 2020.
Kamalika Chaudhuri, Claire Monteleoni, and Anand D Sarwate. Differentially private empirical risk minimization. JMLR, 12(3), 2011.
Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu, and Jingjing Liu. Distilling knowledge learned in bert for text generation. In ACI. pp. 7893-7905, 2020.
Jinshuo Dong, Aaron Roth, and Weijie Su. Gaussian differential privacy. Journal of the Royal Statistical Society: Series B, 2019.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In TCC, pp. 265-284. Springer, 2006.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. TCS, 9(3-4): 211-407, 2014.
Arpita Ghosh and Aaron Roth. Selling privacy at auction. In Proceedings of the 12th ACM conference Electronic commerce, pp. 199-208, 2011.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.0235 2015,
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In ICLR, 2019.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. JMLR, 18 (1):6869 6898, 2017.
Gavin Kerrigan, Dylan Slack, and Jens Tuyls. Differentially private language models benefit from public pre-training. In Workshop in EMNLP, pp. 39-45, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. NeurIPS, 2016.
Xuechen Li, Florian Tramèr, Percy Liang. and Tatsunori Hashimoto. Large language models can be strong differentially private learners. arXiv preprint arXiv:2110.05679 2021.
Yitong Li, Timothy Baldwin, and Trevor Cohn. Towards robust and prreacy-preeervigg text represen- tations. In ACI., pp. 25-30, 2018
Liu, M Ding. S Shaham, W Rahayu, Farokhi, and Z Lin. When machine learning meets privacy. ACM Computing Surveys, 2021.
Wakana Maeda, Yu Suzuki, and Satoshi Nakamura. Fast text anonymization using k-anonyminity. In iiWAS. pp. 340-344, 2016.
H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In ICLR, 2018.
Fatemehsadat Mireshghallah. Huseyin Inan, Marcello Hasegawa, Victor Rühle. Taylor Berg- Kirkpatrick, and Robert Sim. Privacy regularization: Joint privacy-utility optimization language- models. In NAACL, pp. 3799-3807, 2021.
11

---

Under review as a conference paper at ICLR 2022
Yuta Nakamura, Shouhei Hanaoka, Yukihiro Nomura, Naoto Hayashi, Osamu Abe, Shuntaro Yada, Shoko Wakamiya, and Eiji Aramaki. Kart: Privacy leakage framework of language models pre-trained with clinical records. arXiv preprint arXiv:2101.00036, 2020.
Nicolas Papernot, Martin Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi- supervised knowledge transfer for deep learning from private training data. In ICLR, 2017.
Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Ulfar Erlingsson. Scalable private learning with pate. In ICLR, 2018.
Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACI, pp- 31 1-318, 2002.
Mijung Park, James Foulds, Kamalika Chaudhuri, and Max Welling. Variational bayes in private settings (vips). arXiv preprint arXiv:1611.00344 2016.
Venkatadheeraj Pichapati, Ananda Theertha Suresh, Felix X Yu, Sashank Reddi, and Sanjiv Kumar. Adaclip: Adaptive clipping for private sgd. arXiv preprint arXiv:1908.07643 2019.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training, 2018.
Benjamin Rubinstein, Peter Bartlett, Ling Huang, and Nina Taft. Learning in large function space: Privacy-preserving mechanisms for svm learning. Journal of Privacy and Confidentiality (JPC), (1):65-100, 2012.
Reza Shokri and Vitaly Shmatikov. Privacy-preserving deep learning. In CCS, pp. 1310-1321, 2015.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In S&P, pp. 3-18. IEEE, 2017.
Congzheng Song and Vitaly Shmatikov. Auditing data provenance in text-generation models. In KDD, pp. 196-206, 2019.
Shuang Song, Kamalika Chaudhuri, and Anand Sarwate. Stochastic gradient descent with differen- tially private updates. In GlobalSIP, pp. 245-248. IEFE, 2013.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. JMLR, 15(1):1929 1958, 2014.
Yu Suzuki, Koichiro Yoshino, and Satoshi Nakamura. A k-anonymized text generation method. In NBIS, pp. 1018-1026. Springer, 2017.
Aleksei Triastcyn and Boi Faltings. Bayesian differential privacy for machine learning. In ICML, pp. 9583-9592. PMLR, 2020.
Yu-Xiang Wang, Stephen Fienberg, and Alex Smola. Privacy for free: Posterior sampling and stochastic gradient monte carlo. In ICMI. pp. 2493-2502. PMLR, 2015.
Yu-Xiang Wang, Borja Balle, and Shiva Prasad Kasiviswanathan. Subsampled rényí differential privacy and analytical moments accountant. In AISTATS, pp. 1226-1235. PMLR, 2019.
Wei Wei, Quoe Le, Andrew Dai, and Jia Li. Airdialogue: An environment for goal-oriented dialogue research. In EMNLP, pp. 1844-3854,, 2018
Da Yu, Huishuai Zhang, Wei Chen, and Tie- Yan Liu. Do not let privacy overbill utility: Gradient embedding perturbation for private learning. In ICLR, 2020.
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. Differentially private fine-tuning of language models. arXiv preprint arXiv:2110.06500 2021.
Tianwei Zhang, Zecheng He, and Ruby B Lee. obfuscation. arXiv e-prints, pp. a80i 2018
machine learning through data
12

---

Under review as a conference paper at ICLR 2022
Yuqing Zhu, Xiang Yu, Manmohan Chandraker, and Yu- Hi--Xinn Wang. Private-knn' Practical differen- tial privacy for computer vision. CVPR, pp. 11854-1 11854-1186,, 2020.
Alexander Ziller, Dmitrii Usynin, Rickmer Braren, Mareus Makowski, Daniel Rueckert, and Georgios Kaissis. Medical imaging deep learning with differential privacy. Scientific Reports, 11(1):1-8, 2021.
A ALGORITHM FOR THE TRAINING OF SEQPATE
The pseudo codes of ScqPATE's training is shown in Algorithm
Algorithm Training of SeqPATE
Require: Deni, DDPub datasets, GPT: pre-trained GPT model.
[ 1ar Im=1: M teacher models, fe: student model, fe: student model for self pre-training, {6" "o")" +- GPT, θ 4- GPT # Initialize teachers and the student for self pre-training.
3: GPT generates a pscudo dataset Druu hased on DDub
{Dir Im=1 f- Dim # Divide private dataset into m subsets.
for all m in MM do
6: Train teacher f on Dpri
7:
8: Teachers {0r Im=1 conduct inference on Brwh to get Pe (w, Win-1) required in Eq. 2 for all samples.
Train fe on Dowh # self pre-training for the student.
11: Initialize student model.
12:
13: while not converge do batch of samples [S] hatchsize in Boub
14:
Student fe conducts feed-forward on batcbtchizee
for all sample S in the batch {5) butchsize do
16:
for all token in sample S do
18:
Po(ar | (113-1) = 1
Xr >m P (w, w1n-1) Aggregate teachers' outputs
19:
Select only top-k or top-p predicted tokens as student's output.
Obtain L. and C as Eq. 2. Noise is added into L protect the privacy.
Get L by combining L. and LN Efficient knowledge distillation.
22:
end for end for
23:
Update e respect to L.
end for
26: end while
B DETAILED DEDUCTION OF THE SENSITIVITY IN SENTENCE LEVEL DP
We obtain Eq. 3 from Sec. 5.2.
M
A IIf(DD - (DD))l2 < |24 - -2y1la (
(2fe - 2'/1)
(3)
o=1
We know (ærjo )2 ) is smaller than |z,j - x'yul since larjo - xayylll (0, for each Hence, have,
1/2
V
1/2
1/2
Σ ( 22/1))
Σ
5
Σ
Tij s/yull
We know
6l a + when a, (0, 1), have,
1/2
[
|
1/2
1/2
Ø + 2'jul)
Σ
(1+1)
s V2.
x
jv
1=1
13

---

Under review as a conference paper at ICLR 2022
In summary, the upper bound of the sensitivity is,
2/70 = - S(D') 1(2)))) -1yl2 v2,
C DETAILS ABOUT THE DATASETS
The AirDialog dataset (Wei et al., 2018) consists of 402,038 dialogues and each dialogue consists of more than two utterances. We treat each utterance sentence in our sentence completion task. We obtain the Europarl_vó dataset from a machine translation benchmark 6 and we only use the monolingual English dataset with 2,015,440 raw sentences.
For the above datasets, we filter the short sentence with less than eight tokens. Then, the first four tokens act as the prefix, and the rest of the tokens acts as the output (ground-truth), We split each datasets into a private set DR and a public set Deuh. For the AirDialog dataset, the private set contains 0.95M/5K/50K samples for aaiining/aalidaaionnesstgg and the public set contains 40K/5K for training/validation. For the Europarl v6 dataset, the private set contains 1.72M/10K/50K samples for vaaiii/g/vviiaaiioneestngg and the public set contains 40K/5K for training/validation. The vocabulary size for the two datasets is set to 50K and we replace the tokens out of the vocabulary with a special token.
D DETAILS ABOUT THE EXPERIMENTAL SETTING
All the comparing algorithms use the same base model, the GPT-small, which has 12 stacked layers as mentioned in the original paper (Radford et al., 2018). The pre-trained GPT model comes from the official website All the methods are initialized by the pre-trained GPT model. We used the Adam optimization scheme (Kingma & Ba, 2015) and we adjust the initial learning rate for all the comparing methods with range of 0.01 ~ 0.00001, which means we conducted systematic search of the hyper- ppper-aaraetter space among all methods. We set the batch size as 32 for all the comparing baselines. We truncate the sentences with the maximal sentence length of 40. In the top-p strategy, the threshold μ is 0.95. The factor À balancing the KL loss and NLL loss mentioned in Eq. 2 is 20.
E DETAILS ABOUT THE EVALUATION METRICS
To measure the quality of generation, PPL is widely used metrics in text generation, especially language modeling, which measures the perplexity of generating the next token. B-n (Bleu-n) (Papineni et al., 2002) measures the degree of the n-gram matching between the generated results and the ground-truth text.
To measure the performance of privacy protection (we can also treat it as the privacy leak of the models), we aim to give an intuitive understanding the protection on text generation scenario (even if the factor in DP indicates the strength of privacy protection according to the DP theorem). Hence, we propose a new metric, P-N, indicating how many generated n- n-grams can be found in the private set Deri, We define Ny the set of unique n-grams in a generated sentence y and define ND as the set of unique n- n-grams in the private data DPri, As shown in Eq. 4, P-N is the ratio of the n-grams that occurs in the two sets over n-grams occurs only in Ny weighted by idf, where the idf is the inverse frequency of this n-gram indicating the sensitivity of this n-gram (Rare n-grams are likely to carry sensitive information, such SSN number and address). P-N ranges from 0 to and lower P-N indicates the more strong privacy protection (less privacy leak from the models).
For the choice of value N. we note that P-N with a very small value of N can hardly reflect the privacy loss since the uni-grams and bi-grams are very easy to be generated even if the model did not meet it from the private dataset Deri, For example, the N=1 does not make sense because only reflects the vocabulary overlapping between two datasets. Thus, we choose 3 and 4 as the value of N.
'The description is in www.statmt.org/eumpaartt the data comes from statmtt.rg/wmml 1/training- monolingual.tgz ggihub.com/ppenaignn22
14

---

Under review as a conference paper at ICLR 2022
AirDialog
PPL
Bluel
Blue2
Blue3
Blue4
Pri3
Pri4
#teacher=1
19.66
8.52
2.48
0.84
0.30
0.10
0.04
#teacher=10 10
18.29
8.27
2.38
0.81
0.29
0.14
0.06
#teacher=200
16.86
9.01
2.70
0.96
0.33
0.16
0.07
#teacher=2000
13.67
11.56
4.35
1.82
0.78
0.32
0.17
Table 4: Analysis about SeqPATE's performance with different teacher number on AirDialog dataset.
Europarl_v6
PPL
Bluel
Blue2
Blue3
Blue4
Pri3
Pri4
#teacher=1
44.64
12.33
3.64
1.11
0.40
0.56
0.32
#teacher=10
44.22
12.67
3.73
1.16
0.43
0.58
0.33
#teacher=200
43.55
13.10
4.06
1.29
0.49
0.60
0.36
#teacher=2000
37.80
13.64
4.40
1.43
0.55
0.69
0.45
Table 5: Analysis about SeqPATE's performance with different teacher number Europarl v6 dataset.
Ny = {n-gram E }
No {n-gram D
P-N EnENynN idfn EmENy idfm
ANALYSES ABOUT THE NUMBER OF TEACHER MODELS
Table 4 and Table show the analyses about SeqPATE's performance with different teacher number on our two datasets.
G THE COMPUTATIONAL COST OF SEQPATE
It seems that our model requires huge computational resources and a costly infrastructure to run. But in fact, our model able to train and inference on single GPU machine. In this section, will introduce some simple strategies we used in our implementation and also introduce the total computational cost of our model.
Memory usage and hard-disk space usage. Since method uses large number of teachers, the naive implementation of loading all teachers into memory for aggregation is impractical. However, note that algorithm only needs access each teacher's top-k prediction. Therefore, train teacher models sequentially. Once a teacher model trained, we obtain its top-k prediction (k=200 in our experiments) the public training data and save the results (i.e. i probabilities). Then, discard the teacher model. Finally, ScqPATE only needs teacher supervision on small number of samples. our experiments, training on 1~2k teacher labeled samples sufficient. Overall, saving teachers' inference results uses 16. The memory usage is similar to that of GPT2 model because do not load all teacher models into memory and instead run inference sequentially and merge teachers' predictions offline.
Training time. While we have a large number of teachers, each teacher is trained on only a small fraction of the entire dataset. Thus the time it takes to train all teachers is roughly equal to the time of training a single GPT2 model on the full dataset (of 1~2M samples our experiments).
In summary, with simple tricks, the teacher training and aggregation steps not much more expensive than training a GPT2 model. Compared to standard NLG model training, our algorithm does not require special hardware distributed learning.
15

---

Under review as a conference paper at ICLR 2022
H ADDITIONAL EXPERIMENTS ON LARGER AND LARGER Ō
We adopted stronger privacy protection c=2 and 8=1e-9 in Table 1, which results in a quite large utility gap between SeqPATE and the non-private training method (Pri-GPT) on the AirDialog dataset. Actually, our model performance can be increased a lot if we conduct weaker privacy protection (i.e. using larger and larger 8). In Table 6, we show the experimental results with c=3 and 8=1e-6 on AirDialog dataset.
Methods
PPL
Bluel
Blue2
Blue3
Blue4
Pub-GGPT+DDP-SGI+DDDra (==2, d=le-9)
17.65
7.90
2.69
1.14
0.56
Pub-GPT--DP-SGD+DD (==3, d=le-6)
13.87
10.24
4.95
2.83
1.74
SeqPATE(OUrs) (s=2, d=le-9)
13.67
11.56
4.35
1.82
0.78
SeqPATE(OOrs) (F=3, d=le-6)
9.94
13.55
7.82
4.43
2.54
Table 6: SeqPATE's performance with larger and larger on AirDialog dataset.
HANDLE LARGE E: USING FEWER TEACHERS
As shown in Fig. 2, when we use large e, our SeqPATE cannot obtain better performance than DP-SGD. As discussed in Sec. 6.2, the reason that the amount of training samples for each teacher model is quite small in our model (especially, when the teacher number is large). We note that this can be improved by using fewer teachers. As shown in Table 7, we conduct experiments on AirDialog dataset with fewer (i.e. 100) teachers and larger (i.e. 50 and 500), where SeqPATE outperforms DP-SGD. For DP-SGD, we choose Pub-GPT+ DP-SGD instead of Pub-GPT+DD-SGD+DDD since
Pub-GPT+ DP-SGD obtains higher performance when e is large.
Methods
# teacher
PPL
Bluel
Blue2
Blue3
Blue4
Pub-GPT+DP-SCE
50
11.21
9.68
4.45
2.43
1.43
SeqPATE(OUrs)
50
100
10.03
10.50
4.66
2.54
1.47
Pub-GPT+DP-SGD
500
10.86
9.81
4.52
2.49
1.45
SeqsATE(Ours)
500
100
9.86
15.82
8.15
4.64
2.75
Table 7: To handle the large using fewer teachers makes SeqPATE outperforms DP-SGD (on
AirDialog dataset).
J EVALUATING PRIVACY LEAK IN ANOTHER METRIC (EXPOSURE)
We have evaluated the performance of privacy protection in P-3 and P-4. We also conduct experiments on another metric, exposure, evaluate the privacy leak. The exposure is proposed by (Carlini et al., 2019). To calculate the score, we use three kinds of patterns "is <word>> are <word>> <digital>>" The repetitions are 100 for all patterns, and the lengths of sampled words or digits are three. The candidate words are randomly sampled from the whole vocabulary and the digits are randomly sampled from {0, 1..., 9}. We calculate the average exposure score over the three kinds of patterns. We follow the implementation of the exposure score from 8
Methods
AirDialog
EuroparL.v6
Pri-GP SeqPATE(Ours)
3.13
2.83
0,91
1,55
Table 8: Results on exposure of PriGPT and SeqPATE on the two datasets.
Table 8 shows the experimental results, where the results are consistent with the results on P-3 and P-4 mentioned in Table 1. For example, both metrics, the privacy leak in AirDialog is more serious than that in Europarl_ve and the gap between Pri-GPT and SeqPATE on AirDialog is larger than that on Europarl.v6.
16

---

Under review as a conference paper at ICLR 2022
K ADDITIONAL EXPERIMENTS WHEN THE DATASET IS SMALLER
To verify our performance on smaller datasets, we produce a small dataset by randomly sampling 50k samples from the AirDialog dataset. The experimental results shown in Table 9. In such a small dataset, our method still outperforms the DP-SGD with é 2 and 5 le-9.
Methods
PPL
Bluel
Blue2
Blue3
Blue4
DGGTTPD--SG++++ 25.93
7.64
2.26
0.87
0.23
SuqSATE(OOrS)
16.68
9.48
2.96
1.08
0.39
Table 9: The performance on a small dataset (50k samples from AirDialog dataset). All the methods
satisfy (2, -DP.
L THE CONTRIBUTION OF THE PSEUDO PUBLIC DATASET Duub
We conducted experiments to verify the contribution of pseudo dataset Drub to our task on AirDialog dataset. The results in Table 10 shows that using Drub can prompt the performance lot, where the promotion can found in both SeqPATE and DP-SGD methods.
Methods
Dataset
PPL
Bluel
Blue2
Blue3
Blue4
Pub- HibbGPTTDD--SD+DDM
w/ D
17.65
7.90
2.69
1.14
0.56
Pub-GPT-DP-SGD+DD
w/o Deub
18.47
7.17
1.95
0.66
0.24
SeusATE(OOrs))
w/ Drub
13.67
11.56
4.35
1.82
0.78
ScqPATE(OOrs)
w/o Deub
18.09
7.96
2.26
0.78
0.28
Table 10: The comparison between using and not using the pseudo dataset, Bpub
M THE ILLUSTRATION OF A RUNNING EXAMPLE.
Here, we will use example to show our training processing. In this example, the prefix from the public dataset Deub is want to book". We feed the prefix to pre-trained GPT2 to generate a pseudo sentence "I want to book flight from Tokyo to Hawaii.". The pseudo sentence will serve as an example in the pseudo public dataset Doub We feed the pseudo sentence to the teacher models to conduct the teacher inference and also feed it to the student model to conduct the feed-forward the student training. Teacher models output the probability distributions all words (10 words, in total) of the sentence. Then, we aggregate all teachers' probability distributions and add the calibrated noise the aggregated distributions. The student model also generates the corresponding probability distributions on those words and we conduct the efficient knowledge distillation and top-k or top-p selection the student's probability distributions. For example, if the student model can do well on the words ("I", "want", "book". "flight", ", and "Tokyo"), the student will only query the teachers' output distributions on the rest of words ("to", "from". "to", and "Hawai"") through the KL loss mentioned in 2. Besides, the student is always supervised by the NLL loss on the whole pseudo sentence ("I want to book flight from Tokyo to Hawaii."). Finally, the student model conducts mack-propagation according to the above losses.
17


Reviews:
[{"review_id": "XN1TskBXRdf", "reviewer": "Reviewer_pnSu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: This paper is well-written and technically sound. Protecting text data is important given there is a rich body of work on attacks for such datasets; however, the protection technique that can provide provable DP guarantees is somewhat understudied. This is a timely and interesting topic and has many applications, for example, the smart compose. The approach is clearly stated, and the empirical evaluation is sufficient. In terms of novelty, SeqPATE has several non-trivial changes compared to the classical PATE. \n\nWeaknesses: I have a few minor comments.\n\n1. When \\eps is large, DPSGD outperforms SeqPATE. In practice, the choice of \\eps depends on the data, so could you offer any instructions on choosing the algorithm when \\eps is not very small (like between 5-10, which is also common in practice)?\n\n2. I'm confused about how you define P-n and why it can be considered as a privacy metric?\n\n3. In Section 4.3, the sentence `` Eq. 1 shows the top-k selection at i-th step ...'' is confusing. Eq. 1 only shows the renormalization of the probabilities. \n", "summary_of_the_paper": "This paper proposes SeqPATE that adapts PATE to text generation while satisfying differential privacy. To overcome the challenge of obtaining sequence-level supervision for text generation, it first generates pseudo inputs by completing the prefix using GPT-2 (a pre-trained language model) and thus reduces the problem to the next word prediction. Then it applies the PATE framework to privately train a student model with several algorithmic innovations, including (1) aggregating teachers' models by averaging their output distributions; (2) reducing output space by top-k/top-p selection; (3) efficient knowledge distillation by only querying the teachers' models when the student model performs poorly. (4) using both public labels and teachers' aggregation to supervise the learning. The paper gives privacy analysis and also shows it can offer privacy guarantees on user-level and secret n-grams. The paper also provides extensive experimental results and shows that it outperforms other DP learning algorithms when the \\eps is small (<5). The experiments demonstrate the practicality of the algorithm.", "summary_of_the_review": "Overall, I think it's a strong paper. The topic is timely and interesting. The writing is good, and the approach is clearly presented. The proposed approach contains several non-trivial changes compared to PATE. The paper provides extensive empirical evaluation, which demonstrates the practicality of SeqPATE. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "W71Y7i2Fmjw", "reviewer": "Reviewer_wFLx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper provides results in the very interesting direction of private language models. I believe that, given the recent progress in non-private language models, the time is ripe to further investigate this area privately, which has been sorely under-explored. As a result, I commend the authors for taking steps in this direction.\n\nFirst, I will evaluate the algorithmic contributions. As I enumerated above, there are a few modifications to the basic PATE setting. Of these, I believe using the student model to reduce the support size is an interesting idea. Using a pre-trained GPT-2 model to extend the training data is also kind of nice, though it is not 100% clear why this is necessary in the experimental settings considered. In the example provided, I understand: \"Cats sit\" is too short a sentence to perform meaningful training from. However, the datasets used seem to have much larger utterances (indeed, it says in Appendix C that sentences less than 8 tokens long were filtered out). The other two modifications seem relatively straightforward: it is of course natural to consider using multiple prefixes of the same pseudo sentence, rather than performing predictions sequentially. And performing PATE on the output distribution rather than the output prediction is also natural (I would be surprised if it has not been done before). I believe it may also be inspired directly from knowledge distillation (which is uncited and uncredited). Overall, the algorithmic modifications are nice, but nothing offering significantly new insights into the nature of natural language generation.\n\nNext, I will comment on the experimental results. The utility metrics provided in Table 1 seem to indicate that the proposed method performs somewhat better than the previous methods which are private with respect to the sensitive dataset. However, the utility is still very far from the non-private utility. Perhaps this is to be expected, given the general cost of guaranteeing DP, but other recent results seem to show how effective pre-trained models are in terms of maintaining utility even with DP -- see, for example, Tramer-Boneh (https://arxiv.org/abs/2011.11660), which demonstrates the efficacy of transfer learning to image classification (resulting in relatively comparably small drops to the accuracy versus no transfer learning), as well as two recent arXiv papers on the same topic (https://arxiv.org/abs/2110.06500 and https://arxiv.org/abs/2110.05679) which achieve relatively small loss in utility compared to the non-private pre-trained model. Of course the latter two works appeared only after the deadline and thus were impossible to compare with, so perhaps one should disregard them, but the story of both papers seem to be at odds, which is puzzling to me. Regardless, the [TB20] paper still indicates the significant power of transfer learning for private learning). Additionally, I noticed that experiments were only performed on two relatively large datasets. This begs the question of what happens when the datasets are relatively small. Lack of accuracy on smaller datasets may be why the authors did not evaluate on more common GEM benchmarks. Comparison with prior methods seem a little mixed. While SeqPATE indeed outperforms previous methods in various regimes, it is somewhat inconclusive at eps = 5, which is closer to the regime usually considered in DP ML papers. As a final note, my impression is that this is an incredibly costly infrastructure to run. I believe this uses k = 2000 teachers, each of which is GPT-2 Small, resulting in an overall model which is comparable in number of parameters to the full GPT-3. This sounds impractical to run in most settings, and this matter is not discussed at all in the paper. Overall, the experimental results are modest, but given this discussion, I am not sure if they mark a qualitatively impressive shift in our understanding of the problem (which, as I mentioned before, is seemingly quite under-explored).\n\nMy final major comment is on the writing. The quality of the language is rather poor, with issues in almost every sentence, to the effect that the final paper feels very rushed. I tried to be charitable to the authors and fill in the gaps intelligently, but there were certain sentences which were incomprehensible in their current state. Beyond this, other (comparatively minor) writing issues include insufficient discussion of related work, incorrect citations, and unusual choice of content in the body. I list some of these issues below. Overall, presentation issues in the submitted version are to the level where I would not recommend anyone to read this until it is better polished.\n\nFurther points and comments:\n- The citation for differential privacy is Dwork, McSherry, Nissim, and Smith, TCC 2006.\n- Bassily, Smith, Thakurta, FOCS 2014 should be cited as another reference for DPSGD.\n- As mentioned above, what happens if one just uses D^pub instead of ~D^pub? Why is this insufficient for the datasets used in the experiments? This question is true for many of the approaches studied, not just SeqPATE.\n- The term knowledge distillation (KD) is used extensively in the paper. As I am sure the authors are aware, KD refers to a specific technique for training from a teacher. Is the KD used in the paper the same as KD as used in the literature? If so, then it should be cited and discussed (or otherwise, contrasted with). If it is different, then the term KD should be changed.\n- It may be helpful for the presentation to keep going with the running example demonstrated in Figure 1, and how the procedure would work with this concrete instance.\n- Although I am not too concerned with what appears in the body of the paper versus the appendix, some choices that I found unusual include: basics of DP and Equation (1), which may be immediate or already known to most readers.\n- It is not clear what bold and underlined numbers in Table 1 are meant to represent.\n- I am curious what happens for the non SeqPATE methods if one uses D^pub instead of ~D^pub. It feels like you're cheating if you're just using the model to generate more training data, and it may or may not have an effect.\n- The choice of P-3 or P-4 is not clear as a privacy metric. It seems to say something more about the nature of the dataset. See in particular how these numbers differ significantly between AirDialog and Europarl_v6. The canary approach used in the Carlini et al Secret Sharer paper seems more meaningful to me. \n- Broken reference in Appendix D\n- It appears that footnote 7 was never filled out appropriately\n- It seems like more teachers result in monotonically better utility. What is the limit of this phenomenon?\n", "summary_of_the_paper": "This paper proposes a variant of PATE (from ICLR 2018), a method intended for standard supervised learning tasks. The present work instead focuses on natural language generation, an inherently sequential task, which introduces new challenges. Relative to PATE (and assuming knowledge of it), some differences in the method involve: extending the public data sentences into \"pseudo sentences\" using GPT-2, simultaneously performing training on all prefixes of the pseudo sentences, averaging the teacher model output distributions rather than aggregating their max score (i.e., their prediction), and using the student model to reduce the support size for said output distributions. Experimental evaluation is provided for two datasets, AirDialog and Europarl_v6. Perplexity and BLEU scores improve upon those which do not use the private data, and those which use previous methods with the private data.\n", "summary_of_the_review": "A nice direction for investigation. But the approach and findings don't appear significant enough to warrant acceptance in ICLR right now. The paper is further hampered by the very poor presentation.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AUcOnQJwD8I", "reviewer": "Reviewer_386M", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: I think the paper does a good job of adapting PATE to the text generation framework and making it work reasonably well. Given that PATE is a very general framework that works for any learning algorithm and not just deep learning, it is useful to have adaptations of it to sequence/text generation settings.\n\nWeaknesses: \n\nI am not fully convinced by the authors claim that their adaptation of PATE beats DPSGD. \n\n(1) The authors cite Kerrigan et al (2020) to claim that DPSGD doesn't work for text-generation using large models like GPT2 due to high dimensionality of the parameter space. This is debunked conclusively in the two recent papers by Li et al 2021 (https://arxiv.org/abs/2110.05679) and Yu et al 2021 (https://arxiv.org/abs/2110.06500). Li et al (2021) show that by careful choice of hyperparameters, training pretrained GPT2 on private datasets using DPSGD achieves performance comparable to non-private training. We also note that these are very recent and possibly concurrent with this work. But nevertheless, the authors should not be coming to an opposite conclusion as these papers.\n\n(2) I am also not convinced by the experiments in Table 1. The authors compare their approach to the baseline DPSGD+$\\tilde{D}^{pub}$. This means that, from what I understand, a model is trained from scratch on private data $D^{priv}$ using DPSGD and then fine-tuned on $\\tilde{D}^{pub}$. I think that the correct baseline to compete against is Pub-GPT + $\\tilde{D}^{pub}$ + DPSGD on $D^{priv}$ in that order. That is we are fine-tuning the pretrained GPT2 on $\\tilde{D}^{pub}$ and then fine-tuning on $D^{priv}$ using DPSGD. I will be more convinced that their framework works if they compare against this benchmark. \n\n(3) Also to claim that, their method is beating DPSGD, they have to demonstrate they did a systematic search of the hyperparameter space for DPSGD. But I didn't find any evidence of this in the paper. The importance of choosing right hyperparameters for DPSGD is demonstrated in the work of Li et al (2021). \n\nOther remarks: Is the grey line in Figure 2, Pub-GPT + $\\tilde{D}^{pub}$? Is it a typo?\n", "summary_of_the_paper": "This paper tries to adapt the PATE (Private Aggregation of Teacher Ensembles) framework for DP training of machine learning models to the setting of text generation. There are some significant challenges in adapting PATE to this framework.  PATE as originally proposed is designed to work for classification tasks with a small number of labels. Text generation or next word prediction has too many labels and a direct application of PATE doesn't work. This paper solves this issue by aggregating only top-k predictions of each teacher where the top-k is computed using the prediction of the student on public data and so doesn't leak extra privacy. The paper also crucially uses public data and large pretrained models like GPT2 to further improve the performance of DP learning using PATE on private data.\n\nThey claim that their approach improves upon DPSGD/Noisy SGD (which is the standard DP algorithm for deep learning) significantly on some standard benchmarks.", "summary_of_the_review": "Because I am not fully convinced the by the claims of the authors, I rate it marginally below acceptance threshold. I am willing to update my score based on further evidence by the authors in support of their claims.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "APdZbRppfBm", "reviewer": "Reviewer_CVpe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths:\n1. Training a DP language model is a less explored problem, and the authors provide a technically sound solution along with a well-setup empirical evaluation. \n2. The paper is overall well written and the method is easy to follow. \n3. The authors propose a new evaluation metric P-N to indicate how many generated n-grams can be found in the private set, which is an interesting and useful indicator in addition to PPL on the private set.\n4. The authors also provide an interesting discussion on user-level DP and n-grams-level DP for language models.\n\n## Weaknesses:\n1. My main concern of this paper is whether SeqPATE can achieve satisfactory utility preservation given small epsilon (say epsilon=2). From Table 1, the B-4 score of SeqPATE is different from that of the upper bound by 2 orders of magnitude, which is a bit concerning. Moreover, the PPL gap between the upper bound and SeqPATE (13.67 - 3.88) is much larger than the gap between the lower bound and SeqPATE (19.39 - 13.67), which suggests that SeqPATE may barely learn useful information on the private data domain. \n2. The advantage of SeqPATE vanished given a large epsilon compared with DP-SGD, which makes the algorithm less scalable. Although the authors explain it is due to the partitioning of the training data, could this issue be resolved if reducing the number of teachers?\n3. The authors discuss user-level DP and n-grams-level DP. It is better to provide some quantitative evaluation on SeqPATE on these settings.\n4. Some experimental setups are a bit unclear. it is better to include more experimental details in the main paper, e.g., how the hyper-parameters are chosen (say #/ teachers) and the computational overhead of SeqPATE and DP-SGD.\n\n## Additional questions:\n1. I see the paper is adopting a strict DP standard (delta=1e-9). Could raising the delta to 1e-6 or 1e-5 can help improve the utility of SeqPATE?\n2. Why Pub-GPT + \\tilde{D}^{pub} can significantly improve (the largest PPL improvement) the PPL of Pub-GPT, considering both methods use the public records only. This result is a bit weird and counterintuitive. \n\nI am willing to raise my scores if the problems above can be well addressed.\n", "summary_of_the_paper": "This paper proposes to protect the privacy of text generation models, which may leak sensitive information of the training data. Specifically, the authors propose SeqPATE, which applies the PATE framework to large-scale language models such as GPT. By reducing the output space and aggregating the supervision from teacher models, SeqPATE can help reduce the privacy cost. Empirical evaluation on two private text corpus shows that SeqPATE achieves a good trade-off between utility and privacy.\n", "summary_of_the_review": "Strengths:\n1. Training a DP language model is a less explored problem, and the authors provide a technically sound solution along with a well-setup empirical evaluation. \n2. The paper is overall well written and the method is easy to follow. \n3. The authors propose a new evaluation metric P-N to indicate how many generated n-grams can be found in the private set, which is an interesting and useful indicator in addition to PPL on the private set.\n4. The authors also provide an interesting discussion on user-level DP and n-grams-level DP for language models.\n\nWeaknesses:\n1. My main concern of this paper is whether SeqPATE can achieve satisfactory utility preservation given small epsilon (say epsilon=2). \n2. The advantage of SeqPATE vanished given a large epsilon compared with DP-SGD, which makes the algorithm less scalable. \n3. The authors discuss user-level DP and n-grams-level DP. It is better to provide some quantitative evaluation on SeqPATE on these settings.\n4. Some experimental setups are a bit unclear. it is better to include more experimental details in the main paper, e.g., how the hyper-parameters are chosen (say #/ teachers) and the computational overhead of SeqPATE and DP-SGD.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This was a very borderline decision. Here are the major factors involved in the decision. 

1. The concurrent works by Li et al and Yu et al. It is unclear about the relationship/strength between those results and the ones in the present paper. However, in accordance with the ICLR policy on simultaneous work, we ignore them to the extent possible.
2. The novelty of the approach. All reviewers agreed that the modifications to the PATE approach are fairly minor or incremental, and this appears to be largely an application of this method. That said, methods for the natural language setting are important.
3. The strength of the findings. Reviewers were mixed on the strength of the results: they appeared significant in some cases but rather lackluster in others. 
4. The poor quality of the writing. I personally read the original version and found the writing to make it impossible to understand in parts. The writing is still subpar in many places, which I would have expected the authors to fully polish given the substantial time during the response period. 

While in isolation, any of 2, 3, and 4 may be acceptable, their combination makes it difficult to recommend this paper for acceptance at this time.
Decision: Reject