SeqPATE: Differentially Private Text Generation via Knowledge Distillation
Authors: Zhiliang Tian
Yingxiu Zhao
Ziyue Huang
Yu-Xiang Wang
Nevin Zhang
He He
Venue: NeurIPS 2022 Accept (2022)
Keywords: Natural Language Generation
Text Generation
Privacy Protection
Differential Privacy

Abstract: Protecting the privacy of user data is crucial for text generation models, which can leak sensitive information during generation. Differentially private (DP) learning methods provide guarantees against identifying the existence of a training sample from model outputs. PATE is a recent DP learning algorithm that achieves high utility with strong privacy protection on training samples. However, text generation models output tokens sequentially in a large output space; the classic PATE algorithm is not customized for this setting. Furthermore, PATE works well to protect sample-level privacy, but is not designed to protect phrases in samples. In this paper, we propose SeqPATE, an extension of PATE to text generation that protects the privacy of individual training samples and sensitive phrases in training data. To adapt PATE to text generation, we generate pseudo-contexts and reduce the sequence generation problem to a next-word prediction problem. To handle the large output space, we propose a candidate filtering strategy to dynamically reduce the output space, and refine the teacher aggregation of PATE to avoid low agreement due to voting for a large number of candidates. To further reduce privacy losses, we use knowledge distillation to reduce the number of teacher queries. The experiments verify the effectiveness of SeqPATE in protecting both training samples and sensitive phrases.

SeqPATE: Differentially Private Text Generation via Knowledge Distillation
Zhiliang Tian'; Yingxiu Zhao', Ziyue Huang', Yu-Xiang Wang ', Nevin L. Zhang?, He He"
National University of Defense Technology,
2 The Hong Kong University of Science and Technology,
UC Santa Barbara,
New York University
(lanzhilianghi@@@ee com, yzhaocx@connect ust hk, zyhuang940gmail. com, yuxiangu@cs ucsb. edu, 1zhang@cse. ust.hk, hhe@nyu. edu
Abstract
Protecting the privacy of user data crucial for text generation models, which can leak sensitive information during generation. Differentially private (DP) learning methods provide guarantees against identifying the existence training sample from model outputs. PATE is recent DP learning algorithm that achieves high utility with strong privacy protection on training samples. However, text genera- tion models output tokens sequentially in large output space; the classic PATE algorithm is not customized for this setting. Furthermore, PATE works well to protect sample-level privacy, but is not designed protect phrases in samples. In this paper, we propose SeqPATE, an extension PATE to text generation that protects the privacy of individual training samples and sensitive phrases in training data. To adapt PATE to text generation, we generate pseudo-contexts and reduce the sequence generation problem
next-word prediction problem. To handle the large output space, we propose candidate filtering strategy to dynamically reduce the output space, and refine the teacher aggregation of PATE to avoid low agreement due to voting for a large number of candidates. To further reduce privacy losses, we use knowledge distillation to reduce number of teacher queries. The experiments verify the effectiveness of SeqPATE in protecting both training samples and sensitive phrases.
Introduction
Recent work has shown that sensitive user information in training corpora, such as addresses and names, can be extracted from text generation models [6]. Providing privacy guarantees to the training corpora of text generation models has become critical problem. Differential privacy (DP) provides provable guarantees against detecting individuals in datasets. Deep learning models with guarantees ensure that the existence of specific training sample cannot be detected.
is popular DP algorithm for deep learning that adds noise to the gradients. PATE 31 is another type of DP learning algorithm that transfers knowledge from teachers trained on private data to a student model, where noises are added to teacher predictions to satisfy DP. PATE model-agnostic, and its privacy cost derives from the knowledge distillation process instead of the model gradients in NoiDySGD (42 24]. Therefore, the noises required by PATE do not scale with model size. Given this benefit, PATE great potential for text generation, since large language
NoisySGD
'This paper was partially done when Zhiliang Tian was a Ph.D. student at HKUST and visiting scholar at
NYU,
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

models (e.g.. GPT-2 (35) have become the backbone of most text generation models. However, and not customized protect 39 50 Additionally, PATE, originally sensitive phrases in the data with a low privacy cost [22
NoisySGD and PATE are used to protect sample-level privacy [5]
24
designed for classification tasks, is not customized for sequential generation on large output space (i.e., the natural language vocabulary), which is very common in text generation.
In this paper, we propose SeqPATE, a DP learning algorithm for text generation to protect the privacy of training corpora. By satisfying DP, SeqPATE has the guarantee of preventing the existence of training samples and sensitive phrases in the training corpora from being detected. Similarly to PATE, SeqPATE employs teacher-student framework: (i) a student model learns to generate text from non- sensitive samples; and (ii) a number of teacher models, trained on sensitive text, supervise the student through noised outputs of aggregated teachers. The calibrated noise added to the output ensures that SeqPATE satisfies the DP requirements. This framework still faces some challenges in text generation. First, it suffers from the high costs of GPU memory and time. To obtain sentence-level supervision for text generation, the model needs to roll out all teachers to produce a sentence (i.e. all teachers vote to generate word, which is then used as the input for the next word prediction). It results in high inference cost with a large number of teachers (e.g. 2k teachers which are common in PATE), Second, the large output space (i.e., the vocabulary) in text generation leads to (i) low agreement rates among teachers and (ii) large noises required by DP, both of which significantly hurt the task performance.
To address the challenges, we generate pseudo-data using a pre-trained language model so that teachers only need to provide token-level supervision given the pseudo inputs. To handle the large output space and reduce the noise, we propose to dynamically filter the candidate words and select only words with high probabilities. Also, we aggregate teachers' outputs by interpolating their output distributions instead of voting with argmax predictions. DP learning methods provide privacy protection by adding noise, which also reduces the utility of the model. To reduce utility loss, we avoid unnecessary knowledge distillation by selectively applying knowledge distillation to generation steps where the student struggles. Most DP learning methods, including SeqPATE, prevent samples from being extracted. SeqPATE has further advantages in protecting users' secret phrases that occur multiple times in the corpora. We evaluate SeqPATE demonstrates its advantage in protecting samples and phrases compared to the baselines.
sentence completion task, which
Our contribution is twofold: (i) We propose SeqPATE that provides privacy at both the sample level and the phrase level with theoretical analyses. (ii) We propose several strategies for SeqPATE to handle autoregressive text generation models with large vocabulary.
2 Problem Setup
Our goal is to achieve the privacy protection quantified by DP in text generation to prevent attackers from inferring whether a sample or n-gram appears in the training set. Our setting contains two types of textual datasets: (1) a private set Drn from a corpus with sensitive information, (2) a public set Druh that contains no sensitive information or comes from data contributors (e.g., volunteers) who have no objection to publishing their data. We aim to protect the privacy on the private set and can ignore the privacy protection on the public set.
Our application, sentence completion, aims to complete the whole sentence given the prefix. We train a language model to accomplish the task. The public set DDob consists of prefixes, which can hardly contain sensitive information. The private set DPn consists of whole sentences. Such a setting fits some real-world text generation applications: in dialog systems, the training samples from online services consist of questions and responses. The questions from customer service staff or service robots can be public, and the response from users carrying individual information should be private.
3 Background on DP and PATE
Definition 3.1. [Differential privacy (DP) B [4] For any two neighboring datasets D, D' (differ in
only one individual), randomized algorithm M n y is (e, differreetilll private if,
Pr[M(D) e $) s e -PrM((DDD)) $ 6 VS y,, where <>0 0, 8 2 0.
(I)
2

---

word
Teucher Inferros
Student Training
Pleaato
ikll hrat lilt hard.
Swttrrled (no need teather superds ons]
0.33
satisfed (need toacher supervisions
Teachort
Lay 0.10
Ouiput word bution
Akil Ted
Ground tnuth strudent
Output disiiturlors abudent worls
Hal ial
Ape ragae
Fificiert Kresweedge Datitstoon
at.. idl.
11.. Inl.
PMBL 169
Cas
billd Iniii
Convicsio
aal ball
lalal Maa
Piloring
Teacher Training
Student
NIL
Cats
(88)
on
the
Cura
lencher!
Teacher?
Seacher'
Cats sit on the mons
Darai
Dona2
Dasa3
Pre toinnd Model
Privade Dala
Public Dara
Figure I: Overview of SeqPATE. SeqPATE trains teachers on private data. Student models are trained on pseudo-sentences generated by a pre-trained language model given the public prefixes. The student is supervised by aggregated teacher output distributions. SeqPATE benefits from candidate filtering (white block in the top right corner) and efficient knowledge distillation that determines whether teacher supervision is needed (pink block).
By definition, DP quantifiable definition of privacy that provides guarantees on identifications of individual data (preventing an adversary from inferring whether the input is 'D or DD). MI. models with DP ensure that each training sample has a degree of plausible deniability, i.e., the trained model is just as likely to be trained on an alternative dataset without that sample. In SeqPATE, .M is the entire training and inference process, is the vocabulary, and Pr[-] denotes the output distribution of generating word. Attackers cannot tell whether sample is in the training set or not, since the output distributions of the datasets with or without that sample are very similar (bounded by Eq, [
PATE 311, designed for classification tasks, takes advantage of an unlabeled public dataset Druhh and also trains on labeled private set Drd in a supervsper scenario. PATE achieves DP through a teacher-studen framework with M teacher models and student model, where the student learns from the private set via knowledge distillation through teachers. PATE has three parts: (i) The teacher models are trained on the private set Deri, which is shuffled and divided into M disjoint subsets. Each teacher trained on one subset. (ii) Teacher aggregation merges the teachers' outputs. Each of the trained teachers then provides supervision to the student's unlabeled public set Drab We use noised majority votes from teachers as labels to supervise the student. (iii) A student model is trained on the public set Deub with the supervision of the aggregated teachers.
4 Approach
Fig. [ shows an overview of SeqPATE. Given the public prefix (e.g., "Cats sit"), we first obtain the pseudo-inputs by completing the sentence (e.g.. "Cats sit the mats")) using pre-trained language model (Sec.
At each word, we then aggregate the teachers' prediction of the next word as supervision for training the student model (Sec. 4.2 To reduce the noise required by DP for large output space of the size of the vocabulary, we reduce the output space by dynamically filtering unimportant words. To reduce the number of teacher queries that incur privacy losses, propose an efficient knowledge distillation strategy that only queries teacher labels on uncertain examples (Sec. 4.3
We show the training algorithm in App. B and running example in App. K.
4.1 Pseudo Input Generation
Conventional text generation models generate words sequentially from left to right. Thus, naively applying PATE to text generation requires rolling out all teachers word by word, i.e., iteratively sampling the next word from the aggregated teacher prediction. This costly both computation (running inference for hundreds of teacher models) and privacy costs (querying teachers every step).
3

---

To tackle this challenge, we use pre-trained language model to complete the public prefixes into pseudo sentences; thus, we only need query teachers on the next word given a (pseudo) context.
4.2 Teacher Aggregation
PATE aggregates teacher predictions by majority vote. While it works for classification problems with a relatively small number of classes, the output space of text generation models contains all words in the vocabulary. As a result, the number of votes for each candidate word may be very low without clear winner. For example, multiple candidates may tie for the top-1 prediction.
Inspired by Chen et al. 9 [7], wc aggregate teacher results by averaging their output distributions. We first train M teacher models disjoint subsets of the private data. To produce the aggregated next word distribution given a context c, we average the teachers' output distributions, add calibrated noises, and then renormalize the results into proper distribution. Following Papernot et al. [32), we apply the Gaussian mechanism. Formally, let p (- c) be the prediction of the m-th teacher. The
1 ΣM (pT (- (c) +V (((0,)"), o²)). 2 where the Gaussian noise aggregated distribution is Page( cc) x 37 added to the aggregated output distribution. The way of SeqPATE satishie DP guarantee (Eq. D is to add that calibrated noise to the teachers' output as mentioned above (detailed analyses in Sec. 5).
4.3 Training of the Student Model
The student model is trained public pseudo-data and also supervised by the aggregated teachers.
Training objectives. The student model a language model that predicts the next word given prior contexts. Given contexts from the (public) pseudo-data autocompleted by pre-trained language model (GPT-2), the student supervised by both the aggregated teacher predictions and the next word in the pseudo-data (i.e. pseudo label). The pseudo-data acts prior for the student given that the number of teacher queries is limited due to privacy concerns. The student's loss function has two parts:
Liaaher denotes the loss with respect to teacher supervision. Note that the aggregated teacher output is distribution over words. Therefore, minimize the forward KI. divergence between the aggregated teacher distribution Page and the student output distribution po:
Creacher (c. Pags) KL (Pags(
c) I| po(: cc)..
(2)
Epseuto denotes the loss with respect to the pscudo-labels from Drub (i.e. next words generated by generic language model). Similar to standard language modeling, we use the negative log- ligliiielooodd:
Cpsccdo (c, w) = log (o c).
(3)
Eq. shows the complete loss. (A balances the two terms and we discuss the noise scale a in Sec. 5
C(Pagg Doub)
Σ
Lpecudo (c, w) + ALuaacher ( (C, Pagg):
(4)
E
Reducing the output space via candidate filtering. The high-dimensionality of the output of text generation models results in large noise (which is added to each coordinate). To reduce the output dimension (hence the amount of noise), we filter words on the tail of the distribution of the student model (i.e. set their probability to zero), and renormalize the teacher's 's aggregated distribution and the student output distribution over the rest words.
Note that the candidate filtering is based on the student's outputs on public or already released inputs, thus it does not affect the privacy guarantee. This choice improves the privacy-utilin tradeoff by adaptively allocating the privacy budget to release the information most helpful to the task.
We experiment with two filtering strategies: top-k and top-p. In top-k filtering, we retain only the top-k most likely candidates and filter the rest according to the student model. In top-p filtering [8].
"atteemttiaalyy the aggregated distribution with noises may be negative. If so, we renormalize the negative value to 0. Practically, we observed that being negative is an extremely rare event, since the M is usually very large (e.g.. g.. 2k) and the first term dominates the above equation.

---

k is chosen dynamically such that the top-k words are the minimum set whose cumulative probability at least p. The strategy seldom loses good candidates because the student usually does well on top-k predictions since the beginning of the training.
Reducing the number of teacher queries via efficient knowledge distillation. While the aggre- gated teacher model satisfies DP, each query from the student incurs some privacy loss. Therefore, we obtain teacher supervision only on "hard" examples when training the student. Note that the student trained on both the pseudo-data and local supervision from the teachers. We consider an example to be hard if the student cannot imitate the pseudo-label, in which case distilling knowledge from the teachers that are trained on large private data is helpful.
Concretely, we query teachers only when the rank of the pseudo-label is below a certain threshold among words ordered by descending probabilities under the student model. we query the teachers, the student is trained via complete loss [(Prg: DPub) (Eq, 44: otherwise, the student is trained via the fipeeddo (Eq. 3). We note that the selection of tokens relies only on the student and is independent of the teachers; thus, the selection does not cause any additional privacy loss.
Privacy Analyses
5.1 Preliminary of Differential Privacy
Lemma 5.1 (Analytical Gaussian mechanism (2). For numeric query f X" 4 Ra over
dataset D, the randomized algorithm that output f(D) + Z where Z ~ NO0 Ia) satisfies
(e, (e) for all 2 0 and ö(s) 2 )) ( 20 x) where A A
maxp-D (D) f(D') )ll is the global L2 sensitivity of f and t is the CDF function of (0,
We can use the same result for an adaptive composition of a sequence of Gaussian mechanisms.
Gaussian mechanisms [I). The adaptive composition of a sequence of Gaussian mechanisms with noise level a .2.... and global L2 sensitivity A1, A2... satisfies (c, (e)(())) for all ε 2 0 and d(e) S 8m(E) where M is a Gaussian mechanism with noise multiplier a / (A.)). -1/2
Lemma 5.2 (Composition
1/2
Specifically, the adaptive composition of k identical Gaussian mechanism with a noise multiplier satisfies the same privacy guarantee of single Gaussian mechanism with noise multiplier a/ Vi. By fixing k and we can calibrate the noise by choosing an appropriate a in Sec. 4.2
5.2 Differential Privacy for Language Models at the Sample Level
Recall that we partition the private dataset into M disjoint subsets, and train each teacher model on one of the subsets. Let vector x; E [VI denote the probability distribution predicted by the i-th teacher model given some context, where [ is the vocabulary size. The aggregation function f(D) := EM Hy the sum of the probability distributions predicted by all teachers. Since the datasets are disjoint, changing one sample affects only one teacher model. For neighboring datasets D, D', let denote the index of each teacher model; the probability distributions 2,y and x² (derived
from D and D' respectively) are different. Then, the sensitivity A in Lemma 5.1 & (See detailed deductions in App. C).
A = A [I((D) - f(D')ll S ||2, - 2 S 22.
Adding the noises given by Lemma 5.2 to each coordinate (each candidate at each generation step
of SeqPATE) preserves (s. DDP for fD). Finally, when we extract top-k coordinates by top-k candidate filtering (Sec. 4.3). the privacy guarantee also holds due to the post-processing property [14). Therefore, the fact about whether a sample is SeqPATE's private sets is protected (satisfying (e. 6(E))-DP).
'In the first 10 training batches. the top-50 predictions of the student cover 94% "true" labels of pseudo samples.
5

---

5.3 Differential Privacy of Users' Secret Phrases
The above analyses show that we can protect the privacy of each sample (i.e., one occurrence of sentence). However, in practice, we may want protect all occurrences of some secret phrases specific to a user (e.g., names and addresses) R
Consider a secret phrase s that occurs ng times (n, 1) in the private set. According to group privacy [14], the protection on phrase satisfies (ne, - e-1 8) -DP [22], where the privacy loss scales linearly with the number of occurrences of (We discuss and analyze better strategy to reduce the privacy loss of baselines in App. M).
Naively applying a DP algorithm requires larger noise to protect phrases that may occur multiple times. SeqPATE enjoys a stronger guarantee by assigning all data of a single user to one or a few teachers, such that any user-specific phrase occurs in the training data of only one or few teachers. We denote n, as the number of teachers whose data contain the phrase 8. Since adding or removing the phrase affects only n. teachers (ñ, is usually 1 or 2) and thus results in sensitivity of v2n, (See App. for details). In this way, the strength of protection on secret phrases roughly equal to that we have derived for sample-level DP. The exact (e, 8(e, f.)) for the phrase can be obtained according to Lemma 5.1 & 5.2 where ó(c, n..) Φ n other generic DP algorithms such NoisySGD, SeqPATE avoids linear increase in privacy loss (i.e., a linear increase in 8) on user phrases by careful partitioning of the private data.
50 /2n v e ((
Rg /2n .2n,
co Unlike
This effect is complimentary to other generic, but more intrusive, techniques such redaction and deduplication [50] for addressing the same issue. Finally, a user-specific partitioning with SeqPATE also protects multiple secret phrases of the same user (e.g.. combination of SSN, credit card numbers, address, day of jointly without incurring a larger privacy loss a benefit that deduplication does not provide.
5.4 How does DP prevent memorization in SeqPATE?
In practice. the privacy of the language model is usually interpreted as not generating secret phrase in the training data as-is during inference. Thus, one may wonder how DP prevents such unintended memorization of the training data. We remark that the protection against memorization follows the definition of DP. Consider the attack by Carlini al. (6, which uses a language model to predict a secret phrase given prefix. By the closure to post-processing [14), the prediction also satisfies DP. We denote W the undesirable event where SeqPATE generates the phrase s verbatim. The DP definition implies that the probability of W to happen when is in the SeqPATE's private sets is at most ef larger than the probability of an alternative SeqPATE model trained without s those sets. The chances for the latter model to generate text with are astronomically small. Hence, DP implies that the probability of W under the former model (i.e. any SeqPATE model in general) is small.
Experiments
6.1 Experimental Settings
Datasets. We evaluate our model on two datasets. AirDialog [47 consists of 1M utterances from customer service dialog flight booking: Europarl_ v6 consists of 2M English sentences collected from European Parliament (See details about datasets in App. E.)
Baselines. We compare SeqPATE with two DP baselines: (1) standard NoisySGD trained on the
private data with calibrated noise on clipped gradients 2 and further trained on public set D without protection; (2) based on NoisySGD, NoisySGD+GC 24 applies a ghost clipping which enables large batch size with memory saving techniques.
Additionally, we use two non-DP methods as reference: (1) Pri-GPT trained on the private set without any privacy protection; (2) the public pre-trained GPT-2 model Pub-GPT without access private data. For all methods, we can optionally fine-tune on the generated pseudo-data as a warm-up, and the operation is denoted as Pobb
'A formal definition of this is called personalized differential privacy, first seen in 16 www.staatt..rg/europaae

---

Implementation details. All models are fine-tuned from the (public) pre-trained GPT-2 model [35] The batch size 32 for all comparing methods except the GC [24] (GC [24] requires 2048). We use Adam [23] and adjust the initial learning rate with range of 10-3 to 10 for all methods. The d mentioned in Sec. for all DP methods is 10-6.
For SeqPATE, before training the student model with teacher supervision, we first fine-tune it on the public pseudo-data Deub as a warm-up. The coefficient À that balances supervision for the teacher 20, where we have tuned it on the validation set of the public
and the pseudo-data (Eq. [ pseudo-data. The default number of teacher models is 2k, where our model works well according to the experiments in App. H. We designed some strategies to reduce memory and disk usage (See strategies and the computational cost in App. I). We run SeqPATE with 2k teachers on a single GPU in days. Our code publicly accessible. (See details about hyperparameters in App. G.)
Evaluation Metrics. We evaluate the generated text by perplexity (PPL) and Bleu (Bleu-n) 33
6.2 Overall Performance
Table I: The performance on the two datasets with sample-level protections (mentioned in Sec. 5.2 All SeqPATE results are statistically significant compared to the strongest baseline under paired sample t-test (p- 0.05).
AirDialog
Europarl v6
PPL +
Bleu-3
Bleu-4
PPL +
Bleu-3
Bleu-4
Pri-GPT
3.88
21.51
17.16
23.25
1.77
0.86
Non-DP
Pub-GPT
63.16
0.31
0.10
57.40
1.02
0.35
Pub-GPT+DRus
19.39
0.71
0.25
45.40
1.38
0.52
NoisySGI
17.49
1.97
0.96
37.31
1.28
0.46
DP (sample)
NoisySGD+
16.78
2.21
1.09
37.69
1.31
0.42
NoisySGD+GC+2omb
11.17
3.15
1.54
35.77
1.56
0.57
SeqPATE
8.00
5.09
3.24
33.92
1.60
0.61
Protection at the sample level. Tab. show the performance on the two datasets. Among the non-DP baselines, Pri-GPT acts as an upper bound on the performance, since it can fully utilize the private set by discarding privacy protection. Pub-GPT+Douh outperforms Pub-GPT on both datasets, showing that the pseudo data helpful (additional ablation study on the pseudo data App. also verifies this). NoisySGD+GC+Duu surpasses the above two methods, since it uses much larger batch size (2048 vs 32) than NoisySGD. Our method, SeqPATE, significantly outperforms NoisySGD+GC++Dmhh (+59% in Bleu4 on AirDialog and +7.0% in Bleu4 on Europarl_ v6) while ensuring the same level of privacy protection in terms of E.
Protection on the user's secret phrases. We evaluate method for privacy protection of secret phrases mentioned in S.c The key step is to partition the data such that each phrase only occurs in the training data of very few teachers, which is straightforward given the user ID associated with the private data. In general, SeqPATE works with any set of secret phrases. In our experiments, we consider a user's full name as their secret phrase since can be easily recognized from the data. We partition AirDialog's ' private data according to the accompanying user IDs. As a result, there are 96.6% users whose data are assigned to a single teacher (details about the data partition in App. F).
As described in Sec. 5.3 standard DP methods incur larger privacy loss on secret phrases. In Tab. 2 we see that Noisy needs large noise to achieve a satisfactory level of protection on phrases, because E increases linearly with the frequency of the phrase (group privacy [4]). "Batching users" indicates partitioning data into batches according to users, which helps NoisySGD protect users' phrases (more analyses in App. M). For SeqPATE, the number of teachers trained data containing the phrase nt is close to on average after our partition. Thus, SeqPATE provides the same level of protection on users' secret phrases with smaller noise and thus achieves better performance (+70% and +36% in Bleu4) (see more about the protection level on users' secret phrases in App. F).
'We train and conduct the inference on the teachers one-by-one and cache the teachers' outputs. httpps///fthubb.oommmi/hiligggg/
7

---

Table The performance on AirDialog with the protections of users' secret phrases (mentioned in Sec. (.3)). Eavg is the average over all secret phrases, as of each phrase varies with the frequency of the phrase and the number of teachers (see App. for detailed analyses about Eavy). All results of SeqPATE: are statistically significant compared to the strongest baseline under paired sample t-test (p < 0.05).
PPL
Bleu-3 f
Bleu-4
NolsySGDD+GG+DDee
16.75
1.71
0.57
DP (phrase)
NoisySGD+GC+DDmm (batching users)
13.42
3.25
1.45
Savg 3
SeqPATE
10.10
4.20
2.46
NoisySGD+GC+DDau
16.49
1.89
0.69
DP (phrase)
NoisySGD+GC+Dra (batching users)
10.56
4.60
2.87
Favg = 5
SeqPATE
8.06
6.10
3,90
NoisySGD
NoisySGD+
NoisySCD+GC+ Drub
SeqPATE
-20
0.7
Negative PPL
Negative PPL
Bleu-4
Bleu-
40
0.5
50
Figure 2: The private-utility tradeoff in Bleu-4 and PPL different All the results are under sample level protections. Subfigure a show the results on AirDialog; e & d show the results on Europarl v6. The grey lines show the "lower bound" since the method does not access the private set.
Privacy-utilitty tradeoff. In Fig. 2 we show the private-utility tradeoff curve all DP algorithms. 8 Typically, DP with e E [0.1, 10 considered to provide meaningful protection (45). We observe
that SeqPATE outperforms NoisySGD and NoisySGD+GC+DDuu in this range. However, SeqPATE does not work better than the two methods when 10. The reason is that Noisy$GGD+GC+DDDn approaches Pri-GPT as E approaches infinity (i.e. the noise approaches O). However, SeqPATE with infinite is still weaker than Pri-GPT because distillation still incurs performance loss: the teachers cannot completely transfer knowledge from the private data to the student. Therefore, we suggest using SeqPATE if strong privacy protection desirable.
Table 3: Ablation studies.
means not using that strategy.
AirDialog
Europarl
PPL
Bleu-3 1
Bleu-4
PPL 1
Bleu-3 1
Bleu-4 1
SeqPATE
8.00
5.09
3.24
33.92
1.60
0.61
Merge P
11.96
3.14
1.85
39.19
1.40
0.47
KI.
12.08
3.26
1.81
39.81
1.41
0.52
Epsendo
8.11
4.74
3.17
33.81
1.58
0.60
Ethi KD
9.37
4.45
3.02
34.10
1.57
0.57
Gaussian
9.54
4.33
2.78
35.31
1.54
0.55
All
13.21
2.95
1.69
42.74
1.32
0.44
Forr the models without protections, we consider to be zero for baselines using the public data and E to be infinity for baselines using the private data.

---

6.3
Ablation Studies
There are several design choices in SeqPATE and we study the importance of each of them. In Tab. []] we consider the following variants of SeqPATE: (1) - -Merge P: aggregating the teachers by voting instead of averaging their output distributions; (2) KL: training the student using the cross-entropy loss with respect to teachers' top-1 prediction instead of KI. divergence; (3) - -peeudo: not learning from the pseudo label (Eq..) 3) (4) -ffi KD: querying teachers on all samples without selection; (5) - Gaussian: using the Laplace mechanism as the original PATE algorithm instead of the Gaussian mechanism; and (6) All: using none of the above strategies, which is similar (although not equivalent) to the original PATE (the difference is that PATE needs to roll out all teachers (Sec. 4.1).
Aggregating the teachers by voting and training with KL loss are the most important strategies for SeqPATE. The poor performance on -Merge P shows that voting is not suitable for text generation. The reason is that voting over a large output space leads to low agreement rates. The results show that the Epscudo loss makes little contribution to SeqPATE. The reason is that we have pre-trained on the student's $ training set via Lipeeudo before the student's training. The promotion caused by efficient which shows that
knowledge distillation (Effi KD) on AirDialog is larger than that on Europarl_ the "clever" student (e.g.. models on AirDialog with low PPI. and high Bleu) benefits more from this strategy. This is because the "clever" student can dramatically save the privacy cost and transfer it to where would benefit the student most. The poor performance of Alll verifies that the original PATE is not suitable for text generation.
Table 4: Analyses about the candidate filtering strategies.
AirDialog
Europarl v6
PPL +
Bleu-3 f
Bleu-4 f
PPL
Bleu-3 1
Bleu-4 f
top-p
8.00
5.09
3.24
33.92
1.60
0.61
top-k=1
18.23
0.89
0.38
45.15
1.40
0.53
top-k=10 10
12.47
3.47
1.95
35.94
1.55
0.54
top-k=50
7.89
4.96
3.35
33.74
1.59
0.59
top-k=100
8.78
4.64
3.17
34.48
1.60
0.62
top-k=200
9.24
3.77
2.94
34.63
1.57
0.55
6.4 Analyses on Candidate Filtering and Teacher Numbers
To analyze candidate filtering with different filtering strategies, we conduct experiments on top-p and top-k filtering. As shown in Tab. ( our full model employs the top-p filtering (the threshold is 0.95) surpasses most variants with manually chosen k. Top-k filtering (k: =50 or 100) also works well. Filtering with too small &s (k 1 or k 10) implies discarding too much useful information from the supervision (k- = is different from KL in Tab. which uses the Top-1 I of teachers' results). Filtering with oversize i results in unnecessarily large noises. Candidates with very small probabilities should be filtered during generation; however, random noises may increase their probabilities, so models may generate those words that are misled by the noise.
The results in App. H show that more teachers lead to better results when the number of teachers is in the range of 2k. This is because the noise assigned to each teacher drops linearly as the number of teachers increases. Note that SeqPATE cannot always benefit from increasing the teacher numbers, because the scale of each teacher's data is linearly decreased the teacher numbers go up. We choose e = 3 on the sample level protection for all results in Tabs. 3 and App. H.
Additionally, we conduct empirical comparisons and analyses of SeqPATE versus the original PATE in App. N. We show the effects of protections on users' secret phrases in App. O. We compare SeqPATE with another non-DP based baseline (i.e. blacklist based filtering) in App. P. We also conduct a case study in App. Q-
Related Work
Text generation models may leak user information through the generated texts 19 7
One direction
of privacy protection to protect author-level (user-level) information. The methods prevent attackers from inferring the author attributes (c.g., gender, age) (25 and the relationship between information

---

and authors [29]. Some researchers [40] 41 infer the membership (whether samples from given author are used to train the model) given black-box model. Some papers protect user privacy of training data against untrusted servers via federated learning 2 10 Another direction is to prevent
attackers from extracting sensitive information in training sets by analyzing the outputs (30] 22], which is urgently needed 77]. Our SeqPATE focuses on this direction. In this direction, regularization methods [6] 20 samples. Anonymization methods 126 44 detect sensitive text and replace it with non-sensitive text. Unlike DP IEI methods, the above methods do not provide a quantifiable guarantee for privacy protection. Some researchers focus protecting user privacy against untrusted servers via federated learning [0].
restrict the model capacity and prevent the model from memorizing exact training
Some researchers apply DP to text generation. For user-level privacy, ER-AE augments the semantic information in the generated text to hide authors' writing styles from attackers. McMahan et al. 28 propose recurrent language model with a DP guarantee against the identification of users. Note that the user-level privacy (relationships between users and their information) different from the privacy of users' secret phrases in our model: Our model prevents individual user phrases from being detected. Some researchers apply NoD to text generation to prevent sensitive training samples from being extracted: some of them 37
39 50) employ DP to protect part of selected tokens; others [22 49 24] apply DP to protect both samples and all tokens, but the privacy cost on
tokens is very high (Sec. 5.3 Our model falls into the latter category and reduces the privacy cost
of tokens. Kerrigan et al. [22] apply NoisySGD [ to text generation. Yu et al. [49] investigate
fine-tuning strategies clipping to pre-trained language models with NoisySGID and reduce memory usage. Shi et al. [38
pre-trrennn pre-trained language models with NoisySGD. Li et al. 24 apply ghost
apply DP to particular generation steps instead of training samples n-grams. Brown et al. 5
analyze DP based method versus data sanitization of text generation models. Brown et al. 12 propose a efficient NoisySGD to speed up model training.
Differential privacy (DP)
I3 [4] formally defines and quantifies privacy, ML models with DP
guarantee 46
15 52
prevent the existence of individual training examples from being detected
[6]. Some researchers protect the privacy of empirical risk minimization classifiers [8] and SVM
[36 with DP. Following Song et al. [42]. NoiDySGD [ achieves DP on deep learning models by adding noises to gradients. Pichapati et al. [4 adaptively clip the gradient in NoisySGD. PATE transfers the knowledge from teacher models trained on private sets with noises to student model. KNN-PATE (11) refines PATE by accessing only the k-nearest neighbors from the private set. Jordon et al. 21 adversarially learn to generate synthetic data with discriminators trained by PATE. These methods are not customized for text generation models. Xie et al. [48] propose DPGAN to adversarially learn with generator and a discriminator.
31 32
Conclusion
In this paper, we propose a novel framework, SeqPATE, to protect the privacy of the training data for text generation models with DP guarantees. SeqPATE achieves good privacy-utility trade-off by leveraging both private and public data. As an extension of PATE, SeqPATE can handle the sequential generation paradigm with large output space at cach step and is therefore adaptive to text generation models. We avoid rolling out teachers by providing pscudo-inputs for the teacher's inference and the student's training. We further reduce the output space by candidate filtering and limit privacy losses via efficient knowledge distillation. SeqPATE achieves a better performance with the sample-level protection and further provides much stronger protection on users' secret phrases. The limitations, ethical considerations, and social impacts of this paper are in App. A and L.
9
Acknowledgement
Research in this paper was supported by Hong Kong Research Grants Council under grand No. 16204920. HH partly supported by the Samsung Advanced Institute of Technology (Next Gener- ation Deep Learning: From Pattern Recognition to Al). YW is partially supported by NSF Award #2048091. The authors thank Mr. Wei Dong and Dr. Yiping Song for their help and insights on this paper.
10

---

References
[1] Martin Abadi, Andy Chu, lan Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In CCS, pages 308-318, 2016.
[2] Borja Balle and Yu-Xiang Wang. Improving the gaussian mechanism for differential privacy: Analytical calibration and optimal denoising, In ICLM, pages 394-403. PMLR, 2018.
[3] Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In FOCS, pages 464-473, 2014.
[4] Haohan Bo, Steven HH Ding, Benjamin CM Fung, and Farkhund Iqbal. Er-ae: Differentially private text generation for authorship anonymization. NAACL, pages 3997-4007, 2021.
[5] Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. What does it mean for language model preserve privacy? In ACM FAccT, 2022.
[6] Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization neural networks. USENIX Security, pages 267-284, 2019.
[7] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In USENIX Security, pages 2633-2650, 2021.
[8] Kamalika Chaudhuri, Claire Monteleoni, and Anand Sarwate. Differentially private empirical risk minimization. In JMLR, volume 12, 2011.
[9] Ven-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu, and Jingjing Liu. Distilling knowledge learned in bert for text generation. In ACL, pages 7893-7905 2020.
[10] Jieren Deng, Chenghong Wang, Xianrui Meng. Yijue Wang, Ji Li, Sheng Lin, Shuo Han, Fei Miao, Sanguthevar Rajasekaran, and Caiwen Ding. secure and efficient federated learning framework for nlp. In EMNLPs, 2021.
[11] Jinshuo Dong, Aaron Roth, and Weijie Su. Gaussian differential privacy. Journal of RSS, Series B, 2019.
[12] Christophe Dupuy, Radhika Arava, Rahul Gupta, and Anna Rumshisky. An efficient dp-sgd mechanism for large scale nlu models. In ICASSP, pages 41 18-4222.. IEEE, 2022.
[13] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In TCC, pages 265-284. Springer, 2006.
[14] Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. In TCS, volume 9, pages 211-407, 2014.
[15] James Foulds, Mijung Park, Kamalika Chaudhuri, and Max Welling. Variational bayes in private settings (vips). In IJCAI, pages 5050-5054, 2021.
[16] Arpita Ghosh and Aaron Roth. Selling privacy auction. In ACM-EC, pages 199-208, 2011.
[17] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. In IJCV, volume 129, pages 1789-18199 2021.
[18] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In ICLR, 2019.
[19] Dirk Hovy and Shannon I. Spruit. The social impact of natural language processing. In ACL., pages 591-598, 2016.
[20] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniy,, and Yoshua Bengio. Quan- tized neural networks: Training neural networks with low precision weights and activations, In JMLR, volume 18, pages 6869-6898. JMLR. 2017.

---

[21] James Jordon, Jinsung Yoon, and Mihaela Van Der Schaar. Pate-gan: Generating synthetic data with differential privacy guarantees. In ICLR, 2018.
[22] Gavin Kerrigan, Dylan Slack, and Jens Tuyls. Differentially private language models benefit from public pre-training. In Workshop in EMNLP, pages 45, 2020.
[23] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
[24] Xuechen Li, Florian Tramèr, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. In ICLR, 2022.
[25] Yitong Li, Timothy Baldwin, and Trevor Cohn. Towards robust and privacy-preserving text representations. In ACL, pages 25-30, 2018.
[26] Wakana Maeda, Yu Suzuki, and Satoshi Nakamura. Fast text anonymization using k- anonyminity. In iiWSS, pages 340-344, 2016.
[27] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effin -effecciinnnne learning of deep networks from decentralized data. In PMLR, pages 1273-1282, 2017.
[28] Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In ICLR, 2018.
[29] Fatemehsadat Mireshghallah, Huseyin Inan, Marcello Hasegawa, Vietor Rühle, Taylor Berg- Kirkpatrick, and Robert Sim. Privacy regularization: Joint privacy-utility optimization in languagemodels. In NAACL, pages 3799-3807, 2021.
[30] Yuta Nakamura, Shouhei Hanaoka, Yukihiro Nomura, Naoto Hayashi, Osamu Abe, Shuntaro Yada, Shoko Wakamiya, and Eiji Aramaki. Kart: Privacy leakage framework of language models pre- pre-trained with clinical records. In arXiv preprint #rXiv:2101.00036. 2020.
[31] Nicolas Papernot, Martin Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi- supervised knowledge transfer for deep learning from private training data. In ICLR, 2017.
[32] Nicolas Papernot, Shuang Song. Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Ulfar Erlingsson. Scalable private learning with pate. In ICLR, 2018.
[33] Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311-318, 2002.
[34] Venkatadheeraj Pichapati, Ananda Theertha Suresh, Felix X Yu, Sashank J Reddi, and Sanjiv Kumar. Adaclip: Adaptive clipping for private sgd. In arXiv preprint arXiv:1908.0700 07643, 2019.
[35] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018
[36] Benjamin Rubinstein, Peter Bartlett, Ling Huang, and Nina Taft. Learning in a large function space: Privacy-preserving mechanisms for svm learning. In JPC, volume 4, pages 65-100, 2012.
[37] Taisho Sasada, Masataka Kawai, Yuzo Taenaka, Doudou Fall, and Youki Kadobayashi. Differentially-private text generation via text preprocessing reduce utility loss. In ICAIIC, pages 042-047. IEEE, 2021.
[38] Weiyan Shi, Si Chen, Chiyuan Zhang, Ruoxi Jia, and Zhou Yu. Just fine-tune twice: Selective differential privacy for large language models. EMNLP, 2022.
[39] Weiyan Shi, Aiqi Cui, Evan Li, Ruoxi Jia, and Zhou Yu. Selective differential privacy for language modeling. In NAACL.
[40] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In S&P, pages 3-18. IEEE, 2017.
12

---

[41] Congzheng Song and Vitaly Shmatikov. Auditing data provenance in text-generation models. In KDD, pages 196-206, 2019.
[42] Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate. Stochastic gradient descent with differentially private updates. In GlobalSIP, pages 245-248. IEEE, 2013.
[43] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. In JMLR, volume 15, pages 1958-1958. JMLR.org, 2014.
[44] Yu Suzuki, Koichiro Yoshino, and Satoshi Nakamura. A k-anonymized text generation method. NBiS, pages 1018-1026. Springer, 2017.
[45] Aleksei Triastcyn and Boi Faltings. Bayesian differential privacy for machine learning. In ICML, pages 9583-9592, PMLR, 2020.
[46] Yu-Xiang Wang. Stephen Fienberg, and Alex Smola. Privacy for free: Posterior sampling and stochastic gradient monte carlo. In ICML, pages 2493-2502. PMLR, 2015.
[47] Wei Wei, Quoc Le, Andrew Dai, and Jia Li. Airdialogue: An environment for goal-oriented dialogue research. In EMNLP, pages 3844-3854, 2018.
[48] Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou. Differentially private generative adversarial network. In arXiv preprint wxxiv:/822.06:2 2018.
[49] Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. Differentially private fine-tuning of language models. In ICLR, 2022.
[50] Xuandong Zhao, Lei Li, and Yu-Xiang Wang. Provably confidential language modelling. In NAACL, pages 943-955. 2022.
[51] Yuging Zhu, Xiang Yu, Manmohan Chandraker, and Yu- Yu-Xiang Wang. Private-knn: Practical differential privacy for computer vision. In CVPR, pages 11854-11866 2020.
[52] Alexander Ziller. Dmitrii Usynin, Rickmer Braren, Marcus Makowski, Daniel Rueckert, and Georgios Kaissis. Medical imaging deep learning with differential privacy. In Scientific Reports, volume 11, pages 1-8. Nature Publishing Group. 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations your work? (Yes] can be found in the supplemental materials.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] It can be found in the supplemental materials.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs all theoretical results? [Yes] It can be found in the supplemental materials.
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] can be found
in the supplemental materials.
13

---

(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes]
be found in the body paper or the supplemental materials. (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] It can be found in the body paper or the supplemental materials.
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes] It can be found the supplemental materials.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "TUvidruCY_2", "reviewer": "Reviewer_hTcT", "summary": "This paper extends PATE into the field of text generation.  To do so, the following technical challenges must be properly addressed: 1. in addition to protecting individual words, we need to protect phrases too. 2. Compared to other task, the output space is huge for text generation. 3. We need to control the privacy loss.  This paper has done a solid work to address these challenges.", "questions": "This is a solid work.   But making it easier to follow may greatly increase the influence of this work.", "limitations": "Privacy is an area with great social impact.  This paper focuses solely on the technical aspect of privacy.  But it’s too early to give an assessment on its social impact.  So in my humble opinion, it is acceptable that its social impact are absent in the paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is well written.  This work is original.  It is based on the theory of differentially private (DP), so its potential and quality are pretty high. The important difficult points are well explained.\nBut as a reader, I think one area can be improved: the connection between theory of DP and the proposed SeqPATE method can be explained more explicitly.  Doing so can greatly decrease the barriers to new researchers who are interested in this area.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PButlHJjzs7", "reviewer": "Reviewer_b5Sy", "summary": "In this paper, authors propose a novel framework, SeqPATE, an extension of PATE on text generation, as a Differentially private (DP) learning algorithm for text generations.\nSeqPATE aims to protect the privacy of both training samples and sensitive phrases in samples,\nand  employs a teacher-student framework.\nAdditionally, authors propose several strategies for SeqPATE to handle text generations with a sequence of classifications over large spaces.", "questions": "*How about applying a simple approach such as a word blacklist to a benchmark?\n\n*Can you explain the rationale for the superiority of this framework compared to the case where teacher models and student model are studied with $D^{pub}$ and $D^{pri}$, respectively?\n\n*Show some examples or qualitative evaluation of how SeqPATE achieves utility with strong privacy protections on training samples.", "limitations": "Without qualitative analysis, \nit is a quantitative comparison of similar models and does not support the author's claim.\nOnly the usual text generation model metrics (i.e., PPL and Bleu)are used.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n+Privacy protections is important for text generation models and other tasks\n\n+Motivation and problem setting are clear\n\n+Previous work survey is enough\n\n\nWeaknesses\n\n-Several claims have not been adequately verified. For example, \"the effectiveness of SeqPATE in protecting both samples and sensitive phrases\", \"training corpora with a moderate privacy cost\".\n\n-No qualitative and error analysis\n\n-Runtime analysis is lacked. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FEC0Z0yZdja", "reviewer": "Reviewer_o7Ks", "summary": "This paper extends the PATE approach to the text generation problem. The authors introduce additional steps to help boosting the performance of PATE in the text generation setting as the oiriginal approach does not bode well with the large output space of vocabulary. The paper further studies phrase-level privacy beyond the regularly studied sample-level privacy.", "questions": "See strengths and weaknesses.", "limitations": "NA.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written and easy to follow. Unfortunately my main concern is on the novelty of the paper. The approach is heavily based on the PATE algorithm with few tricks to work better for the text generation task. It utilizes a pre-trained LM to generate pseudo completions and reduces the output space by filtering the tail distribution without a privacy requirement and finally the privacy loss is reduced by acquiring the teacher supervision only when the student is not good at a certain prediction. The latter idea has also appeared in the more recent PATE paper. While I believe these extensions are valuable in improving the performance of PATE in this scenario, I do not think they provide sufficient novelty for this venue.I have one critical comment about the users' secret phrases section. The authors took the route of group privacy for this scenario, which I do not think might bethe effective way with DP. DP-SGD algorithm can easily be adapted to have \"user-level privacy\" by batching users instead of samples. I find it an unfair comparison in the sense that the authors have not employed this approach but took the naive way of applying group privacy on user-level.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AWn_3KJsDh", "reviewer": "Reviewer_h8Jn", "summary": "The paper proposes an extension of PATE, a private learning algorithm, to text generation tasks. The extensions are simple yet effective: they generate pseudo inputs and reduce the sequence generation problem to next word predictions. They also propose a strategy to dynamically filter out candidates to reduce the large output space in the text decoder. Experiments in the sentence completion task show that the proposed model is effective in protecting samples and sensitive phrases.", "questions": "None.", "limitations": "As the work focuses on privacy, I think it would be nice to have a specific section on limitations and societal impact. This is currently absent in the main paper.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* The proposed extension is very simple yet intuitive and effective for differentially private text generation.\n\nWeaknesses\n* The paper could have been more convincing if the model is tested on multiple text generation tasks such as dialog response generation (generate a response given previous utterances) where privacy is more crucial.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept