Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tries to adapt the PATE (Private Aggregation of Teacher Ensembles) framework for DP training of machine learning models to the setting of text generation. There are some significant challenges in adapting PATE to this framework.  PATE as originally proposed is designed to work for classification tasks with a small number of labels. Text generation or next word prediction has too many labels and a direct application of PATE doesn't work. This paper solves this issue by aggregating only top-k predictions of each teacher where the top-k is computed using the prediction of the student on public data and so doesn't leak extra privacy. The paper also crucially uses public data and large pretrained models like GPT2 to further improve the performance of DP learning using PATE on private data.

They claim that their approach improves upon DPSGD/Noisy SGD (which is the standard DP algorithm for deep learning) significantly on some standard benchmarks.
main_review: Strengths: I think the paper does a good job of adapting PATE to the text generation framework and making it work reasonably well. Given that PATE is a very general framework that works for any learning algorithm and not just deep learning, it is useful to have adaptations of it to sequence/text generation settings.

Weaknesses: 

I am not fully convinced by the authors claim that their adaptation of PATE beats DPSGD. 

(1) The authors cite Kerrigan et al (2020) to claim that DPSGD doesn't work for text-generation using large models like GPT2 due to high dimensionality of the parameter space. This is debunked conclusively in the two recent papers by Li et al 2021 (https://arxiv.org/abs/2110.05679) and Yu et al 2021 (https://arxiv.org/abs/2110.06500). Li et al (2021) show that by careful choice of hyperparameters, training pretrained GPT2 on private datasets using DPSGD achieves performance comparable to non-private training. We also note that these are very recent and possibly concurrent with this work. But nevertheless, the authors should not be coming to an opposite conclusion as these papers.

(2) I am also not convinced by the experiments in Table 1. The authors compare their approach to the baseline DPSGD+$\tilde{D}^{pub}$. This means that, from what I understand, a model is trained from scratch on private data $D^{priv}$ using DPSGD and then fine-tuned on $\tilde{D}^{pub}$. I think that the correct baseline to compete against is Pub-GPT + $\tilde{D}^{pub}$ + DPSGD on $D^{priv}$ in that order. That is we are fine-tuning the pretrained GPT2 on $\tilde{D}^{pub}$ and then fine-tuning on $D^{priv}$ using DPSGD. I will be more convinced that their framework works if they compare against this benchmark. 

(3) Also to claim that, their method is beating DPSGD, they have to demonstrate they did a systematic search of the hyperparameter space for DPSGD. But I didn't find any evidence of this in the paper. The importance of choosing right hyperparameters for DPSGD is demonstrated in the work of Li et al (2021). 

Other remarks: Is the grey line in Figure 2, Pub-GPT + $\tilde{D}^{pub}$? Is it a typo?

summary_of_the_review: Because I am not fully convinced the by the claims of the authors, I rate it marginally below acceptance threshold. I am willing to update my score based on further evidence by the authors in support of their claims.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a variant of PATE (from ICLR 2018), a method intended for standard supervised learning tasks. The present work instead focuses on natural language generation, an inherently sequential task, which introduces new challenges. Relative to PATE (and assuming knowledge of it), some differences in the method involve: extending the public data sentences into "pseudo sentences" using GPT-2, simultaneously performing training on all prefixes of the pseudo sentences, averaging the teacher model output distributions rather than aggregating their max score (i.e., their prediction), and using the student model to reduce the support size for said output distributions. Experimental evaluation is provided for two datasets, AirDialog and Europarl_v6. Perplexity and BLEU scores improve upon those which do not use the private data, and those which use previous methods with the private data.

main_review: This paper provides results in the very interesting direction of private language models. I believe that, given the recent progress in non-private language models, the time is ripe to further investigate this area privately, which has been sorely under-explored. As a result, I commend the authors for taking steps in this direction.

First, I will evaluate the algorithmic contributions. As I enumerated above, there are a few modifications to the basic PATE setting. Of these, I believe using the student model to reduce the support size is an interesting idea. Using a pre-trained GPT-2 model to extend the training data is also kind of nice, though it is not 100% clear why this is necessary in the experimental settings considered. In the example provided, I understand: "Cats sit" is too short a sentence to perform meaningful training from. However, the datasets used seem to have much larger utterances (indeed, it says in Appendix C that sentences less than 8 tokens long were filtered out). The other two modifications seem relatively straightforward: it is of course natural to consider using multiple prefixes of the same pseudo sentence, rather than performing predictions sequentially. And performing PATE on the output distribution rather than the output prediction is also natural (I would be surprised if it has not been done before). I believe it may also be inspired directly from knowledge distillation (which is uncited and uncredited). Overall, the algorithmic modifications are nice, but nothing offering significantly new insights into the nature of natural language generation.

Next, I will comment on the experimental results. The utility metrics provided in Table 1 seem to indicate that the proposed method performs somewhat better than the previous methods which are private with respect to the sensitive dataset. However, the utility is still very far from the non-private utility. Perhaps this is to be expected, given the general cost of guaranteeing DP, but other recent results seem to show how effective pre-trained models are in terms of maintaining utility even with DP -- see, for example, Tramer-Boneh (https://arxiv.org/abs/2011.11660), which demonstrates the efficacy of transfer learning to image classification (resulting in relatively comparably small drops to the accuracy versus no transfer learning), as well as two recent arXiv papers on the same topic (https://arxiv.org/abs/2110.06500 and https://arxiv.org/abs/2110.05679) which achieve relatively small loss in utility compared to the non-private pre-trained model. Of course the latter two works appeared only after the deadline and thus were impossible to compare with, so perhaps one should disregard them, but the story of both papers seem to be at odds, which is puzzling to me. Regardless, the [TB20] paper still indicates the significant power of transfer learning for private learning). Additionally, I noticed that experiments were only performed on two relatively large datasets. This begs the question of what happens when the datasets are relatively small. Lack of accuracy on smaller datasets may be why the authors did not evaluate on more common GEM benchmarks. Comparison with prior methods seem a little mixed. While SeqPATE indeed outperforms previous methods in various regimes, it is somewhat inconclusive at eps = 5, which is closer to the regime usually considered in DP ML papers. As a final note, my impression is that this is an incredibly costly infrastructure to run. I believe this uses k = 2000 teachers, each of which is GPT-2 Small, resulting in an overall model which is comparable in number of parameters to the full GPT-3. This sounds impractical to run in most settings, and this matter is not discussed at all in the paper. Overall, the experimental results are modest, but given this discussion, I am not sure if they mark a qualitatively impressive shift in our understanding of the problem (which, as I mentioned before, is seemingly quite under-explored).

My final major comment is on the writing. The quality of the language is rather poor, with issues in almost every sentence, to the effect that the final paper feels very rushed. I tried to be charitable to the authors and fill in the gaps intelligently, but there were certain sentences which were incomprehensible in their current state. Beyond this, other (comparatively minor) writing issues include insufficient discussion of related work, incorrect citations, and unusual choice of content in the body. I list some of these issues below. Overall, presentation issues in the submitted version are to the level where I would not recommend anyone to read this until it is better polished.

Further points and comments:
- The citation for differential privacy is Dwork, McSherry, Nissim, and Smith, TCC 2006.
- Bassily, Smith, Thakurta, FOCS 2014 should be cited as another reference for DPSGD.
- As mentioned above, what happens if one just uses D^pub instead of ~D^pub? Why is this insufficient for the datasets used in the experiments? This question is true for many of the approaches studied, not just SeqPATE.
- The term knowledge distillation (KD) is used extensively in the paper. As I am sure the authors are aware, KD refers to a specific technique for training from a teacher. Is the KD used in the paper the same as KD as used in the literature? If so, then it should be cited and discussed (or otherwise, contrasted with). If it is different, then the term KD should be changed.
- It may be helpful for the presentation to keep going with the running example demonstrated in Figure 1, and how the procedure would work with this concrete instance.
- Although I am not too concerned with what appears in the body of the paper versus the appendix, some choices that I found unusual include: basics of DP and Equation (1), which may be immediate or already known to most readers.
- It is not clear what bold and underlined numbers in Table 1 are meant to represent.
- I am curious what happens for the non SeqPATE methods if one uses D^pub instead of ~D^pub. It feels like you're cheating if you're just using the model to generate more training data, and it may or may not have an effect.
- The choice of P-3 or P-4 is not clear as a privacy metric. It seems to say something more about the nature of the dataset. See in particular how these numbers differ significantly between AirDialog and Europarl_v6. The canary approach used in the Carlini et al Secret Sharer paper seems more meaningful to me. 
- Broken reference in Appendix D
- It appears that footnote 7 was never filled out appropriately
- It seems like more teachers result in monotonically better utility. What is the limit of this phenomenon?

summary_of_the_review: A nice direction for investigation. But the approach and findings don't appear significant enough to warrant acceptance in ICLR right now. The paper is further hampered by the very poor presentation.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to protect the privacy of text generation models, which may leak sensitive information of the training data. Specifically, the authors propose SeqPATE, which applies the PATE framework to large-scale language models such as GPT. By reducing the output space and aggregating the supervision from teacher models, SeqPATE can help reduce the privacy cost. Empirical evaluation on two private text corpus shows that SeqPATE achieves a good trade-off between utility and privacy.

main_review: ## Strengths:
1. Training a DP language model is a less explored problem, and the authors provide a technically sound solution along with a well-setup empirical evaluation. 
2. The paper is overall well written and the method is easy to follow. 
3. The authors propose a new evaluation metric P-N to indicate how many generated n-grams can be found in the private set, which is an interesting and useful indicator in addition to PPL on the private set.
4. The authors also provide an interesting discussion on user-level DP and n-grams-level DP for language models.

## Weaknesses:
1. My main concern of this paper is whether SeqPATE can achieve satisfactory utility preservation given small epsilon (say epsilon=2). From Table 1, the B-4 score of SeqPATE is different from that of the upper bound by 2 orders of magnitude, which is a bit concerning. Moreover, the PPL gap between the upper bound and SeqPATE (13.67 - 3.88) is much larger than the gap between the lower bound and SeqPATE (19.39 - 13.67), which suggests that SeqPATE may barely learn useful information on the private data domain. 
2. The advantage of SeqPATE vanished given a large epsilon compared with DP-SGD, which makes the algorithm less scalable. Although the authors explain it is due to the partitioning of the training data, could this issue be resolved if reducing the number of teachers?
3. The authors discuss user-level DP and n-grams-level DP. It is better to provide some quantitative evaluation on SeqPATE on these settings.
4. Some experimental setups are a bit unclear. it is better to include more experimental details in the main paper, e.g., how the hyper-parameters are chosen (say #/ teachers) and the computational overhead of SeqPATE and DP-SGD.

## Additional questions:
1. I see the paper is adopting a strict DP standard (delta=1e-9). Could raising the delta to 1e-6 or 1e-5 can help improve the utility of SeqPATE?
2. Why Pub-GPT + \tilde{D}^{pub} can significantly improve (the largest PPL improvement) the PPL of Pub-GPT, considering both methods use the public records only. This result is a bit weird and counterintuitive. 

I am willing to raise my scores if the problems above can be well addressed.

summary_of_the_review: Strengths:
1. Training a DP language model is a less explored problem, and the authors provide a technically sound solution along with a well-setup empirical evaluation. 
2. The paper is overall well written and the method is easy to follow. 
3. The authors propose a new evaluation metric P-N to indicate how many generated n-grams can be found in the private set, which is an interesting and useful indicator in addition to PPL on the private set.
4. The authors also provide an interesting discussion on user-level DP and n-grams-level DP for language models.

Weaknesses:
1. My main concern of this paper is whether SeqPATE can achieve satisfactory utility preservation given small epsilon (say epsilon=2). 
2. The advantage of SeqPATE vanished given a large epsilon compared with DP-SGD, which makes the algorithm less scalable. 
3. The authors discuss user-level DP and n-grams-level DP. It is better to provide some quantitative evaluation on SeqPATE on these settings.
4. Some experimental setups are a bit unclear. it is better to include more experimental details in the main paper, e.g., how the hyper-parameters are chosen (say #/ teachers) and the computational overhead of SeqPATE and DP-SGD.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes SeqPATE that adapts PATE to text generation while satisfying differential privacy. To overcome the challenge of obtaining sequence-level supervision for text generation, it first generates pseudo inputs by completing the prefix using GPT-2 (a pre-trained language model) and thus reduces the problem to the next word prediction. Then it applies the PATE framework to privately train a student model with several algorithmic innovations, including (1) aggregating teachers' models by averaging their output distributions; (2) reducing output space by top-k/top-p selection; (3) efficient knowledge distillation by only querying the teachers' models when the student model performs poorly. (4) using both public labels and teachers' aggregation to supervise the learning. The paper gives privacy analysis and also shows it can offer privacy guarantees on user-level and secret n-grams. The paper also provides extensive experimental results and shows that it outperforms other DP learning algorithms when the \eps is small (<5). The experiments demonstrate the practicality of the algorithm.
main_review: Strengths: This paper is well-written and technically sound. Protecting text data is important given there is a rich body of work on attacks for such datasets; however, the protection technique that can provide provable DP guarantees is somewhat understudied. This is a timely and interesting topic and has many applications, for example, the smart compose. The approach is clearly stated, and the empirical evaluation is sufficient. In terms of novelty, SeqPATE has several non-trivial changes compared to the classical PATE. 

Weaknesses: I have a few minor comments.

1. When \eps is large, DPSGD outperforms SeqPATE. In practice, the choice of \eps depends on the data, so could you offer any instructions on choosing the algorithm when \eps is not very small (like between 5-10, which is also common in practice)?

2. I'm confused about how you define P-n and why it can be considered as a privacy metric?

3. In Section 4.3, the sentence `` Eq. 1 shows the top-k selection at i-th step ...'' is confusing. Eq. 1 only shows the renormalization of the probabilities. 

summary_of_the_review: Overall, I think it's a strong paper. The topic is timely and interesting. The writing is good, and the approach is clearly presented. The proposed approach contains several non-trivial changes compared to PATE. The paper provides extensive empirical evaluation, which demonstrates the practicality of SeqPATE.
review_title: null

Meta-review: This was a very borderline decision. Here are the major factors involved in the decision. 

1. The concurrent works by Li et al and Yu et al. It is unclear about the relationship/strength between those results and the ones in the present paper. However, in accordance with the ICLR policy on simultaneous work, we ignore them to the extent possible.
2. The novelty of the approach. All reviewers agreed that the modifications to the PATE approach are fairly minor or incremental, and this appears to be largely an application of this method. That said, methods for the natural language setting are important.
3. The strength of the findings. Reviewers were mixed on the strength of the results: they appeared significant in some cases but rather lackluster in others. 
4. The poor quality of the writing. I personally read the original version and found the writing to make it impossible to understand in parts. The writing is still subpar in many places, which I would have expected the authors to fully polish given the substantial time during the response period. 

While in isolation, any of 2, 3, and 4 may be acceptable, their combination makes it difficult to recommend this paper for acceptance at this time.
Decision: Reject