Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method for answering next-token queries in a privacy-preserving manner. As opposed to training models with DP guarantees, it adds privacy-preserving guarantees during the prediction time. The approach works by getting a public model and then fine-tuning it on disjoint parts of the private dataset. This way it obtains, several models. During prediction, it queries each model and combines the outputs by proportionally adjusting the contribution from each model. The experiments how that utility is better than that pf DP training and also protects against extraction attacks of sentences.
main_review: The paper is well written and experiments show that the method works in practice. The visualizations are also great. The paper studies a difficult problem as utility of next word prediction suffers a lot when trained with differential privacy.

Experimental section does not provide details on whether other methods with DP were trained using an already pre-trained models and not only using private corpora. This needs to be made clear that the improvement in the utility is not only from the user of a pre-trained model (for example, why would one even use DP-SGD and not the pretrained model?)

The comparison with PATE seems a bit brief as the methods are quite alike albeit PATE was used for classification tasks. It would be interesting to see how SUBMIX performs on classification tasks.

Please explain while splitting into two parts is needed after the data is already split in k.

Minor: Fig 1 b calls the second model differently than in Alg 1 and 2 (h and lambda)
summary_of_the_review: The paper presents an interesting idea for privacy-preserving prediction for fine-tuned models. However, as written now it is not clear if the benefit in experiments is due to fine-tuning or due to the new method of training/prediction proposed in the paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a new approach to simultaneously preserve user level-DP and prevent text extraction attacks in prediction. Experimental results were conducted on Wikitext-103 and BigPatent-G datasets showing some promising results. However, there are several critical concerns detailed in the main review regarding the application contexts, theory analysis, and experimental settings. 
main_review: First, the reviewer does not understand why the proposed approach defends against the text extraction attacks without releasing the trained models? The prediction setting provided in the proposed approach may be suitable for ML as a service (MLaaS). However, why next token is an MLaaS application and in which real-world scenarios? Also, the connection between user level-DP with text extraction attacks is weak. Why is user level-DP needed to defend against text extraction attacks?

Second, the threat model is missing. The reviewer is not sure how the adversaries will choose the query $x_t$ to extract sensitive text data. It is unclear from the writing.

Third, the user level-DP provided in the proposed approach by partitioning users’ text data into different parts and sub-parts is incorrect. For the fine-tuning models to work well, data in a single part must be sufficiently large, requiring a certain number of users included in the part. As a result, the neighboring part definition (excluding one part) does not provide a precise user level-DP. We can only have user level-DP if a user is a part. This scenario will not work well for fine-tuning large NLP models, such as GPT-2, given a limited amount of text data of a user. 

Also, the assumption that if two models return similar results when they do not memorize the context of a query $x_t$ is very vague. It is unclear to me when this could happen and how we can quantify the similarity here. No evidence was provided. The same issue given the two models return dissimilar results when they memorize the query $x_t$ given that they do not share common users. 

Fifth, experimental results are unconvincing. DP-SGD does not provide either user level-DP or solutions to prevent text data extraction attacks. Why is DP-SGD being considered a baseline here? At least, the user level-DP from McMahan et al. (2017) and its variations (Kairouz et al., 2019; Ramaswamy et al., 2020) should be considered as appropriate baselines. Improving the perplexity in the fine-tuning does not indicate that the model is secure against data extraction attacks. 

Regarding the attack evaluation, why randomly generating $m$ codes with each code being an $l$-digit number (for example, representing a user’s age, ZIP-code, phone number, SSN, etc.) can be a representative measure for information leakage under data extraction attacks? Sensitive information can be extremely broader than these numbers, such as name entity, correlation among them, etc. What would be a good way to evaluate the model under the attacks and under which threat models?
summary_of_the_review: Overall, this is an important research direction. The proposed approach is interesting. However, there is room for improvement.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper attempts at introducing a privacy-preserving method for predicting next words, for large language models. The problem setup for the paper is when pretrained large language models are available, and they are to be fine-tuned on private data for the task of next word prediction. If DP-SGD is used here, due to the large number of parameters in these models, the authors claim that the noise that is added would be too much and utility (i.e. perplexity) would suffer gravely. Therefore, they introduce an ensemble-based approach: partitioning the data into disjoint subsets, such that each user's data appears in only one partition. Then, finetuning a separate model on each partition. For decoding (i.e. inference, guessing the next word), if all the partition-LMs agree, then they assume that there is no leakage and release it. If there is disagreement, the output of these models is mixed, along with the output of a publicly pretrained LLM, so as to bound the private leakage (kind of similar to PATE, but for language modeling). Then the paper proceeds to attempt at empirically testing this method on two datasets. 
main_review: I applaud the authors for focusing on the problem of privacy in language modeling, however, I have some major concerns about the assumptions made and the actual applicability of the proposed method, that I will list below:

1. Faulty assumptions in the experiments: The paper has defined their setup to be "finetuning" a pretrained language model. In finetuning tasks in NLP, it is often the case that training data is scarce. However, here for the experiments, Wikipedia is used as "private" dataset, which is far from a realistic scenario, especially when there are many realistic available datasets that NLP/privacy researchers use, such as Enron email dataset,  sentiment 140, Shakespeare, etc (I have listed multiple related missing privacy in NLP reference papers below, there are more appropriate datasets there as well). I strongly believe that if a small dataset is used, the utility of the proposed method will also suffer gravely, given how it partitions the data and leaves few training samples for each large model. I think an experiment on a small dataset could clear the outcome of this issue. 

2. Assumption on the correlation between users: I highly disagree with the assumption that if all the LM partitions agree on the next word for a given sequence, then there is no leakage. There is the possibility that users are correlated, like in an email dataset in a company, and a large group of them might all be talking about a secret, like a secret serial number. If different users talk about this, there is a chance that each partition ends up with at least one user that has that serial number in their data. Therefore, they might all predict it, and agree on it, and this could leak that string. Please clarify on how such a scenario can be prevented in your setup. 

3. Huge computation/storage costs: Although the limitations section of the paper mentions this and proposes using a smaller model as a solution, I highly disagree. The proposed method's functionality depends on prior pretraining and then finetuning. This is not the usage mode for small LSTM models. So the proposed solution isn't really feasible. 

4. The entire NLP-related body of work on this problem is missing. This is not  just a few missing references. I have provided a list below:


1. Gopi S, Gulhane P, Kulkarni J, Shen JH, Shokouhi M, Yekhanin S. Differentially private set union. InInternational Conference on Machine Learning 2020 Nov 21 (pp. 3627-3636). PMLR.

2. Mireshghallah F, Inan HA, Hasegawa M, Rühle V, Berg-Kirkpatrick T, Sim R. Privacy Regularization: Joint Privacy-Utility Optimization in Language Models.

3. Lyu L, He X, Li Y. Differentially Private Representation for NLP: Formal Guarantee and An Empirical Study on Privacy and Fairness. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings 2020 Nov (pp. 2355-2365).

4. Xu Z, Aggarwal A, Feyisetan O, Teissier N. A Differentially Private Text Perturbation Method Using Regularized Mahalanobis Metric. InProceedings of the Second Workshop on Privacy in NLP 2020 Nov (pp. 7-17).

5. Li Y, Baldwin T, Cohn T. Towards Robust and Privacy-preserving Text Representations. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) 2018 Jul (pp. 25-30).

6. Kim K, Gopi S, Kulkarni J, Yekhanin S. Differentially Private n-gram Extraction. arXiv preprint arXiv:2108.02831. 2021 Aug 5.


summary_of_the_review: To summarize the points from above, I believe the main shortcomings of this paper are:

1. unrealistic assumptions in the experiments and about the problem setup in general (relating to the size of data and correlation between users).
2. Huge computation and storage costs which are not explicitly studied.
3. Not providing any comparisons with prior existing work in NLP.  

If the authors show that this method is effective in a setup where the user data is scarce, address my concern on corrolations, and provide comparisons (qualitatively) with prior work I am willing to update my score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Language models keep sentence-level information of training samples in detail. This feature jeopardizes data privacy by data-extraction attacks, which can retrieve full sentences of training samples by directly querying models. Current solution to this issue is based on differential privacy, such as DP-SGD. However, existing approach suffers from information leakage due to full access to model’s parameter/gradients and only can be applied to small-scale RNNs, which also cannot balance privacy-utility trade-off. In this work, the proposed approach called SUBMIX focuses on private prediction for answering next-token queries with regular language models fine-tuned on the private corpus. Submix as an ensemble based on models by private corpus and public pre-trained models, and only mix them if they disagree. Privacy leakage is measured by Renyi divergence, and operational privacy is guaranteed. 
main_review: Pros:
1. Submix does not need to modify training algorithm and existing large-scale language models are available.
2. Submix utilize probabilistic nature of next-token sampling to protect privacy.
3. Submix does not need to add noise for privacy of model prediction.

Cons:
1. Fail to show runtime comparison of proposed approach and existing work (e.g., DP-SGD). Except effectiveness analysis, efficiency analysis may convince reader more since less efficient approach may weaken its application in the real world. Authors can provide a chart with respect to runtime vs. epsilon.

Question:
1. Since this work uses non-private language models for fine-tuning, is it under risk of member inference attack?
2. About user-level corpus, since young people sometimes use similar internet words, phrases and sentences on their social media posts, D_i and D_j sometimes may not be disjoint set. How should Submix deal with this scenario?
summary_of_the_review: Sentence-level privacy in the language model is a very challenge problem since LMs applied to many tasks in NLP. To this end, Submix provides a scalable and flexible approach to address the issue by focussing on next-token prediction, which is popular for many downstream NLP tasks. Hence, I recommend to accept this paper.
review_title: null

Meta-review: This paper develops a new mechanism SubMix that provides next-token prediction under a variation of the differential privacy constraint. There is disagreement among the reviewers when assessing the quality of this work. Even though the study of private predictions in large language models is new, the reviewers raised several issues in the proposed approach. First, the formulation of partition-level DP created confusion about the privacy guarantees provided by the mechanism. Given the similarity to PATE, it might be useful to articulate if there is any difference between the privacy guarantee in this paper and the one of PATE. Second, the authors might want to further clarify the reason for having two sub-parts, which has also created some confusion. Even after reading the author's response and the updated revision, the AC still could not understand the relevant privacy argument. In summary, the paper may require further clarification and revision before it is ready for publication.
Decision: Reject