Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper studies the privacy issue of widely used neural language models in the current literature. The authors consider the privacy implication phenomena of two model snapshots before and after an update. The updating setting considered in this paper is kind of interesting. However, the contribution of the current paper is not strong enough and there are many unclear experimental settings in the current paper.

According to the current paper, the privacy implication seems to be defined in terms of general sequences in training datasets. If this is the case, I don’t think such privacy implication is meaningful because our language models should memorize some general information to achieve their tasks. 

There are some unclear settings in the experiments:
1.In the experiments, why there are only 20000 vocabulary size for Wikitext-103 datasets?
2.It is unclear how to construct canary phrases. 
3.After constructing the new dataset, the model is retrained or trained in the online way?
4.Since the results for Wikitext-103 is not finished, the authors should remove the results on this dataset.
5.What is the perplexity of the trained models?
6.How to choose initial sequence in real data experiments?
7.When you applying DP mechanism, how did you define the neighboring datasets, and how did you implement it (what is the clipping level, how did you calculate privacy loss for language models)?
8.$\epsilon=111$ seems that the model will provide no privacy guarantee according to the definition of differential privacy?
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper provides an empirical evaluation of the privacy implications of releasing updated versions of language models. The authors show how access to two sequential snapshots of a trained language model can reveal highly specific information about the content of the data used to update the model, even when that data is in-distribution.

The paper contains easy to understand, concrete experiments and results, but seems altogether a little underdeveloped. The methodology is sound, but the synthetic experiments around which much of the paper is based may not be sufficiently novel and give little indication of broader implications. It would have been more convincing if these results replicated with two splits of the same dataset, rather than identical datasets with one augmented by canary tokens. 
 
The qualitative evaluation of subject-specific updates is also not sufficiently informative. It would have been useful to define a specific attack and see under what circumstances such an attack would succeed. In the current results, I am not convinced that any of the phrases in Table 3 represent a privacy violation. 

The differential privacy experiment seems to be missing many details: what dataset was this trained on? Are the accuracy values for the training set or a separate testing set? Other works have shown that it is possible to train a differentially private language model without large sacrifices in accuracy, so it would be helpful to know what differentiates this experiment. 

I would also note that the motivation, a predictive keyboard, is not a situation in which maximizing accuracy is generally desirable: users tend to find this creepy rather than helpful.

This is a nice idea but would benefit from some more polishing and more extensive testing.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary: This paper looks at privacy concerns regarding data for a specific model before and after a single update. It discusses the privacy concerns thoroughly and look at language modeling as a representative task. They find that there are plenty of cases namely when the composition of the sequences involve low frequency words, that a lot of information leak occurs.

Positives: The ideas and style of research is nice. This is an important problem and I think this paper does a good job investigating this in the context of language modeling. I do hope the community (and I think the community is) moving towards being aware of these sorts of privacy issues.

Concerns: I don't know how generalizable these results would be on really well-trained language models (rnn, convolution-based, or transformer-based). The related work section doesn't seem particularly well put together, so its difficult to place the work in appropriate context and gauge its impact.

Other Thoughts: I'd like more thorough error analysis looking at exactly what kinds of strings/more nuanced properties of sequences that get a high differential score.

Overall I think this work is interesting and I would encourage the authors to try and add as much quantitative evaluation as possible, but also try and include qualitative information regarding specific sequences after prodding the models. Those could go a long way in strengthening the paper.
review_title: Official Blind Review #1

Meta-review: This paper report empirical implications of privacy ‘leaks’ in language models. Reviewers generally agree that the results look promising and interesting, but the paper isn’t fully developed yet. A few pointed out that framing the paper better to better indicate broader implications of the observed symptoms would greatly improve the paper. Another pointed out better placing this work in the context of other related work. Overall, this paper could use another cycle of polishing/enhancing the results. 

Decision: Reject