Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The submission titled "Recovering Private Text in Federated Learning of Language Models" discusses privacy attacks in federated learning that recover private information from user gradients. The proposed attack first recovers a bag of words from the user update and then applies a series of beam searches to uncover sentences of user data. The approach is validated for a GPT-2 model with a batch size of up to 128 and sentences from enron and wikitext103.
strengths_and_weaknesses: Overall I like the general approach proposed by the authors. The idea to re-use the language model itself to score its sentences and guide towards sentences likely contained in the gradient update is sensible and executed well. The submission features a range of interesting case studies, especially considering partial memorization and intermediate model states that I especially like. I do have a few smaller concerns regarding experimental evaluation,  ablation studies and related work that I would like to discuss:

* The ROUGE scores in Sec.5 are computed by comparing the recovered sentence to all ground truth sentences as possible references if  my understanding is correct. I wonder whether this measure is overestimating the effectiveness of the attack as the number of references increases. It would be good to include baselines of a) a random sentence generated by beam search from the current model and b) a sentence generated only from beam-search starting from the bag-of-words (so only the first step of the proposed search). I also wonder what the peak accuracy would be, i.e. comparing the recovered sentence to all reference sentences in direct accuracy and choosing the highest accuracy.

* Relatedly, this submission could be greatly improved with an ablation study of the sequence of improvements discussed in Sec.4, which I could not find in the appendix.

* This work mentions related work in Deng et al. "Tag: Gradient attack on transformer-based language models." and Dimitrov et al., "Lamp: Extracting text from  gradients with language model priors", but does not return to comparisons to these more recent works in the experimental evaluation. While this is (to me) understandable for the case of Dimitrov et al., given the complexity of the attack therein and the recency of that paper, but implementing Deng et al. should only be a minor change to the already present evaluation of the Zhu et al. baseline. In Dimitrov et al., the attack of Deng et al. was able to noticeably increase the baseline effectiveness of this type of matching attack.


Overall, these are the reasons why I currenly only score this work as a weak accept, but I am interested in discussing these and increasing my score in the future.





questions: Some questions for the author response:

* Is it not clear to me why this submission restricts the attack to only single sentences. What happens when the full range of 1024 tokens in GPT is filled by a user update, instead of only short sentences? Relatedly, what is the average sequence length evaluated in the experimental section? 

* Another question I have is concerning memorization. How many epochs are each of these models finetuned? Is it possible that the enron example leaks more data because every example is seen more often during training? Would it be possible to mark on Fig.4 when the data is first repeated? 

* The proposed approach seems limited to autoregressive models to my understanding? This would be ok, but I wonder if the authors comment on the feasibility of using this attack on a masked language model?



Other minor remarks: 
* There is a typo in line 285.

* The panels in Fig.5 are flipped (or at least referenced in text in the opposite way?)

* I further think this submission could be improved by discussing Dimitrov et al. in a bit more detail in the related work, given that both approaches use language priors and gradient information, just in different stages of the recovery (but I do not think a comparison is necessary).

* Minor: The submission discusses the oracle case where word frequencies are known to the attacker, and generally approximates word frequencies using the procedure proposed in Sec.4. However, word frequencies can also be recovered from the user gradients in addition to the basic bag-of-words by applying the approach of Wainakh et al, "User Label Leakage from Gradients in Federated Learning". This is also described in Fowl et al., " Decepticons: Corrupted transformers breach privacy in federated learning for language models". I wonder if this would be helpful to improve the attack?
limitations: The submissions discusses some limitations and ethical considerations and does so adequately.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this work, the authors present a model inversion attack for language models in the context of federated learning, termed FILM. A white-box attack scenario is considered, where an adversary has access to the global model computed by the central server and the local updates (gradients) of an (arbitrary) participating client. The proposed attack is carried out through three key pivotal steps. First, a bag of words is extracted through the non-zero rows of the client's embedding gradients. In the second step, sentences are reconstructed through a beam search algorithm. During the third, and final step, tokens appearing in the extracted sentence are ordered using a scoring mechanism that takes into account sentences' perplexity and gradient norm. Overall, this is an interesting and well-motivated work, with a clear structure, background, and contributions. However, elaboration is needed in the final empirical evaluation that is critical for replicating the presented results in federated settings and improving results completeness. 
strengths_and_weaknesses: Strengths:
1. Well-written paper, easy-to-follow with clear motivation and contributions.
2. Interesting language model inversion attack that challenges existing language models training approaches.
3. Empirical results demonstrate the effectiveness of the attack in challenging natural language tasks.    

Weaknesses:
1. Some details are missing from the attack setting.
2. Further discussion is needed in the evaluation section.
questions: W1: Why a larger batch size is important in Language Model (LM) tasks? It would be great if more background is given on the importance of the larger batch size. Moreover, does the attacker have full access to the entire vocabulary/embedding matrix during the white-box attack? It would be better to test the susceptibility of local datasets on a per-learner basis. For instance, what is the vulnerability (private text leakage) for each client? Do different data distributions (e.g., unbalanced, IID, or non-IID) affect the effectiveness of the attack? Is the attack method equally efficient across all clients' partitions? Moreover, what are some other federated domains that the proposed attack can be applied out of the box?     

W2: Even though this work is placed in the federated setting, it is written as if the attack is tailored for centralized settings. Please elaborate on the federated environments you have evaluated. How many clients have you considered? How were the data/sentences distributed across clients? Are the data distributions easy to replicate so that future research can build on top and construct defensive mechanisms? Also, how many federation rounds did you train the models for? What are the update frequency and the aggregation method of local models? Training iterations should be expressed in terms of federation rounds (e.g., ref.2 below). To demonstrate model vulnerability due to overfitting another dimension would be attack success (at every federation round) versus the average training loss of the global model across all clients' local datasets.


Suggestions / Text Edits:
- missing $\theta$ from prob. distribution in equation 1
- line 69, 89: remove extra space
- line 71, 76: change work to works
- line 134: iterations to iteration
- Table 3 shows the standard deviation for precision and recall but the number of conducted trials is missing
- line 20, 61: consider adding the following references as well:

	ref.1: https://www.ndss-symposium.org/wp-content/uploads/2020/04/diss2020-23004-paper.pdf 

	ref.2: https://proceedings.mlr.press/v143/gupta21a/gupta21a.pdf 
limitations: The authors have discussed proposed approach's limitations.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposed FILM, an inversion attack that reconstructs text given gradients and language model parameters in the federated learning setting. The threat model considered an adversarial eavesdropper who can extract gradients and model parameters. FILM first recovers the bag-of-words from gradients and then exploits the model memorization to reconstruct the order of the text. The attack was evaluated empirically on benchmark datasets and outperformed prior attacks that were proposed to recover images. No defense was evaluated against the proposed attack.

strengths_and_weaknesses: Strengths:
1. The paper is well-organized. The threat model and the attack methods are explained clearly and easy to follow.
2. The attack method is simple yet exploits the key vulnerabilities in training language models (i.e. memorization so that model would rank training data higher). The comparison with prior work also demonstrated that the attack is effective. 

Weaknesses:
1. Although the author acknowledged this as well, there was no evaluation on any mitigation. I think it is critical to know the privacy-utility tradeoff for different mitigation strategies to understand the real impact of any attack. 
2. Seems like there are also other easy defenses which the author did not discuss. 1). If the server and client encrypts their traffic then the eavesdropper could not decode anything from the cipher-text. 2). If the model is fine-tuned on all parameters except the word embedding layer then the adversary could not get the bag-of-words in the first place.
3. In Section 4.2, the author mentioned that they could not determine the frequency of words. I don’t think this claim holds as one could learn frequency information from the norm of word embedding gradients, i.e. more frequent words tend to have seen more SGD updates and thus larger norms.
4. The precision number in Table 3 seems to suggest that recovering multiple sentences would be hard with FILM and results in quite a number of false positives. 

questions: 1. Another common threat model is to have a malicious server who knows only about the gradients aggregated on multiple users on multiple batches. How would FILM work in such a threat model?
2. I would appreciate it if authors could provide an intuition behind using gradient norm in the scoring function in Section 4.4 instead of appendix so that readers could understand the method more easily.

limitations: The authors acknowledged their limitations but I think the paper can be improved if there were quantitative measurements on the limitations.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a new attack named FILM for extracting text from federated learning models.
The key idea is to use gradients with respect to embedding matrix in order to identify words occurring in the batch,
and then use beam search and token reordering to reconstruct a sentence from the batch.
The method is experimentally evaluated on several datasets which show benefits compared to the baseline.

strengths_and_weaknesses: I believe that this paper considers quite relevant problem as many practical applications of federated learning are in NLP space, while
existing attacks have mostly focused on reconstructing images.

However, I have found some issues with the paper which I hope authors can clarify. Due to these issues I currently think the paper is below the acceptance bar.

First, there is a serious limitation to this approach which I think should be emphasized.
This is the fact that the extraction approach described in Section 4.2 assumes that token embeddings are also being trained.
When embeddings are not trained, this approach doesn't work at all as it cannot compute bag of words in the batch.
Furthermore, this limitation does not exist in other papers that consider this setting, e.g. DLG (Zhu et al.), TAG (Deng et al.), LAMP (Dimitrov et al.) as they can utilize other elements of the gradient.
Therefore, since this attack is bypassed by just not training embeddings (which is sometimes even done by default), it is unclear how significant it is.

First, comparison with TAG (Deng et al.) and DLG (Zhu et al.) does not seem to match results in respective papers. Results in Table 1 show that DLG and TAG essentially can't reconstruct anything even for batch size 1.
Moreover, authors claim that TAG doesn't improve over DLG (footnote in Table 1). This all clashes with results in Deng et al. and Dimitrov et al. where DLG has much better reconstruction than shown in this submission,
and using TAG further improves the reconstructions.
In this paper, authors evaluate on completely new datasets than the prior papers (Zhu et al., Deng et al., Dimitrov et al.) so it is difficult to compare to those works, and might be a reason for this discrepancy.
I would suggest authors to experiment with datasets used in other papers (e.g. CoLA, SST-2, Rotten Tomatoes) so that we can compare the actual performance of different methods.

In Section 4.4 I find it a bit strange to optimize for the sentence which minimizes combination of prior and gradient norm.
Why should we not combine prior with L2 distance between the current gradient and the ground truth gradient, as in Zhu et al.?
Could authors experimentally demonstrate that their loss function works better?

Furthermore, Section 4.4 seems basically the same approach as discrete optimization from Dimitrov et al.
The only difference I can see is that this paper is using gradient norm instead of the L2 distance between the gradients (which is not clear why this works better, as discussed above).
Given that authors are aware of that work, they should acknowledge in Section 4.4 that this approach comes from Dimitrov et al., and discuss the differences.

As a more general comment, the paper at few places does not acknowledge contributions from other paper and tends to overclaim their own contributions:

- As mentioned earlier, Section 4.4 is basically the same as in LAMP (Dimitrov et al.), and yet there is no acknowledgment of this.
- In the paragraph starting on Line 244 authors write that they propose using ROUGE as a metric, but this was already used in TAG (Deng et al.) and should be acknowledged there.
- In L327, authors write "This paper presents the first attack that can recover sentences from gradients in federated learning of language models", but this is not true as such reconstruction has already been performed by Zhu et al., Deng et al., Dimitrov et al.

According to this, writing should change to account from contribution from other papers.

Typo:

L71: both work -> both works

questions: 1) The authors say they focus on extracting 1 sentence from the batch, but could they evaluate how well can they recover full batches, as that is the setting in other papers?

2) It would be good to see some analysis of which sentence in the batch is actually recovered. Is it always the shortest sentence?

3) As mentioned earlier, this attack is easily bypassed by a defense which does not train word embeddings which prevents reconstructing bag of words. Could you experiment with this defense and some other defenses (e.g. differential privacy based)?

limitations: Yes.
review_title: null

Meta-review: metareview: This paper proposes a novel gradient inversion attack to recover private text in FL training under the honest-but-curious server threat model. Through extensive experimentation, the authors demonstrate that their attack achieves improved attack performance compared to prior work under the same threat model, and performed plenty of ablation studies to analyze the effect of batch size, stage of model training, hyperparameters, etc.

One major weakness is that the method requires the token embedding layer to be trained, whereas real-world applications often use pre-trained embeddings. The authors showed that their attack performance degrades significantly when the token embeddings are frozen. Other weaknesses include limitation of the attack to autoregressive models, and inadequate discussion regarding the honest-but-curious threat model and secure aggregation (reviewer FFKN).

After the discussion phase, most reviewers agreed that the above weaknesses are minor and that the paper’s contribution warrants acceptance. AC therefore recommends acceptance to NeurIPS, but strongly encourages the authors to revise their draft to address concerns regarding frozen embedding—by performing more extensive experiments to demonstrate the method’s potential limitations, and more clearly position their attack under the setting of an honest-but-curious server with/without secure aggregation.

confidence: Less certain
Decision: Accept