Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents an attack on federated learning of Transformers which allows recovering the training sequences, based on a threat model in which the server is corrupt and submits a specially designed weight set which, once applied to the gradient computation of training sequences, will store all information necessary for the recovery of the training sequences in the gradient vectors. 
strength_and_weaknesses: The paper is very strong and convinces through its clarity despite many algorithmic steps requiring the solution of non-trivial problems, as well as the strength of its results: it leaves no doubt that in the absence of any other security measure, federated learning, with models even as complicated as Transformers, widely allows recovering training sequences.

On principle, it does not come as a surprise that this recovery should be possible; but the practical demonstration of this conjecture, together with a careful analysis, was needed and is provided by this paper.
clarity,_quality,_novelty_and_reproducibility: The paper is very clearly written, with a natural structure presenting the attack design before the recovery algorithm, and contains all necessary details. Notation is clear. Diagrams and illustrations are carefully executed and helpful. The paper is entirely self-contained so that only interesting but optional discussions are offloaded to the supplementary material.

The algorithm presented results from several, carefully assembled techniques (separating gradient signal, sequence reassignment, sequence vocabulary recovery, sequence clusters). 

The experimental evaluation is careful, well described and informative in that it explores other threat models, attack confidence, factors affecting attack accuracy.

The paper is largely reproducible: code is provided with notebooks and a CLI; runs on CPU; uses very standard datasets.

This work is novel as it departs from previous, weaker threat models. 
summary_of_the_review: The paper is of very good quality and established an important result, with consequences on future research and practical considerations.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces an attack that corrupts the transformer models to extract private data from users' updates in federated learning. The threat model assumes that the server can send a malicious update to corrupt the user-side models and then can extract the original words and sentences entered from users' updates. To mount the attack, the adversary needs to be in charge of the server, send one malicious model update to the user side, and then extract private information. 
strength_and_weaknesses: **Strength:**
- The paper is well presented, where comprehensive background information is provided. Technical details are well documented and explained.
- One main contribution of this work is the threat model. As a conceptual contribution, the threat model is well-motivated and explained in detail. The attack scenario, as explained by the authors, is also interesting. However, there is also a concern regarding this threat model. Please check below. 

**Weakness:**
- About the threat model contribution: will the malicious update be obvious to be detected by users themselves? After disabling all attention layers and most outputs of each feed-forward layer, will it be obvious for the users to realize the performance change of the transformer model? And, consequently, will it make the users stop adding new inputs? If so, this raises another relevant question: will the historical data be used to update the gradients? 
- Given the different threat models, the results provided by Figure 9 are interesting but less informative, since the attack budgets are different in TAG and the proposed approach.


clarity,_quality,_novelty_and_reproducibility: This work is well structured and presented. 
summary_of_the_review: This paper introduces an attack that corrupts the transformer models to extract private data from users' updates in federated learning. The threat model is novel and interesting. Comprehensive details are provided.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper focuses on the privacy problem under the setting of federated learning (FL), where a global model is trained by using the gradients computed by each private client. Although clients do not give out their private data but return the gradients computed using their data, the gradients may leak the private training data. The paper proposes an attack method that reconstructs the input text from the gradients. In this approach, the model weights will be reprogrammed such that by using the gradients, the attackers can solve the input text. Experimental results have shown that the proposed attack works well with different models, batch sizes, and sequence lengths.

strength_and_weaknesses: Strengths:
- The paper proposes a novel attack method to reconstruct input text from gradients by carefully reprogramming the weights in transformer models. The attack method is interesting and makes sense to me. 
- The paper is well-motivated and well-positioned. The attacking settings and threat models have been discussed well. The experimental results show the vulnerability of transformer models in a practical setting. I believe these results are insightful and are of interest to the NLP privacy/security community. 
- The paper is presented in a good structure. I like the presentation in which a (simple) single-sentence case is presented first and then multi-sentence cases. 

Weaknesses:
I don’t see a crucial weakness; but I would like to make the following suggestions which I think the paper can be improved upon.
- Figure 3 is not easy to follow, as long as the methods for reducing the probability of a collision.
- The authors are encouraged to provide more insights based on the experiments. For example, why is the 3-layer transformer more vulnerable?
- Is “total accuracy” a token-level metric or a sentence-level one?

clarity,_quality,_novelty_and_reproducibility: Clarity & Quality: The paper is presented in a fairly good flow. The work is solid.
Novelty: The setting/method is new in the NLP scope, although it is based on Fowl et al. (2021).
Reproducibility: The results should not be hard to reproduce.

summary_of_the_review: In summary, I think this work is novel and solid. I hold a positive attitude towards accepting this paper.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper presents a new data reconstruction attack for text data in federated learning. In the proposed attack, a dishonest server drafts corrupted model parameters and sends them to the clients. The server can reconstruct sequences of text by observing the gradients returned by the clients using the corrupted model parameters. The proposed attack is tailored to small Transformers. Experimental results were conducted on several text datasets, including wikitext, Shakespeare, and StackOverflow.
strength_and_weaknesses: Strength

+ The research topic is critical.
+ The proposed attack is somewhat new for text data.

Despite an important topic, the paper has several crucial problems, as follows:

Weaknesses

- The federated learning setting in this paper is unrealistic. There is no evidence to support that clients train BERT or Transformers on their local devices for keystroke prediction tasks in real-world applications (deployment of such a setting). Training BERT or Transformers requires computational power on the training devices. It will incur communication costs for the local devices as well. Assuming local devices could be personal computers or mobile devices, how practical is the setting in the real world? Are there any deployed systems for such applications? If this is a cross-silo setting, what is the benefit of using FL compared with centralized training independently among clients?

- The attacks appear to be easily detectable since many model weights are set to either 1 or 0. For instance, the $W_K$ matrix is set to identity 1, $b_K = 0$, $W_ Q = 0$, $W_V=1$, and $b_v = 0$. In the real world, when such conditions are blended into the actual parameter distribution making the attack stealthy? If clients check one of these conditions, they can easily ignore the training round, thus avoiding the attack. That poses a fundamental question: How do we theoretically and empirically understand the stealthiness of the proposed attack? In addition, the attacker updates the model weights to deactivate the attention and feed-forward layers. That introduces a significant gap between model utility before, during, and after the attack. This gap can be visible to the clients to detect the attack. How could the proposed attack avoid or minimize this gap? What is the fundamental advantage to make the attack severe?

- The paper does not provide any empirical or theoretical analysis of the proposed attack against privacy-preserving mechanisms, such as DP and local DP. It is unrealistic to assume that the clients naively follow the protocol set up by the server. Therefore, a data reconstruction attack needs to be evaluated against privacy-preserving mechanisms. The critical point is not the attack but the trade-off between the attack, privacy protection, and model utility. That will inform the practicability of the attack in real-world applications.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written, and I enjoy reading the paper. The implementation is available, indicating that the result can be reproduced with some effort. However, the paper does not advance my knowledge in data reconstruction attacks v.s. privacy-preserving and model utility in real-world applications. It appears that the paper will bring more confusion to the community instead of answering already-known but open questions.
summary_of_the_review: The paper studies a fundamental data reconstruction problem for text data in federated learning. However, there is room for improvement.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces a novel gradient inversion attack against NLP transformer models in a federated learning setting. The attack is constructed around a new threat model that allows the attacker (server) to deploy the model on user devices with poisoned model parameters, which are downloaded from the server and replaced by benign parameters with the next update. This poisoned model update disables the attention layers and (most) FF layers to facilitate the separation and recovery of input tokens. The proposed attack is then evaluated on different transformer-based models of varying model sizes.
strength_and_weaknesses: Strengths:
+ clearly written paper, which is easy to follow. Especially the given intuitions and visualizations of the attack are helpful for understanding.
+ interesting new attack setting (server sends poisoned model weights to facilitate the attack)
+ approach combines various ideas of previous research but also offers its own improvements
+ empirical results state strong attack success

Weaknesses:
- The interpretation of the empirical results could use a little more detail. For example, the paper shows that random data is more vulnerable than natural language inputs. This is a very interesting finding, and discussing possible reasons would improve the paper.
- I miss some discussion of the limitations of the approach, particularly the fact that the model after the malicious update won't produce any meaningful predictions on the user side (see next section for more details)
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is clearly written and easy to follow.

Quality: The overall quality of the paper is quite high. The provided figures illustrate the proposed concepts and make it easier to understand the approach. However, the font size in some figures (e.g., Fig. 2) is a bit small, and increasing it would improve readability.

Novelty: Whereas some parts of the attacks seem to be already known, the attack combines different approaches and adds its own developments to them. Also, a novel threat setting is introduced, which opens an interesting avenue for future research.

Reproducibility: Hyperparameters, model and dataset details and source code are provided. I did not run the code but I expect at least the reproduction of the attack is possible.

Some concerns regarding the limitations: The attacker first updates the model weights of a victim to deactivate the attention layers and most feed-forward layers to make gradient inversion and token reconstruction feasible. The victim then performs an update, which is sent to the server and inverted. So let us assume the model is used for something like keystroke prediction or translation. Whereas the model should behave as expected on benign weights, after replacing the model weights with the malicious server state, the model should in this particular state basically be useless in terms of prediction performance. I assume that any user then will then note this significant drop in model performance, either by a strong increase in the training loss or because the model produces meaningless outputs. So the attack is not as secretly performed as other attacks that are only based on the gradient updates and do not alter the victim's model. If this is true, it should at least be discussed in the paper as a limitation and maybe also a possible attack detection. However, I do not think this fact affects the overall contribution of the paper in a significantly bad way.

Small remarks:
- The term "fedAVG" should be introduced with a small sentence for readers not familiar with the term.
- Fig. 7: The "Loading MathJax" in the images should be removed.
- The ethics statement mainly discusses possible mitigations. However, a discussion on potentially negative impacts on the security and privacy of models and user data is missing.
summary_of_the_review: I like the proposed attack based on the idea of facilitating gradient inversion attacks by applying a malicious update to the victim model. It offers a new perspective to the research in this area and is well presented throughout the paper. I only have small concerns with the rather short interpretation of the evaluation results. Overall, I think the community will benefit if the paper will get accepted.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a novel federated learning attack, whereby a server sends weights to clients  appropriately designed so that it can recover the client inputs from client gradient computations. Several reviewers found it strong and convincing, particularly in (a) solving several constituent of non-trivial problems, (b) weakening assumptions/strengthening the attack compared to  previous work, (c) having a well motivated and interesting threat model and (c) establishing that even with models as complicated as Transformers, the attack allows recovering training sequences. The attack is tested on state of the art FL systems using transformers.

On the negative side, the most important criticism is that it is possible that the attack is severely hampered by the use of DP. Even though the paper includes an experimental study of DP defenses, this avenue is worth exploring further; seeing this explored more thoroughly in the paper would certainly improve the contribution, but discussions of attacks without countermeasures are also important contributions to the community-even for the purpose of exposing the inherent privacy issue, but also to further motivate DP research on the subject. Reviewers also would like to see more discussion and interpretation of certain phenomena observed in experiments, as well as a discussion on the visibility of the attack due to degraded model performance. The authors have provided responses to these issues; it would be good if they are also included in the paper.


justification_for_why_not_higher_score: A better exploration of DP defenses was the main reason why a single reviewer maintained a very negative score. 
justification_for_why_not_lower_score:  Another reviewer chimed in and pointed out that attacks are of independent interest and have appeared in several papers in the past even without explicit defenses. All in all, even though a better exploration of DP defenses would add value to the paper, at the very least the paper has the potential of motivating exploration of exactly this topic. All other reviewers liked the paper.
Decision: Accept: poster