Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new attack on federated learning using text transformers, where the assumption is that the server is not honest-but-curious, but instead malicious and cannot be trusted. Current existing attacks in this scenario try to recover all input tokens of all users. The paper argues that this is not necessary, as an attacker is only interested in specific private information, which in reality is most likely only a small part of the whole sequence. To only reconstruct sequences containing sensitive data, the paper modifies the multi-head attention blocks as well as the linear layers of the transformer and filter for sequences containing specific keywords (e.g., like "credit card" or "social security number"). 
strength_and_weaknesses: Strengths:
- The idea of the paper to only reconstruct parts of sequences that are relevant for an attacker is novel.
- This is an important area of research, as the current real-world implementations are based on trustworthiness of the server. Showing that such an attack is possible is showing that changing the implementations and protocols might be necessary to ensure privacy.
- The paper is well written and most of the time easy to follow.

Weaknesses:
- It should be clearer how many heads are needed in total for the attack, as this might be a major limitation. Sudden drastic changes in the architecture when sending the modified model to the users might allow the users to detect this kind of attack.
- It is a bit hard to understand the reasoning behind some of the actions for the attack. For example, what is the reasoning behind shifting the entries to the first d' positions? Without deep knowledge of the related work this paper is building on, it is hard to follow the mathematical reasoning. A short sentence to explain the intuition would be nice and benefit the reader a lot.
- Experimental evaluation is limited. The experiments were conducted only on 2 models Fowl et al. (2022) is for example using another model (Transformer-3) for evaluating their approach.
- The metrics used for evaluating the approach are not explained in detail. This makes it hard to interpret the results.

Questions to Authors:
- Do we need one head for each keyword and additionally one for each of the positional imprinting of each keyword?
- If you write "We fix the sequence length to 32" do you mean that every sequence has the length of exactly 32 tokens or is it possible that sequences have less than 32 tokens which are then padded using padding tokens?
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is clearly written. However, the approach is quite complicated and some aspects of the approach could be explained in more detail without relying on related work.

Quality: The paper is well written and is mostly easy to follow. Parts where the paper is building on previous work could be explained better. However, the idea is clearly explained and supported by mathematical deductions.

Novelty: The idea of the paper is novel. In previous works the whole input sequence is reconstructed whereas in this paper only parts of the sequence which are relevant to the attacker are reconstructed.

Reproducibility: The experiments seem reasonable. However, no code was shared and a statement that the code will be made public upon acceptance was missing.
summary_of_the_review: The paper proposes a novel privacy attack on federated learning using text transformers. The problem statement is well motivated and it is shown that reconstructing only part of sequences that are relevant for the attacker is increasing the performance in contrast to previous approaches. The paper is well written and most of the time easy to follow. If the reader is not very familiar with the work that this paper is building on, it is hard to follow the reasoning.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper is about attacking the privacy of users when they do federated learning in specific settings. The setting is for training (large) language models that specifically use the “transformer” architecture. Moreover, the attack is done in the “malicious” setting, in which the central server who is running the federated learning does *not* behave honestly (& curiously) but it rather sends fabricated models to the users with the goal of extracting personal data from them. The paper is focused on extraction of *targeted* information; e.g., a credit card number as the string that follows the text “credit card”.

The paper compares its findings with recent works that also performs data extraction attacks on federated learning for the same architecture. Since the paper’s focus is on extracting targeted sentences it will do better in doing the exact goal (of this paper) because the other papers aim to extract texts rather indiscriminately. 
strength_and_weaknesses: Strength:
I like the aspect of “targeted” extraction of the texts. Here the attacker plants specific traps to extract specific texts from the model. This is a plus for this paper’s attack.

Weakness:
The fact that an attack is done in a malicious setting is a downside. This means that the users can *potentially notice* that the server is *not* following the protocol honestly. The paper has (as far as I can see) zero discussions on this important aspect, and I would like to hear from the authors about this, if possible: Can you argue that the (malicious) changes to the model are indistinguishable for the users?
clarity,_quality,_novelty_and_reproducibility: The writing quality is not great. The paper assumes a lot of familiarity with the transformer architecture and explains things bottom up. Indeed, the reader is facing a lot of technical terms and details without being given the big picture that is crucial.
summary_of_the_review: In summary, I find the targeted aspect of the attacks of this paper interesting and I think this will lead to more papers like this. On the other hand, the paper suffers from a presentation that is not suitable for general audience and does not investigate the limitations of their attacks; e.g., against users who inspect the models sent to them.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel privacy attack against FL systems, where the proposed attack extracts text sequences that contain targeted, privacy critical phrases. The targeted attack mechanism is achieved by maliciously modifying the model parameters on the server. In particular, the attack consists of three steps: tagging target sequences, filtering out irrelevant tokens and post-processing. According to the experimental results, the proposed method is more powerful than the selected Deception baseline.
strength_and_weaknesses: #### **Strengths**:
- The targeted attack setting is novel, well-motivated and interesting

#### **Weaknesses**:
- The designed three steps of the proposed attack match the intuition to perform targeted attack, but it seems to me that the proposed techniques are largely inspired by or directly adopted from Fowl et al. (2022), which makes the technical contribution of this work incremental. 
- Based on my understanding, this method requires the model gradients returned by the users. Is this the reason that the authors only performed experiments on FedSGD? What if the gradients are not available, how could the proposed attack work (for instance on FedAVG)? This raises my concern on the generality of the proposed method. Maybe the author could provide some discussions and insights on this point as in Fowl et al. (2022) and Fowl et al. (2021).
- In my opinion, the experimental setup is not very sufficient. Firstly, as mentioned above, only FedSGD is studied. Secondly, the selected evaluation metrics seem to be too general and could not really reveal whether the attack is indeed ‘targeted’. In the worst-case, the performance gain could be simply attributed to the additional attention weights added instead of the actually proposed method. It would be more convincing if the author can show that the proposed attack could recover the sequences with **targeted** key words.
- Finally, this might be not so important. But I am curious about whether the proposed attack could surpass very simple defenses? For instance, the users could just add some noises to the gradients when returning the gradients to the server. Will this break this attack? Not to mention more advanced defenses discussed in Fowl et al. (2022).

clarity,_quality,_novelty_and_reproducibility: - Novelty: This work presents a novel idea of targeted privacy attack against FL systems. This targeted setting is novel and well-motivated.
- Clarity: This work is well organized. The concepts are either clearly explained or highlighted with proper references.
- Quality: This work has some minor issues. Such as my concerns about the generality of the method as well as the experimental design on FedSGD.
- Reproducibility: I didn’t find the code of this paper. But I still trust the authors regarding their results.

summary_of_the_review: Overall, I think this work is interesting and I rated this work as 6. I do have some concerns and I am not an 100% expert on attacking FL systems, but I am willing to increase my score if the authors could address my concerns above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers the potential privacy leakage in federated learning settings. The paper argues that previous attacks are too ambitious to extract as much data as possible and this is at the expense of fidelity, so that previous attacks do not work under large-scale settings. The paper proposes an attack on FL which extracts input text that contains targeted privacy-critical words in language modeling training. In the proposed method, the parameters of the transformer are carefully programmed to filter relevant sequences from user data and encode the information in the gradients. The conducted experiments show that the attack works well and outperforms previous approaches in different settings; they show that the attack works even when the batch size is as large as 2000.

strength_and_weaknesses: Strengths:
- The paper is well-motivated. I agree with the authors’ arguments on the existing approaches and also believe that focusing on only privacy-sensitive data is a practical setting for attackers.
- The attacking setting (e.g., malicious server scenario), background, and related work has been discussed well. Generally, I like the positioning of this paper.
- Although the method is based on a couple of existing works, it is relatively novel. 
- The paper conducts extensive experiments to evaluate the attacks with different setups. The results really show the vulnerabilities of federated language model training in a practical large-scale setting.

Weaknesses:
- The paper can be presented better. The method is complicated and section 3.3 is not easy to follow. Since the approach is built based on existing approaches, some details are not included in this paper, which may confuse readers.
- The paper can be written better by discussing possible defense methods and it will also benefit the community and facilitate more research in this direction.
- The paper lacks qualitative attacking examples reconstructed by the approach. It is hard to have a sense about how strong the attack is by only looking at those metrics.
- What is the “total accuracy” metric exactly?

clarity,_quality,_novelty_and_reproducibility: Clarity: the paper can be presented better.

Quality: the paper is somehow solid.

Novelty: the paper is technically novel.

Reproducibility: the paper provides some experiment details in the appendix. In general, the method should not be very hard to reproduce.

summary_of_the_review: In general, I am positive towards accepting the paper. However, I also believe the paper can be better if there is another round of review.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the problem of obtaining private text data from federated transformer models. This paper studies the malicious server setting, where the server can arbitrarily tamper with the model sent to clients. This paper also studies a unique attack setting --- it focuses only on several privacy sensitive key words and the tokens following the key words, such as 'card number'. The proposed method elaborately designs transformer parameters such that the gradients directly reveal the token embeddings following the key words. Experiments are performed on real-world transformer models, where the proposed method outperforms a state-of-the-art baseline, Decepticon (Fowl et al. 2022). 
strength_and_weaknesses: Strength: 
- The problem setting studied in this paper is unique and practical. This paper studies malicious servers instead of honest-but-curious servers studied in existing works. This makes sense as one can never assume that an adversary is 'honest'. Also, the paper studies how to extract tokens following specific key words. This is also practical, since most sentences typed by users are not highly related to privacy. 

Weaknesses: 
- This paper does not seem self-contained and hard to understand without sufficient background. This paper heavily refers to related works (especially (Fowl. et al. 2022)) in the designs and equations. However, the intuition/implications of them are not sufficiently introduced. I list several examples here. (It is suggested that the authors number the equations). 
    - Eqn. 1 is derived from Phong et al. (2017). However, although I knew the related work and understood it, it is not immediate for those who are unfamiliar with the related work. Also, it seems that $\frac{\partial b}{\partial L}$ should be $\frac{\partial L}{\partial b}$. 
    - The equation following Eqn. 1 (Page 4) from Fowl et al. 2022 is also not immediate and hard to understand. Also, it seems that the equation only works for one layer of transformer block (otherwise, the input should not be token embeddings $e$ but output features from the previous layer). 
    - The first equation on Page 6 from Fowl et al. 2022 is hard to understand without interpretation. The authors should interpret what the equation intuitively means. (e.g. organizing $w^Tx_j$ into bins). 
    - The equation above Eqn. 5 from Fowl et al. 2022 is also hard to understand. I know it is more or less similar to Eqn. 2, but as they serve different purposes, it is still beneficial to explain it.   
    - Thus, the paper does not seem self-contained. This paper is hard to understand without sufficient knowledge on (Fowl et al. 2022). 

- It is not sure that the proposed method can be used in practice. I list several doubts on the applicability. 
    - First, it is not sure that the proposed method is compatible with model training. If the local model is training smoothly (local loss decreasing) but the global model is not training (e.g. loss significantly increases after server aggregation), clients will not have the motivation to participate. 
    - Second, the proposed method leads to non-negligible and weird parameter distribution. For example, a client can simply compute the norm of token embeddings and decide that the server model is weird and potentially malicious. By comparison, backdoor attacks on FL requires that the model outputs normally on normal data (Bhagoji et al. 2019, Bagdasaryan et al. 2019). 
    - Third, for the motivating example of credit card numbers, the sensitive information (i.e. the card number) is a string of numbers which is almost certainly out-of-vocabulary. Therefore, the card number may not even have a token embedding, making the retrieval impossible. 
    - Finally, the experiments use FedSGD (i.e. communication every iteration) as the attacked method. However, FedSGD is rarely used in practice for its high communication cost. In practice, FedAvg, which averages the gradients for multiple local steps (e.g. ~100) is used. It is not sure how this method works with FedAvg. Furthermore, if secure aggregation is used, the server only obtains aggregated gradients. Then, it is not clear how the retrieved information can be traced back to specific users. 
    - Of course, as I am not very familiar with (Fowl et al. 2022), a central related work of this paper, I may make misunderstandings. Please correct me if there are any. 
clarity,_quality,_novelty_and_reproducibility: Clarity: The clarity of this paper is poor. It is hard to understand for readers without sufficient knowledge of (Fowl et al. 2022). 

Novelty: The paper is novel. It identifies a practical and novel attack setting, i.e. malicious server and tokens following specific keywords. 

Quality: It is not very clear how the proposed method can be put in practice. Thus, I am doubtful about the quality of this paper. 

Of course, as I am not very familiar with (Fowl et al. 2022), a central related work of this paper, I may make misunderstandings. Please correct me if there are any. 
summary_of_the_review: From my perspective, the main problems of this paper are i) Not sufficiently self-contained; 2) Hard to be deployed in practice. Therefore, although this paper studies a practical and unique problem setting, I cannot recommend acceptance at this stage.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this work, the authors propose the first attack on federated learning (FL) systems that achieves targeted extraction of sequences that contain privacy-critical phrases. They use maliciously modified parameters to allow the transformer to filter relevant sequences from aggregated user data and encode them in the gradient update. The authors claim that their attack can effectively extract sequences of interest even against extremely large-scale aggregation. According to the authors this represents an improvement over current attacks, which are sequence-agnostic and aim to extract as much data as possible from an FL update at the expense of fidelity for any particular sequence. The reviewers thought that the paper presents an attack in a new and more difficult scenario and interesting area of research "as the current real-world implementations are based on trustworthiness of the server". 

The reviewers raised some concerns: (1) need more clarity on the number of heads (2) drastic changes in the architecture might make detection of attack easy (3) lack of clear intuition behind some of the logic (4) difficulty in following the mathematical reasoning (5) limited experiments. The authors provided a thorough rebuttal which addressed some of the concerns that were raised. However the reviewers did not change their scores. My own reading of the paper is similar to the reviewers. The paper studies an interesting novel setting but has some flaws. In my view the benefits of the paper and discussing it outweigh its flaws and therefore I recommend acceptance. However, I encourage the authors to address the technical issues raised by the reviewers in the final version of their manuscript.

justification_for_why_not_higher_score: The paper clearly has some flaws alluded to by the reviewers so does not merit a higher score.
justification_for_why_not_lower_score: This paper is borderline but I think the setting is new and interesting and more critical work like this needs to be done for federated learning so I lean towards accepting the paper.
Decision: Accept: poster