Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose TESSERACT, an aggregation scheme that is robust to the directed deviation attack (proposed in Fang et. al. 2020). 
main_review: Pros:
a. The defense is based on an interesting observation that, for a sufficiently small learning rate, as the model approaches optima in the benign setting, a large number of gradients do not flip their direction with a large magnitude. And if such a behavior is observed it is indicative of an attack. The paper defines a Flip Score for every received gradient update and uses it to identify and either reward or penalize the update.

b. The history of rewards and penalties for a client is maintained as a reputation score. Normalized reputation scores are then used to compute the global model for the next round (Algorithm 1).

c. In addition to defending against the directed deviation attack, the paper also proposes two adaptive attacks (Adaptive-Krum and Adaptive-Trim) in which colluding attackers knowing the parameters of TESSERACT adjust their attack vectors to escape the cut-off flip scores.

d. The paper provides a convergence analysis sketch and compares the performance of TESSERACT with a set of Byzantine Resilient defenses (Krum, Bulyan, Trimmed Mean and Median, etc.).

Weaknesses:

1. The paper proposes a defense against a specific form of attack and does not provide any guarantees or justification about why it should generalize against other more powerful forms of model poisoning attacks. On pg. 2, the paper argues (without justification) that other attacks are less damaging and essentially weaker than the attack in Fang et. al. However, one can argue that while the directed deviation attack, is designed to prevent model convergence, there are model poisoning attacks that not only insert targeted backdoors but also allow the model to converge (Bhagoji et. al., 2019: https://arxiv.org/pdf/1811.12470.pdf, Bagdasaryan et. al. 2020: https://arxiv.org/pdf/1807.00459.pdf). These attacks are thus more powerful and can also maintain stealth. 

2. While the authors provide two very interesting adaptive attacks, it is hard to generalize the strength of defense without any formal guarantees, In other words, can there be no other adaptive attack that can bypass this defense. For example, can an attacker target the reputation update mechanism over time?

3. Typically, for reasons of privacy, the gradient updates are usually encrypted before being sent to the server (secure aggregation scheme by Bonawitz et. al. 2017, https://eprint.iacr.org/2017/281.pdf). TESSERACT in its current form will be difficult to implement with such schemes.

4. In Fig. 2, both sub-figures are labeled (a) and there is an inconsistency between the sub-figure caption and the image caption. Figure caption describes (a) as an aggregation of benign updates and sub-figure caption describes (a) as an aggregation of malicious updates. Finally, the Flip Scores of the figures(y-axes) are very different, but what is confusing is that there are both benign and malicious clients in both. 
summary_of_the_review: Please see the detailed comments above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This submission with the title "Tesseract: Gradient Flip Score to Secure Federated Learning against Model Poisoning Attacks " discusses defenses against data poisoning in federated learning. The authors propose a novel defense against the recently popularized attack "Tesseract: Gradient Flip Score to Secure Federated Learning against Model Poisoning Attacks " by Yang et al. This attack reduces model availability by sending malicious updates from compromised client that maximize sign flips in the global model gradient. 
This defense then proposes a measure of change in gradient direction that can be evaluated for each local update and used to dynamically down-weight clients with a large number of flips in direction. 
main_review: This submission presents a reasonable and timely defense against a strong attack against data poisoning in federated learning, which is nice. The design of the defense is well-executed and the inclusion of a dynamic reputation is a good addition to its robustness. I do have a few comments,  mostly regarding the part of the paper concerning adaptive attacks:

* The submission discussses *an* adaptive attack against this defense, but I would like to understand and see more discussion on why this is a strong adaptive attack specifically. It is currently not clear to me that this is a strong adaptive attack. It seems that the attacker could mount a stronger adaptive attack by keeping track of its own reputation and send local updates are optimized under an additional linear constraint that includes its own reputation? It would be great if the authors could clarify my understanding in this matter.

* How does this defense perform compared to the other considered defenses when the number of malicious clients is unknown? For example, assuming c_max is fixed to m/5, but the actually number of malicious clients is decreased from m/5 down to 1. This would be especially interesting to compare to defense algorithms that require no knowledge of c_max.

And some minor comments and questions:
* GM and LM are used in the text before the variables are formally explained
* "We halve the reputation score of every client if any of the values grows so large that the softmax operation causes an overflow" -> I am surprised that this could even happen, given the sizes of m, n_r and mu?
* page 7: "here to how the effect on diverse datasets" ->here to show the effect on diverse datasets
* "We find empirically (result not shown) that FABA degrades fast" -> it would be better if the authors could include this result in their appendix (or clarify the sentence - I think a partial answer to this statement is contained in Fig3b?)
* The caption of Fig.2 says "(a) shows the results where only benign updates were aggregated using FEDSGD, and (b) shows the case where only malicious updates were aggregated", but the figure headings say the opposite
* For CIFAR-10 and Shakespeare only two clients are malicious, is the attack too strong to be mitigated if more malicious clients exist?
summary_of_the_review: In summary, I think this is a decent submission. I have some questions (the central one is the strength of the adaptive attack) which I would like to discuss with the authors and I am open to changing my evaluation accordingly.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studied a very important topic in the field of federated learning: how to efficiently resist untargeted model poisoning attacks. In order to defend against such a poisoning attack, the authors developed TESSERACT, an aggregation algorithm that assigns reputation scores to participating clients based on their behavior in the training phase and weights the client's contribution. Extensive case studies have verified the effectiveness of the algorithm. In particular, the experimental results show that TESSERACT provides robustness against even a white-box version of the attack. 
main_review: The strengths and weaknesses of this paper are summarized as follows:
Strengths:
+ The problem studied in this paper is important and needs to be solved in federated learning
+ Good writing

Weaknesses:
- Need to include more related work that is highly important
- Need more justifications about the novelty claims 
- Need to add some experiments under non-IID settings
- Need to unify the attack paradigm
- Insufficient theoretical analysis

Comments:
1. The following important references are missing:
[1] Kang J, Xiong Z, Niyato D, et al. Incentive mechanism for reliable federated learning: A joint optimization approach to combining reputation and contract theory[J]. IEEE Internet of Things Journal, 2019, 6(6): 10700-10714.
[2] Awan S, Luo B, Li F. CONTRA: Defending against Poisoning Attacks in Federated Learning[C]//European Symposium on Research in Computer Security. Springer, Cham, 2021: 455-475.
[3] Zhang J, Wu Y, Pan R. Incentive Mechanism for Horizontal Federated Learning Based on Reputation and Reverse Auction[C]//Proceedings of the Web Conference 2021. 2021: 947-956.
2. The untargeted model poisoning attacks (i.e., Full-Krum attack and Full-Trim attack) designed in this paper are vague. It would be better if the authors could formally define these attacks. Second, the authors need to explain how these attacks in this paper differ from [4].
[4] Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. Local model poisoning attacks to byzantine-robust federated learning. In 29th USENIX Security Symposium (USENIX Security 20), Boston, MA, August 2020.
3. The reputation-based techniques used in this paper are quite common and non-surprising. A lot of previous work (e.g., [1-3]) has used reputation-based techniques to mitigate the poisoning attacks. Therefore, readers may think that the authors just applied the reputation-based methods in federated learning. The authors need to provide more details to justify the novelty of this paper.
4. There are some grammatical errors and inappropriate formula symbols in the context. For example, “k=1,2,…,m” should be “$k=1,2,\ldots,m$.” 
5. The editorial quality of this paper is not always satisfactory. It contains quite a lot of inconsistent/non-precise descriptions, as also reflected in the above comments.
6. Theorem 1 lacks rigorous proof and complete theoretical analysis. It would be better if the author could give complete proof of Theorem 1.
7. The number of attackers has always been a very important hyperparameter (or factor) in model poisoning attacks. Therefore, it would be better if the authors could conduct more case studies to explore the influence of the number of attackers on different defense algorithms.
8. In addition, non-IID data seems also to affect the gradient direction (or value) of the client. Therefore, the authors need to add some experiments to illustrate the effectiveness of the proposed algorithm under non-IID settings.
9. In fact, the poisoning attack defended by the baselines chosen by the authors is different from the attacks designed in this paper. Then, it would be better if the authors tested the proposed defense method on the poisoning attack involved in the baseline schemes.
10. In general, the authors need to add more theoretical analysis and verification experiments.

summary_of_the_review: For now, the authors need to add more theoretical analysis and verification experiments. The reason is that there are still unfair comparisons in the comparative experiments in this paper. If the author can address the reviewer’s comments, I will consider giving it a score of "6".
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper tackles the problem of adversarial attacks in federated learning settings. The main proposal is a defensive technique to address the “byzantine generals” problem in federated learning: how to ensure that the general ML model is not affected by “poisonous” attempts made by corrupted clients.
The proposed technique is experimentally validated on four datasets, outperforms previous defensive methods, and the evaluation also considers adaptive adversaries with increasing degrees of knowledge. 

Overall, the presentation of the paper is very good.
The quality of the English text is good.
Figures are appropriate, Tables require some editing.
The topic addressed by the manuscript is trendy and in-line with ICLR’s scope.
The references should be improved
The contribution is significant

STRENGTHS:
+ Adaptive adversary 
+ Trendy subject (federated learning)
+ Evaluation on multiple datasets
+ Technically sound

WEAKNESSES
- Unclear assumptions and threat model.
- Problem or Feature space attacks?
- Lack of a concrete use-case
- Tradeoff?

main_review: I enjoyed reading the paper. In contrast to many research efforts on adversarial ML, this paper makes many security assumptions that set it apart with respect to the existing body of literature. I also praise the the consideration of attackers with varying “strength” and the different datasets. All these points make me lean to recommend acceptance.

Nonetheless, there are some issues that the authors could solve to further improve their paper. Let me elaborate on the above-mentioned weaknesses, starting from the most significant ones.

**Assumptions and Threat Model?**
This is probably the only “true” problem of the paper, which should be absolutely rectified.
I was not fully able to understand the assumptions made by Tesseract. Does it work “only” against the “directed deviation attack” proposed by Fang et al.? Or does it also protect against different attacks?
In general, Section 2.2, Threat Model, is not very comprehensive. The authors should better expand this section by clearly pointing out all the assumptions and requirements of the proposed method. This is especially true because the Fang et al. attack was proposed in 2020, and some of its assumptions are not yet well-known. Specifically, this statement is suspicious: “We assume a full-knowledge (white-box)
attack where the attackers have access to the current benign gradients.”. Does it mean that Tesseract only works under this assumption? I.e., the attacker knows, and exploits, the current benign gradients? This is a rather “unrealistic” assumption: I understand the willingness to work against “worst case” scenarios; yet, if such “worst case” scenarios are not realistic in the first place, then what is the purpose of the proposed mechanism? What benefit is there in protecting against an attack that will never happen in the first place?
I invite the authors to restructure this section by using the common taxonomies adopted in adversarial ML papers [I]. 


**Problem or Feature Space attacks?**
The authors perform their experiments on four well-known datasets: MNIST, CIFAR, Shakespeare, FEMNIST; for each dataset, a different (deep) ML model is targeted. Three of these datasets are of images, whereas Shakespeare contains text data.
There are different ways to create “adversarial examples”, depending on the ‘space’ where the perturbation is applied. As far as I am aware, the adversarial examples considered in this paper to perform the poisoned updates are created in the feature space. It would be a lot more interesting if at least one evaluation included adversarial examples generated in the “problem” space [A]—or, at the very least, considered samples generated by “physically realizable” adversarial perturbations [B]. I acknowledge that the method should work even in these circumstances, as the proposed Tesseract defense is agnostic of the process used to apply the perturbation. However, considering the strong relationship with (real) security that permeates the paper, I believe that a more convincing use-case would dramatically improve the quality of the paper. This is also motivated by the current state-of-the-art: after almost a decade of adversarial attacks, more recent efforts are leaning towards evaluation that consider more realistic circumstances, where the attacker is constrained by the limitations of the real world; this is even more true in “distributed system” scenarios, such as Network Intrusion Detection Systems, which bear a strong relationship with federated learning (e.g., [C, D, E, F]).
As such, I invite the authors to perform an additional “proof-of-concept” experiment where they consider adversaries with constrained capabilities. This is also motivated by the fact that some perturbations may yield different effects when created in the problem space (as shown in [A]).

**Tradeoff?**
A common problem in adversarial ML countermeasures is that they may degrade baseline performance [G, H]. Hence, I am interested in knowing how the proposed method responds when there are no “malicious” clients. Even if the baseline performance does not decrease, what is the overhead of the proposed method? For instance, in Table 2 the authors report some results for “Attack=None”, which I assume represent the accuracy when no attack takes place. However, all the rows of these experiments (namely, FedSGD, Tesseract, Faba, FoolsGold, FLTrust) consider hardening FL techniques; for instance, on MNIST the proposed Tesseract has an accuracy of 92.52 when no attack takes place—the best among all other defences. Despite being appreciable, I am interested in knowing the performance when NO defense is applied. Surely, the test accuracy in a “fully trusted” FL setting should be superior than 92.52. Hence, I ask: what is the ‘cost’ of Tesseract?

**Lack of a concrete use-case.**
I believe that the paper could be further improved with a concrete use-case, where the authors explain, step-by-step, how a (single, or multiple) attacker can compromise a federated learning system, and how the proposed method can help in solving such problem. Hence, I request the description of a concrete use-case explaining the abstract scenario reported in Figure 1. Such use-case can be at the basis of the “constrained” attack that I invite the authors to perform in my "problem space perturbations" suggestion.





Some additional issues:

•	In the Introduction, the authors state: “To counter this threat, a set of approaches has been developed for countering Byzantine clients in FL…”. I believe that “Byzantine Clients” is a wrong term: what is countered by Tesseract are not byzantine clients, but "unloyal" clients, that are “against” the byzantine clients (at least by referring to the well-known problem of the byzantine generals, which should agree on a method to reach consensus in the presence of unloyal generals).

•	The caption of Figure 1 has a typo “c out of m clients maybe be malicious”.

•	In Figure, the gradient “LM_{c-1}” is out of place. 

•	In Section 2, the authors state “Our simulation of federated learning consists of m clients, each with its own local data, but the same model architecture and SGD optimizer, out of which c are malicious, as shown in Figure 1”. Is there a minimum amount of “m”?

•	Figure 1 appears before Figure 2, but in the text it is referenced after Figure 2.

•	Putting Figure 2 so early on is very confusing. The “flip score” is a measure introduced for the first time in this paper. As such, any reader would be thrown off by such graphs before reading the paper, meaning that the findings of Figure 2 are difficult to interpret---during the Introduction---, as the flip score has not been defined yet. As such, such graphs are ultimately meaningless: I have to trust the authors that they correspond to “interesting” observations and “fair” experiments, which is not scientific.

•	The presentation and notation in the “Flip-score” (page 5) is very ugly and difficult to follow.

•	Section 5 should be merged in Section 6

•	W.r.t. Table 2, the authors state “We see that TESSERACT is the winner or 2nd place finisher in 7 of the 12 cells (benign + two attacks * 4 datasets)”. This should be better highlighted. I only see three bold values for Tesseract in Table 2.

•	W.r.t. Table 2, the authors state “We have not shown the test loss curve for Krum aggregation because of the large loss values.”. I invite the authors to report such values in Table 2, because the different “formats” of the three subtables (None, Full-Krum, Full-Trim) make this table very hard to interpret.



EXTERNAL REFERENCES

[A]: "Intriguing properties of adversarial ml attacks in the problem space." 2020 IEEE Symposium on Security and Privacy (SP). IEEE, 2020.

[B]: "Improving robustness of ML classifiers against realizable evasion attacks using conserved features." 28th {USENIX} Security Symposium ({USENIX} Security 19). 2019.

[C]: "Modeling Realistic Adversarial Attacks against Network Intrusion Detection Systems." ACM Digital Threats: Research and Practice. 2021.

[D]: "Constrained concealment attacks against reconstruction-based anomaly detectors in industrial control systems." ACM Annual Computer Security Applications Conference. 2020.

[E]: "Conaml: Constrained adversarial machine learning for cyber-physical systems." Proceedings of the 2021 ACM Asia Conference on Computer and Communications Security. 2021.

[F]: "Resilient networked AC microgrids under unbounded cyber attacks." IEEE Transactions on Smart Grid 11.5 (2020): 3785-3794.

[G]: "Adversarial example defense: Ensembles of weak defenses are not strong." 11th {USENIX} workshop on offensive technologies ({WOOT} 17). 2017.

[H]: "Deep reinforcement adversarial learning against botnet evasion attacks." IEEE Transactions on Network and Service Management 17.4 (2020): 1975-1987.

[I]: "Wild patterns: Ten years after the rise of adversarial machine learning." Pattern Recognition 84 (2018): 317-331.

summary_of_the_review: The paper tackles a very interesting problem and the many security considerations as well as the experiments on various datasets and comparisons with existing defenses are commendable.

Some issues (unclear threat model, flexbile perturbations in the feature space) still prevent me from recommending complete acceptance. More clarifications are necessary, and by adding some more "realistic" experiments I believe that the paper could be turned into a significant submission of ICLR.

I am recommending a "5", but my score can be easily increased to 6 by addressing the many clarifications expressed in my review. Further experiments and the concrete use-case would further increase my score.

___

AFTER REBUTTAL: score increased to 6 which I will increase additionally to 8 if the authors are willing to support the claim that Tesseract is a "secure by design" defense.

FURTHER UPDATE: score increased to 8, and I stand by my decision unless other reviewers point out that the claim of Tesseract being "secure-by-design" is flawed.
review_title: null

Meta-review: The paper presents a defense against the gradient sign flip attacks on federated learning. The proposed method is novel, technically sound and well evaluated. The crucial issue of the paper is, however, that this defense is specific to gradient-flip attacks. The authors show the robustness of their method against white-box attacks adhering to this threat model and claim that "an adaptive white-box attacker with access to all internals of TESSERACT, including dynamically determined threshold parameters, cannot bypass its defense". The latter statement does not seem to be well justified, and following the extensive discussion of the paper, the reviewers were still not convinced that the proposed method is secure by its design. The AC therefore feel that the specific arguments of the paper should be revised - or the claim of robustness further substantiated - in order for the paper to be accepted.  

Furthermore, as a comment related to ethical consideration, the AC remarks that the paper's acronym, Tesseract, is used by an open source OCR software (https://tesseract-ocr.github.io/) as well as in a recent paper: Pendlebury et al., TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time, USENIX Security 2019.

All of the above mentioned reservations essentially add up to a "major revision" recommendation which, given the decision logic of ACLR, translates into the rejection option.
Decision: Reject