Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper studies the vulnerability of Federated Learning (FL) Algorithm for untargeted poisoning attacks. In this direction, it formulates the problem of poisoning FL systems as a model-based Reinforcement Learning.  

The RL based attack consists of three key steps: learning the distribution of the data, learning the optimal attack policy by building a simulator from the learned distribution, and finally executing the attack policy. The work presents an upper bound on the attack performance as a function of fraction of malicious nodes in the FL setting and learning rate of the FL algorithm. 

strengths_and_weaknesses: The paper provides a three-phased framework for poisoning attacks, however, the novelty in each component is limited. The methods used are previously known and studied in the literature. 

Since the problem is modeled as RL, the paper does not compare the poisoning attack proposed in the paper with the poisoning attack proposed in the online RL setting. A similar comparison with the corresponding attack cost in terms of theoretical guarantees is also missing. 

This paper is also closely related to the misspecification of distribution of a prior in a Bayesian setting. This line of work is independently studied. Authors should consider a comparison and a literature survey in that direction, which is currently missing. 

Overall, the writing of the paper is unclear, and the contributions of the paper are limited. 


questions: Please see the weaknesses associated with the comparison with the literature, and address them.
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: - The authors propose a RL-based attack framework against federated learning (FL) systems. 
- The framework learn distribution of the dataset using the data of malicious workers and conduct model-based RL using the learned distribution. 
- The authors provide a theoretical analysis of the effect of inaccurate distribution learning on the difference in the optimal return. 
- The authors conduct model poisoning attack to FL system of MNIST and EMNIST learning under Krum and Clipping Median scheme. The experimental results show that the proposed RL-based method outperform the baseline methods.
strengths_and_weaknesses: # Strengths
- The experimental results on MNIST and EMNIST under Krum and Clipping Median show that the proposed RL-based attack method clearly outperforms the baseline methods.
- The authors theoretically analyze the effect of inaccurate distribution learning and explain how the attack performance loss depends on the ratio of benign workers, the inaccuracy of learned distribution. 
- The proposed attack method require less knowledge compared to existing attack methods on FL domain. 
- They suggest a state and action space compression technique to make the method scalable. 

# Weaknesses
- Lack of some important ablation studies such as actual runtime comparison. 
- It seems difficult to apply this method to datasets of large dimension such as ImageNet. Experiments were also conducted on small datasets such as MNIST and EMNIST. In other words, applications of the method seem limited.
- There is no related work section. 

# Other Comments
- typo) line 128 A^{\tau(r)} )
questions: - To apply the attack method on an online FL system, the actual runtime of the attack method  is very important. Even though they suggest a technique for scalability, many RL-based applications suffer from a large amount of computation and runtime. I think an analysis on the actual runtime (during the overall attack process) of the proposed RL method and baseline methods is needed.
- The distribution learning becomes harder as the dimension becomes larger due to the curse of dimension. Hence, It seems difficult to apply the proposed method to datasets of large dimensions such as ImageNet. If possible, I want to see the experimental results on larger datasets.  If there is not enough time to do this, I would like to see the performance of distribution learning in a large dataset.
- I think a related work section can help readers to understand the difference from the previous works and clarify the contributions of the paper.


POST REBUTTAL COMMENTS:  The authors answer the questions. My concerns have been addressed. They prove that they can apply their method to the larger datasets such as CIFAR-10. Also, they show that their proposed method without online search which has comparable performance in Krum is fast enough. Moreover, the proposed method with online search is fast enough to be executed in a real-world setting. So I decided to adjust my score from 5 to 6.
limitations: I wish that the authors include ethics or broader-impact statement in the paper or appendix.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The works aims to propose an adaptive model poisoning attack method against federated learning systems. To this end, the paper proposes a model-based reinforcement learning attack framework. To utilize the limited global knowledge, the proposed framework first conduct the distribution learning step, which approximates the distribution of the clients’ aggregated data. Then in the policy learning step, the malicious clients get the attack policy based on the learned FL environment and reinforcement learning, and finally generate attack actions. The authors also theoretically analyze the performance loss of the attack policy due to the inaccurate distribution learning and data heterogeneity. Through experiments with real-world datasets, the authors validates that the proposed framework could successfully poison the global model even when the server adopts a robust aggregation rule.
strengths_and_weaknesses: Strengths
* Privacy related issues in federated learning is getting more important. The paper proposes an adaptive attack to poison the global model by integrating distribution learning and RL-based policy learning.
* The paper provides theoretical analysis on the effect of inaccurate distribution learning and heterogeneous local data distributions on the performance loss of the attack policy.
* The proposed RL-based attack framework could be potentially extended to different attacks and be applied to other learning scenarios.

Weaknesses
* There have been proposed several defenses against gradient leakage attacks, e.g., adding noise, and most gradient leakage attacks are only effective on small batches. Both leads to lower estimation accuracy in the distribution learning step. Besides, the Theorem 1 shows that the performance loss is linearly related to the estimation accuracy $\delta$, which makes the distribution learning step crucial.
* Besides the robust aggregation rules, several defenses are proposed to defend against targeted/untargeted poisoning attacks based on abnormal detection (e.g., SPECTRE [1]). The experiments should consider this kind of defenses.
* More complicated datasets should be considered (e.g., CIFAR-10) in the experiments since the distribution learning could be less accurate.

[1] Hayase, Jonathan, et al. "SPECTRE: defending against backdoor attacks using robust statistics." arXiv preprint arXiv:2104.11315 (2021).

questions: * It seems that the robust aggregation oracle proposed in [1] have better defense performance against the data/model poisoning attacks. Is there any reason not including it in the experiments?

[1] Pillutla, Krishna, Sham M. Kakade, and Zaid Harchaoui. "Robust aggregation for federated learning." arXiv preprint arXiv:1912.13445 (2019).


POST REBUTTAL COMMENTS: Part of my concerns have been addressed and the authors promised to solve the others in the future.
limitations: As mentioned in the weaknesses, the distribution learning step could lead to significant attack performance loss. The defenses against gradient leakage attacks should be considered.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to simulate a RL environment to learn a poisoned gradient, which will be sent to the server for aggregation by some attackers. This paper achieves SOTA attack performance on MNIST and Fashion-MNIST. 
strengths_and_weaknesses: Major weakness: 
-	The novelty is limited. Why do we need RL here is not clear, compared with some adversarial manipulation on the gradients, e.g., directly maximizing the “reward” function w.r.t. gradients. RL is unstable and hard to train, the necessity should be addressed, otherwise, it seems like simply an adaptation of some techniques. 
-	The evaluation is far from enough. This paper should also include some results on colored images, e.g. CIFAR10. 
-	 In practice, noisy gradients may also be considered to provide better privacy. This paper proposes a technique heavily rely on the accuracy of gradients, so, an ablation study is suggested. 
-	This paper proposes a technique that seems to require a large amount of malicious users. An ablation study on the number of malicious users is suggested. 
-	Given the limited contribution of this paper, it is suggested that this paper provides some investigation on why gradients generated by RL can be better, what kind of properties it possesses, what is its advantages over other methods, or under what settings this method has an overwhelming advantage. This paper also addresses some potential future works, many of them can be integrated organically in this paper to increase this paper’s contribution, e.g backdoor attacks. 
-	Achieve SOTA on this task on some non-real world dataset is less significant, this paper is suggested to dig out some insights from other aspects, instead of just a SOTA. 

Minor weakness: 
- Typos & Grammars: Line 128; Line 304; 
- Writing: The reviewer doesn’t see any insights w.r.t data heterogeneity in section 4, while that is in the section title. 
- It seems that selected malicious clients are required to perform much much much more computations compared with other clients. Although this paper includes a paragraph discussing (line 266- line 274). Some quantitative measure may be preferred, e.g. seconds for FL epoch, seconds for RL epoch.


questions: -	In figure 3, the RL curve converges to random guess and has no oscillation, which is very weird to the reviewer. It seems like a NaN in gradient. 
-	In figure 4 c), the RL histogram performs very weird, especially 0.3, RL here seems to have no std? This is nearly impossible for RL. 
-	In experiments settings, there are in total 100 works with 20 attackers in it, this setting is so weird.  

limitations: N/A
review_title: null

Meta-review: metareview: The recommendation is based on the reviewers' comments, the area chair's personal evaluation, and the post-rebuttal discussion. 

This paper proposed a model-based reinforcement learning framework for data poisoning attacks on federated learning. All reviewers find the results convincing and valuable. The authors' rebuttal has successfully addressed the reviewers' concerns. Given the unilateral agreement, I am recommending acceptance
confidence: Certain
Decision: Accept