Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper describes a federated reinforcement learning architecture that is expected to be tolerant to adversarial attacks or faulty agents. The architecture assumes a percent of the nodes are Byzantine agents that are actively trying to fool the central server, and assesses algorithm convergence and performance. Theoretical guarantees support empirical performance evaluation on some benchmark domains.

I have read the other reviews and the author responses. I agree that its difficult to do anything about the exploration policy without having further information. Although I still think this will be a limitation in the real world, drowning out interesting or unique outputs. My score remains borderline on this one.
main_review: The main idea of the paper seems to be to cluster good agents and bad agents using some form of variance based clustering. This idea is pretty intuitive,  and the paper is well written. Overall, I think this paper does a good job of advancing federated RL. However, I do have some questions:

1. In the situation when the agents are exploring, that is in the early stage of learning, it is possible that some agents will find very different (anomalous) outcomes than others. Not in all cases are those outcomes bad. What happens here?
2. Similarly, what is there is diversity in the agent experience, and this leads to seemingly anomalous outcomes, which are then dominated out by the others that cluster together.

It seems that if the above considerations are true, then the federated RL agent will end up ignoring critical outliers or learning experiences. Is there a way the authors could evaluate against this? I did not see discussion about the exploration phase in the experiments. 

The sample complexity guarantees look good, but again the thing that confuses me with those guarantees is what is the role of exploration policy? Obviously a bad exploration policy will destroy the guarantees. Are these upper bounds? I would have liked to see some more insights into what the theoretical results mean.


limitations_and_societal_impact: The authors do not seem to discuss societal impacts. Although the paper is theoretical and restricted to benchmark, it is true that federated RL will probably be used in a server node setting with many agents. Some of whom will be diverse and behave differently. It is important to know whether the idea of variance based clustering and rejecting of Byzantine agents based on that will hold in this case. Although I do not think that this is a major issue for this paper as is since its not focused on applications  currently.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors propose a federated reinforcement learning approach that combines existing stochastic variance-reduced policy gradient method with a robust aggregation method to achieve fault tolerance. Theoretically, the authors provide convergence and sample complexity results under reasonable assumptions. The proposed approach is tested empirically on various benchmark tasks.
main_review: Overall, the presentation of this work is clear and easy to read.

My main concern is regarding novelty. Without the fault-tolerance aspects, the algorithm and analysis would be very similar to that of [Xu et al 2020]. So the incorporation of fault-tolerance in terms of Algorithm 1.1 (ScsPG-Aggregate) is the only aspect that makes it "Federated". Other aspects of FL, such as heterogeneity of agents are not covered here.

Algorithm 1.1, however, seems quite similar to the works by Alistarh et al and Khanduri et al. I wonder whether the author can elaborate on the novelty introduced in this work? Perhaps there are additional challenges involved in integrating these works that I missed?


limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper aims to propose an algorithm for federated reinforcement learning with fault tolerance against Byzantine failure. The paper mainly considers the policy gradient method and proposes a server aggregation method with a Byzantine filter to reduce the effect of gradients from malicious agents. The paper also provides a theoretical analysis to support the design decision of the filter. 
main_review: It’s nice to see a fault-tolerant algorithm for FRL that is based on a theoretical guarantee. The algorithm can support up to (less than) 50% of agents being malicious which should be sufficient for real-world use cases. The empirical demonstrates the sample complexity does not degrade too much even some agents are malicious. 

Although this seems to be working well. I question whether the problem setting is realistic in the real world. All the agents are assumed to have identical MDP and transition probability, which seems to be very similar to a distributed RL training setting. If the environments are all the same among all the agents, why do we need to do federated learning, we can just train everything in a centralized fashion. Federated RL often contains some kind of data heterogeneity, where either the transition probability is different or the MDP differs slightly. Would the algorithm still work well if the agents are heterogeneous (e.g. have different transition functions)? Maybe, in this case, some of the good gradients would be quite different gradients from other agents. Can you provide more insights here?

In the experiment section, you selected three attack methods: random noise, random action, flip gradient. Although these methods seem to be valid, more sophisticated attacks should be considered to support the effectiveness of the algorithm. Some examples are: [1][2]. These attack may not requires large weight variance but be able to find specific values that would degrade the performance of the RL policy. This would even worst if the malicious agent collude when optimizing for their adversarial weights.

The aggregation method seems to require a pairwise computation among the weights of all the agents which may result in O(n^2) compared to O(n) in FedAVG. This may limits the scalability of the system. Is this true? Or is there ways to mitigate this?

[1] Fang, Minghong, et al. “Local Model Poisoning Attacks to Byzantine-Robust Federated Learning”
[2] Bhagoji, Arjun Nitin, et al. "Analyzing federated learning through an adversarial lens" 
[3] Bagdasaryan, Eugene, et al. "How to backdoor federated learning"


limitations_and_societal_impact: See the main review for the suggestion for limitations.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a federated reinforcement learning (FRL) algorithm that uses a variance-reduced federated policy gradient framework. The proposed framework is proven to have convergence and robustness guatantees (can tolerant the failures of up to half of the agents), with some assumptions on the environment and the agents. Both theoretical results and empirical results justify that the proposed federation can boost the sample efficiency of RL agents.
main_review: This paper proposes the first FRL method with formal convergence and robustness guarantees, which I think is a significant contribution to the community. The theoretical analysis seems to be correct, and the experiment results do support the theory. This paper is in generally well-written and easy to follow. I am not very familar with the related work, so it is hard to justify the novelty of the proposed idea. But this paper provides explanations about technical challenges that can distinguish this paper from prior works.

My main concern is that, although the analysis makes intuitive sense, is mainly built on some assumptions which might not be realistic. For example, assumption 2 is important to the algorithm and the guarantees, and the variance bound $\sigma$ is also used as a hyperparameter in Algorithm 1.1. How can we know the value of $\sigma$ in practice? How possible is that $\sigma$ exists and is not too large? If I am understanding it correctly, the gradient estimator lives in a high-dimensional space, and this assumption should hold for any trajectory generated from the environment given a certain policy, so $\sigma$ can be very large, which also makes the sample complexity high.

limitations_and_societal_impact: The experiment environments used in this paper is relatively simple, so it is not clear how the proposed algorithm scales to large environments and a large number of agents.
review_title: null

Meta-review: The reviewers in general agree that the paper considers an interesting new setting and this is one of the very first works that consider federated RL with theoretical guarantees, which make the reviewers in general lean towards an acceptance. But reviewers still have some concerns on the novelty of the proposed approach and whether or not the assumptions are realistic. 
Decision: Accept (Poster)