Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors propose a FACtored Multi-Agent Centralised PG (FACMAC) method to solve the fully cooperative MARL problems.  The proposed method is under the paradigm of centralized training with decentralized execution.  The critic of the proposed method is centralized and factored, in the sense that the joint action-value function is a combination of the utility of each agent. A new benchmark MAMuJoCo (multi-agent variant of MuJoCo) is proposed to measure the performance of FACMAC. And the proposed method behaves the best compared with several other popular MARL algorithms. 



main_review: The reviewer is not an expert in this area. From the reviewer's point of view, the paper combines the technique of several MARL algorithms such as MADDPG and QMIX. The novelty is that compared to MADDPG which optimizes each agent separately while fixing other agents, the FACMAC directly optimizes the joint action. Compared to QMIX, FACMAC allows the user to apply richer value factorizations, which can be nonmonotonic. Because this work is not a theoretical study that focuses on the sample complexity of a certain algorithm, the practical performance of the proposed framework will be the main judging criteria. From this perspective, FACMAC shows advantages in various tested benchmark problems against the compared algorithms. Therefore, the reviewer considers this as a good submission. 
limitations_and_societal_impact: This work is an algorithm that solves certain high-level abstract RL problems, it does not focus on any specific problems that may have negative societal impacts.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper presents a centralized multi-agent actor-critic method with a factored critic, which combines per-agent utilities (Q values) into a mixing function, as in QMIX. The difference is that it removes the constraint of monotonicity (using non-negative weights in the mixer) for higher representational capacity. It also proposes a centralized policy gradient estimator to fully take the advantage of the centralized critic, which updates policies over joint action space instead of optimising each agent's policy separately. Along with the FACMAC algorithm, Multi-agent MuJoCo, a new multi-agent benchmark suite, is introduced for cooperatively tasks with continuous control based on single-agent MuJoCo.
main_review: Generally, FACMAC is similar to a combination of MADDPG and QMIX in structure, e.g. the QMIX with actors or the MADDPG with a mixer. But this combination is valuable from the experimental results.

The author claims there are no monotonic constraints in FACMAC but no directly theoretical proof is found in the paper, which is said to be important for maintaining consistency between policies in QMIX. Is this because of the use of Centralised Policy Gradients? If so, rigorous proof will be appreciated.

In terms of Centralised Policy Gradients, the author has not provided the theoretical proof for why optimising joint policy is better than optimising each agent’s policy separately as well.

The Multi-agent MuJoCo is quite useful when I trying to test it with some benchmark algorithms, all agents have a clear division of labor and their cooperation is well emphasized.

In the experiment, there are many classic scenarios in MuJoCo but only three of them are selected to be tested. If more scenarios are tested, the results on continuous control will be more persuasive.

Pros:
Critic factorisation is more flexible without monotonic constraints.
The Multi-agent MuJoCo is a practical suite.
The Paper is overall well-written and clear.

Cons:
Lack of theoretical proof. e.g. Why monotonic constraints could be removed? Why do Centralised Policy Gradients help?
Novelty may not be enough. What's the key difference comparing to QMIX with actors/MADDPG with a mixer?
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposed the FACMAC method, an actor-critic approach for the cooperative MARL tasks. Under the centralized training with decentralized execution setting, FACMAC learns a centralized but factored critic and updates the decentralized policy of each agent using a centralized gradient estimator. Comparing to the existing approaches, the proposed approach 1) enjoys the flexibility to use nonmonotonic factored critics to handle the tasks that cannot be solved with monolithic or monotonically factored critics; and 2) using the centralized policy gradients for decentralized policy training prevents being stuck in suboptimal solutions.
main_review: The main novelty of the work is to use a centralized but factored critic and a centralized gradient for training decentralized policies. Some of the ideas are brought from existing approaches, but the authors are able to explain and compare the differences clearly. The extensive empirical studies show the potential power of the proposed approach, and also validates the advantages claimed in previous sections. The Multi-Agent MuJoCo benchmark built for continuous cooperative MARL can be meaningful. Overall, the materials are presented in a clear and precise way.

Major Questions:
1. Can the authors comment on what kinds of tasks cannot be handled by FACMAC and may require a FACMAC-nonmonotonic approach?
2. As there're no restrictions on the factored critics, is it possible for the proposed algorithm to go beyond cooperative tasks?
limitations_and_societal_impact: The authors addressed them.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed an actor-critic-based multi-agent deep reinforcement learning approach in a fully cooperative setting. Specifically, instead of using a monolithic centralized critic, the authors proposed to use a factorized critic, where the individual agent’s value function is nonlinear combined into a joint value function. In addition, the authors proposed a centralized policy gradient estimator where joint actions are sampled from current policies. The approach is evaluated on MPE, SMAC, and  multi-agent MuJoCo, which is a new multi-agent environment. 
main_review: The paper applied the factorized value function, which is often used in value-based approaches such as QMIX, to actor-critic based methods such as MADDPG. It is impressive that the proposed approach achieves good experimental results on SMAC which was mostly dominated by value-based approaches. The paper is generally well written and easy to follow. However, the reviewer has some concerns about the motivation of the centralized gradient and the scalability of the proposed factorized critic. Please see the detailed comments and questions below. 

1. As the motivation for the centralized gradient, the authors mentioned that assuming all other agent’s action is fixed while optimizing one agent’s policy will lead to suboptimal policies. The reviewer doesn’t see any intuitive or theoretical reason why this leads to suboptimal policies. Could you elaborate?

2. The authors claim that sample other agents’ action from the replay buffer leads to suboptimal policies. Again, it is unclear to the reviewer why this leads to suboptimal policies. 

3. The authors claim that the factorized critic is more scalable. How does it compare with GCN-based centralized critic, such as  [1, 2]? [1, 2] scales to hundreds of agents in the MPE environment. In this paper, only nine agents are considered in the MPE environment. 

[1] Graph Convolutional Reinforcement Learning, Jiang et. al. ICLR 2020

[2] PIC: Permutation Invariant Critic for Multi-Agent Deep Reinforcement Learning, Liu et. al. CoRL 2019



limitations_and_societal_impact: Yes, the authors discuss the limitation and  societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a centralized policy gradient actor-critic method for multiagent cooperation environments. The critic network follows the CTDE value-based training model, like VDN and QMix, but with flexible factorization forms. The actor network is updated by a centralized policy gradient. The proposed method is not just suitable for the continuous action space but also extends to the discrete action space. The method shows superior performance over other baselines in three domains, including a novel MAMujoco environment, which is first released with this paper.
main_review: Policy gradient methods have achieved success in the single-agent task, and promote it to multi-agent tasks is a promising and important direction. This paper takes advantage of two kinds of existed multi-agent methods and proposes a novel combination.  A multiagent benchmark MAMujocu is also released, which may as a good testbed for multi-agent research.

The paper is written well and easy to follow, related work and existing methods are well placed and cited.

The experiments are conducted in three domains both in continuous and discrete action space. The evaluations are fairly sufficient and encourage. I believe it could drive more research to further explore the policy gradient methods in multi-agent tasks.

Some weaknesses and questions:

1. Some experiment results in figures are not consistent with the original paper,  such as in the MMM2 map, DOP reaches about 50% win in the original paper but only 20% in this paper. I don't check all the plots so I can't guarantee there existed no more mismatches. The author should explain how the experimental data are acquired? Why not directly use the data from other related papers.

2. In my view, the whole method is essentially more like a centralized training method, if so, why not compare it with the centralized policy gradients methods, such as centralized DDPG or PPO? Does that may work? 

3. The DOP method utilizes offline training to enhance the efficiency of samples,  and the Q value target is combined with the online q target and tree-backup q target. I know FACMAC follows the deterministic policy gradient and extends to discrete tasks via Gumbel-softmax trick, but I still wonder why it works so nicely? Are there some advanced tricks or more analysis?

4. Different critic factorization leads to inconsistent superiority in different tasks. I suggest the author make a deeper study and give more reasonable explanations.
limitations_and_societal_impact: None
review_title: null

Meta-review: FACMAC uses the ideas of MADDPG and QMIX, where its structure has both a mixer and an actor. The proposed algorithm was empirical tested on MPE, SMAC, and multi-agent MuJoCo, and demonstrated superior performances. Reviewers appreciate the algorithm's empirical performances, especially because policy gradient-type methods didn't work well on such tasks (multi-agent mojuco) prior to this submission. Although the algorithmic idea is rather natural, the empirical results are strong for multi-agent cooperative RL. 

One remaining comment is that the paper is purely empirical and lacks theoretical proof to justify central policy gradient and policy consistency without monotonicity constraint. Although theoretical proof is not a must, we strongly recommend that the authors provide more theoretical justifications and explanations, so that people can learn useful insight from the pure empirical work. Another remaining comment is the method relies on centralized training and only the execution is decentralized. ers 
Decision: Accept (Poster)