FACMAC: Factored Multi-Agent Centralised Policy Gradients

Abstract: Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Boehmer, Shimon Whiteson

Reviews and author responses:

This paper proposes a new multi-agent deterministic policy gradient method, called FACMAC, which is essentially MADDPG + QMIX. Although the combination is rather trivial, the author claims that they improve MADDPG in the sense that FACMAC optimise in the joint-action space rather than each individual agent's action space as MADDPG do, and they also claim that FACMAC improves QMIX by removing the monotonic constraints on the critic decomposition.   Overall, I have trouble agreeing with the many claims that authors make about the MADDPG and QMIX method. More importantly, the idea of FACMAC =  QMIX + MADDPG is trivial, from both a theoretical perspective and an empirical perspective. I have concerns on the two critical tricks that FACMAC introduce (i.e. the nonmonotonic critic decomposition (Eq. 4) and the policy gradient over joint action space (Eq. 7)). Moreover, I am not convinced why such a combination is an important direction and worth attention from the MARL community, neither does the reported empirical results support that. Therefore, I cannot recommend for acceptance at this stage. 


1. In line 11, the author mentioned "FACMAC uses a centralised policy gradient estimator that optimises over the entire joint action space, rather than optimising over each agent’s action space separately as in MADDPG.". 

I believe this shall not be considered as a benefit of FACMAC over MADDPG. The idea of MADDPG to learn an individual policy without the need to learn a big joint Q-network. Following this logic, the centralised Q-learning method should be used in all MARL problems. Also, I don't think it is fair to claim that MADDPG only optimises in each agent's action space, after all the critic value that guides the policy to learn is the centralised critic, so presumably, what MADDPG is doing is searching over a joint-strategy space. 

Furthermore, FACMAC cannot optimise in the entire joint action space as the author claim. The main trick that FACMAC introduces is through Equation 7 in which all agents share the same policy parameter (i.e., CPG trick), so updating one agent's policy naturally equals to updating multiple agents' policy. However, this trick only holds for the case when all agents share the same set of policy parameters. Although the author claims that it also appleis in the non-sharing cases, however, if agents each have different set of parameters for its policy, then Equation 7 needs to take derivative at the same time for each agent, which again loops back to the issue of each agent only optimise in its own action space.  And importantly, it is commonly known that taking simultaneous gradients will lead to convergence issue. [On Gradient-Based Learning in Continuous G] https://arxiv.org/pdf/1804.05464.pdf
Moreover, the author also mentions in the line 187 that FACMAC-vdn is in fact optimising each agent's actions, which self-contradicts what the previous claims are about. 

The effectiveness of CPG is also questionable. Although in the toy example in Figure 2a, the CPG works with MADDPG. However, this could be because of the different exploration strategies in such an toy example, as pointed out by [Multi-agent Soft Q-learning, AAAI]. I suspect even by changing the payoff value of the toy example, it would show a completely different results. Most importantly,   Later in Figure, 7, the CPG trick almost makes no difference whether or not it is activated. 

2. In line 7, the author says "unlike QMIX, there are no inherent constraints on factoring the critic." In line 44, the author says "solve some tasks that cannot be solved with monolithic, or monotonically factored critics." I am not convinced how using a neural network-based mixing network in Equation 4 achieve the claim of releasing the monotonic assumption in meeting the individual-global-max condition. To me, FACMAC, which is QMIX+MADDPG, cannot claim anything beyond what QMIX has claimed. 

3. I also have major concerns on the reported results.

First, the SMAC tasks and Mujoco tasks share different baseline algorithms, even in the same task different scenarios, baseline methods are different. For example, the Fig 4c and Fig 5 c have different citations in [5] and [7], both of which although are irrelevant to the COMIX, COVDN, IDDPG methods. 

Secondly, across almost all results on MUJoCO, there is no evidence showing that FACMAC is in fact outperforming other baselines. 
Similar observations also hold on Fig6 on SMAC tasks. Either all baselines converge to 100% winning rate, or, the baseline methods are not trained to the best of the other reported results [cross reference the results in MAPPO]. 

Moreover, the SMAC task has many maps, can the author explain why the six specific maps have been chosen among the many possibilities, especially under these maps, the performance is not really discriminative. I would like to see the performance on the super-hard maps if the author tries to claim the state-of-the-art.


4. In line 75, "the first multi-agent actor-critic method to outperform state-of-the-art valued-based methods on SMAC.". This is an incorrect claim given the IPPO and MAPPO work. 

IPPO: [Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?]
MAPPO: [the surprising effectiveness of Mappo in cooperative, multi-agent games]

5. In line 160, it is unclear to me what is the motivation behind developing FACMAC-vdn and FACMAC-vdn-s, apart from trial-and-error testing different MARL architectures. 

6. The gumbel-max trick is notoriously hard to tune during many empirical studies, I was wondering if the author meets similar issues, and if so, mroe details need to be provided for future result replication. 

> ***Post-rebuttal***: I read all reviews, and the rebuttal to my review. I am not convinced by most of the questions, and during the rebuttal phase I found more problems in the replication of the results by running the submitted code. See my detailed comments.  I will maintain my score of strong rejection. Scores are adjusted post-rebuttal.

 The claim for MADDPG and QMIX need changing. 

The baselines need to stay consistent throughout different tasks (I am actually surprised why a naive TRPO/PPO is not a baseline in your mujoco tasks. I understand you design a multi-agent mujoco setting, but what is the point of using FACMAC if it cannot outperform TRPO. Maybe on many agent swimmer, TRPO would fail due to scalability).

The selected tasks cannot show the effectiveness of FACMAC (Figure 4 5 6 fail to show at all). On both Mujoco and SMAC, more solid results need providing.  



This paper proposes to apply the idea of value function factorization from recent value-based multi-agent RL work (e.g., VDN, QTRAN, QMIX) and apply it in the multi-agent actor-critic setting. Specifically, the paper proposes to use a centralized factorized critic, with either a monotonic (as in QMIX) or non-monotonic mixing of the individual value terms. The proposed approach applies the deterministic policy gradient method and is applicable to both discrete and continuous action spaces. The approach is compared to baselines in a series of particle world and SMAC benchmarks, and a new multi-agent MuJoCo benchmark proposed in the paper. Ablation studies reveal that scalability to more agents or actions is improved over MADDPG, and that nonmonotonic mixing of the value terms can be useful in some domains.  The main novelty and potentially most significant insight of the paper in my opinion is the concept of non-monotonic mixing. I have some concerns on originality, and some minor technical questions. Overall, the empirical results are thorough, and the paper is very clearly written.

Strengths:
The paper convincingly argues that non-monotonic mixing is applicable in a centralized critic in an actor-critic setting, as the monotonicity constraint required in value-based methods is no longer necessary. This is empirically demonstrated to be quite useful in some domains (Fig. 8). The empirical evaluation is thorough. The MAMuJoCo benchmark is a nice additional contribution, complementing the continuous particle environments proposed in the literature so far.

Limitations:
The idea of a factorized critic proposed in the paper is certainly reasonable, but I am not sure about the novelty. In [1], a similar critic factorization approach is investigated in the actor-critic setting. This paper should make clear what the similarities and differences to [1] are, and provide appropriate empirical comparisons. This being said, it does seem to me that the non-monotonicity of mixing proposed in this paper has not been considered before.

The paper identifies some domains where the non-monotonic mixing is useful. One limitation however is that the paper is a bit light on theory or insights about what such domains are like, and when one could expect non-monotonic critic factorization to be useful. The empirical results on non-monotonic mixing also have a minor role in the paper.

If the concerns above could be addressed, I am happy to revise my score.

[1] Jianyu Su, Stephen C. Adams, Peter A. Beling. Value-Decomposition Multi-Agent Actor-Critics. AAAI 2021


Detailed comments:
I have a technical concern regarding the centralized gradient estimator (Equation (7)). As far as I understand, the two gradients on the right hand side will be vectors of possibly different dimensions. How is it possible to multiply them as presented?

In several places in the paper, sampling from the policy is mentioned (e.g., Fig. 1, Lines 129-130, 174, 182). This is somewhat confusing, given that the paper learns deterministic policies. This should be clarified overall.

Minor:
Line 116-117: I don't think MADDPG learns reward functions.
Fig. 4-8: consider removing or explaining meaning of numbers in square brackets in the legend
Fig. 5 (right): legend placement is a bit unfortunate with overlap with the plots.

Post-rebuttal: I thank the authors for their detailed rebuttal. It addresses most of my concerns and I am increasing the overall score to reflect this. The rebuttal has made clear what the difference to Su et al. is. I hope to see a discussion and the empirical results in a revised version of the paper. I also hope to see future work providing actionable insights about which kinds of domains indeed have non-monotonic value functions fitting for the proposed method. Having briefly reviewed the provided source code, I would suggest to include specific commands and configuration files to replicate the experimental results provided in the paper. Yes

 **NOTE: there are *no* problems with the code that affect our experimental results. The problems encountered by the reviewer are due to setting some (hyper-)parameters to different or invalid values.**  

Thank you for your comments. 

”Plus, I have spent several hours running the code in the supplementary material, and find troubles repeating the results (even for the baselines)” \
\> The code we have provided is exactly the same code we used to run the experiments. We can investigate this issue if you provide a specific example of the command you ran and the results you could not reproduce.

“It becomes even unclear what is the motivation and findings of the whole paper after reading the rebuttal.” \
\> Our motivations are clearly stated in the introduction. The ‘basket of algorithms’ you mention are all introduced for a reason, and are not the result of a trial and error search. \
As mentioned in our earlier response, FACMAC-vdn and FACMAC-vdn-s are introduced to see what kind of a factorisation is necessary. The motivations for using a centralised policy gradient (CPG) are clearly outlined in Section 3.2. FACMAC (without CPG) is an ablation to study the importance of our proposed CPG. We clearly demonstrate both theoretical and empirical improvements when using CPG, as mentioned in our earlier response.

“Although this paper introduces an interesting setting of multi-agent Mujoco, but the author seems to be against adding comparisons even against the most naive TRPO/PPO method.” \
\> Multi-agent MuJoCo is a benchmark for continuous control in the Dec-POMDP setting. In order to compare against TRPO/PPO we would need to formulate it as a single-agent task that is not relevant to the subject of the paper, which is multi-agent RL.

Bug1 and Bug2 \
\> This has *no* effect on our experiments, which were all run with batch_size_run=1. All the config files (in src/config/algs/) we have provided for running our experiments use batch_size_run=1. We will update the code to prevent issues should any future experiments use larger values. 

Bug3 \
\> Again, this has *no* effect on our experiments. In PyMARL batch.max_seq_length cannot be 1. Even in an environment which terminates after a single timestep, the observations and states at the next timestep will be stored making max_seq_length > 1 always. Hence, even in the single-step matrix game experiments there are no problems.

“Although multi-threaded sampling is used in the code” \
\> PyMARL provides the ‘parallel’ runner which uses multi-*processing* in order to run multiple environments in parallel. We did not use this functionality for our experiments. Importantly, PyMARL's parallel runner makes a copy of the environment, one for each parallel process so that any given environment need not be aware it is being run in parallel.

“...the sampling efficiency is not improved...” \
\> Gathering experiences from multiple environments in parallel (in the manner suggested) is extremely unlikely to improve sample efficiency (performance compared to the total number of environmental interactions used). Hence, we only use a single environment to gather experiences.

“...the training time is too long.” \
\> We do not see why the speed of the simulator should be used as a criterion to evaluate our method. We already clearly demonstrate that our method is more sample efficient than competing approaches when considering the performance against the total number of training timesteps, which is an extremely common criterion used to evaluate deep MARL algorithms. 

“Let alone the reported results are much worse than the [MAPPO] and [IPPO] across all SMAC tasks.” \
\> As mentioned in our earlier response, MAPPO and IPPO use very different experimental setups. As such, you cannot directly compare the results, especially if the SC2 version differs (which it most likely does).

“Although the author insists that their method does not need a QMIX asssumption on the monotonic value function. However, by using a neural network-based mixing network in Equation 4, I don't see how the claim is solved. In fact, there are tons of MARL work that try to release this assumption, but in this particular work, there shows no new insights of how this can be addressed.” \
\> As mentioned in our earlier response, we are using an actor-critic setup and therefore there are *no* restrictions on the form of the critic. If there is a particular theoretical argument the reviewer is referencing, please mention it so that we can respond with the relevant context.

 I would like to thank the author team to provide a detailed rebuttal to my questions. However, I am not convinced for most of the questions I raised. Plus, I have spent several hours running the code in the supplementary material, and find troubles repeating the results (even for the baselines). I believe there is still a substantial amount of work that needs improving. Therefore, I would remain my score of *strong rejection*. Here are more details:

1. I am not convinced where the novelty of this paper lies. This paper essentially combines QMIX + MADDPG, which are two well-known baselines in MARL domain. With all respect, this paper is more like a technical report of experiments results. There is neither new theoretical insights nor substantial empirical performance improvement that are introduced on top of either QMIX or MADDPG. Especially, this paper has tried to introduce a basket of algorithms (e.g., FACMAC-vdn, FACMAC-vdn-s, ..FACMAC-CPG, FACMAC-w/o CPG, ...) by combining different components from QMIX and MADDPG, this makes me even concerned if there is any insight apart from trials and errors on different tricks. It becomes even unclear what is the motivation and findings of the whole paper after reading the rebuttal. 

2. Although this paper introduces an interesting setting of multi-agent Mujoco, but the author seems to be against adding comparisons even against the most naive TRPO/PPO method. 


3. Since this paper mainly sells its empirical performance, I spent 10 hours looking into the code that the author provides, but unfortunately, it is rather far away from being able to replicate the reported results, even I cannot replicate the MADDPG baseline results.    Overall, I am quite suspicious of the performance boost claimed in the paper.  Here is subtlety:

    - The code is ***not bug-free***, as follows: 

        - bug1: In src/controller/cqmix_controller.py line24, the parameter of self.forward  hidden_states=self.hidden_states[bs] is passed, but line114 still uses self.hidden_states in the forward function, and does not use the passed-in parameter hidden_states, which will cause the problem that the dimension of the hidden_state of the input agent does not match the dimension of the input when batch_size_run>1.

        - bug2: After fixing bug1, a RuntimeError: unsupported operation: more than one element of the written-to tensor refers to a single memory location. Please clone() the tensor before performing the operation.
bug3: src/learners/facmac_learner.py line 68-72 did not deal with the case of batch.max_seq_length==1. In fact, it will sample the sample of max_seq_length=1, and will report RuntimeError: stack expects a non-empty TensorList.

    - Although multi-threaded sampling is used in the code, the provided gym environment can only handle a single action meanwhile, the sampling efficiency is not improved, and the training time is too long.

    - The network structure diagram in the text should be a vector diagram.



4. In terms of experiment settings, the author seems to be selective on the map. For example, on SMAC tasks, they choose maps “MMM2” and “27m_vs_30m” for super-hard task, with no reasons explained why these two chosen among the many. I believe to report the SOTA results, a comprehensive list of experiments are needed rather than cheery picking. See similar practise in [MAPPO] and [IPPO] paper. Let alone the reported results are much worse than the [MAPPO] and [IPPO] across all SMAC tasks.

5. Although the author insists that their method does not need a QMIX asssumption on the monotonic value function. However, by using a neural network-based mixing network in Equation 4,  I don't see how the claim is solved. In fact, there are tons of MARL work that try to release this assumption, but in this particular work, there shows no new insights of how this can be addressed. 





 Thank you for your review. 

“This paper should make clear what the similarities and differences to [1] are, and provide appropriate empirical comparisons. This being said, it does seem to me that the non-monotonicity of mixing proposed in this paper has not been considered before.” \
\> We will include a discussion of this in a future revision as well as an empirical comparison. To summarise, the main differences between FACMAC and VDAC are: 1) FACMAC uses *deterministic* policy gradients and follows our novel centralised policy gradient to learn policies, while VDAC uses per-agent *stochastic* policy gradients, 2) FACMAC factors the joint action-value function $Q_{tot}$, while VDAC factors the joint value function $V_{tot}$, 3) FACMAC is off-policy (and hence avoids problems related to non-stationarity when training on older data), while VDAC is on-policy, and 4) FACMAC works for both discrete and continuous cooperative tasks, while VDAC only works for discrete cooperative tasks. \
In addition, our FACMAC framework allows for a more flexible factorisation of the joint-action $Q$. We have explored a new non-monotonic factorisation and demonstrated its advantages in some tasks as shown in Figure 8 to take advantage of this, whereas [6] only consider monotonic factorisation. We will make this clear in the revised version. 
 
We have also run VDAC-mix on SMAC. Our results show that FACMAC performs significantly better than VDAC-mix on *all* 6 SMAC maps we tested. We will include these results in a future revision, but cannot update the paper with these new experiments during the review period.

“One limitation however is that the paper is a bit light on theory or insights about what such domains are like, and when one could expect non-monotonic critic factorization to be useful.” \
\> Non-monotonic critic factorisation can be useful in tasks with *non-monotonic* value functions, as demonstrated by our experimental results in Figure 8. Joint action-value functions are characterised as non-monotonic if an agent’s ordering over its own actions depends on other agents’ actions [1] (as discussed in Lines 552-554 in Appendix A).

The consequences of a monotonic factorisation, including theoretical insights and discussions are included in other works: [1, 2, 3, 4]. We will highlight this in future revisions.

“The empirical results on non-monotonic mixing also have a minor role in the paper.” \
\> Extensively exploring the use of non-monotonic critics is an interesting avenue for future work. In this paper we first wanted to establish the benefits of a centralised policy-gradient estimator and to demonstrate that factored critics can achieve great performance in many tasks both discrete and continuous.

“I have a technical concern regarding the centralized gradient estimator (Equation (7)). As far as I understand, the two gradients on the right hand side will be vectors of possibly different dimensions. How is it possible to multiply them as presented?” \
\> This equation follows the standard deterministic policy gradient equation, an application of the chain rule, in which $\mathbf{\mu}$ is the policy. [5] provides details on its derivation.

“In several places in the paper, sampling from the policy is mentioned (e.g., Fig. 1, Lines 129-130, 174, 182). This is somewhat confusing, given that the paper learns deterministic policies. This should be clarified overall.” \
\> Figure 1 features an explicit sampling step because we sample from the categorical distribution when using discrete actions (described in Lines 218-220). Additionally, during training we need a sampling step to allow for exploration, e.g., using Gaussian noise for continuous actions, or epsilon-greedy for discrete actions. We will clarify this in future revisions.

“Line 116-117: I don't think MADDPG learns reward functions.” \
\> Thanks for pointing this out, we will fix it. We meant to say that each agent can have its own arbitrary reward function.

“Fig. 5 (right): legend placement is a bit unfortunate with overlap with the plots.” \
\> We will utilise the extra page allowed for accepted papers in order to improve the presentation of our figures, specifically the overlaps in the legends you mentioned. 

[1] MAVEN: Multi-Agent Variational Exploration. Mahajan et al. NeurIPS 2019. \
[2] QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning. Son et al. ICML 2019. \
[3] QPLEX: Duplex Dueling Multi-Agent Q-Learning. Wang et al.  ICLR 2021. \
[4] Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. Rashid et al. NeurIPS 2020. \
[5] Deterministic Policy Gradient Algorithms. Silver et al. ICML 2014. \
[6] Value-Decomposition Multi-Agent Actor-Critics. Su et al. AAAI 2021.

 “The gumbel-max trick is notoriously hard to tune during many empirical studies, I was wondering if the author meets similar issues, and if so, mroe details need to be provided for future result replication.” \
\> We will include the values of the temperature hyper-parameter setting we used in our gumbel-softmax sampling. We did not encounter any specific difficulties utilising it. In our experiments, it is trivial to switch between continuous and discrete action spaces for FACMAC, which is another strength of our method. We have provided significant additional details in the full code for reproducing all experiments. Additionally, Section 3.3 provides all of the details (once we include the temperature hyper-parameter, thank you for pointing this out) for independent reproduction.

“The baselines need to stay consistent throughout different tasks (I am actually surprised why a naive TRPO/PPO is not a baseline in your mujoco tasks. I understand you design a multi-agent mujoco setting, but what is the point of using FACMAC if it cannot outperform TRPO. Maybe on many agent swimmer, TRPO would fail due to scalability).” \
\> Our Multi-Agent MuJoCo benchmark is different to the standard MuJoCo benchmark that is commonly used in single-agent continuous control benchmarks. The purpose is not to outperform a single-agent setup, its purpose is to introduce a benchmark for continuous control 
in the multi-agent setting.  

[1] Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. Rashid et al. JMLR 2020. 

 Thank you for your review.

“Moreover, I am not convinced why such a combination is an important direction and worth attention from the MARL community, neither does the reported empirical results support that.” \
\> We believe that the factorisation of the critic and the centralised policy gradient estimator are worthy of attention in the MARL community, particularly to those researchers focused on the CTDE cooperative setting. First, we demonstrate in our results the factorisation of the critic can be crucial to ensuring good performance, which is definitely of interest to the Deep MARL community. Second, the centralised policy gradient estimator is necessary to fully reap the benefits of a centralised critic in our cooperative setting, which again is of interest to Deep MARL researchers.

“In line 11, the author mentioned "FACMAC uses a centralised policy gradient estimator that optimises over the entire joint action space, rather than optimising over each agent’s action space separately as in MADDPG." I believe this shall not be considered as a benefit of FACMAC over MADDPG.” \
\> Section 3.2 discusses the benefits of our centralised policy gradient estimator in detail, and our empirical results in Figures 2(a) and 7 demonstrate that it improves performance on three different domains (continuous matrix game, SMAC, and MAMuJoCo). Thus, we see no reason to discredit its benefits.

“Also, I don't think it is fair to claim that MADDPG only optimises in each agent's action space, after all the critic value that guides the policy to learn is the centralised critic, so presumably, what MADDPG is doing is searching over a joint-strategy space.” \
\> MADDPG uses a per-agent policy gradient (Eqn 3) in which the policy of each agent is optimised *individually* assuming that the other agents’ policies are fixed. We demonstrate a simple failure case for this in Figure 2, which clearly shows MADDPG without the centralised policy gradient is *not* optimising over the joint-action policy as a whole.

“Furthermore, FACMAC cannot optimise in the entire joint action space as the author claim … However, this trick only holds for the case when all agents share the same set of policy parameters.” \
\> The reviewer is mistaken. Eqn 7 optimises over the entire joint-action space irrespective of whether the agents are sharing parameters. One easy way to see this, is to imagine a $\mathbf{\mu}$ as being a single-agent’s policy that has a particular architecture (shown in Figure 1a). Then taking the gradient of the policy’s parameters, $\theta$, optimises over the policy's entire action space.

“And importantly, it is commonly known that taking simultaneous gradients will lead to convergence issue.” \
\> The paper referenced refers to a non-cooperative setting. In our particular fully cooperative setting, since there is a *single* shared reward function such pathologies do not arise. 

“Moreover, the author also mentions in the line 187 that FACMAC-vdn is in fact optimising each agent's actions, which self-contradicts what the previous claims are about.” \
\> There is no contradiction, because this particular claim *only* applies to FACMAC-vdn. If we use another form of factorisation for the critic, as FACMAC does, then it is no longer equivalent. Please see the paragraph starting at Line 187 for the description of this. 

“The effectiveness of CPG is also questionable. Although in the toy example in Figure 2a, the CPG works with MADDPG. However, this could be because of the different exploration strategies in such an toy example” \
\> Both methods are using *exactly* the same exploration in these experiments. Figure 2b, clearly demonstrates the differences between a per-agent policy gradient (which MADDPG uses) and our proposed centralised policy gradient. 

“Most importantly, Later in Figure, 7, the CPG trick almost makes no difference whether or not it is activated.” \
\> Figure 7 clearly shows that FACMAC significantly outperforms FACMAC (without CPG) on SMAC map MMM2, and has lower variance across seeds on 2c_vs_64zg. Furthermore, on ManyAgent Swimmer with 10 agents, FACMAC performs significantly better than FACMAC (without CPG). The only difference between these 2 methods is the use of our centralised policy gradient, so it clearly shows that CPG does significantly affect the performance. 

“I am not convinced how using a neural network-based mixing network in Equation 4 achieve the claim of releasing the monotonic assumption in meeting the individual-global-max condition.” \
\> The reviewer is mistaken. We do not claim to enforce consistency between the argmax of the centralised critic, and our greedy agent policies (which you refer to as the IGM condition). In fact, a huge benefit of our method, that we emphasise repeatedly, is that we do *not* need to enforce this consistency in the same way that QMIX does. Thus, we are free to factorise our critic in any manner we choose without restriction. 

“First, the SMAC tasks and Mujoco tasks share different baseline algorithms, even in the same task different scenarios, baseline methods are different. For example, the Fig 4c and Fig 5 c have different citations in [5] and [7], both of which although are irrelevant to the COMIX, COVDN, IDDPG methods.” \
\> The reviewer is severely mistaken. In our figures the number in square brackets is the number of random seeds. This is explained in the captions for Figures 4 and 5, that you mentioned. Some baseline algorithms are different on SMAC and MAMuJoCo since SMAC features discrete action spaces while MAMuJoCo features continuous action spaces. This is also why we introduce two new baselines COVDN and COMIX, to compare FACMAC against value-based approaches in continuous cooperative tasks. 

“Secondly, across almost all results on MUJoCO, there is no evidence showing that FACMAC is in fact outperforming other baselines.” \
\> The Multi-Agent MuJoCo benchmark, which we introduce in this paper, demonstrates the importance of factorising the critic. In these experiments, FACMAC is one of the best performing algorithms on *every* tested scenario. It performs significantly better than FACMAC-vdn, MADDPG, COVDN, and IDDPG. 

“Moreover, the SMAC task has many maps, can the author explain why the six specific maps have been chosen among the many possibilities, especially under these maps, the performance is not really discriminative.” \
\> The SMAC maps can be grouped into three categories including Easy, Hard, and Super-Hard, based on the overall performance difference of all algorithms [1]. In our paper, we consider 2 Esay maps (2s3z, MMM), 2 Hard maps (2c_vs_64_zg, bane_vs_bane), and 2 Super-Hard maps (MMM2, 27m_vs_30m), which cover *all* different difficulty levels of the maps on SMAC. 

“I would like to see the performance on the super-hard maps if the author tries to claim the state-of-the-art” \
\> We already present results on 2 super-hard maps in Figure 6, for maps “MMM2” and “27m_vs_30m” (and we mention they are super-hard maps in the caption of Figure 6 and Lines 307-308).

“Either all baselines converge to 100% winning rate, or, the baseline methods are not trained to the best of the other reported results [cross reference the results in MAPPO]” \
\> The paper referenced utilises a significantly different training setup and performs extensive hyperparameter tuning for their results. Nonetheless, our results are comparable at the 2 million timestep mark. Additionally, we utilise SC2 version 4.10 in our experiments which is not directly comparable to results obtained using 4.6. The paper referenced does not specify which version they use. We utilise the PyMARL framework for all of our experiments, and run all of our lines ourselves since it is extremely important to maintain a consistent experimental setup when comparing methods. Furthermore, the point of these experiments is to produce relative comparisons between methods and not merely present a higher bolded number in a table. 

“In line 75, "the first multi-agent actor-critic method to outperform state-of-the-art valued-based methods on SMAC.". This is an incorrect claim given the IPPO and MAPPO work.” \
\> This specific claim is not from our paper, it is from DOP (and we make this clear in Lines 73-75).

“In line 160, it is unclear to me what is the motivation behind developing FACMAC-vdn and FACMAC-vdn-s, apart from trial-and-error testing different MARL architectures.” \
\> This is done in order to “better understand the advantages of factoring a centralised critic”, as was remarked in the previous sentence on Line 159. These different critic architectures are important in order to better understand what kind of a factorisation is necessary or required. Moreover, these kinds of factorisations are extremely prevalent throughout the literature on value-based methods and have not been merely conjured up through trial-and-error testing. 

 Thank you for your review.

“I’m curious as to why QTran (Son et al., 2019) was not also included as a baseline?” \
\> QTran was not included since it is almost always outperformed by the other baselines on SMAC, especially more recent methods such as QPLEX which we do compare to. 

“In line 292 … It is true that FACMAC converges faster, but so does MADDPG and DOP, so this seems more a reflection on the policy gradient approach in general and not something specific to FACMAC.” \
\> Even though FACMAC and QMIX use exactly the same type of architecture for representing the joint-action Q, FACMAC is able to easily solve the task whereas QMIX struggles. Since the major difference between the 2 methods is in the difference between a policy gradient based approach and Q-learning, this demonstrates that a policy gradient based approach can better take advantage of the same factored architecture. \
In this particular map (bane_vs_bane) although factoring the architecture isn’t required (since MADDPG performs well), our other experiments demonstrate its benefits and show that in general it is very beneficial to factorise the critic. And so it is important to see that our method is able to perform more consistently than QMIX across maps. \
We do not think that the performance in bane_vs_bane is solely due to the use of a policy gradient based approach, since both COMA and Central-V do not perform well here. We will make this clearer. 

“Regarding the results in Figure 8 for the matrix game.” \
\> In this small environment, the extra architectural complexity of FACMAC-nonmonotonic can indeed slow down the speed of convergence compared to the much simpler MADDPG with CPG. Given enough time we would expect FACMAC to also solve the task, however due to its monotonic factorisation it cannot exactly represent the matrix game which then makes convergence to the optimal solution more difficult.

The paper proposes a new algorithm for cooperative multi-agent reinforcement learning (MARL) applicable in both discrete and continuous settings. The main innovation is an adaptation of multi-agent deep deterministic policy gradient (MADDPG). MADDPG trains decentralised policies using a centralised critic for each agent in the centralised training with decentralised execution (CTDE) paradigm. In this work, the authors specifically introduce a version of MADDPG with a centralised critic that learns a factored value function. Each factor is an agent-specific utility function with learned weights determined by a mixing network, as is done in popular value factorisation methods in MARL such as VDN and QMIX. In addition, the approach makes use of a centralised policy gradient estimator, where all agent’s current policy actions are considered during gradient updates to guard against pathologies such as over-generalisation. The authors refer to their algorithm as factorised multi-agent centralised policy gradients, or FACMAC for short. Extensive experimental results are presented to showcase the effectiveness of the proposed algorithm, along with multi-agent MuJoCo, a new multi-agent environment suite for continuous control, and ablation studies highlighting the specific benefits of value function factorisation, centralised policy gradients and the expressiveness of the factorisation function when applied to MADDPG. 
  **Originality**: In essence, the work represents a novel combination of MADDPG (Lowe et al., 2017) and QMIX (Rashid et al., 2018) with the addition of the idea of using centralised policy gradients. Combining both innovations to leverage their strength and applicability to discrete and continuous tasks is shown to be useful in settings not previously tested, using a large number of agents. The introduced multi-agent MuJoCo environment for continuous control builds on the single-agent MuJoCo engine, but is new and seems promising and useful as a benchmark for future research in continuous MARL.

**Quality**: The work is technically sound. Most of the claims in the paper are supported by a decent set of experimental results. Experiments are done over multiple seeds and include a wide variety of baselines and environments. The author’s motivation for their approach, citing pathologies such as over-generalisation, could perhaps be made more concrete but overall the presentation comes across as honest and the details provided are sufficient for a reader, interested in MARL, to follow and understand the work.

**Clarity**: The paper is well-written and the overall structure of the paper makes it easy to follow. Some minor remarks. Section 3.2 presents some experimental results before the experiments section, which was slightly odd, but I can also understand why the authors thought to present it earlier to motivate their argument for CPG. On a similar note, the authors reference Figure 11 in appendix D for this experiment, but the same figure is already present in the center of Figure 2(b), so they could just refer to that. Also, throughout the paper, the authors use “a” as the index symbol for agents and “u” for actions, whereas it is perhaps more the convention to use “i” for agents and “a” for actions. This change might make it easier for readers familiar with other MARL papers and notation to read the paper. 

**Significance**: In my opinion, the work will definitely be of interest to the community. The results seem to indicate that the approach shows significant promise at scale and nicely ties together different streams of investigation in MARL, namely, value decomposition methods and multi-agent policy gradient methods. Furthermore, the newly introduced multi-agent MuJoCo suite could greatly benefit future research as a benchmark for continuous control. 

**Questions**:

* The paper references PyMARL for specific implementations. I’m curious as to why QTran (Son et al., 2019) was not also included as a baseline? Since the library already has an implementation. In my opinion, this should not be held against the authors, since I think the number of baselines is already sufficient, especially with the introduction of COVDN and COMIX. It is just out of interest that I ask the question.  
* In line 292 it is stated: “For instance, on bane_vs_bane, a task with 24 agents, while QMIX struggles to find the optimal policy with 2 million timesteps, FACMAC, with exactly the same value factorisation, can quickly recover the optimal policy and achieve 100% test win rate. This shows the convergence advantages of policy gradient methods in this type of multi-agent settings”. It is true that FACMAC converges faster, but so does MADDPG and DOP, so this seems more a reflection on the policy gradient approach in general and not something specific to FACMAC. Do you think this might be a more honest reflection? Or is the statement more specific to factorisation not being useful in this case, i.e. PG with and without factorisation having no difference which places the focus on QMIX vs FACMAC and FACMAC benefiting from PG? In any case, perhaps worth rephrasing the claim to be less focused on FACMAC. 
* Regarding the results in Figure 8 for the matrix game. It is interesting that in Figure 2(a) it seems MADDPG with CPG can achieve convergence in just 50k steps, whereas FACMAC fails to converge and FACMAC-nonmonotonic takes around 200k steps to converge. Curious to know whether the authors might have any explanation as to why this might be the case? Is it perhaps that FACMAC also has to learn the mixing and hypernetworks, which significantly slows down convergence speed in such a small environment. Also, do the authors expect FACMAC to converge eventually, given enough time? 

**References**

- Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. In Advances in Neural Information Processing Systems, pages 6379–6390, 2017.
- Tabish Rashid, Mikayel Samvelyan, Christian Schroeder Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pages 4292–4301, 2018.
- Son, K., Kim, D., Kang, W.J., Hostallero, D.E. and Yi, Y., 2019, May. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning (pp. 5887-5896). PMLR.
 The authors highlight some cases where their approach fails to outperform specific baselines but makes no noticeable mention of the core limitations of their algorithm. In terms of societal impact and negative consequences, these are also not stated in the main text. However, given the nature of the work, I don’t feel too strongly about these remarks although it might be interesting for the authors to think about the downsides of FACMAC. There is a section in the appendix detailing some of the social impacts that might stem from the multi-agent MuJoCo suite. 

Meta-review: This paper generated an involved discussion between the reviewers and the authors, as well as between the reviewers themselves.
The paper essentially combines two well-known baselines in MARL domain, and therefore was judged as a report of experiments for which reproducibility is of particular importance. The reviewers had an intense discussion about reproducibility, two of them tried the code provided by the authors and one of them commented on it based on their experience. The analysis raised significant doubts about reproducibility (it is unfortunate that the authors did not provide configs on their experiments, to enable to rerun them exactly).

Decision: accept