FACMAC: Factored Multi-Agent Centralised Policy Gradients
Authors: Bei Peng
Tabish Rashid
Christian Schroeder de Witt
Pierre-Alexandre Kamienny
Philip Torr
Wendelin Boehmer
Shimon Whiteson
Venue: NeurIPS 2021 Poster (2021)
Keywords: multi-agent reinforcement learning
multi-agent policy gradients

Abstract: We propose FACtored Multi-Agent Centralised policy gradients (FACMAC), a new method for cooperative multi-agent reinforcement learning in both discrete and continuous action spaces. Like MADDPG, a popular multi-agent actor-critic method, our approach uses deep deterministic policy gradients to learn policies. However, FACMAC learns a centralised but factored critic, which combines per-agent utilities into the joint action-value function via a non-linear monotonic function, as in QMIX, a popular multi-agent $Q$-learning algorithm. However, unlike QMIX, there are no inherent constraints on factoring the critic. We thus also employ a nonmonotonic factorisation and empirically demonstrate that its increased representational capacity allows it to solve some tasks that cannot be solved with monolithic, or monotonically factored critics. In addition, FACMAC uses a centralised policy gradient estimator that optimises over the entire joint action space, rather than optimising over each agent's action space separately as in MADDPG. This allows for more coordinated policy changes and fully reaps the benefits of a centralised critic. We evaluate FACMAC on variants of the multi-agent particle environments, a novel multi-agent MuJoCo benchmark, and a challenging set of StarCraft II micromanagement tasks. Empirical results demonstrate FACMAC's superior performance over MADDPG and other baselines on all three domains.

FACMAC: Factored Multi-Agent Centralised
Policy Gradients
Bei Peng" University of Liverpool
Tabish Rashid" University of Oxford
Christian A. Schroeder de Witt" University of Oxford
Pierre-Alexande Kamienny
Philip H. S. Torr University Oxford
Wendelin Böhmer Delft University of Technology
Facebook Al Research
Shimon Whiteson University of Oxford
Abstract
We propose FACtored Multi-Agen Centralised policy gradients (FACMAC), method for cooperative multi-agent reinforcement learning in both discrete and con- tinuous action spaces. Like MADDPG, popular multi-agent actor-critic method, our approach uses deep deterministic policy gradients to learn policies. However, FACMAC learns centralised but factored critic, which combines per-agent util- ities into the joint action-value function via non-linear monotonic function, as in QMIX, popular multi-agent Q-learning algorithm. However, unlike QMIX, there are no inherent constraints on factoring the critic. We thus also employ a nonmonotonic factorisation and empirically demonstrate that its increased rep- resentational capacity allows it to solve some tasks that cannot be solved with monolithie, or monotonically factored critics. In addition, FACMAC uses a cen- tralised policy gradient estimator that optimises over the entire joint action space, rather than optimising over each agent's action space separately as in MADDPG. This allows for more coordinated policy changes and fully reaps the benefits of a centralised critic. We evaluate FACMAC on variants of the multi-agent particle environments, a novel multi-agent MuJoCo benchmark, and a challenging set of StarCraft II micromanagement tasks. Empirical results demonstrate FACMAC's superior performance over MADDPG and other baselines all three domains.
Introduction
Significant progress has been made in cooperative multi-agent reinforcement learning (MARL) under the paradigm of centralised training with decentralised execution (CTDE) 27 18 in recent years, both value-based [41 31 37 32 44, 30) and actor-critic [23] 9 36 13 approaches. Most popular multi-agent actor-critic methods such as COMA [9] and MADDPG 23 learn centralised critic with decentralised actors. The critic centralised to make use of all available information (i.e., it can condition the global state and the joint action) to estimate the joint action-value function Qiot, unlike a decentralised critic that estimates the local action-value function Qa based only on individual observations and actions for each agent Even though the joint action-value function
* 'Equal contribution. Correspondence to: Bei Peng <bei. wiwereeoleep.c....o
1 Work done while the authors were at the University of Oxford.
'COMA learns single centralised critic for all cooperative agents due parameter sharing. For each agent the critic has different inputs can thus output different values for the same state and joint action.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

these actor-critic methods can represent is not restricted, in practice they significantly underperform value-based methods like QMIX [31] on the challenging StarCraft Multi-Agent Challenge (SMAC)
K5
benchmark
R3 32
In this paper, propose novel approach called FACtored Multi-Agent Centralised policy gradients (FACMAC), which works for both discrete and continuous cooperative multi-agent tasks, Like MADDPG, our approach uses deep deterministic policy gradients [20] to learn decentralised policies. However, FACMAC learns a single centralised but factored critic, which factors the joint action-value function Qio into per-agent utilities Qu that are combined via a non-linear monotonic function, as in the popular Q Q-learning algorithm QMIX [3I]. While the critic used in COMA and MADDPG is also centralised, it monolithic rather than factored Compared to learning a monolithic critic, our factored critic can potentially scale better to tasks with larger number of agents and/or actions. In addition, in contrast to other value-based approaches such as QMIX, there are no inherent constraints on factoring the critic. This allows us to employ rich value factorisations, including nonmonotonic ones, that value-based methods cannot directly use without forfeiting decentralisability or introducing other significant algorithmic changes. We thus also employ a nonmonotonic factorisation and empirically demonstrate that its increased representational capacity allows it to solve some tasks that cannot be solved with monolithic, or monotonically factored critics.
In MADDPG, a separate policy gradient derived for each agent individually, which optimises its policy assuming all other agents' actions are fixed. This could cause the agents to converge to sub-optimal policies in which no single agent wishes to change its action unilaterally. In FACMAC, we use a new centralised gradient estimator that optimises over the entire joint action space, rather than optimising over each agent's ss action space separately as in MADDPG. The agents' policies are thus trained as single joint-action policy, which can enable learning of more coordinated behaviour, as well as the ability to escape sub-optimal solutions. The centralised gradient estimator fully reaps the benefits of learning centralised critic, by not implicitly marginalising over the actions of the other agents in the policy-gradient update. The gradient estimator used in MADDPG is also known to be vulnerable to relative overgeneralisation [7). To overcome this issue, in our centralised gradient estimator, we sample all actions from all agents' current policies when evaluating the joint action-value function. We empirically show that MADDPG can quickly get stuck in local optima in simple continuous matrix game, whereas our centralised gradient estimator finds the optimal policy. While Lyu et al. [24] recently show that merely using centralised critic (with per-agent gradients that optimise over each agent's 's actions separately) does not necessarily lead to better coordination between agents, centralised gradient estimator re-establishes the value of using centralised critics.
Most recent works on continuous MARL focus on evaluating their algorithms on the multi-agent particle environments [23] which feature simple two-dimensional world with some basic simulated physics. To demonstrate FACMAC's s scalability to more complex continuous domains and to stimulate more progress in continuous MARL, introduce Multi-Agent MuJoCo (MAMuJoCo), a new, comprehensive benchmark suite that allows the study of decentralised continuous control. Based on the popular single-agent MuJoCo benchmark [5], MAMuJoCo features a wide variety of novel robotic control tasks in which multiple agents within a single robot have to solve a task cooperatively.
We evaluate FACMAC on variants of the multi-agent particle environments 23 and our novel MAMuJoCo benchmark, which both feature continuous action spaces, and the challenging SMAC benchmark (35), which features discrete action spaces. Empirical results demonstrate FACMAC's superior performance over MADDPG and other baselines on all three domains. In particular, FACMAC scales better when the number of agents (and/or actions) and the complexity of the task increases. Results on SMAC show that FACMAC significantly outperforms stochastic DOP (46), which recently claimed to be the first multi-agent actor-critic method to outperform state-of- -het-rt valued-based methods on SMAC, in all scenarios we tested. Moreover, our ablations and additional experiments demonstrate the advantages of both factoring the critic and using our centralised gradient estimator. We show that, compared to learning a monolithic critic, learning a factored critic can: 1) better take advantage of the centralised gradient estimator to optimise the agent policies when the
In MADDPG, each agent learns its own centralised critic, as it is designed for general multi- learning problems, including cooperative, competitive, and mixed settings.
'We use "centralised and monolithic critic" and "monolithic critic" interchangeably to refer to the centralised critic used in COMA and MADDPG, and "centralised but factored critic" and "factored critic" interchangeably refer to the critic used in approach.
2

---

number of agents and/or actions is large, and 2) leverage a nonmonotonic factorisation to solve tasks that cannot be solved with monolithic or monotonically factored critics.
2 Background
We consider a fully cooperative multi-agent lask in which a team of agents interacts with the same environment to achieve some common goal. It can be modeled as a decentralised partially observable Markov decision process (Dec-POMDPP) [28] consisting of tuple G = (N,, S, U, P. S2 7) Here N = [l..................... n denotes the fimite set of agents and E S describes the true state of the environment. At each time step, each agent E N selects discrete or continuous action u. E U, forming a joint action u E U = U This results in a transition to the next state according to the state transition function P(s'|s, u) S x U x 5 - [0, 1] and team reward r(s. u). [0, 1) is discount factor. Due to the partial observability, each agent a E W draws an individual partial observation Og E Q from the observation kernel ((s.ol a). Each agent learns stochastic policy Ta (Ma (Mal or deterministic policy (a (Ta), conditioned only on its local action-observation history Ta E 7 (9t x U)" The joint stochastic policy π induces a joint action-value function: QT (st, Ut) 1 1 [Re|st, Uz),
where R2 = -i= T'T++ is the discounted return. Similarly, the joint deterministic policy μ induces a joint action-value function denoted (st, Uz). We adopt the centralised training with decentralised execution (CTDE) paradigm 27 [8] where policy training can exploit extra global information that might be available and has the freedom to share information between agents during training. However, during execution, each agent must act with only access to its own oostrv-observttion history.
VDN and QMIX. VDN [4]] and QMIX 33 are Q-lcarning algorithms for cooperative MARL tasks with discrete actions. They both aim efficiently learn a centralised but factored action-value function Q using CTDE. To ensure consistency between the centralised and decentralised policies, VDN and QMIX factor Th assuming additivity and monotonicity, respectively. More specifically, VDN factors Qto into sum of the per-agent utilities: Q (T, u: Φ) = Σ ===1 QT (Ta: Na: Φa). QMIX, however, represents Quo as a continuous monotonic mixing function of each agent's 's utilities:
(T, u, Φ v)) = f (s, Q (T1, "1: 11)....,,.9 (Tn, Un; 2n)). where afc 2 V, E N. This TQ. is sufficient to ensure that the global arg max performed on Quo yields the same result as a set of individual arg max performed on each 05e Here f is approximated by monotonic mixing network, parameterised by sb. Monotonicity can be guaranteed by non non-negative mixing weights. These weights are generated by separate hypernetworks [[2) which condition on the full state s. QMIX is trained end-to-end to minimise the following loss:
C(d, ()) = ED (1000
u, u, 11))
(1)
where the bootstrapping target yu = r + 7 maxu' u', (To ,' Here r is the global reward, and o= and v/- are parameters of the target Q and mixing network, respectively, as in DQN (26.. The expectation is estimated with a minibatch of transitions sampled from an experience replay buffer D (1I. During execution, each agent selects actions greedily with respect to its own Q
(T',
23 is an extension of DDPG (00 to multi-agent settings. It is an actor- critic, off-policy method that uses the paradigm of CTDE learn deterministic policies in continuous action spaces. In MADDPG, a separate actor and critic is learned for each agent, such that each agent can have its own arbitrary reward function. It therefore applicable either cooperative, competitive, or mixed settings. We assume each agent has a deterministic policy Ma (Ta: fa), parameterised by 0, (abbreviated as pa), and let μ = (Ta: 0=) be the set of all agent policies. In MADDPG, centralised and monolithic critic that estimates the joint valuue function Q! (s. ui...., . Uy 6g) learned for each agent a separately. The critic is said to be centralised as it utilises information only available to it during the centralised training phase, the global state and the actions of all agents, U1, " Un, to estimate the joint action-value function Qu, which is parameterised by Qa- This joint action-valus function is trained by minimising the following loss:
MADDPG. MADDPG
[(da)= = ED (
Qu ui, Uni oo))
'If the global state is not available, the centralised and monolithic critic can condition on the joint observations or ocserniobservation histories.
3

---

where y" = + QQth u4y ui....,, (P Here Ta is the reward received by each agent a, u4..... uf, is the set of target policies with delayed parameters 07 and 05 are the parameters of the target critic. The replay buffer DD contains the transition tuples (s, M,,...., Uny .i...., Tn).
The following policy gradient can be calculated individually to update the policy of each agent at
Vo ((u)) = ED (a. V., Qa Ma,...,, ", (x))
where the current agent a' 's action ua is sampled from its current policy Hı when evaluating the joint action-value function Qhe while all other agents' actions are sampled from the replay buffer D.
3 FACMAC
In this section, we propose a new approach called FACtored Multi-Agent Centralised policy gradients (FACMAC) that uses centralised but factored critic and a centralised gradient estimator to learn continuous cooperative tasks. We start by describing the idea of learning a centralised but factored critic. We then discuss our new centralised gradient estimator and demonstrate its benefit in simple continuous matrix game. Finally, we discuss how we adapt our method to discrete cooperative tasks.
3.1 Learning a Centralised but Factored Critic
Learning centralised and monolithic critic conditioning on the global state and the joint action can be difficult and/or impractical when the number of agents and/or actions large [33. We thus employ value function factorisation in the multi-agent actor-critic framework to enable scalable learning of a centralised critic Dec-POMDPs. Another key advantage of adopting value factorisation in an actor-critic framework is that, compared to value-based methods, it allows for more flexible factorisation as the critic's ' design not constrained. One can employ any type of factorisation, including nonmonotonic factorisations that value-based methods cannot directly use without forfeiting decentralisability or introducing other significant algorithmic changes.
Specifically, in FACMAC, all agents share centralised critic Q that is factored as:
(ha u, 9; Φ v²) = go (s, 1Q (Tas Uai & )=)))=
(4)
where & and o. are parameters of the joint action-valt value function Q and agent-wise utilities QK- respectively. our canonical implementation which we refer to as FACMAC, Dr a non-linear monotonic function parametrised as mixing network with parameters 1h, in QMIX (31. To evaluate the policy, the centralised but factored critie is trained by minimising the following loss:
[(d, 1b) ED (1V Qloer (T, 8; Φb 15)
(5)
where fot = + ( u(T': 0 5'; &f ..). Here the replay buffer, and 0_, "T and 2b are parameters of the target actors, critic, and mixing network, respectively.
Leveraging the flexibility of our approach, namely the lack of restrictions the form of the critic, we also explore a new nonmonotonic factorisation with full representational capacity. The joint action-value function Q is represented utilities Q#a This nonmonotonic mixing function is parameterised as a mixing network, with a similar architecture G in FACMAC, but without the constraint of monotonicity enforced by using non-negative weights. We refer to this method as D.CMMAC-nommoootonicccc Additionally, to better understand the advantages of factoring a centralised critic, we also explore two additional simpler factorisation schemes. These include factoring the centralised critic Q into a sum of per-agent utilities Qu in VDN (FACMAC-vdn), and as a sum of Q!- and a state-dependent bias (FACMAC-vdn-s) Our value factorisation technique is general and can be readily applied to any multi-agent actor-critic algorithms that learn centralised and monolithic critics 23
non-linear nonmonotonic mixing function of per-agent
3.2 Centralised Policy Gradients
To update the decentralised policy of each agent, a naive adaptation of the deterministic policy gradient used by MADDPG (shown in 3s)
Jo ED Vo, (Ta) nn Qua (T, un, ..., Un. 8) (u. (Ta)
(6)

---

Computing centralised gradient
Quo (T, u)
Quor (T, u)
MILP
Sampling
Sampling
Mixing Function
MLP
Tri (ut (m)
Tip (ur
On (T,
MLP
Actor
Actor n
Critic
Critic n
W
MLP
(of 1)
(oh, u
(of of
(oh wir
(Qu ud
(a)
(b)
(c)
Figure 1:
The overall FACMAC architecture. (a) The decentralised policy networks. There is a sampling step since we sample from the categorical distribution when using discrete actions. (b) The centralised but factored critic. (c) The non-linear monotonic mixing function.
Compared to the policy gradient used in MADDPG, the updates of all agents' individual deterministic policies now depend on the single shared factored critic Q as opposed to learning and utilising a monolithic critic Q! for each agent. However, there two main problems in both policy gradients. First, each agent optimises its own policy assuming all other agents' actions are fixed, which could cause the agents to converge to sub-optimal policies in which no single agent wishes to change its action unilaterally. Second, both policy gradients make the corresponding methods vulnerable to relative overgeneralisation [47] as, when agent a ascends the policy gradient based on Q or Q only its own action UA is sampled from its current policy Ha. while all other agents' actions are sampled from the replay buffer D. The other agents' actions thus might be drastically different from the actions their current policies would choose. This could cause the agents to converge to sub-optimal actions that appear to be better choice when considering the effect of potentially arbitrary actions from the other collaborating agents.
In FACMAC, action space, rather than optimising over each agent's 's actions separately as in both (3) and (6). to achieve better coordination among agents. In addition, overcome relative overgeneralisation, when calculating the policy gradient we sample all actions from all agents' current policies when evaluating Qte Our centralised policy gradient can thus be estimated
new centralised gradient estimator that optimises over the entire joint
VOJ(H) - ED Vou Qtior (T, (T) (T))
where μa (Mi(T); ............. nn (Tn; 0n)) is the set of all agents' current policies and all agents share the same actor network parameterised by 0. However, it is not a requirement of our method for all agents to share parameters in this manner.
the critic factorisation is linear, as in FACMAC-vdn, then the centralised gradient is equivalent to the per-agent gradients that optimise over each agent's 's actions separately. This is explored in more detail by DOP [46], which restricts the factored critic to be linear to exploit this equivalence. A major benefit of our method then, is that it does not place any such restrictions on the critic. As remarked by Lyu et al. [24], merely using a centralised critic with per-agent gradients does not necessarily lead to better coordination between agents due to the two problems outlined above. Even with factored critic, methods that use stochastic policy gradients can still suffer from the problems caused by the per-agent gradients if a fully factored stochastic policy is used. Our centralised gradient estimator, which uses deterministic policies and optimises over the entire joint action space, is required in order fully take advantage of a centralised critic.
Figure Dillustrates the overall FACMAC architecture. For each agent a, there is one policy network that decides which individual action (discrete or continuous) to take. There is also one critic network for each agent a that estimates the individual agent utilities Qa, which are then combined into the joint action-valle value function Qtoe via non-linear monotonic mixing function as in QMIX. Qios is then used by our centralised gradient estimator to help the actor update its policy parameters.
To show the benefits of our new centralised gradient estimator, we compare MADDPG with the centralised policy gradient (CPG) against the original MADDPG on a simple continuous cooperative matrix game with two agents. Figure 10 in Appendix D.I illustrates this matrix game. There is
5

---

MADDPG [10]
MACDPG CPG) (10)
14 Q
(4))
V Q (2))
90
50
100
150
200
(k)
(a)
(b)
Figure 2: (a) Mean test return on Continuous Matrix Game (b) Left: Per-agent policy gradient at the origin. For agent (similarly for agent it since the gradient term assumes the other agent's action to be fixed and thus it only considers the relative improvements along the dotted line (agent I's own action space). Right: Our centralised policy gradient correctly determines the gradient for improving the joint action.
narrow path (shown in red) starting from the origin (0,0) to (1, 1), in which the reward gradually increases. Everywhere else there is small punishment moving away from the origin, increasing in magnitude further from the origin. Experimental results are shown in Figure 2(a) MADDPG quickly gets stuck in the local optimum within 200k timesteps, while MADDPG (with CPG) robustly converges to the optimal policy. Figure 2 visualises the differencess between the per-agent and centralised policy gradients, demonstrating that the per-agent policy gradient can be wrong in continuous matrix game and our centralised policy gradient is necessary to take advantage of the centralised critic. In Section [ we further demonstrate the benefits of our centralised gradient estimator in more complex tasks.
3.3 Discrete Policy Learning
As FACMAC requires differentiable policies and the sampling process discrete actions from a cate- gorical distribution is not differentiable, we the Gumbel-Softmax estimator [14] to enable efficient learning of FACMAC on cooperative tasks with discrete actions. The Humbel-Softmall estimator is a continuous distribution that approximates discrete samples from categorical distribution to produce differentiable samples. Moreover, we use the Straight-Through Gumbel-Softtax Estimator [14] to ensure the action dynamics during training and evaluation are the same. Specifically, during training, we sample discrete actions ua from the original categorical distribution in the forward pass, but the continuous Gumbel-Softmas sample Tg in the backward pass to approximate the gradients: Vo, R Vo_ To- We can then update the agent's 's policy using our centralised policy gradient:
VaJ(x) ED V (T, 21........... s) where x = (. In. is the set of continuous samples that approximates the discrete agent actions. The softmax temperature hyperparameter is set to be in our experiments.
4 Multi-Agent MuJoCo
The evaluation of continuous MARL algorithms has recently been largely limited to the simple multi-agent particle environments 233. We believe the lack of diverse continuous benchmarks is one factor limiting progress in continuous MARL. To demonstrate FACMAC's scalability to more complex continuous domains and to stimulate more progress in continuous MARL,, we develop Multi-Agent MuJoCo (MAMuloCo), a novel benchmark for continuous cooperative multi-agent robotic control. Starting from the popular fully observable single-agent robotic MuJoCo [4] control suite included with OpenAI Gym (5), create wide variety of novel scenarios in which multiple agents within a single robot have to solve a task cooperatively.
This design offers important benefits. It facilitates comparisons to existing literature on both the fully observable single-agent domain 29], as well as settings with wwwbbandiithh communication [45]. More importantly, it allows for the study of novel MARL algorithms for decentralised coordination isolation (scenarios with multiple robots may add confounding factors such as spatial exploration), which is currently gap in the research literature. MAMuJoCo also includes scenarios with larger and more flexible number of agents, which takes inspiration from modular robotics [49 [9]. Compared traditional robots, modular robots are more versatile, configurable, and scalable. We thus

---

A)
8:
Legend
Slite jaint (glatse))
Hinge juint
12-11
Hige (gubbl))
Hj
K]
agty
Figure 3: Agent partitionings for MAMuJoCo environments: A) Manyagent Swimmer, B) 3- Agent Hopper [3x1], C) 2-Agent HalfCheetah [2x3], D) 6-Agent HalfCheetah [6x E) 2-Agent Humanoid and 2-Agent HumanoidStandup (each [1x9.I88), F) 2-Agent Walker [2X3]. G) 2-Agent 2-Agent Ant [2x4], 1) 2-Agent Ant Diag [2x4], 4-Agent Ant [4x2], and K) Manyagent Ant. Colours indicate agent partitionings. Each joint corresponds to a single controllable motor. Split partitions indicate shared body segments. Square brackets indicate [(number of agents) x (joints per a.ent)]. Joint IDs are in order of definition in the corresponding OpenAl Gym XML asset files [5]. Global joints indicate degrees of freedom of the center of mass of the composite robotic agent.
Reacher [2x1], H)
develop two scenarios named ManyAgent Swimmer and Many Ant, which one can configure an arbitrarily large number of agents (within the memory limits), each controlling consecutive segment of arbitrary length.
Single-robot multi-agent tasks in MAMuJoCo arise by first representing a given single robotic agent body graph, where vertices (joints) are connected by adjacent edges (hody segments), as shown in Figure 3] We then partition the body graph into disjoint sub-graphs, one for each agent, each of which contains one or more joints that can be controlled. Note that in ManyAgen Swimmer (see Figure 3A) and ManyAgen Ant (see Figure 3K) the number of agents are not limited by the given single robotic agent. See Appendix for more details about MAMuloCo.
Experimental Results
In this section we present our experimental results on our cooperative variants of the continuous
simple tag environment introduced by Lowe et al. 23 (we refer to this environment as Continuous Predator-Prey), our novel continuous benchmark MAMuloCo, and the challenging SMAC (35] benchmark with discrete action spaces. In discrete cooperative tasks, we compare with state-of-the-ae
multi-agent actor-critic algorithms MADDPG (23) COMA 19, CentralV (). DOP (46), VDAC-mix [4). and value-based methods QMIX (31) and QPLEX [44). In continuous cooperative tasks, we compare with MADDPG 23 and independent DDPG (IDDPG), as well as COVDN and COMIX. two novel baselines described below. We also explore different forms of critic factorisation to better understand the advantages of factoring centralised critic. More details about the environments, experimental setup, and training details are included in Appendix D and Code is available at https://gi thub. com/ wowo////hil//acaaaa
COVDN and COMIX. We find that not many multi-agent value-based methods work off the shelf with continuous actions. To compare FACMAC against value-based approaches in continuous cooperative tasks, we use existing continuous Q-learning approaches in single-agent settings to extend VDN and QMIX continuous action spaces. Specifically, we introduce COVDN and COMIX, which use VDN-style and QMIX-style factorisation respectively and both perform approximate greedy action selection using the cross-entropy method (CEM) 77. CEM a sampling-base based derivative-fres free
"We utilise 10,,4.10., which is used by the latest PyMARL 35 framework. The original results reported in Samvelyan et al. [35] and Rashid et al. [35] use SC2.4.6. Performance not always comparable across versions.
7

---

Continuous Predator-Prey
Continucus Predalor-Prey
Continuous Predator-Prey
Eplsode Returss
200
Episode Returns
300
Episode Returne
300
FACMAG [5]
FACMACC vdn
200
MADDPG 15]
COMIX [S]
100
COVON [S]
100
100PG
8.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
T (mi)
T (mil)
(mil)
(a) agents and prey
(b) 6 agents and preys
(c) 9 agents and preys
Figure 4: Mean episode return Continuous Predator-Pres with different number of agents and preys. The mean across seeds is plotted and the 95% confidence interval is shown shaded. The numbers in square brackets in the figure legend represent the number of random seeds used to run each method (similarly for all other figures).
2-Agent Humanoid
2-Agent HumanoidStandup
ManyAgent Swimmer [10x2)
600
150000
HACMMC [1]
Episad Retums
Epsariee Reluams
Epsode Returss
400
FACHACWVI
MADORG 10
400
100000
200
DONER 177
200
50000
1.0 (mill
1.5
15
1.0
3.0
(mil)
(mil)
Figure 5: Mean episode return on different MAMuJoCo tasks. In ManyAgent Swimmer, we configure the number of agents to be 10, each controlling consecutive segment of length 2. The mean across 7 seeds is plotted and the 95% confidence interval shown shaded.
heuristic search method that has been successfully used to find approximate maxima of nonconvex Q-networks in single-agent robotic control tasks [15). The centralised but factored Qtot allows us to use CEM to sample actions for each agent independently and to the per-agent utility Qa to guide the selection of maximal actions. We do not consider COVDN and COMIX significant algorithmic contributions but instead merely baseline algorithms. See Appendix for more details about them.
FACMAC outperforms MADDPG and other baselines in both discrete and continuous action the mean episode return attained by different methods on Continuous
tasks. Figure 4 Predator-Prey with varying number of agents and different MAMuJoCo tasks, respectively. We can see that FACMAC significantly outperforms MADDPG on all these continuous cooperative tasks, both in terms of absolute performance and learning speed. On discrete SMAC tasks, Figure 6 shows that FACMAC performs significantly better than MADDPG 4 out of 6 maps tested, and achieves similar performance to MADDPG on the other maps. Additionally, on all 6 SMAC maps, FACMAC significantly outperforms all multi-agent actor-critic baselines (COMA. CentralV, DOP, and VDAC-mix), while DOP is recently claimed to be the first multi-agent actor-critic method that outperforms ofate-of-the-arr valued- salued-based methods SMAC. FACMAC is also competitive with state-of-the-art based methods (QMIX and QPLEX), with significantly better performance on MMM, bane bane, MMM2, and 27m _sp__ss_____________ 30mm These results demonstrate the benefits of our method for improving performance in challenging cooperative tasks with discrete and continuous action spaces.
and and
In Continuous Predator- Prey, FACMAC-vdn scales better than FACMAC when the number of agents increases. However, on MAMuJoCo, FACMAC-vdn performs drastically worse than FACMAC in 2-Agent Humanoid and ManyAgent Swimmer (with 10 agents). demonstrating the necessity of the non-linear mixing of agent utilities and conditioning on the central state information in order to achieve competitive performance in such tasks. Furthermore, on SMAC, Figure [3] in Appendix Flshows that FACMAC is noticeably more stable than FACMAC-vdn and FACMAC-vdn-s across different maps, and achieves significantly better performance on the super hard map MMM2.
Interestingly, we find that FACMAC performs similarly to COMIX on both Continuous Predator- Prey and MAMuJoCo tasks. FACMAC and COMIX use the same value factorisation as QMIX and are both off- off-policy, this suggests that, in these continuous cooperative tasks, factorisation of the
8

---

2832
MMM
2c. vs 642g
100
100
100
Modian Tost Win 16
Median Toat Win %/
Median Toxt Win %/s
60-
60
60
60
40
-0
20
8.0
0.5
1.0
1.5
2.0
8.0
0.5
1.0
1.5
2.0
8.0
0.5
1.0
1.5
2.0
(mii)
T (mil)
(mil)
bane vs bane
MMM2
27m vs. 30m
100
100
100
FACZANG [S]
Median Test Win %6
MADDPN (9)
Median Testt Win
Modian Tost Win
DOP (S)
DOMA
60
600
60
FI)
40
40
60
(B
VDNG-mm4
20
8,
0.5
1.0 (mil)
1.5
2.0
0.5
1.0 T (mil)
1.5
2.0
8.
0.5
1.0 T (mil)
1.5
2.0
Figure 6: Median test win % on six different SMAC maps, including 2s3z (easy), MMM (easy), 2c vs_ ps_642g (hard), bane vs bane (hard), MMM2 (super hard), and 27m_ vs 3_30mm (super hard). The median across 5 seeds is plotted and the 25 75% percentiles is shown shaded. The performance of the heuristic-based algorithm is shown as a dashed line. We report the median instead of the mean as recommended by Samvelyan et al. 35 in order to avoid the effect of any outliers.
joint Q-value function plays a greater role in performance than the underlying algorithmic choices. On SMAC, however, FACMAC performs significantly better than QMIX on MMM, bune_vs vs_bane, MMM2, and 27m_vs_30mm
For instance, on sane_vs_bane a task with agents, while QMIX struggles to find the optimal policy with 2 million timesteps, FACMAC, with exactly the same value factorisation, can quickly recover the optimal policy and achieve 100% test win rate. This shows the convergence advantages of policy gradient methods in this type of multi-agent settings [43].
FACMAC scales better as the number of agents (and/or actions) and the complexity of the task increases. As shown in Figure 4(b and 4(c) MADDPG performs poorly if we increase the number of agents in Continuous Predator- Prey,, while both FACMAC and FACMAC-vdn achieve significantly better performance. The monolithic critic in MADDPG simply concatenates all agents' observations into a single input vector, which can be quite large when there are many agents and/or entities and make it more difficult to learn good critic. Factoring the critic enables scalable critic learning, by combining individual agent utilities that condition on much smaller observations into joint action- value function. On MAMuJoCo (shown in Figure similarly, the largest performance gap between FACMAC and MADDPG can be seen on Many Agent Swimmer (with 10 agents), a task with the largest number of agents among three MAMuJoCo tasks tested.
On SMAC (shown Figure op. the largest performance gap between FACMAC and MADDPG can be seen on the challenging MMM2 and 27m_ vs_30m with large number of agents, which are classified as super hard SMAC maps due to current methods' poor performance 35. We can see that FACMAC is able to learn to consistently defeat the enemy, whereas MADDPG fails to learn anything useful in both tasks. The second largest performance gap between FACMAC and MADDPG can be seen on the hard map 2c_vs_ vw__4gg where MADDPG not only performs significantly worse but also exhibits significantly more variance than FACMAC across seeds. While there are only agents in this scenario. the number of actions each agent choose is the largest among all 6 maps tested as there are 64 enemies. These results further demonstrate that FACMAC scales better when the number of agents (and/or actions) and the complexity of the tasks increases.
Factoring the critic can better take advantage of our centralised gradient estimator to optimise the agent policies when the number of agents and/or actions is large. We conduct ablation experiments to investigate the influence of factoring the critic and using the centralised gradient estimator in our method. FACMAC (without CPG) is our method without the centralised policy gradient. It uses naive adaptation of the deterministic policy gradient used in MADDPG (shown in (6). Thus, the only difference between FACMAC (without CPG) and MADDPG is that the previous

---

one learns a non-linearly factored critic while the latter one learns a monolithic critic. We also evaluate MADDPG with our centralised policy gradient and refer as MADDPG (with CPG).
Figure 7 shows the ablation results on SMAC and MAMuJoCo. We can see that FACMAC (without CPG) performs significantly better than MADDPG on both SMAC maps tested, demon- strating the advantages of factoring the critic in challenging coordination problems. With the cen- tralised policy gradient. MADDPG (with CPG) performs significantly better than MADDPG on 2c vs 64Ig. However, on the harder map MMM2, MADDPG with both policy gradients fail to learn anything useful. By contrast, FACMAC signifi- cantly outperforms FACMAC (without CPG) on MMM2, and has lower variance across seeds on vs_64gg. Furthermore, on Many Agent Swim- mer with 2 agents, our centralised gradient esti- mator does not affect the performance of both MADDPG and FACMAC. However, on the same task with 10 agents, using the centralised gradient estimator significantly improves the learning performance when learning a factored critic. These results demonstrate that factoring the critic can better take advantage of our centralised gradient estimator optimise the agent policies when the number of agents and/or actions is large.
Ze_ vs B42g
300
FACMAC [S]
MMM2
Meddan Text Win
75
CPCC
MADDING (with CIrqu
50
MADIPO
26
8a
0.5
ManyAgent Swimmer [2x2]
Marryggent Swimmer [30x2]
Egisode Retums
300
Episone Returss
FACMAN (1)
FACMAC FACAAO We
MADORS 1ith (Po
-1PLLO
10
2.0 [mil)
30
40
06
1.0
2.0
2.
4.2
(ml))
Figure 7: Ablations for different FACMAC com-
ponents on SMAC and MAMuloCo tasks.
Nonmonotonically factored critics can solve tasks that cannot be solved with monolithic or monotonically factored crities. In our
Episode Retums
Continusus Matrix Game
FACMAC [1o]
Episote Retums
40
Continoous Prodator Prey
FACMAC P]]
FACMAC ronmapononic [10)
FACAACarannnnaaaa P1
MADOPG 110
22
MADUPU
multi-agent actor-critic framework, there are no
inherent constraints on factoring the critic, we thus also employ a nonmonotonic factorisation and refer to it as MCCMAC-nommoootee discussed in Section 3. As shown in Figure (8 (left), our continuous matrix game (as dis- AACMAC-nonmontommi robustly learn the optimal policy, while both FACMAC and MADDPG converge to some sub-optimal policy within 200k timesteps. On a variant of Continuous Predator-Prey task (see Appendix D.2 for more details about this task) with non- monotonic value functions (i.e., an agent's ordering over its own actions depends on other agents' actions (25 ), Figure & (right) shows that both FACMAC and MADDPG fail to learn anything useful, successfully learns to capture the prey. These results demonstrate that nonmonotonically factored critics can solve tasks that cannot be solved with monolithic or monotonically factored critics.
%
150
200
0.0
0.5
1.0
1.5
T (mil)
(as
Figure 8: Mean episode return on (Left) Contin- uous Matrix Game and (Right) a variant of our Continuous Predator-Prey task (with 3 agents and 1 prey) with nonmonotonic value functions.
cussed in Sectionn.2))
while
It is important to note that the relative performance of FACMAC and FACMAC-nonmonooonis is task dependent. On the original Continuous Predator-Prey task (with 3 agents and 1 prey), FACMAC- nonmonotonic yields similar performance FACMAC (see Figure 12 in Appendix [F. On SMAC (see Figure 13 in AFpenddx.FFFACACCCCmoCoo. FACMMAC-nommonotonic performs similarly to FACMAC on easy maps, but exhibits significantly worse performance on harder maps. This shows that, in this type of tasks, using an unconstrained factored critic could lead to an increase in learning difficulty. We thus be more useful in tasks with nonmonotonic value functions.
expect FACMAC-nonmonoton
6 Conclusion
This paper presented FACMAC, multi-agent actor-critic method that learns decentralised policies with centralised but factored critic, working for both discrete and continuous cooperative tasks. We showed the advantages of both factoring the critic and using the new centralised gradient estimator in our approach. We also introduced a novel benchmark suite MAMuJoCo to demonstrate FACMAC's scalability to more complex continuous tasks. Our results three different domains demonstrated FACMAC's superior performance over existing MARI. algorithms. Future work will explore more forms of nonmonotonic factorisation to tackle tasks with nonmonotonic value functions.
10

---

Acknowledgements
We would like to thank Dr Martin Strohmeier and the members of the Whiteson Research Lab for their helpful feedback. We would also like to thank the anonymous reviewers for their constructive ments during the reviewing process. This project has received funding from the European Research Council (ERC), under the European Union's 's Horizon 2020 research and innovation programme (grant agreement number 637713). It was also supported by an EPSRC grant (EP/M5081 1/1, EP/N50971 1/1). Christian Schroeder de Witt is generously funded by Cyber Defence Campus, Sci- ence and Technology, Armasuisse, Switzerland. The experiments were made possible by generous equipment grant from NVIDIA.
References
[1] Johannes Ackermann, Volker Gabler, Takayuki Osa, and Masashi Sugiyama. Reducing overestimation bias in multi-agent domains using double centralized critics. arXiv preprint 1rXiv:1910001465, 2019.
[2] Brandon Amos, Lei Xu, and Zico Kolter. Input convex neural networks. Proceedings of the 34th International Conference Machine Learning- Volume 70, pages 146-155. JMLR. org, 2017.
[3] Ronald C Arkin. Motor schema- based mobile robot navigation. The International journal of robotics research, 8(4):92-112, 1989.
[4] Marilena Bescuca. Factorised critics in deep multi-agent reinforcement learning. In Master Thesis, University of Oxford, 2019.
[5] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 160.00 2016.
[6] Rodney Brooks. A robust layered control system for mobile robot. IEEE journal roboties and automation, 241): 1986.
[7] Pieter-Tjert De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. tutorial on the cross-entropy method. Annals of operations research, 134(1):19-67, 2005.
[8] Yali Du, Lei Han, Meng Fang, Ji Liu, Tianhong Dai, and Dacheng Tao. LIIR: Learning individual intrinsic reward in multi-agent reinforcement learning. In Advances in Neural Information Processing Systems 32, pages 4405-4416, 2019.
[9] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon White- son. Counterfactual multi-agent policy gradients. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[10] Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In International Conference on Machine Learning, pages 1829-2838 2016.
[11] Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems, pages 66 83. Springer, 2017.
[12] David Ha, Andrew Dai, and Quoe V Le. Hypemetworks. arXiv preprint 16099:160. 2016.
[13] Shariq Iqbal and Fei Sha. Actor-atientionncritte for multi-agent reinforcement learning. In International Conference on Machine Learning, pages 2961-2970, 2019.
[14] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax arXiv preprint 1rXiv:1611.01144 2016.

---

[15] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et Qt-opt: Scal- able deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv: 1806, 10293, 2018.
[16] Hiroaki Kitano, Minoru Asada, Yasuo Kuniyoshi, Itsuki Noda, Eiichi Osawa, and Hitoshi Matsubara. Robocup: A challenge problem for ai. Al magazine, 18(1):73-73, 1997.
[17] Daphne Koller and Ronald Parr. Computing factored value functions for policies in structured mdps. In Proceedings of IJCAI, pages 1332-1339, 1999.
[18] Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing, 190:82 2016.
[19] Haruhisa Kurokawa, Kohji Tomita, Akiya Kamimura, Shigeru Kokaji, Takashi Hasuo, and Satoshi Murata. Distributed celf-reeonfiguratiinn of m-tran iii modular robotic system. The International Journal of Robotics Research, 27(3-4):373-331 2008.
[20] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In 4th International Conference on Learning Representations, ICLR, 2016.
[21] Long-Ji -j Lin. self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8(3-4):293-32) 1992.
[22] Siqi Liu, Guy Lever, Josh Merel, Saran Tunyasuvunakoo, Nicolas Heess, and Thore Graepel. Emergent coordination through competition. arXiv preprini arXiv:/902.07/555 2019.
[23] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch, Multi- agent actor-critic for mixed eeoperative-competttie environments. In Advances in Neural Information Processing Systems, pages
2017.
(24] Xueguang Lyu, Yuchen Xiao, Brett Daley, and Christopher Amato. Contrasting centralized and decentralized critics in multi-agent reinforcement learning. In Proceedings of the 20th International Conference on Autonomous Agents and Multi-Agent Systems, 2021.
[25] Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi- agent variational exploration. In Advances in Neural Information Processing Systems, pages 7613-7624, 2019.
[26] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland. Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533 2015.
[27] Frans A. Oliehoek, Matthijs T. J. Spaan, and Nikos Vlassis. Optimal and approximate Q-value functions for decentralized pomdps. JAIR, 32:289-353, 2008.
[28] Frans Oliehoek, Christopher Amato, et al. A concise introduction to decentralized POMDPs. volume 1. Springer, 2016.
[29] OpenAl. openai/baselines, May 2020. original-date: 2017-05-24T01:58.133
[30] Ling Pan, Tabish Rashid, Bei Peng. Longbo Huang, and Shimon Whiteson. Softmax with regularization: Better value estimation in multi-agent reinforcement learning. arXiv preprins wwXi:2203.///88 2021.
[31] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pages 4292 4301, 2018.
[32] Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation. In Advances neural information processing systems, 2020.
12

---

[33] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt. Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. JMIR, 11:177:1-1188:0 2020.
[34]
Martin Riedmiller, Thomas Gabel, Roland Hafner, and Sascha Lange. Reinforcement learning for robot soccer. Autonomous Robots, 27(1):55-73, 2009.
[35] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung. Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. CoRR, abs/ 0bss/1902.04043, 2019.
[36] Christian Schroeder de Witt, Jakob Foerster, Gregory Farquhar, Philip Torr, Wendelin Boehmer, and Shimon Whiteson. Multi-Agent Common Knowledge Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[37] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. arXiv preprint met.::/005000000 2019.
[38] Kyunghwan Son, Sungsoo Ahn, Roben Delos Reyes, Jinwoo Shin, and Yung Yi. Qtran+++ Improved value transformation for cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2006. 12200066.10010 12010, 2020.
[39] Peter Stone and Richard S. Sutton. Scaling reinforcement learning toward RoboCup soccer. In leml, volume pages 537-544, Citeseer, 2001.
[40] Jianyu Su, Stephen Adams, and Peter A Beling. Value-decomposition multi-agent actor-critics. In Proceedings of the Thirty-Fifih AAAI Conference on Artificial Intelligence, 2021.
[41] Peter Sunchag. Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-deccompssiioon networks for cooperative multi-agent learning based on team reward. In AAMAS, pages 2085-200 2087, 2018.
[42] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033, 2012.
[43] Jianhao Wang, Zhizhou Ren, Beining Han, and Chongjie Zhang. Towards understanding linear value decomposition in cooperative multi-agent q-learning. arXiv preprint 20X1v:2006.00587, 2020.
[44] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent g-laaniig. arXiv preprint arXiv:2008.01062 2020.
[45] Tingwu Wang. Renjie Liao, Jimmy Ba, and Sanja Fidler. NerveNet: learning structured policy with graph neural networks. In 6th International Conference on Learning Representations, ICLR, 2018
[46] Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. Dop: Off- policy multi-agent decomposed policy gradients. In 9th International Conference on Learning Representations, ICLR, 2021.
[47] Ermo Wei and Sean Luke. Lenient learning in independent-learer stochastic cooperative games. The Journal of Machine Learning Research, 17(1):2914 2955, 2016.
[48] Yaodong Yang. Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.0332 2020.
[49] Mark Yim, Ying Zhang, and David Duff. Modular robots. IEEE Spectrum, 39(2):30-34 2002.
[50] Meng Zhou, Ziyu Liu, Pengwei Sui, Yixuan Li, and Yuk Ying Chung. Learning implicit credit assignment for multi-agent actor-critic. In Advances in neural information processing systems, 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We mention that the relative
performance of FACMAC and FACMAC-nommonotonic is task dependent and discuss
some empirical results in the last paragraph of Section [
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Appendix [.T]]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material the code and instructions needed to reproduce the main experimental results in the supplemental material.
URL)? [Yes] We include
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix D
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] For all results on Continuous Matrix Game, Continuous Predator-Pres Prey, and MAMuJoCo, dence interval. For all results on SMAC, we report the median test win rate as well as the 25-75% percentiles (as in (B5)).
report the mean episode return and the 95% confi-
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We describe the type of GPUs used in Appendix E
4. If you using existing assets (e.g., code, data, models) curating/releusing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] The license is already included in the
original code resource we cited and could change in the future.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "p5ITSEL_Gtz", "reviewer": "Reviewer_Byep", "summary": "This paper proposes to apply the idea of value function factorization from recent value-based multi-agent RL work (e.g., VDN, QTRAN, QMIX) and apply it in the multi-agent actor-critic setting. Specifically, the paper proposes to use a centralized factorized critic, with either a monotonic (as in QMIX) or non-monotonic mixing of the individual value terms. The proposed approach applies the deterministic policy gradient method and is applicable to both discrete and continuous action spaces. The approach is compared to baselines in a series of particle world and SMAC benchmarks, and a new multi-agent MuJoCo benchmark proposed in the paper. Ablation studies reveal that scalability to more agents or actions is improved over MADDPG, and that nonmonotonic mixing of the value terms can be useful in some domains.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main novelty and potentially most significant insight of the paper in my opinion is the concept of non-monotonic mixing. I have some concerns on originality, and some minor technical questions. Overall, the empirical results are thorough, and the paper is very clearly written.\n\nStrengths:\nThe paper convincingly argues that non-monotonic mixing is applicable in a centralized critic in an actor-critic setting, as the monotonicity constraint required in value-based methods is no longer necessary. This is empirically demonstrated to be quite useful in some domains (Fig. 8). The empirical evaluation is thorough. The MAMuJoCo benchmark is a nice additional contribution, complementing the continuous particle environments proposed in the literature so far.\n\nLimitations:\nThe idea of a factorized critic proposed in the paper is certainly reasonable, but I am not sure about the novelty. In [1], a similar critic factorization approach is investigated in the actor-critic setting. This paper should make clear what the similarities and differences to [1] are, and provide appropriate empirical comparisons. This being said, it does seem to me that the non-monotonicity of mixing proposed in this paper has not been considered before.\n\nThe paper identifies some domains where the non-monotonic mixing is useful. One limitation however is that the paper is a bit light on theory or insights about what such domains are like, and when one could expect non-monotonic critic factorization to be useful. The empirical results on non-monotonic mixing also have a minor role in the paper.\n\nIf the concerns above could be addressed, I am happy to revise my score.\n\n[1] Jianyu Su, Stephen C. Adams, Peter A. Beling. Value-Decomposition Multi-Agent Actor-Critics. AAAI 2021\n\n\nDetailed comments:\nI have a technical concern regarding the centralized gradient estimator (Equation (7)). As far as I understand, the two gradients on the right hand side will be vectors of possibly different dimensions. How is it possible to multiply them as presented?\n\nIn several places in the paper, sampling from the policy is mentioned (e.g., Fig. 1, Lines 129-130, 174, 182). This is somewhat confusing, given that the paper learns deterministic policies. This should be clarified overall.\n\nMinor:\nLine 116-117: I don't think MADDPG learns reward functions.\nFig. 4-8: consider removing or explaining meaning of numbers in square brackets in the legend\nFig. 5 (right): legend placement is a bit unfortunate with overlap with the plots.\n\nPost-rebuttal: I thank the authors for their detailed rebuttal. It addresses most of my concerns and I am increasing the overall score to reflect this. The rebuttal has made clear what the difference to Su et al. is. I hope to see a discussion and the empirical results in a revised version of the paper. I also hope to see future work providing actionable insights about which kinds of domains indeed have non-monotonic value functions fitting for the proposed method. Having briefly reviewed the provided source code, I would suggest to include specific commands and configuration files to replicate the experimental results provided in the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "hqSf-per0xr", "reviewer": "Reviewer_Ve1n", "summary": "The paper proposes a new algorithm for cooperative multi-agent reinforcement learning (MARL) applicable in both discrete and continuous settings. The main innovation is an adaptation of multi-agent deep deterministic policy gradient (MADDPG). MADDPG trains decentralised policies using a centralised critic for each agent in the centralised training with decentralised execution (CTDE) paradigm. In this work, the authors specifically introduce a version of MADDPG with a centralised critic that learns a factored value function. Each factor is an agent-specific utility function with learned weights determined by a mixing network, as is done in popular value factorisation methods in MARL such as VDN and QMIX. In addition, the approach makes use of a centralised policy gradient estimator, where all agent’s current policy actions are considered during gradient updates to guard against pathologies such as over-generalisation. The authors refer to their algorithm as factorised multi-agent centralised policy gradients, or FACMAC for short. Extensive experimental results are presented to showcase the effectiveness of the proposed algorithm, along with multi-agent MuJoCo, a new multi-agent environment suite for continuous control, and ablation studies highlighting the specific benefits of value function factorisation, centralised policy gradients and the expressiveness of the factorisation function when applied to MADDPG. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: In essence, the work represents a novel combination of MADDPG (Lowe et al., 2017) and QMIX (Rashid et al., 2018) with the addition of the idea of using centralised policy gradients. Combining both innovations to leverage their strength and applicability to discrete and continuous tasks is shown to be useful in settings not previously tested, using a large number of agents. The introduced multi-agent MuJoCo environment for continuous control builds on the single-agent MuJoCo engine, but is new and seems promising and useful as a benchmark for future research in continuous MARL.\n\n**Quality**: The work is technically sound. Most of the claims in the paper are supported by a decent set of experimental results. Experiments are done over multiple seeds and include a wide variety of baselines and environments. The author’s motivation for their approach, citing pathologies such as over-generalisation, could perhaps be made more concrete but overall the presentation comes across as honest and the details provided are sufficient for a reader, interested in MARL, to follow and understand the work.\n\n**Clarity**: The paper is well-written and the overall structure of the paper makes it easy to follow. Some minor remarks. Section 3.2 presents some experimental results before the experiments section, which was slightly odd, but I can also understand why the authors thought to present it earlier to motivate their argument for CPG. On a similar note, the authors reference Figure 11 in appendix D for this experiment, but the same figure is already present in the center of Figure 2(b), so they could just refer to that. Also, throughout the paper, the authors use “a” as the index symbol for agents and “u” for actions, whereas it is perhaps more the convention to use “i” for agents and “a” for actions. This change might make it easier for readers familiar with other MARL papers and notation to read the paper. \n\n**Significance**: In my opinion, the work will definitely be of interest to the community. The results seem to indicate that the approach shows significant promise at scale and nicely ties together different streams of investigation in MARL, namely, value decomposition methods and multi-agent policy gradient methods. Furthermore, the newly introduced multi-agent MuJoCo suite could greatly benefit future research as a benchmark for continuous control. \n\n**Questions**:\n\n* The paper references PyMARL for specific implementations. I’m curious as to why QTran (Son et al., 2019) was not also included as a baseline? Since the library already has an implementation. In my opinion, this should not be held against the authors, since I think the number of baselines is already sufficient, especially with the introduction of COVDN and COMIX. It is just out of interest that I ask the question.  \n* In line 292 it is stated: “For instance, on bane_vs_bane, a task with 24 agents, while QMIX struggles to find the optimal policy with 2 million timesteps, FACMAC, with exactly the same value factorisation, can quickly recover the optimal policy and achieve 100% test win rate. This shows the convergence advantages of policy gradient methods in this type of multi-agent settings”. It is true that FACMAC converges faster, but so does MADDPG and DOP, so this seems more a reflection on the policy gradient approach in general and not something specific to FACMAC. Do you think this might be a more honest reflection? Or is the statement more specific to factorisation not being useful in this case, i.e. PG with and without factorisation having no difference which places the focus on QMIX vs FACMAC and FACMAC benefiting from PG? In any case, perhaps worth rephrasing the claim to be less focused on FACMAC. \n* Regarding the results in Figure 8 for the matrix game. It is interesting that in Figure 2(a) it seems MADDPG with CPG can achieve convergence in just 50k steps, whereas FACMAC fails to converge and FACMAC-nonmonotonic takes around 200k steps to converge. Curious to know whether the authors might have any explanation as to why this might be the case? Is it perhaps that FACMAC also has to learn the mixing and hypernetworks, which significantly slows down convergence speed in such a small environment. Also, do the authors expect FACMAC to converge eventually, given enough time? \n\n**References**\n\n- Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. In Advances in Neural Information Processing Systems, pages 6379–6390, 2017.\n- Tabish Rashid, Mikayel Samvelyan, Christian Schroeder Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pages 4292–4301, 2018.\n- Son, K., Kim, D., Kang, W.J., Hostallero, D.E. and Yi, Y., 2019, May. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning (pp. 5887-5896). PMLR.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors highlight some cases where their approach fails to outperform specific baselines but makes no noticeable mention of the core limitations of their algorithm. In terms of societal impact and negative consequences, these are also not stated in the main text. However, given the nature of the work, I don’t feel too strongly about these remarks although it might be interesting for the authors to think about the downsides of FACMAC. There is a section in the appendix detailing some of the social impacts that might stem from the multi-agent MuJoCo suite. ", "review_text": "", "extra_scores": {"ethical_concerns": "To the best of my knowledge, there are no ethical concerns with this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "gfWUtEJTei6", "reviewer": "Reviewer_wmCS", "summary": "This paper proposes a new multi-agent deterministic policy gradient method, called FACMAC, which is essentially MADDPG + QMIX. Although the combination is rather trivial, the author claims that they improve MADDPG in the sense that FACMAC optimise in the joint-action space rather than each individual agent's action space as MADDPG do, and they also claim that FACMAC improves QMIX by removing the monotonic constraints on the critic decomposition. ", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I have trouble agreeing with the many claims that authors make about the MADDPG and QMIX method. More importantly, the idea of FACMAC =  QMIX + MADDPG is trivial, from both a theoretical perspective and an empirical perspective. I have concerns on the two critical tricks that FACMAC introduce (i.e. the nonmonotonic critic decomposition (Eq. 4) and the policy gradient over joint action space (Eq. 7)). Moreover, I am not convinced why such a combination is an important direction and worth attention from the MARL community, neither does the reported empirical results support that. Therefore, I cannot recommend for acceptance at this stage. \n\n\n1. In line 11, the author mentioned \"FACMAC uses a centralised policy gradient estimator that optimises over the entire joint action space, rather than optimising over each agent’s action space separately as in MADDPG.\". \n\nI believe this shall not be considered as a benefit of FACMAC over MADDPG. The idea of MADDPG to learn an individual policy without the need to learn a big joint Q-network. Following this logic, the centralised Q-learning method should be used in all MARL problems. Also, I don't think it is fair to claim that MADDPG only optimises in each agent's action space, after all the critic value that guides the policy to learn is the centralised critic, so presumably, what MADDPG is doing is searching over a joint-strategy space. \n\nFurthermore, FACMAC cannot optimise in the entire joint action space as the author claim. The main trick that FACMAC introduces is through Equation 7 in which all agents share the same policy parameter (i.e., CPG trick), so updating one agent's policy naturally equals to updating multiple agents' policy. However, this trick only holds for the case when all agents share the same set of policy parameters. Although the author claims that it also appleis in the non-sharing cases, however, if agents each have different set of parameters for its policy, then Equation 7 needs to take derivative at the same time for each agent, which again loops back to the issue of each agent only optimise in its own action space.  And importantly, it is commonly known that taking simultaneous gradients will lead to convergence issue. [On Gradient-Based Learning in Continuous G] https://arxiv.org/pdf/1804.05464.pdf\nMoreover, the author also mentions in the line 187 that FACMAC-vdn is in fact optimising each agent's actions, which self-contradicts what the previous claims are about. \n\nThe effectiveness of CPG is also questionable. Although in the toy example in Figure 2a, the CPG works with MADDPG. However, this could be because of the different exploration strategies in such an toy example, as pointed out by [Multi-agent Soft Q-learning, AAAI]. I suspect even by changing the payoff value of the toy example, it would show a completely different results. Most importantly,   Later in Figure, 7, the CPG trick almost makes no difference whether or not it is activated. \n\n2. In line 7, the author says \"unlike QMIX, there are no inherent constraints on factoring the critic.\" In line 44, the author says \"solve some tasks that cannot be solved with monolithic, or monotonically factored critics.\" I am not convinced how using a neural network-based mixing network in Equation 4 achieve the claim of releasing the monotonic assumption in meeting the individual-global-max condition. To me, FACMAC, which is QMIX+MADDPG, cannot claim anything beyond what QMIX has claimed. \n\n3. I also have major concerns on the reported results.\n\nFirst, the SMAC tasks and Mujoco tasks share different baseline algorithms, even in the same task different scenarios, baseline methods are different. For example, the Fig 4c and Fig 5 c have different citations in [5] and [7], both of which although are irrelevant to the COMIX, COVDN, IDDPG methods. \n\nSecondly, across almost all results on MUJoCO, there is no evidence showing that FACMAC is in fact outperforming other baselines. \nSimilar observations also hold on Fig6 on SMAC tasks. Either all baselines converge to 100% winning rate, or, the baseline methods are not trained to the best of the other reported results [cross reference the results in MAPPO]. \n\nMoreover, the SMAC task has many maps, can the author explain why the six specific maps have been chosen among the many possibilities, especially under these maps, the performance is not really discriminative. I would like to see the performance on the super-hard maps if the author tries to claim the state-of-the-art.\n\n\n4. In line 75, \"the first multi-agent actor-critic method to outperform state-of-the-art valued-based methods on SMAC.\". This is an incorrect claim given the IPPO and MAPPO work. \n\nIPPO: [Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?]\nMAPPO: [the surprising effectiveness of Mappo in cooperative, multi-agent games]\n\n5. In line 160, it is unclear to me what is the motivation behind developing FACMAC-vdn and FACMAC-vdn-s, apart from trial-and-error testing different MARL architectures. \n\n6. The gumbel-max trick is notoriously hard to tune during many empirical studies, I was wondering if the author meets similar issues, and if so, mroe details need to be provided for future result replication. \n\n> ***Post-rebuttal***: I read all reviews, and the rebuttal to my review. I am not convinced by most of the questions, and during the rebuttal phase I found more problems in the replication of the results by running the submitted code. See my detailed comments.  I will maintain my score of strong rejection. Scores are adjusted post-rebuttal.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The claim for MADDPG and QMIX need changing. \n\nThe baselines need to stay consistent throughout different tasks (I am actually surprised why a naive TRPO/PPO is not a baseline in your mujoco tasks. I understand you design a multi-agent mujoco setting, but what is the point of using FACMAC if it cannot outperform TRPO. Maybe on many agent swimmer, TRPO would fail due to scalability).\n\nThe selected tasks cannot show the effectiveness of FACMAC (Figure 4 5 6 fail to show at all). On both Mujoco and SMAC, more solid results need providing.  \n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "25"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper generated an involved discussion between the reviewers and the authors, as well as between the reviewers themselves.
The paper essentially combines two well-known baselines in MARL domain, and therefore was judged as a report of experiments for which reproducibility is of particular importance. The reviewers had an intense discussion about reproducibility, two of them tried the code provided by the authors and one of them commented on it based on their experience. The analysis raised significant doubts about reproducibility (it is unfortunate that the authors did not provide configs on their experiments, to enable to rerun them exactly).
Decision: Accept (Poster)