QPLEX: Duplex Dueling Multi-Agent Q-Learning
Authors: Jianhao Wang
Zhizhou Ren
Terry Liu
Yang Yu
Chongjie Zhang
Venue: ICLR 2021 Poster (2021)
Keywords: Multi-agent reinforcement learning
Value factorization
Dueling structure

Abstract: We explore value-based multi-agent reinforcement learning (MARL) in the popular paradigm of centralized training with decentralized execution (CTDE). CTDE has an important concept, Individual-Global-Max (IGM) principle, which requires the consistency between joint and local action selections to support efficient local decision-making. However, in order to achieve scalability, existing MARL methods either limit representation expressiveness of their value function classes or relax the IGM consistency, which may suffer from instability risk or may not perform well in complex domains. This paper presents a novel MARL approach, called duPLEX dueling multi-agent Q-learning (QPLEX), which takes a duplex dueling network architecture to factorize the joint value function. This duplex dueling structure encodes the IGM principle into the neural network architecture and thus enables efficient value function learning. Theoretical analysis shows that QPLEX achieves a complete IGM function class. Empirical experiments on StarCraft II micromanagement tasks demonstrate that QPLEX significantly outperforms state-of-the-art baselines in both online and offline data collection settings, and also reveal that QPLEX achieves high sample efficiency and can benefit from offline datasets without additional online exploration.

Published as a conference paper at ICLR 2021
QPLEX: DUPLEX DUELING MULTI-AGENT Q- D-LARNNNNG
Jianhao Wang"!, Zhizhou Ren"l Terry Liu', Yang Yu², Chongjie Zhang' Institute for Interdisciplinary Information Sciences, Tsinghua University, China PPolixir Technologies, China [wih19, r2z16, 1iudr18] @mails tes.s.siin tsinghua. www.ghuaaad..cc... yuyênju. edu.cn on mhongjielte ts waw.inga..eeeeee edu.cn
ABSTRACT
We explore value-based multi-agent reinforcement learning (MARL) in the popular paradigm of centralized training with decentralized execution (CTDE). CTDE has an important concept. Individual- Saaiaa--Goobl--MMa (IGM) principle, which requires the consistency between joint and local action selections to support efficient local socision-making. However, in order to achieve scalability, existing MARL methods either limit representation expressiveness of their value function classes or relax the IGM consistency, which may suffer from instability risk may not perform well in complex domains. This paper presents a novel MARL approach, called duPLEX dueling multi-agent Q-learning (QPLEX), which takes a duplex dueling network architecture to factorize the joint value function. This duplex dueling structure encodes the IGM principle into the neural network architecture and thus enables efficient value function learning. Theoretical analysis shows that QPLEX achieves a complete IGM function class. Empirical experiments on StarCraft II micromanagement tasks demonstrate that QPLEX significantly outperforms state- of-the-art baselines in both online and offline data collection settings, and also reveal that QPLEX achieves high sample efficiency and can benefit from offline datasets without additional online exploration!
INTRODUCTION
Cooperative multi-agent reinforcement learning (MARL) has broad prospects for addressing many complex real-world problems, such as sensor networks (Zhang & Lesser 2011). coordination of robot swarms (Hüttenrauch et al., 2017), and autonomous cars (Cao et al. 2012). However, cooperative MARL encounters two major challenges of scalability and partial observability in practical applications. The joint state-action space grows exponentially as the number of agents increases. The partial observability and communication constraints of the environment require each agent to make its individual decisions based on local action-observation histories. To address these challenges, a popular MARL paradigm, called centralized training with decentralized execution (CTDE) (Olichock et al. 2008 Kraemer & Banerjee 2016). has recently attracted great attention, where agents' policies are trained with access to global information a centralized way and executed only based on local histories in a decentralized way.
Many CTDE learning approaches have been proposed recently. among which value-based MARL algorithms (Sunehag et al.) 2018 Rashid et al. 2018 Son et al. 2019 Wang et al.) (019b) have shown state-of-the-ar performance on challenging tasks, e.g., unit mieromanagement StarCraft (Samvelyan et al. 2019). To enable effective CTDE for multi-agent Q-learning, it is critical that the joint greedy action should be equivalent to the collection of individual greedy actions of agents, which is called the IGM Saadvvidual-llobal--e
principle (Son et al. 2019). This IGM principle provides two advantages: 1) ensuring the policy consistency during centralized training (learning the joint fufuction)) and decentralized execution (using individual Q-functions) and 2) enabling
"quul contribution.
Videos available

---

Published as a conference paper at ICLR 2021
scalable centralized training of computing one-step TD target of the joint Q-function (deriving joint greedy action selection from individual O-functions). To realize this principle, VDN (Sunehag et al. 2018) and QMIX (Rashid et al. 2018) propose two sufficient conditions of IGM to factorize the joint action-value function. However, these two decomposition methods suffer from structural constraints and limit the joint action-value function class they can represent. As shown by Wang et al. (2020a), the incompleteness of the joint value function class may lead to poor performance or potential risk of training instability in the offline setting (Levine et al., 2020). Several methods have been proposed to address this structural limitation. QTRAN (Son et 2019) constructs two soft regularizations to align the greedy action selections between the joint and individual value functions. WQMIX (Rashid et al. 2020) considers weighted projection that places more importance on better joint actions. However, due to computational considerations, both their implementations are approximate and based on heuristics, which cannot guarantee the IGM consistency exactly. Therefore, achieving the complete expressiveness of the IGM function class with effective scalability remains an open problem for cooperative MARL.
To address this challenge, this paper presents a novel MARL approach, called duPLEX dueling multi- agent Q-learning (QPLEX), that takes duplex dueling network architecture to factorize the joint action-value function into individual action-value functions. QPLEX introduces the dueling structure Q Q= A (Wang et al. 2016) for representing both joint and individual (duplex) action-value functions and then reformalizes the IGM principle as an advantage-base baage IGM. This reformulation transforms the IGM consistency into the constraints on the value range of the advantage functions and thus facilitates the action-value fuunction learning with linear decomposition structure. Different from QTRAN and WQMIX (Son et al. 2019 Rashid et al., 2020) losing the guarantee of exact IGM consistency due to approximation, QPLEX takes advantage of a duplex dueling architecture to encode it into the neural network structure and provide guaranteed IGM consistency. To our best knowledge, QPLEX is the first multi-agent Q-learning algorithm that effectively achieves high scalability with a full realization of the IGM principle.
We evaluate the performance of QPLEX in both didactic problems proposed by prior work (Son et al. 2019; Wang et al., 2020a) and a range of unit micromanagement benchmark tasks in StarCraft II (Samvelyan et al. 2019). In these didactic problems, QPLEX demonstrates its full representation expressiveness, thereby learning the optimal policy and avoiding the potential risk of training instability. Empirical results more challenging StarCraft II tasks show that QPLEX significantly outperforms other multi-agent Q-learning baselines in online and offline data collections. It is particularly interesting that QPLEX shows the ability support offline training, which is not possessed by other baselines. This ability not only provides QPLEX with high stability and sample efficiency but also with opportunities efficiently utilize multi-source offline data without additional
online exploration (Fujimoto et al. 2019 Fu et al. 2020
Levine et al. 2020 Yu et al. 2020).
2
PRELIMINARIES
2.1
DECENTRALIZED PARTIALLY OBSERVABLE MDP (DEC-POMDP)
We model fully cooperative multi-agent task as Dec-POMDP Olichock et al., 2016) defined by tuple M (V, A. P. f2, 0, p,r,7), 7), where A = (1.2...., n. is a finite set of agents and E S is finite set of global states. At each time step, every agent: i E ' chooses an action a, E A= (4(1)
A(AI)) } on a globaal state s, which forms a joint action = [ results in a joint reward r(s, a) and transition to the next global state s' P(- a). E 1), is discount factor. We consider a partially observable setting, where each agent i receives an individual partial observation o; E f2 according to the observation probability function ((o a.). Each agent i has an action-observation history T. E T = (Ω x A)* and constructs its individual policy n, (alT.) to jointly maximize team performance. We use T E TT== = T to denote joint action-observation history. The formal objective function is to find joint policy π (I Tn) that maximizes joint value function (*(() = E
E A= A".
I>> t 7rri = s, TT.. Another quantity of interest in policy search is the joint valiun-value function QT
(s, a) r(s, a) + 7.,, (V*(('))
2.2 DEEP MULTI-AGENT Q-LEARNING IN DEC-POMDP
Ollearning algorithms is a popular algorithm to find the optimal joint action-value function (s) = r(s, a) +>B [maxa Q" (s', a²)). Deep Q-learning represents the action-value function
2

---

Published as a conference paper at ICLR 2021
with deep neural network parameterized by 0. Mutli-agent Q-learning algorithms (Sunehag et al. 2018) Rashid et al. 2018: Son et al.) 2019; Yang et al. 2020) use a replay memory D to store the transition tuple (T, r, J. where
the reward for taking action at joint action-observation his- tory 7 with transition to T'. Due to partial observability, Q (T, a; 0) is used in place of Q (8, a; 0).
Thus, parameters 0 are learnt by minimizing the following expected TD error:
() E (( [I+ V (r':0 Q (T, a; (T)
where V (T':0-)= = maxa Q (T', a'; 0-) is the one-step expected future return of the TD target and 0 are the parameters of the target network, which will be periodically updated with 0.
2.3 CENTRALIZED TRAINING WITH DECENTRALIZED EXECUTION (CTDE)
CTDE is popular paradigm of cooperative multi-agent deep reinforcement learning (Sunehag et al., 2018: Rashid et al. 2018: Wang et al., 2019a 2020b cld). Agents are trained a centralized way and granted access to other agents' information or the global states during the centralized training process. However, due to partial observability and communication constraints, each agent makes its own decision based on its local action-observation history during the decentralized execution phase. IGM (Individual- Sundividul-GGloba---e Global-Max, Son et al. 2019) is a popular principle to realize effective value-based CTDE, which asserts the consistency between joint and local greedy action selections in the joint action-value Qroe (T, a) and individual action-values (T,(Ti,a:))==:
ds)];=1
VT E T arg max Qtoe(T, a) = arg max QT 01); ...,....... arg max Qu(Tn; (n) E. Two factorization structures, additivity and monotonicity, has been proposed by VDN (Sunehag ct al. 2018) and QMIX (Rashid et al., (2018), respectively, as shown below:
a
EA
aQuMIN (T.a) > 0.
YoO (T, = ((((T))))) and ViE aQ,(T.,;:)
Qatten (Yang et al. 2020 is variant of VDN, which supplements global information through known that, these structures implement sufficient but not
a multi-head attention structure, necessary conditions for the IGM constraint, which limit the representation expressiveness of joint action-value functions (Mahajan et al. 2019). There exist tasks whose factorizable joint action-value functions can not be represented by these decomposition methods, shown in Section 4 In contrast, QTRAN (Son et al. 2019) transforms IGM into a linear constraint and uses it as soft regularization constraints. WOMIX (Rashid et al.) 2020) introduces weighting mechanism into the projection of monotonic value factorization, in order to place more importance on better joint actions. However, these relaxations may violate the exact IGM consistency and may not perform well in complex problems.
QPLEX: DUPLEX DUELING MULTI- Q-LEARNING
In this section, we will first introduce advantage-based IGM, equivalent to the regular IGM principle, and, with this new definition, convert the IGM consistency of greedy action selection to simple constraints advantage functions. We then present novel deep MARL model, called duPLEX dueling multi-agens Q-learning algorithm (QPLEX), that directly realizes these constraints by a scalable neural network architecture.
3.1 ADVANTAGE-BASED IGM
To ensure the consistency of greedy action selection on the joint and local action- value functions, the IGM principle constrains the relative order of Q-values over actions. From the perspective of dueling decomposition structure Q V + A proposed by Dueling DQN (Wang et al.) 2016), this consistency should only constrain the action-dependent advantage term A and be free of the state-value function V. This observation naturally motivates us to reformalize the IGM principle as advantage-bass IGM, which transforms the consistency constraint onto advantage functions.
Definition (Advantage-be based IGM). For a joint action-value function Qiat: T x A + R and individual action-value functions QQ; T A + RE where VT E T, Va E A, Vi EN,
(Joint Dueling) Qiat ( a) = Vio((T) (+++) + Atoe(T a) and Vioe(T) = max Qiot (T,a'), (3) a'
(Individual Dueling) Qi(T,a))) = Vi(T:) + Ar(Te, at) and V.(T:) max (T)))
(4)
af
3

---

Published as a conference paper at ICLR 2021
Ptot (T, a)
TD Loss
(()))))))))))))
Cior (r,
Duplex Dueling
Weighted Sum
Dueling Mixing
IV 0,6
A1(
(t,,
MLP
(1) (to
Dot Product
Transformation
Transformation
St
St
Qi(1.
(t, (T)
(TTTn). (n
MLP
MLP
(,(T,,a))
(n(T
hg --
GRU
Agent
Agent
MLP
f
MOOE: [A,(r, =
aff
(of, 11
(of.a)
(a)
(b)
(c)
Figure I: (a) The dueling mixing network structure. (b) The overall QPLEX architecture. (c) Agent network structure (bottom) and Transformation network structure (top).
such that the following holds
-(arg arg max A1 (T), arg max A, (T (Tr, (Txx an))
arg max Acot (T, a)
(5)
a
aiÊA
dn E.A
then, we can say that [Q:] satisfies advantage-based IGM for Qtot-
As specified in Definition I advantage-based IGM takes duplex dueling architecture, Joint Dueling and Individual Dueling, which induces the joint and local (duplex) advantage functions by A == QQV.. Compared with regular IGM, advantage-bard based IGM transfers the consistency constraint action- value functions stated in Eq. (2) to that on advantage functions. This change is an equivalent transformation because the state-value terms V do not affect the action selection, as shown by Proposition !
Proposition The advantage-based IGM and IGM function classes are equivalent.
One key benefit of using advantage- based IGM is that its consistency constraint can be directly realized by limiting the value range of advantage functions, as indicated by the following fact.
Fact 1. The constraint of advantage-based IGM stated in Eq. (5) is equivalent to that when VT T. Va E (T), Va E (T). Vi E, Aror (T,a") Ai(Ti, a;) = and A a) < <0) A,(r,, 0,
(6)
where (T( = {a|a E A, Qroe(TT a) = Vioe(T))-
To achieve full expressiveness power of advantage-based IGM or IGM, Fact ennables us develop an efficient MARI. algorithm that allows the joint state-value function learning with any scalable
decomposition structure and just imposes simple constraints limiting value ranges of advantage functions. The next subsection will describe such MARL algorithm.
3.2 THE QPLEX ARCHITECTURE
In this subsection, we present novel multi-agent Q-learning algorithm with duplex dueling architecture, called QPLEX, which exploits Fact and
and realizes the advantage-based IGM constraint. The overall architecture of QPLEX is illustrated Figure [ which consists of two main components as follows: (i) an Individual Action- Action-Value Function for each agent, and (ii) Duplex Dueling component that composes individual action-value functions into a joint action-value function under the advantage-bon based IGM constraint. During the centralized training, the whole network is learned in an and-to-end fashion to minimize the TD loss as specified in Eq. ([. During the decentralized execution, the duplex dueling component will be removed, and each agent will select actions using its individual Q-fucction based on local observvttio history.
Individual Action- Value Function is represented by recurrent Q-network for each agent i, which takes previous hidden state h! 11-1 current local observations o/, and previous action of d-1 as inputs and outputs local (:(T;, az).
4

---

Published as a conference paper at ICLR 2021
Duplex Dueling component connects local and joint action-value functions via two modules: (i) a Transformation network module that incorporates the information of global state or joint history into individual action-valus functions during the centralized training process, and (ii) a Dueling Mixing network module that composes separate action-value functions from Transformation into joint action-value function. Duplex Dueling first derives the individual dueling structure for each agent i by computing its value function (I(T,) = maxa, Qi a:) and its advantage function (, 0.) (Ti a.) Vi(T:), and then computes the joint dueling structure by using individual dueling structures.
Transformation network module uses the centralized information transform local dueling structure [V(()), (T,,, 4:]l==1 to [Vi(T), (, a;)l==ı conditioned on the joint action-observatioo history, as shown below, for any agent i, i.e., Qi(T, 05) = wy (T) a;) + (T), thus,
Vi(T) = w;(T) + (T),,, and A,(T, Qi(T, Vi(T) = (T) a;), (7)
where w;(r) > 0 is positive weight. This positive linear transformation maintains the consistency of the greedy action selection and alleviates partial observability in Dec-POMDP (Son et al. 2019;
Yang et al. 2020). As used by QMIX (Rashid et al. 2018), QTRAN (Son et al. 2019), and Qatten ( (Yang et al. 2020), the centralized information can be the global state s, if available, or the joint action-observatton history T.
Dueling Mixing network module takes the outputs of the transformation network as input, e.g.. [V;,A,]3 = and produces the values of joint Qtot, as shown in Figure Ib. This dueling mixing network uses individual dueling structure transformed by Transformation to compute the joint value (T) and the joint advantage A a). respectively, and finally outputs Qtor (T, (To - Aroe(e (T, a) by using the joint dueling structure.
Based on Fact T the advantage-based IGM principle imposes constraints on value functions. Therefore, to enable efficient learning, we use a simple sum structure to compose the joint value:
Veoe = Σ Vi(T)
(8)
To enforce the IGM consistency of the joint advantage and individual advantages, as specified by Eq. ( QPLEX computes the joint advantage funetion follows:
ATo T, a) = Σ
a:), where >r 0.
The joint advantage function Atot the dot product of advantage functions [A,] and positive importance weights [N,] with joint history and action. The positivity induced by X will continue to maintain the consistency flow of the greedy action selection and the joint information of X, provides the full expressiveness power for value factorization. To enable efficient learning of importance weights X; with joint history and action, QPLEX uses scalable multi-head attention module (Vaswani
et al. 2017):
K
Ar a) =
Σ k=1
(10)
where is the number attention heads, (T, a) and (),x(() are attention weights activated by sigmoid regularizer, and (T) is positive key of each head. This sigmoid activation of A; brings sparsity to the credit assignment of the joint advantage function to individuals, which enables efficient multi-agent learning (Wang et al.) 2019b).
With Eq. (8) and (9), the joint action-value function Qiot can be reformulated as follows:
Q a)) = Vio((T) + Aroe (T, a) = Σ Qi((,,TT a.) + Σ (Ai(T, - 1) A.(T, az).
(11)
1=1
It can be seen that Qtor consists of two terms. The first term is the sum of action-value functions [QI which is the joint action-value value function Q of Qatten (Yang et al. 2020) (which is the Qiot of VDN (Sunehag et al.) 2018) with global information). The second term corrects for the discrepancy between the centralized joint action-value function and Quatten which is the main contribution of QPLEX to realize the full expressiveness power of value factorization.
5

---

Published as a conference paper at ICLR 2021
Median Test Retum
Mediun Test Return
M
81
A(1
A(2)
A
02
A0
8
-12
-12
QPLEX
Qenzo
GPLEX: S.I0E
A
-12
-12
(0)
6
(9)
0
QIRAN
QMIX
VDN
CW -W--XMXXX
Optimal
OW-QMIX
GULEX-22411
GPLEX-L44
QPLEX-IL.10E
Optanal
100
200
500
150
300
450
600
750
Iterations
Iterations
(a) Payoff of a harder matrix game
(b) Deep MARL algorithms
(c) Learning curves of ablation study
Figure 2: (a) Payoff matrix for a harder one-step game. Boldface means the optimal joint action selection from the payoff matrix. The strikethroughs indicate the original matrix game proposed by QTRAN. (b) The learning curves of QPLEX and other baselines.
(c) The learning curve of QPLEX, whose suffix aLbH denotes the neural network size with layers and b heads (multi-head attention) respectively.
for learning importance weights A (see Eq. ( and (10
Given the universal function approximation of neural networks, the action-value function class that QPLEX can realize is equivalent to what is induced by the IGM principle.
Proposition 2.
In practice, QPLEX can utilize common neural network structures (e.g.. multi-head attention modules) to achieve superior performance by approximating the universal approximation theorem (Csáji et al. 2001]. We will discuss the effects of QPLEX's duplex dueling network with different configurations in Section 4.1 As introduced by [Son et al. (2019) and Wang et al. (2020a), the completeness of value factorization is very critical for multi-agent Q-learning and we will illustrate the stability and state-of arr performance of QPLEX in online and offline data collections in the next section.
EXPERIMENTS
In this section, we first study didactic examples proposed by prior work (Son al. 2019; Wang et al. 2020a) to investigate the effects of QPLEX's complete IGM expressiveness on learning optimality and stability. To demonstrate scalability on complex MARL domains, we also evaluate the performance of QPLEX on a range of StarCraft II benchmark tasks (Samvelyan et al. (019). The completeness of the IGM function class can express richer joint action-value function classes induced by large and diverse datasets training buffers. This expressiveness can provide QPLEX with higher sample efficiency to achieve state-of-the-art performance online and offline data collections. We compare QPLEX with
state-of-the-an baselines: QTRAN (Son et al. 2019). QMIX (Rashid et al. 2018). VDN (Sunchag ct al. 2018). Qatten (Yang et al. 2020). and W WQMIX (OW-QMIX and C CW-QMIX; Rashid et al. 2020p. In particular, the second term of Eq. 11) is the main difference between QPLEX and Qatten. Thus, Qatten provides a natural ablation baseline of QPLEX to demonstrate the effectiveness of this discrepancy term. The implementation details of these algorithms and experimental settings are deferred to Appendix B We also conduct two ablation studies to study the influence of the attention structure of dueling architecture and the number of parameters on QPLEX, which are deferred be discussed in Appendix E Towards fair evaluation, all experimental results are illustrated with the median performance and 75% percentiles over random seeds.
4.1 MATRIX GAMES
QTRAN (Son et al. 2019) proposes hard matrix game, shown in Table 4a of Appendix C In this subsection, we consider a harder matrix game in Table 2a which also describes a simple cooperative multi-agent task with considerable miscoordination penalties, and its local optimum is more difficult to jump out. The optimal joint strategy of these two games is to perform action A(1) simultaneously. To ensure sufficient data collection in the joint action space, we adopt uniform data distribution. With this fixed dataset, perspective, ignoring the challenge of exploration and sample complexity.
study the optimality of multi-agent Q-learning from optimization
As shown in Figure 2b QPLEX, QTRAN, and WQMIX, which possess richer expressiveness power of value factorization can achieve optimal performance, while other algorithms with limited expressiveness (e.g.. QMIX, VDN, and Qatten) fall into local optimum induced by miscoordination penalties. In the original matrix proposed by QTRAN, QPLEX and QTRAN can also successfully converge to optimal joint action-value functions. These results are deferred to Appendix C QTRAN

---

Published as a conference paper at ICLR 2021
500
T=0
@1 202, r=0
400
QPLEX
QTRAN
QMIX
41 22 =A
VDN
200
Qatien
r=0
OW-QMEX
100
CW-QMIX
Oytimal
(1=0==A()) r=1
40
80
100
Iteration
(a) Two-state MMDP
(b) Learning curves of MARL algorithms
Figure 3: (a) A special two-state MMDP used to demonstrate the training stability of the multi-agent Q-learning algorithms. r is shorthand for r(s, a). (b) The learning curves of 1Qt oo in a specific two-state MMDP
achieves superior performance in the matrix games but suffers from its relaxation of IGM consistency in complex domains (such StarCraft II) shown in Section 4.3
In the theoretical analysis of QPLEX, Proposition exploits the universal function approximation of neural networks. QPLEX allows the scalable implementations with various neural network capacities
(different layers and heads of attention module) for learning importance weights X; (see Eq. ( and (TO).. As shown in Figure 2c] by increasing the neural network size for learning A; (e.g.. OPLEX-3L1OH) QPLEX possesses more expressiveness of value factorization and converges faster. However, learning efficiency becomes challenging for complex neural networks. To effectively perform StarCraft II tasks ranging from 2 to 27 agents, we use small multi-head attention module (i.e.. QPLEX- IL4H) in complex domains (see Section 4.3p. Please refer to Appendix B for more detailed configurations.
4.2 TWO-STATE MMDP
In this subsection, focus Multi-agent Markov Decision Process (MMDP) (Boutilier 1996) which is a fully cooperative multi-agent setting with full observability. Consider a two-state MMDP proposed by /Wang et al. (2020a) with two agents, two actions, and single reward (see Figure Gap. Two agents start at state 82 and explore extrinsic rewards for 100 environment steps. The optimal policy of this MMDP is simply executing the action (12) at state $2, which is the only coordination pattern to obtain the positive reward. To approximate the uniform data distribution, uniform exploration strategy (i.e., e-greedy exploration with e = 1). We consider the training stability of multi-agent Q-learning algorithms with uniform data distribution in this special MMDP task. As shown in Figure 3b) the joint state-value function Qtot. learned by baseline algorithms using limited function classes, including QMIX, VDN, and Qatten, will diverge. This instability phenomenon of VDN has been theoretically investigated by Wang et al. (2020a). By utilizing richer function classes, QPLEX, QTRAN, and WQMIX can address this numerical instability issue and converge to the optimal joint vtlue function.
adopt
4.3 DECENTRALIZED STARCRAFT II MICROMANAGEMENT BENCHMARK
A more challenging set of empirical experiments are based on StarCraft Multi-Agent Challenge (SMAC) benchmark (Samvelyan et al. 2019). We first investigate empirical performance a popular experimental setting with c-greddy exploration and limited (Samvelyan et al., 2019), named online data collection setting. To demonstrate the offline training potential of QPLEX, we also adopt the offline data collection setting proposed by Levine et al. (0000). which can be granted access to given dataset without additional online exploration.
first-out (FIFO) buffer
4.3.1 TRAINING WITH ONLINE DATA COLLECTION
We evaluate QPLEX in 17 benchmark tasks of StarCraft II, which contains 14 popular tasks proposed by SMAC (Samvelyan et al. 2019) and three new super hard cooperative tasks. To demonstrate the overall performance of each algorithm, Figure [guure4ploor plots the averaged median test win rate across all 17 scenarios and the number of scenarios in which the algorithm outperforms, respectively. Figure 4a shows that, compared with other baselines, QPLEX constantly and significantly outperforms baselines
7

---

Published as a conference paper at ICLR 2021
Averuged Median Text Win Rate
7.5
Maps Best (out of 1f
60
mm
QIRAN
QPLEX
QMIX
VON
30
Qalien
1.5
OW-QMIX
CW-QMIX
6.0M
6.4M
0.3M
1.2M
LoM
2
0.3M
0.4M
0.5M
1.2M
L6M
2,0M
Timesteps
Timestops
(a) Averaged test win rate
(b) Maps best of 17 scenarios
Figure 4: (a) The median test win %, averaged across all 17 scenarios. (b) The number of scenarios in which the algorithms' median test win % is the highest by least 1/32 (smoothed).
over the whole training process and exceeds at least 10% median test win rate averaged across all 17 scenarios. Moreover, Figure 4b illustrates that, among all tasks, QPLEX is the best performer on up to eight tasks, underperforms on just two tasks, and ties for the best performer on the rest tasks. After 0.8M timesteps, the number of tasks that QPLEX achieves the best performance gradually decreases to five, because, in several easy tasks, other baselines also reach almost 100% test win rate as shown in Figure The overall evaluation diagram of the original SMAC benchmark (14 tasks) corresponding to Figure i deferred to Figure 8 in Appendix [
Figure B the learning curves on nine tasks in the online data collection setting and the results of other eight maps are deferred to Figure 7 in Appendix D From Figure [ we can observe that QPLEX significantly outperforms other baselines with higher sample efficiency. On the super hard map 5s10z, the performance gap between QPLEX and other baselines exceeds 30% in test win rate, and the visualized strategies of QPLEX and QMIX in this map are deferred to Appendix [F) Most multi-agent Q-learning basclines including QMIX, VDN, and Qatten achieve reasonable performance (see Figure 5) However, as Figure suggests, QTRAN performs the worst in these comparative experiments, even though it performs well the didactic games. From theoretical perspective, the online data collection process utilizes e-greedy exploration process, which requires individual greedy action selections to build an effective training buffer. QTRAN may suffer from its relaxation of IGM consistency (soft constraints of IGM) in the online data collection phase, while the duplex dueling architecture of QPLEX (hard constraint of IGM) provides effective individual greedy action selections, making it suitable for data collection with e-greedy exploration.
Moreover, although WQMIX (OW-QMIX and CW-QMIX) outperforms QMIX in some tasks (il- lustrated in Figure 5 e.g.. 2c vs 64zg and bane_ sa__vv_baaa _s_ba WQMIX show very similar overall performance as QMIX across 17 StarCraft II benchmark tasks. In contrast, QPLEX achieves signifi- cant improvement in convergence performance in lot of hard and super hard maps and demonstrates high sample efficiency across most scenarios (see Figure 5)
4.3.2 TRAINING WITH OFFLINE DATA COLLECTION
Recently, offline reinforcement learning has been regarded as a key step for real-world RL applications (Dulac-Arnold et al. 2019 Levine et al. 2020). Agarwal et al. (2020) presents optimistic perspective of offline Q-learning that DQN and its variants can achieve superior performance in Atari 2600 games (Bellemare al. 2013) with sufficiently large and diverse datasets. In MARL, StarCraft II benchmark has the same discrete action space as Atari. We conduct a lot of experiments on the StarCraft II benchmark tasks to study offline multi-agent Q-learning in this subsection. We adopt a large and diverse dataset to make the expressiveness power of value factorization become the dominant factor to investigate. We train a behavior policy of QMIX and collect all its experienced transitions throughout the training process (see the details in Appendix 9 As shown in Figure [3] in Appendix G QPLEX significantly outperforms other multi-agent Q-learning baselines and possesses the -fthe-art value factorization structure for offline multi-agent Q-learning. QMIX and Qatten cannot always maintain stable learning performance, and VDN suffers from offline data collection and leads to weak empirical results. QTRAN may perform well in certain cases when its soft constraints, two f2-penalty terms, are well minimized. With offline data collection, individual greedy action selections do not need build training buffer, but they still need to compute the one-step TD target
8

---

Published as a conference paper at ICLR 2021
QPLEX
QTRAN
QMIX
Qatten
OW-QMIX
CW-QMIX
3s52
Median Test Win Rate %
1c3s5z
10m_vs, jom__s____
Median Test Win Rate
100
100
100
00
00
Median Win
40
20
20-
0.01
0.5M
1.0M
1.5M
2.0M
0.0M
0.5M
1.0M
1.5M
2.0M
0.0M
0.5M
1.0M Timesteps
1.5M
2.0M
Timesteps
Timesteps
2c_vs_ [c_vs_6440 ___44g
35_ 3s__s__ _5
bane vs. bane
Median Test Win Rate
100
Median Test Win Rate
100
100
80
80
80:
Madian Win
1.0M
2.0M
0.5M
1.5M
2.0M
0.0M
1.0M Timesteps
Timesteps
Timesteps
5s10z
Mediaa an Test Win Rate %
7sz
1c358z_ve ls_.vs_llcc vs_1c3s92
Med an Test Win Raze
100
100
100
80
80
00
Median Win
60
60
60
1.0M Timesteps
2.0M
0.5M
1.5M
2.0M
1.5M
Timesteps
Timesteps
Figure 5: Learning curves of StarCraft II with online data collection.
for centralized training. Therefore, compared with QTRAN, QPLEX still has theoretical advantages regarding the IGM principle in the offline data collection setting.
CONCLUSION
In this paper, we introduced QPLEX, novel multi-agent Q-learning framework that allows central- ized end-to-end training and learns to factorize joint action value function to enable decentralized execution. QPLEX takes advantage IGM consistency constraint on joint and individual greedy action selections. Our theoretical analysis shows that QPLEX achieves a complete IGM function class. Empirical results demonstrate that significantly outperforms state-of-the-at baselines in both online and offline data collection settings. In particular, QPLEX possesses strong ability of supporting offline training. This ability provides QPLEX with high sample efficiency and opportunities of utilizing offline multi-source datasets. It will be an interesting and valuable direction to study offline multi-agent reinforcement learning in continuous action spaces (such MuJoCo (Todorov al. 2012 with QPLEX's value factorization.
duplex dueling architecture that efficiently encodes the
ACKNOWLEDGEMENT
We would like to thank the anonymous reviewers for their insightful comments and helpful sugges- tions. This work is supported in part by Science and Technology Innovation 2030 "New Generation Artificial Intelligence" Major Project (No. 201 2018AAA0100904)) and grant from the Institute of Guo Qiang, Tsinghua University.
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference Machine Learning, 2020.

---

Published as a conference paper at ICLR 2021
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Craig Boutilier. Planning, learning and coordination in multiagent decision processes. In Proceedings of the 6th Conference on Theoretical Aspects of Rationality and Knowledge, pp. 195-210. Morgan Kaufmann Publishers Inc., 1996.
Yongcan Cao, Wenwu Yu, Wei Ren, and Guanrong Chen. An overview of recent progress in the study of distributed multi-agent coordination. IEEE Transactions on Industrial Informatics, 9(1): 427-438, 2012.
Balázs Csanád Csáji et al. Approximation with artificial neural networks. Faculty of Sciences. Ervs Lornd University, Hungary, 24(48):7, 2001.
Gabriel Aannac- Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. arXiv preprint arXiv: 2019.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072/9, 2020.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. International Conference on Machine Learning, pp. 2052-2062, 2019.
Maximilian Hüttennauch, Adrian Šošié, and Gerhard Neumann. Guided deep reinforcement learning for swarm systems. arXiv preprint orXiv:1709.066 07709006011 0,6611 1, 2017.
Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning rehearsal for decentralized planning. Neurocomputing, 190:82-94, 2016.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. arXiv preprint arXiv:2005.01644 2020.
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. In Advances in Neural Information Processing Systems, pp. 7611-7622, 2019.
Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289-353, 2008
Frans A Olichock, Christopher Amato, al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pp. 419242224449 43011, 2018.
Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 33, 2020.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung. Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft chaleegg In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2186-2188. International Foundation for Autonomous Agents and Multiagent Systems, 2019.
Kyunghwan Son, Daewoo Kim, Wan Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, pp. 5887-5896, 2019.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. vecoppdecooiosition networks for cooperative multi aagtt learning based on team reward. Proceedings of the 17th International Conference Autonomous Agents and MultiAgent Systems, pp. 2085-2087 2018.
10

---

Published as a conference paper at ICLR 2021
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-basee control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998-6008 2017.
Jianhao Wang. Zhizhou Ren, Beining Han, and Chongjie Zhang. Towards understanding linear value decomposition in cooperative multi-agent q-learning. arXiv preprint arXiv: 00587, 2020a.
Tonghan Wang, Jianhao Wang. Yi Wu, and Chongjie Zhang. Influence-based multi-agent exploration. arXiv preprint arXiv: 2019a.
Tonghan Wang. Jianhao Wang, Chongyi Zheng, and Chongjie Zhang. Learning nearly decomposable value functions via communication minimization. arXiv preprint arXiv:1910.05366, 2019b.
Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Multi-agent reinforcement learning with emergent roles. arXiv preprint arXiv:2003 08039, 2020b.
Tonghan Wang. Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523,, 2020c.
Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. Off-policy multi-agent decomposed policy gradients. arXiv preprint arXiv:2007.12322 2020d.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International Conference on Machine Learning, pp. 1995-2003, 2016.
Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.03999 2020.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-basee offline policy optimization. arXiv preprint arXiv:2005.13239 2020.
Chongjie Zhang and Victor Lesser. Coordinated multi-agent reinforcement learning in networked distributed pomdps. Twenty-Fifth AAAI Conference on Artificial Intelligence, 2011.

---

Published as a conference paper at ICLR 2021
A OMITTED PROOFS IN SECTION
3
Definition (Advantage-based IGM). For a joint action-value function Qiat: T x A
+ AA
and
individual action-value functions [Q; T A Ri] where VT T, Ya E A, Vi E.N
(Joint Dueling)
Quol a. Vio((T) (T, a) and (iot(T) = max Qior (T, a²), (3)
a'
(Individual Dueling)
Qi(Ts, Vi(Ta) Ac(Ti, as) and Vi(Ti) max Qi(Ti,
of
such that the following holds
arg max Atot (T, .) arg max A acA
arg max An (Tn; An(Tr:an)) an)
(5)
on
then, we can say that [Qili=1 satisfies advantage-base IGM for Qror.
Let the action-valus function class derived from IGM is denoted by
{ ERITIAI," [ E ATTII "
Eq. 2 is satisfied
where Qiot and @ i=1 respectively. Similarly, let
denote the joint and individual action-value functions induced by IGM,
@ ē= ( ERITIAI"
[Q. [Q.ERITIAA]]
Eq.
(3)
5 are satisfied
denote the action-valle value function class derived from advantage-baser IGM. Veot and Asst denote the joint state-value and advantage functions, respectively. [
and A i=1 denote the individual
1=1
state-value and advantage functions induced by advantage IGM. respectively. According to the duplex dueling architecture Q V + A stated in advantage-bar based IGM (see Definition we derive the joint and individual action-valu functions following: VT T, Va A, Vi E N.
IGM.
QI a) = (To + Aroe a) and Qi(Ti, a.) = Vi(Ts)) + As(Ti, az). Proposition The advantage-based IGM and IGM function classes are equivalent.
Proof. We will prove @ = the following two directions.
cc ã For any (Qror, @ -
2, we construct Qeot = Qiot and [Q.] i=1
= @
The
i=1
joint and individual vaate--alue/advantage functions induced by advantage-IGiM
(T) = max Qtor ( (T, a²) and (To a) = =Qoo (To a) VIo)(T), ÂI(T,, (T) -()))))))V(TT)) ViEW, V.( = max Q 4/) and
a'
are derived by Eq. and Eq. action selection, VT E T. Va A.
respectively, Because state-value functions do not affect the greedy
arg max 21oe(T.a0 =
arg max Q1 (T (T).01), arg max Qn (Tn, an) 0.EAA
asA
an EA
arg max Qrai (T, a) aEA
arg max (2( arg max Qn (Tn, (Tn, a.EA
a. E.
(Quel(T, a) -Vio((T)
= arg max aEA
(Qi(T), - vi((1))
arg max (Ta(Ta Va V(((m))
arg max
aIEA
a.EA
+ arg max (T, asA
arg max (1 aled
arg max (n am))
an EA
Thus,
(Qrot:
Qi n1
②, which means that @c Ó
:=1
12

---

Published as a conference paper at ICLR 2021
2 Q ②
We will prove this direction the same way. For any (Qrot, @ (
Ô we construct
E
Qiot Quat and Q n
@
Because state-value functions do not affect the greedy action
selection, VT T. va i=1 A.
arg max Arot (T, aEA
arg
mag max Āi (T ai), *aEAA
arg max (Tn; an)
a,EA
(ALoC(T,a) -Vooe(T)
arg max aEA
+ ((i.aai)+(((ii))))))))
(An(Tn: + V ((a(t))))
arg max
arg max anEA
a1EA
= arg max (T,
arg max Q1(
arg max 2n (T an) anEA
acA
aIEA
((rg)) arg max Q1(T1, a1EA
arg max Qioc(T,a)=
arg max Qu a.EA
(Tar (n(Taaan)) an)
aeA
Thus, (Qtot: Q i=1
E 2. which means that 2 C Ô. The action-value function classes derived
from advantage-base IGM and IGM are equivalent.
Fact The constraint of advantage-based IGM stated Eq. is equivalent to that when VT E T, va E (T), Va E A' (T), Vi E. Atoc(T, Ai(T,, =0 and Ator(T, < (., 50,
(6)
where (²(T) faaa E A, QuocQ
Vior(T))-
Proof. We derive that VT E TT, Va E A, E., Asor(T, a) S 0 and Ac(Ts, a:) S from Eq.
and Eq. (4) of Definition [ respectively. According to the definition of arg operator, Eq.
and Eq. (4). VT T, let
(T) denote arg max Aror(T (T,a) a) follows:
A'(7) (T) = arg max And (T, a) = arg max (roe(T,,0) (T,
aEA
aEA
(ala= E A, (roe(T,T)) = Vior(T))
{ala E A, Qioe(T, (T, a) - Vioe(T) = =0}
(ala E A, Aror (T, = }). Similarly, VT T.W. Vi N, let (T)) denote arg max,, EA A, a:) as follows:
(12)
A( arg max As(T;, ai) arg max Qi(Ti, a:) a,A
a.EA
{ailss A, Qi(T. = (.(T.))
{alls E A, Ac a;) = 0}.
(13)
Thus, VT E Ā (T), Va E A (7),).
Aror(T, a*) = and ATor(T, a) <
(14)
VT E T.E.EN.. E (Tt)). Va, E (T)).
A at) = 0 and Å, az) <0.
(15)
Recall the constraint stated in Eq. 5
VT E T.
arg max An (Tn (nrnan))) a, EA
arg max A (T, a) acA
arg max A (7), 01), 0,
13

---

Published as a conference paper at ICLR 2021
We rewrite the constraint of advantage-biin based IGM stated in Eq. 5 as VT T
A" (T) = { (o1.....,,aau an) E (T)))
(16)
Therefore, combining Eq. Va E A A' (T), ViE
14). Eq. 15). and Eq. (6). we can derive VT E T, va (T).
Acoe (T, a²) aT)
(T,T,, 05) and ( a) < < &, 0;) S.
(17)
In another way, combining Eq. (1). Eq. 15 and Eq. (7]. we can derive Eq. 16 by the definition
of A and A' (see Eq. 12) and Eq. 1. more detail, the closed set property of Cartesian
i=1
product [a;] has been encoded into the Eq. (1 and Eq. 17 simultaneously.
Proposition 2. Given the universal function approximation of neural networks, the action- value function class that QPLEX can realize is equivalent to what is induced by the IGM principle.
Proof. We assume that the neural network of QPLEX can be large enough to achieve the universal function approximation by corresponding theorem (Csáji et al. 2001). Let the action-value function class that QPLEX realize is denoted by
2 0= IIITIAA, @2, R RITII
Eq.
77,,
(8),
09,
10
11
are satisfied]
1==1/
In addition, Qrat V tot, Atnts
[
A
i-1 Q.1 [. 1-1 and [A.] denote
the corresponding (joint, transformed, and individual) (action-value, state-value, and advantage) functions, respectively. In the implementation of QPLEX, we ensure the positivity of important weights of Trunsformation and joint advantage function, [wr]]__ and [Ai]_=]- which maintains the greedy action selection flow and rules out these non non-interesting points (zeros) optimization. We will prove Q = Q in the following two directions.
C Q For any (Qroe:
Q
Ô we construct Qioe -Qrot and [Q:]i = [
E
and derive V tor Acots construction of QPLEX,
V
i== and
A | by Eq. and Eq. 4 respectively. Note that in the
= ((()))))))(())+bbb and (
and
a
Quoe(T, a) Vioe(T) Ator a) Σ Vi(T) + Σ ((T.O)
In addition, we construct transformed functions connecting joint and individual functions as follows: VT ET A. N.
a)(T,a)= Q V((T)
arg arg C'(TT, a), and A((T, = -
n
asA
which means that according to Fact
A((T,, 00
when Ar(Ti ai) 0,
(x(T) = (x(T) = V - Vi(T,), and As(T..
A,(Te, ay)
1,
when Ae(Ti, a;) = 0.
Thus, (Qrat: [Q.] E Q. which means that Ô Q.
14

---

Published as a conference paper at ICLR 2021
Q C 2 For any (otor: [ [2i
E 2, with the similar discussion of Fact
VT E T, Vi N, let
A (7)) denote arg a E.A A (T, ai), where
Ai(T.) fasla; E A, A,,,,T, =
Combining the positivity of [wi]i=1 and [Al]i=] with Eq. 7 8
(9
and
11
can derive
T.EN, Vaz E A (ri). Va, E A \ (Ti).
A, a.) = 0 and
A, a:) <
= A( 0)) : d;) = and
= w: (T) o,) <0
Atoe(T >r(T a [ a.) 0 and an). Notably, these a" forms an)
A \r (u(T.a).1 (T) <,
where a" = (ai,.... (2 and
A (T) = (a).....,a))( E A. (T).
(18)
which is similar to Eq. (16 in the proof of Fact We construct Qiot Qiot and Q
Q
n
i=1'
According to Eq. (18). the constraints of advantage-based IGM stated in Fact (Eq. Eq. Eq. (6)) are satisfied, which means that (Qrat:
4 and
@
ê and Q ².
Thus, when assuming neural networks provide universal function approximation, the joint action-value function class that QPLEX can realize is equivalent to what is induced by the IGM principle.
B EXPERIMENT SETTINGS AND IMPLEMENTATION DETAILS
B.1 STARCRAFT II
We consider the combat scenario of StarCraft II unit micromanagement tasks, where the enemy units are controlled by the built-in AI, and each ally unit is controlled by the reinforcement learning agent. The units of the two groups can be asymmetric, but the units of each group should belong to the same race. At each timestep, every agent takes action from the discrete action space, which includes the following actions: no-op, move [direction]. attack [enemy id], and stop. Under the control of these actions, agents move and attack continuous maps. At each time step, MARL agents will get global reward equal to the total damage done to enemy units. Killing each enemy unit and winning the combat will bring additional bonuses of 10 and 200, respectively. We briefly introduce the SMAC challenges of our paper in Table
B.2 IMPLEMENTATION DETAILS
We adopt the PyMARL (Samvelyan et al.
2019) implementation of state- of thetherrrt baselines: QTRAN
(Son al. 2019), QMIX (Rashid et al.) 2018). VDN (Sunehag al. 2018). Qatten (Yang al. 2020), and WQMIX (OW-QMIX and CW- NW-QMIX: Rashid et al. 2020). The hyper-parameters of these algorithms are the same as that in SMAC (Samvelyan et al. 2019) and referred in their source codes. are illustrated in Table 2 and other common hyper-parametern are adopted by the default implementation of PyMARL (Samvelyan et al. 2019). Especially in the online data collection, we take the advanced implementation of Transformation of Qatten in QPLEX. To ensure the positivity of important weights of Transformation and joint advantage function, In addition, we stop gradients of local advantage function A; to increase the optimization stability of the max operator of dueling structure. This instability consideration due to max operator has been justified by Dueling DQN (Wang et al. 2016). We approximate the joint action-valuu function as
QPLEX is also based on PyMARL, whose special
add sufficiently small amount e' 10-10 on [w:]/_n and [>,)
T
Qior a)
(((TT +
Σ (M(T,a)-1 (T,T,a))
where A, denotes variant of the local advantage function A, by stoping gradients.
15

---

Published as a conference paper at ICLR 2021
Map Name
Ally Units
Enemy Units
2x3z
Stalkers 3 Seallos
Stalkers 3 Zealots
3s52
3 Stalkers & 5 Zealots
3 Stalkers & 5 Zealots
1c3s52
1 Colossus, 3 Stalkers & 5 Zealots
1 Colossus, 3 Stalkers & 5 Zealots
5m_ vs_ 6m
5 Marines
6 Marines
10m_vs_lm vs___m
10 Marines
11 Marines
27m vs_ 30m
27 Marines
30 Marines
3s52_ vs_______ 33_6z
3 Stalkers & $ 7ealots
3 Stalkers & 6 Zealots
MMM2
1 Medivac, 2 Marauders & 7 Marines 1 Medivac, 2 Marauders & 8 Marines
2s_vs_ s_vvs_le Isc
2 Stalkers
Spine Crawler
3s. vs 5z
3 Stalkers
Zealots
6h vs 8z
Hydralisks
Zealots
bane_vs, vs bane
20 Zerglings & 4 Banclings
20 Zerglings & 4 Banelings
2e_vs_ e_vs_64zg
2 Colossi
64 Zerglings
corridor
Zealots
24 Zerglings
5s10z
5 Stalkers 10 Zealots
5 Stalkers & 10 Zealots
7sz
7 Stalkers & 7 Zealots
7 Stalkers & 7 Zealots
13382z_v__l3$62
1 Colossus, 3 Stalkers & 8 Zealots
1 Colossus, 3 Stalkers & 9 Zealots
Table I: The StarCraft multi-Agent challenge (SMAC: Samvelyan et al.
2019 benchmark.
QPLEX's architecuture configurations
Didactic Examples
StarCraft II
The number of layers in w. b, A, Φ, The number of heads in the attention module Unit number in middle layers of w, b, X, ø,
2 or
4 or 10
4
64
Ø
Activation in the middle layers of w, Activation in the last layer of Activation in the middle layers of Activation in the last layer of b
Relu
Ø
Absolute
Absolute
Relu
Ø
None
None
Activation in the middle layers of X, d Activation in the last layer of X
Relu
Ø
Sigmoid
Sigmoid
Table 2: The network configurations of QPLEX's architecture. architetuer
Our training time on an NVIDIA RTX 2080TI GPU of each task is about 6 hours to 20 hours, depending on the agent number and the episode length limit of each map. The percentage episodes in which MARL agents defeat all enemy units within the time limit is called test win rate. We pause training every 10k timesteps and evaluate 32 episodes with decentralized greedy action selection to measure test win rate of each algorithm. After training every 200 episodes, the target network will be updated once. We call this update period an Iteration for didactic tasks. In the two-state MMDP, Optimal line of Figure 3b is approximately Ei 7² = 63.4 one episode of 100 timesteps.
Training with Online Data Collection We have collected a total of 2 million timestep data for each task and test the model every 10 thousand steps. We use e-greedy exploration and limited first-infirst--uut (FIFO) replay buffer of size 5000 episodes, where linearly annealed from 1.0 to 0.05 over 50k timesteps and keep it constant for the rest training process. To utilize the training buffer more efficiently, we perform gradient updates twice with a batch of 32 episodes after collecting every episode for each algorithm.
Training with Offline Data Collection To construct diverse dataset, we train a behavior policy of QMIX (Rashid et al., 2018) or VDN (Sunehag et al., (018)) and collect its 20k or 50k experienced episodes throughout the training process. The dataset configurations are shown in Table 3 We evaluate QPLEX and four baselines over six random seeds, which includes three different datasets and tests two seeds each dataset. We train 300 epochs to demonstrate our learning performance, where each epoch trains 160k transitions with a batch of 32 episodes. Moreover, the training process of behavior policy is the same as that discussed in PyMARI. (Samvelyan et al. 2019).
16

---

Published as a conference paper at ICLR 2021
Map Name
Replay Buffer Size
Behaviour Test Win Rate
Behaviour Policy
2s3z
20k episodes
95.8%
QMIX
3s5z
20k episodes
92.0%
QMIX
Ic3s5z
20k episodes
90.2%
QMIX
2s_vs_Isc
20k episodes
98.1%
QMIX
3s_vs, 5z
20k episodes
94.4%
VDN
2c vs 64zg
50k episodes
80.9%
QMIX
Table 3: The dataset configurations of the offline data collection setting.
C
OMITTED FIGURES AND TABLES IN SECTION
AND 4.2
40)
A ((3)
d1
A
A
A
01
4(1)
A
A(3)
02
02
02
A
8
-12
-12
A0))
8.0
-12.1
-12.1
A
8.0
-1..0
-2..0
4
-12
0
0
A (2)
-1.22
-0.0
-0.0
A
-1..0
+0.0
0.0
A3
-12
A (8)
-12.1
-0.0
-0.0
A
-12.0
0.0
0.0
(a) Payoff of matrix game
(b) Que of QPLEX
(c) Stor of QTRAN
A(4)
A(2)
A(3)
A(1)
A(2)
A
A(4)
A(2)
A
02
02
02
A0
-8,0
-8.0
-8.0
A
-6,2
-49
-4.9
A
66.2
-4.9
-4.9
A(2)
-8.0
-0.0
-0.0
A(2)
-.9
-3.6
-3.6
A
-4.9
-3.5
-3,5
A3
-8.0
-0.0
-0.0
A3
-4.9
-3.6
-3.6
A3
-4.9
-3.5
-3.5
(d) Qroc of QMIX
(e) Qeot of VDN
(f) Qroe of Qatten
Table (a) Payoff matrix the one-step game. Boldface means the optimal joint action selection from payoff matrix. (b-f) The joint action-value functions Qtat of QPLEX, QTRAN, QMIX, VDN, and Qatten. Boldface means greedy joint action selection from joint action-value functions.
Median Text
QPLEX
QTRAN
Qutin
200
300
400
500
Iterations
Figure 6: The learning curves of QPLEX and other baselines the origin matrix game.
17

---

Published as a conference paper at ICLR 2021
D EXPERIMENTS ON STARCRAFT II WITH ONLINE DATA COLLECTION
QPLEX
QTRAN
QMIX
VDN
Qatten
OW-QMIX
CW-QMIX
2s_vs_ 1sc
Median Test Win Rate %
253z
5m_vs 3m_vs_6m
Median Test Win Rate
100
100
100
80
00
Mn Win
40
20
20
0.0M
0.5M
1.0M
1.5M
2.0M
0.0M
0.5M
1.0M Timesteps
1.5M
2.0M
0.0M
0.5M
1.0M
1.5M
2.0M
Timesteps
Timesteps
3s52_ vS_ 3s62
6h vs 8z
27m vS_30m
Median Test Win Rate
100
Median Test Win Rate
100
100
80
80
20
0.0M
0.5M
1.0M
1.5M
2.0M
0.0M
0.5M
1. OM
1.5M
2.0M
0.0M
0.5M
1.0M Timesteps
1.5M
2.0M
Timesteps
Timesteps
MMM2
Median Test Win Rate %
corridor
Median Test Win Rtate
100
100
80
80
60
60
1.0M Timesteps
2.0M
0.5M
1.5M
2.0M
Timesteps
Figure 7: The learning curves of StarCraft II with online data collection on remaining scenarios.
Avarged Median Test Win Rare %/6
75
Mape Best 14)
60
QPLEX
QTRAN
QMIX
30
OW-QMIX
CW- XW-QMIX
088
1.6M
20M
Timestops
Timesteps
Averaged test rate
(b) Maps scenarios
Figure 8: (a) The median test win %, averaged across all 14 scenarios proposed by SMAC (Samvelyan iet al. 2019). (b) The number of scenarios in which the algorithms' median test win the highest by at least 1/32 (smoothed).
18

---

Published as a conference paper at ICLR 2021
E ABLATION STUDIES WITH ONLINE DATA COLLECTION
In this section, we conduct two ablation studies to investigate why QPLEX works, which includes: (i) QPLEX without the multi-head attention structure of dueling architecture, and (ii) QMIX with the same number of parameters as QPLEX. For these studies, Figure plots the averaged median test win rate % over the tasks of the StarCraft II benchmark mentioned Section 4.3.1 Detailed learning curves on each task are shown in Figure 10 I1 respectively.
2
Multi-head attention structure of importance weights A (see Eq. ( and (T0) allows QPLEX adapt its scalable implementation to different scenarios, e.g., didactic games or StarCraft II benchmark tasks. Section D the importance of this attention structure, which can provide QPLEX more expressiveness of value factorization to perform better on the didactic matrix games. In this ablation study, we aim to test whether this multi-head attention is necessary for this StarCraft domain. Specifically, we use one-layer forward model instead of this multi-head attention structure in QPLEX, which called QPLEX- OPLEe-wo-duel- atten. Figure 9a shows that QPLEX-wo-duel achieves similar performance QPLEX, which indicates that the the superiority of QPLEX over other MARL. methods is largely due to the duplex dueling architecture (see Figure T) instead of the multi-head attention trick.
Averaged Median Tost Win Rate %
Averaged Median Test Win Rate %
100
100
QELIX QMIX
60
Large QMIX
40
40
QMI
Qaten
0.8M
0.434
0.55
1.2M
1AM
2.0
0.0M
0.4M
0,804
1.2M
1.6M
2.0M
Timesteps
Timestenns
(a) Without dueling attention
(b) QMIX with similar neurons
Figure Ablation studies on QPLEX with the median test win %, averaged benchmark scenarios.
QPLEX uses more parameters in the value factorization architecture of neural networks due to its multi-head attention structure. We introduce Large QMIX with similar number of parameters with QPLEX, to investigate whether the superiority of QPLEX over QMIX is due to the increase in the number of parameters. Figure 9b shows that QMIX with larger network cannot fundamentally improve its performance, and QPLEX still significantly outperforms Large QMIX by large margin. This result also confirms that the main effect of QPLEX comes from novel value factorization structure (duplex dueling architecture) rather than the number of parameters.
19

---

Published as a conference paper at ICLR 2021
QPLEX
QMIX
Qatten
vs 1sc
2s3z
3s5z
Medlan Test Win Rate
100
Median Test Win Rate
100
Medlan Test Win Rate
100
25-
25-
25-
0.0M
0.5M
1.0M
1.5M
2.0M
0.0M
0.5M
1.0M
1.M
2.0M
0.0M
0.5M
1.0M
1.5M
2.0M
Timesteps
Timesteps
Timesteps
Median Test Win Rate %
1c3s5z
Median Test Win Rate %
10m vs 11m
Median Test Win Rate %
2c vs 6__4
100
100
100
75
50
50
50
25-
25-
25
0.0M
0.5M
1.0M
2.0M
0.0M
0.5M
1.0M
15M
2.0M
0.0M
0.5M
1.5M
2.0M
Timesteps
Timesteps
Timesteps
Median Test Win Rate %
5m_ _s_ vs_6m
Median Test Win Rate %
3s vs. 5z
Median Test Winn Rate %
MMM2
100
100
100
75
75
75
1.0M Timesteps
Timesteps
Timesteps
5s10z
7sz
1c3s8z_vs vs_cv_ss2212222
Median Test Win Rate
100
Median Test Win Rate
100
Median Test Win Rate
100
0
0.0M
0.5M
1.0M
1 5M
2.0M
0.0M
0.5M
1.0M
1.5M
2.0M
0.0M
0.5M
1.0M
1.5M
2.0M
Timesteps
Timesteps
Timesteps
Figure 10: The learning curves of median test win rate % for QPLEX, QPLEX's ablation QPLEX- wo-duel-atten QMIX, and Qatten with online data collection.
20

---

Published as a conference paper at ICLR 2021
QPLEX
QMIX
Large QMIX
vs_ 1sc
2s3z
3s5z
Median Test Win Rate
100
Median Test Win Rate
100
Medlan Test Win Rate
100
25-
25
25
0.0M
0.5M
1.0M
1.5M
2.0M
0.0M
0.5M
1.0M
1.M
2.0M
0.0M
0.5M
1.0M
1.5M
2.0M
Timesteps
Timesteps
Timesteps
Median Test Win Rate %
1c3s5z
Median Test Win Rate %
10m vs 11m
Median Test Win Rate %
2c vs vs_64
100
100
100
75
50
50
50
25-
25-
25
0.0M
0.5M
1.0M
2.0M
0.0M
0.5M
1.0M
15M
2.0M
0.0M
0.5M
1.5M
2.0M
Timesteps
Timesteps
Timesteps
Median Test Win Rate %
5m vs 6m
Median Test Win Rate %
3s vs. 5z
Median Test Win Rate %
MMM2
100
100
100
75
75
75
Timesteps
Timesteps
Timesteps
5s10z
7sz
1c3s8z_vs 1esi2_v__133$99 s__ss999
Median Test Win Rate
100
Median Test Win Rate
100
Median Test Win Rate
100
0
0.0M
0.5M
1.0M
1. 5M
2.0M
0.0M
0.5M
1.0M
1.5M
2.0M
0.0M
0.5M
1.0M
1.5M
2.0M
Timesteps
Timesteps
Timesteps
Figure 11: Learning curves median test win rate for QPLEX, QMIX, and Large QMIX with online data collection.
21

---

Published as a conference paper at ICLR 2021
F A VISUALIZATION OF THE STRATEGIES LEARNED IN 5s10Z
(a) Strategy of QPLEX 5s10z
Strategy of QMIX 5s10z
Figure 12: Visualized strategies of QPLEX and QMIX on 5s10z map of StarCraft II benchmark. Red marks represent learning agents, and blue marks represent build-in Al agents.
As shown in Figure 12 both MARL agents and opponents contain ranged soldiers (denoted by circle) and 10 melee soldiers (denoted by line) on 5s map. The ranged soldiers have stronger combat capabilities and need to be protected strategically. QPLEX uses 10 melee soldiers to build lines of defense against the enemy, while QMIX fails coordinate melee soldiers such that ranged soldiers have to fight against the enemy directly.
G EXPERIMENTS ON STARCRAFT II WITH OFFLINE DATA COLLECTION
QPLEX
QTRAN
QMIX
VDN
Qatten
3s_vs_ vs_vs_52
1c3s52
2s _s_vs_ 1sc
Median Test Win Rate
100
Median Test Win Rate
100
Median Test Win Rate
100
0
100
200
300
100
200
300
100
200
300
Epoches
Epoches
Epoches
Median Test Win Rate %
2s3z
3s5z
Median Test Win Rate %
2c ____s___4gggg
100
100
100
Median Test Win Rate
80
60-
60-
60
100
200
300
100
200
30D
100
200
300
Epoches
Epoches
Epoches
Figure 13: Deferred learning curves of StarCraft with offline data collection on tested scenarios.
22

---

Published as a conference paper at ICLR 2021
H ABLATION STUDIES ABOUT QPLEX WITH DIFFERENT NETWORK CAPACITIES IN STARCRAFT II
We trade off the expressiveness and learning efficiency of the multi-head attention network module for estimating importance weights X (see Eq. (9) and (TO) in Section
is generally sufficient for QPLEX use simple multi-head attention with small number of heads and layers (as evaluated in StarCraft II tasks) to achieve the state-of-the-an performance shown in Figure 4 However, in some didactic comer-cases with an adequate and uniform dataset, harder matrix game illustrated in Figure 2a requires very high precision in estimating the action-value function in order to differentiate the optimal solution from the sub-optimal solutions. For this matrix game, a multi-head attention structure with more layers and heads has more representational capacity and results in better performance, as demonstrated by the ablation study illustrated in Figure 2c in Section 4.1
In contrast, StarCraft II micromanagement benchmark tasks contain much more complicated agents with large state-action spaces and range from 2 to 27 agents. To support the superior training scalability of QPLEX, we used multi-head attention with just one layer and four heads. To evaluate the effect of the choice of layer and the number of heads, we conducted
EverggedMMdiingg Tem Win Rate
100
QPI QPLEX-ILLES
QPLEX-2L48
4.0
ablation study in
Starcraft II benchmark tasks. For simplicity, we follow the notation of Figure 2 i.c., use QPLEX-aLóH note QPLEX with a layers and b heads of importance weights A₁ As shown in Figure 14, using more heads, ILLEX-ILIOH, does not change the performance, but using more layers, QPLEX- may slightly degen- erate the learning efficiency of QPLEX in this complex domain. Detailed learning curves on these tasks are shown in Figure [5] This is because using more layers significantly increase the number of parame- ters and requires more samples for learning, which may offset the benefits of higher expressiveness brings. This ablation study shows that simple attention neural network with just layer and four heads has enough expressiveness to handle these complex StarCraft II tasks.
2.0
de-
0.0M
0.4M
0.3M
1.2M
1.6M
20M
Timestops
Figure 14: Ablation study about QPLEX with different network capacities Star- Craft II.
23

---

Published as a conference paper at ICLR 2021
QPLEX 1L4H
QPLEX-1L1OH
QPLEX-2L4H
Median Test Win Rate %
25_vS vs 1sc _sc
2s3z
Median Test Win Rate %
3s5z
100
Median Test Win Rate
100
100
75-
75
0.0M
0.5M
1.0M
1.5M
2.0M
0.0M
0.5M
1.0M
1.5M
2.0M
0.0M
0.5M
1.0M
1.5M
2.0M
Timesteps
Timesteps
Timesteps
1c3s5z
10m vs_ 1s_11m
Median Test Win Rate 96
2c_vs 2c_vs_64zg __2z
Median Test Win Rate
100
Median Test Win Rate
100
100
25
25
25
0.0M
0.5M
1.0M
1.5M
2.0M
0.0M
0.5M
1.0M
15M
2.0M
0.0M
0.5M
1.0M
1.5M
2.0M
Timesteps
Timesteps
Timesteps
Median Test Win Rate sf
5m_ 6m
Median Test Win Rate %
3s vs. vs_5z
Median Test Win Rate %
MMM2
100
100
100
75
75
50-
50
50
25-
25-
25-
0.0M
0.5M
1.0M
1. LM
2.0M
0.0M
0.5M
1.0M
1.5M
2.0M
0.0M
0.5M
1.0M
1.5M
2.0M
Timesteps
Timesteps
Timesteps
Median Test Win Rate %
5s10z
Median Test Win Rate %
7sz
Median Test Winn Rate %
1c3s8z_vs vs_.sss_i___ 1c3s92
100
100
100
75
75
75
0.0M
1.0M Timesteps
Timesteps
Timesteps
Figure 15: Deferred figures of median test win rate % for QPLEX-IL4H, QPLEX-ILIIOH and
1PLEX-2144 with online data collection.
24

---

Published as a conference paper at ICLR 2021
1 ABLATION STUDIES ABOUT QTRAN
Both QPLEX and QTRAN aim to provide a richer factorized acton-value function class. The main difference between QPLEX and QTRAN that QPLEX uses duplex dueling architecture to realize the IGM principle (a hard constraint). In contrast, QTRAN uses two penalties as soft constraints to approximate the IGM principle. Moreover, from the perspective of implementation, QTRAN does not have a Transformation module (see Section (32) on the individual Q-functions and cannot utilize a multi-head attention module on the joint Q-function directly (because QTRAN does not take the duplex dueling architecture as QPLEX).
To test whether QPLEX outperforms QTRAN because of these factors, we conducted an ablation study by re- moving the Transformation module and replacing the multi-head attention module with simple one-layer forward model in the QPLEX's dueling architecture, which is denoted QPLEX-wo-trans atten. In addi- tion, we also introduce a variant of QTRAN, which also the Transformation module for individual Q- functions, denoted as QTRAN-w-trans, Our experiments evaluated on the tasks of the StarCraft benchmark mentioned in Section 4.3.I
Averaged Median Test Win Rate
100
QULEX
80
QTTAN o w0
60
60
0.0M
1.2M
1.6M
200
Timestops
Figure 16: Ablation study about QTRAN with online data collection.
Figure T6 illustrates the averaged median test win rate % over all tested scenarios. Detailed learning curves on these tasks are shown in Figure 17 These empirical results show that QPLEX-wo- wo-trans--teen significantly outperforms QTRAN and QTRAN-w-trans, which implies that the outperformance of QPLEX over QTRAN is largely due to its duplex dueling architecture. can also be seen that
OTRAN-w-trans cannot significantly improve the performance of QTRAN, which implies that QTRAN cannot benefit a lot from an extra Transformation module empirically.
As discussed in Section D. Figure 9a can be regarded as ablation study of QPLEX-wo-aties which just removes the multi-head attention module from dueling architecture. That ablation study shows that simple neural network implementation of QPLEX's dueling structure has enough expressiveness to handle these StarCraft II tasks the online data collection setting, even if this structure provide QPLEX with excellent performance in didactic matrix games using an adequate and uniform dataset (see Section 4.1). Thus, compared with -oo-atenn in Figure 9a] Figure 16 shows that Transformation (abbreviated -trans) is useful module for QPLEX empirically, which indicates an another QPLEX's advantage, i.c., QPLEX can equip a Transformation module to improve QPLEX's empirical performance, whereas QTRAN may not by directly using it. Moreover, we would like to emphasize that, unlike the multi-head attention module, the Transformation module is actually a necessary module for QPLEX to realize IGM, as shown in Figure I and by the proof of Proposition 2 Therefore, it is actually fair to evaluate QPLEX with the Transformation module.
25

---

Published as a conference paper at ICLR 2021
QPLEX
QPLEx-wo-trans-atten
QTRAN
QTRAN-w-trans
1sc
2s3z
Medlan Test Win Rate %
3s5z
Median Test Win Rate
100
Median Test Win Rate
100
100
50
25
25
25
0.0M
0.5M
1.0M
1.5M
2.0M
0.0M
0.5M
1.0M
1.5M
2.0M
0.0M
0.5M
1.0M
1.5M
2.0M
Timesteps
Timesteps
Timesteps
Median Test Win Rate %
1c3s5z
Median Test Win Rate %
10m vs 11m
Median Test Win Rate %
2c 2c_vs_ 2__4
100
100
100
75
50
50
50
0.0M
0.5M
1.0M
1.5M
2.0M
0.0M
0.5M
1.0M
15M
2.0M
0.0M
0.5M
1.0M
1.5M
2.0M
Timesteps
Timesteps
Timesteps
Mediann Test Win Rate %
5m_vs_ _S__v__6m 6m
3s_vs vs__2__ 52
Median Test Win Rate %
MMM2
100
Median Test Win Rate
100
100
75-
75
75
0.0M
1.0M
1.5M
2.0M
0.0M
0.5M
1.5M
0.0M
0.5M
1.0M
1.5M
2.0M
Timesteps
Timesteps
Timesteps
5s10z
7sz
Mealan Test Win Rate %
233s8z_vs_1c3s90
Median Test Win Rate
100-
Median Test Win Rate
100
100
25
25
25
0
0.0M
0.5M
1.0M
1.5M
2.0M
0.0M
0.5M
1.0M
1.5M
2.0M
0.0M
0.5M
1.0M
1.5M
2.0M
Timesteps
Timesteps
Timesteps
Figure 17: Figures of median test win rate % for QPLEX, QPLEX's ablation QPLEX-wo-trans-atten QTRAN, and QTRAN- w-trass with online data collection.
26

---

Published as a conference paper at ICLR 2021
J A COMPARISON TO WQMIX IN PREDATOR-PREY
WOMIX (Rashid et al. 2020 is a recent advanced multi-agent Q-learning algorithm. We compare QPLEX with WQMIX toy game, predator-prey, which is introduced by WQMIX and aims to Test coordination between agents in partially-observabe setting.
Median Text Renum
-40
OPLEX
120
GMIX
OW-OMIX
160
CW-QNIX
0.3M
Timesteps
Figure 18: Learning curves of median test return for QPLEX, QTRAN, QMIX, and WQMIX (OW-QMIX and CW-QMIX) in the toy predator-prey task.
Predator-prey a multi-agent coordinated game used by WQMIX (Rashid et al. 2020) with miscoor- dination penalties. In order collect the experience with the positive reward of agents' coordinated actions, extensive exploration can benefit multi-agent Q-learning algorithms to solve this kind of tasks. WQMIX shapes the data distribution with an importance weight to boost efficient learning, which can also be regarded as type of biased exploration. To support QPLEX with effective exploration, we use an c-greedy strategy which also discussed in the paper WQMIX. This strategy's is linearly annealed from to 0.05 over million timesteps, increased from the 50k used in SMAC (Samvelyan) jet al. 2019). shown Figure 18. besides WQMIX and QTRAN, QPLEX can solve this task by using the introduced e-greedy exploration strategy. Moreover, QMIX can also solve this task by using the same e-greedy strategy as QPLEX, but QPLEX enjoys higher sample efficiency.
27


Reviews:
[{"review_id": "mY4_-gTZOY", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a new value-based method in cooperative multi-agent reinforcement learning. This paper proposes a duplex dueling network that can solve both the limited representation power problem existing in QMIX, and the approximation error problem due to neural network separation, existing in QTRAN. The authors show the novelty of the proposed method through theoretical proof and experimental results. This paper is well written, but there are two significant concerns:\n\n1. The authors argue that QTRAN does not perform well in complex domains because it uses soft constraints, but QPLEX needs to demonstrate this more precisely. QPLEX uses two additional neural network technologies, and It seems the performance gains through these technologies are significant. First, QTRAN uses summation of individual action-values, while QPLEX uses summation of transformed action-values, and QPLEX constructs an advantage function using a scalable multi-head attention module. The authors need to show that performance is better than QTRAN even when these factors are removed. Also, in figure 2(b), QTRAN learns the global optimal policy quickly, but QPLEX learns slowly, so it seems necessary to explain. On the graph, in a matrix game, QTRAN learns more than five times faster than QPLEX.\n\n2. An additional explanation of the authors' environmental selection criteria is needed. In the online data collection setting, the authors use environments such as 5s10z and 7sz, which seem to be environments that were not well used in existing MARL papers, and most environments only use stalker and zealot as agents. Also, the online data collection experiment and the offline data collection experiment use different environments. If the authors use a unified environment, the experiment results will be more enjoyable.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "JSl5fFMBE-t", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel value decomposition approach to learn decentralized Q function in multi-agent setting. This idea is to follow Individual-Global-Max (IGM) principle. The main contribution is to use dueling structure (Q_i = V_i+A_i) for each agent i, and separately combining advantage/value terms respectively to form a centralized Q and A terms for training. Such combinations keep the positive correlation constraint between Q_tot and individual agent Q_i in QMix (i.e,. \\partial Q_{tot} / \\partial Q_i > 0) via positive trainable module in neural network, implemented by multi-head attention, etc. \n\nWhile this paper seems to be a reasonable contribution, a bunch of critical issues exist in the experiments so I vote for rejection. \n\n1. The idea is quite similar/incremental to QMIX, but instead of a mixing network, the authors use two-stages (first a linear weight that depends on all states, then a linear mixture over all agents with attention) to enforce the positive correlation how an individual Q would affect Q_tot. I wonder what’s the key component that makes QPLEX much better than QMIX? Is that purely because you have more parameters in the mixing? There don’t seem to be ablation studies (the video has sub-captions called “ablation study” but no sentence on what’s going on in the two videos) and discussions. \n\n2. Qatten uses attention while the remaining baselines (e.g., QMix, VDN, QTRAN) do not have attention. Is QPLEX working because of attention rather than the dueling structure? How is QPLEX compared with a version of itself without attention?\n\n3. In StarCraft Multi-agent Challenge, 5s10z is neither a hard or a super hard map. The hard maps are 2c_vs_64zg, bane_vs_bane, 5m_vs_6m and 3s_vs_5z, and the super hard maps are MMM2, corridor, 3s5z_vs_3s6z, 6h_vs_8z and 27m_vs_30m (See Tbl. 1, Fig. 5 and Fig. 6 in SMAC paper). I wonder why Fig. 4 and Fig. 5 are tested on different set of maps and how does QPLEX perform in hard/super hard maps (e.g., no comparison for online setting in 2c_vs_64zg, and for offline setting, QPLEX doesn’t see to perform well on 2c_vs_64zg). Note that the Qatten paper shows performance on these hard/super hard tasks. \n\nMinor: \n1. What does the strikethrough mean in Fig. 2a (pay off matrix)?", "extra_scores": {}, "extra_text": {}}, {"review_id": "HcGXbCc-JV", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n \nSummary:\n \nThe paper proposed a multi-agent Q Learning algorithm with an entire IGM function class for cooperative games. The key idea is to leverage a duplex dueling network architecture to factorize the joint action-value function into individual action-value functions. The main contributions of the work lie in that the proposed method offered an highly scalable algorithms for cooperative tasks. Empirical results show that the method could achieve significant improvement in StarCraftII tasks.\n##########################################################################\n\nReasons for score: \n \nOverall, I'd vote for acceptance to the paper. I like the idea that is using a Transformer to learn to decouple the relationship between different agents. However, the main concerns of mine lie in the details of the proposed model(see cons below).\n##########################################################################\nPros: \n \nOverall, the paper is well written. The motivation of the proposed method is straightforward and sound. This paper provides comprehensive experiments, including didactic problems and complex benchmarks, to demonstrate the efficacy and efficiency of the proposed methods.\nThe scalability of previous methods is one of the most critical issues of a cooperative multi-agent setting. The proposed methods give a practical solution for that.\nThe credit assignment problem is another exciting problem of a multi-agent system.\n \n \n \n##########################################################################\nCons: \n Although the main motivation of the proposed method is to address the scalability of previous methods, some details are unclear to me\nWhy does \\lambda take joint-action and joint-observation as a parameter in eq 10. \nWhat is the computation cost to calculate of \\lambda for every agent?\nAlthough the proposed method conducts a few experiments, I still suggest the authors conduct the following ablation studies to enhance the quality of the paper: \nWhy do the choice of layer and the number of heads are not aligned on matrix and StarCraft II tasks?\nThe ablation studies on Qatten\n \n \n##########################################################################\n\nQuestions during the rebuttal period: \n \nPlease address and clarify the cons above \n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "5vgDLbSkdS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach towards multi-agent reinforcement learning (MARL) in the context of centralized training with decentralized execution (CTDE). This paper considers the Individual Global Max (IGM) principle that involves consistency between the local action of every agent and joint action of all agents. \n\nThe paper presents an approach that enforces IGM through an appropriate selection of a neural network architecture that transforms the IGM principle into constraints on the output range of an appropriate neural network. These constraints enable enforcing consistency that is necessary for enforcing the IGM principle.\n\nOverall, I find the paper to be well written, and is well motivated. I went through the theoretical claims but haven’t been able to completely verify the claims of Proposition 2. In terms of experimental results, the paper presents results on a variant of a matrix game considered by prior work, two state MMDP and in both online and offline settings of a micro-management benchmark of StarCraft. \n\nWhile I believe the paper presents good contributions to MARL, I am not an expert in this area, so, I will mark my review with low confidence since I am unable to offer constructive pointers on possible improvements etc.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The majority of reviewers recommend acceptance for this paper, and the average score is in the acceptance rate. Only one reviewer (reviewer 2) recommend rejection, and from the reading the review, the authors answer, and the paper, I think it is possible that the reviewer missed the motivation behind this architecture, which is partly reason for rejection. Unfortunately the reviewer did not answer the authors so I cannot be sure if the reviewer is aware of that. Therefore, I am confident in my recommendation to follow the majority of the reviewers.

The reviewers generally believe this paper is well written. The paper has a good structure and although quite technical, is still easy to follow.

Concerns were raised about the scale of the experiments and motivations for some experimental choices. 
I appreciate that the authors have extended the set of evaluations on the Starcraft task, and added ablations studies, which partly address some of these concerns.

Decision: Accept (Poster)