QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning
Authors: Kyunghwan Son
Sungsoo Ahn
Roben D. Delos Reyes
Jinwoo Shin
Yung Yi
Venue:  (2021)
Keywords: multi-agent reinforcement learning

Abstract: QTRAN is a multi-agent reinforcement learning (MARL) algorithm capable of learning the largest class of joint-action value functions up to date. However, despite its strong theoretical guarantee, it has shown poor empirical performance in complex environments, such as Starcraft Multi-Agent Challenge (SMAC). In this paper, we identify the performance bottleneck of QTRAN and propose a substantially improved version, coined QTRAN++. Our gains come from (i) stabilizing the training objective of QTRAN, (ii) removing the strict role separation between the action-value estimators of QTRAN, and (iii) introducing a multi-head mixing network for value transformation. Through extensive evaluation, we confirm that our diagnosis is correct, and QTRAN++ successfully bridges the gap between empirical performance and theoretical guarantee. In particular, QTRAN++ newly achieves state-of-the-art performance in the SMAC environment. The code will be released.

Under review as a conference paper at ICLR 2021
QTRAN++: IMPROVED VALUE TRANSFORMATION FOR COOPERATIVE MULTI-AGENT REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
QTRAN is a multi-agent reinforcement learning (MARL) algorithm capable of learning the largest class of joint-action value functions up to date. However, despite its strong theoretical guarantee, has shown poor empirical performance in complex environments, such as Starcraft Multi-Agent Challenge (SMAC). In this paper, we identify the performance bottleneck of QTRAN and propose a substantially improved version, coined QTRAN++. Our gains come from (i) stabilizing the training objective of QTRAN, (ii) removing the strict role separation between the action-value estimators of QTRAN, and (iii) introducing multi- head mixing network for value transformation. Through extensive evaluation, we confirm that our diagnosis correct, and QTRAN++ successfully bridges the gap between empirical performance and theoretical guarantee. In particular, QTRAN+ newly achieves state-of-the of-the-art performance in the SMAC environment. The code will be released.
INTRODUCTION
Over the decade, reinforcement learning (RL) has shown successful results for single-agent tasks (Mnih et al., 2015; Lillicrap et al., 2015). However, the progress in multi-agent reinforcement learning (MARL) has been relatively slow despite its importance in many applications, e.g., controlling robot swarms (Yogeswaran et al., 2013) and autonomous driving (Shalev-Shward) et al., 201 6). Indeed, naîvely applying single-agent algorithms demonstrated underwhelming results (Tan, 1993; Lauer and Riedmiller, 2000; Tampuu et al., 2017). The main challenge is handling the non non-stationarity of the policies: a small perturbation in one agent's policy leads to large deviations of another agent's policy.
Centralized training with decentralized execution (CTDE) is popular paradigm for tackling this issue. Under the paradigm, value-based methods (i) train a central action- value estimator with access to full information of the environment, (ii) decompose the estimator into agent- wise utility functions, and (iii) set the decentralized policy of agents to maximize the corresponding utility functions. Their key idea is design the action-value estimator et al., 2019), i.e., individual policies maximizing the estimate of central action-valus For example, celue-decompositionn networks (VDN, Sunehag et al. 2018) decomposes the action-value estimator into a summation of utility functions, and QMIX (Rashid et al., 2018) use a monotonic function of utility functions for the decomposition instead. Here, the common challenge addressed by the prior works is how to design the action-value estimator as flexible as possible, while maintaining the execution constraint on decentralizaaiility
decentralizable function (Son
Recently, Son et al. (2019) proposed QTRAN to eliminate the restriction of value-based CTDE methods for the action-value estimator being decentralizable, To be specific, the authors introduced true action-value estimator and a transformed action-value estimator with inequality constraints imposed between them. They provide theoretical analysis on how the inequality constraints allow QTRAN to represent larger class of estimators than the existing value-based CTDE methods. However, despite its promise, other recent studies have found that QTRAN performs empirically worse than QMIX in complex MARL environments (Mahajan et al., 2019; Samvelyan al., 2019; Rashid et al., 2020a). Namely, a gap between the theoretical analysis and the empirical observation is evident, where we motivated to fill this gap.

---

Under review as a conference paper at ICLR 2021
Contribution. In this paper, we propose QTRAN++, novel value-based MARL algorithm that resolves the limitations of QTRAN, i... filling the gap between theoretical guarantee and empirical performance. Our algorithm maintains the theoretical benefit of QTRAN for representing the largest class of joint action-value estimators, while achieving state-of-the -art---rrtt--tttt-- performance in the popular complex MARL envirommet StarCraft Multi-Agent Challenge (SMAC, Samvelyan al. 2019).
At a high-level, the proposed QTRAN+ improves over QTRAN based on the following critical modifications: (a) enriching the training signals through the change of loss functions, (b) allowing shared roles between the true and the transformed joint action-value functions, and (c) introducing multi-head mixing networks for joint aation-value estimation. To be specific, we achieve (a) through enforcing additional inequality constraints between the transformed action-value estimator and using a non-fixed true action-valus estimator. Furthermore, (b) helps to maintain high expressive power for the transformed action-value estimator with representation transferred from the true action- value estimator. Finally, (c) allows an unbiased credit assignment in tasks that are fundamentally non-decentralizzble.
We extensively evaluate our QTRAN++ in the SMAC environment, where we compare with 5 MARL baselines under 10 different scenarios. We additionally consider a new rewarding mechanism which promotes "selfish" behavior of agents by penalizing agents based on their self-interest. To highlight, QTRAN+ consistently achieves state-of-ther
performance in all the considered experiments. Even in the newly considered settings, our QTRAN+ successfully trains the agents to cooperate with each other to achieve high performance, while the existing MARL algorithms may fall into local optima of learning selfish behaviors, e.g., QMIX trains the agents to run away from enemies to preserve their health conditions. We also construct additional ablation studies, which reveal how the algorithmic components of QTRAN+ are complementary to each other and crucial for achieving state-of-te the-art art performance. We believe that QTRAN++ method can be a strong baseline when other researchers pursue the MARL tasks in the future.
2
PRELIMINARIES
2.1 PROBLEM STATEMENT
In this paper, we consider decentralized partially observable Markov decision process (Oliehoek et al., 2016) represented by a tuple g (S, P,r,O. N, N. 7). To be specific, we let s E S denote the true state of the environment. At each time step, agent E N {1, NN. selects action u; E 21 as an element of the joint action vector u. == [un," ... u. It then goes through a stochastic transition dynamic described by the probability P(s'I u.) All agents share the same reward r(s. u) discounted by a factor of y. Each agent is associated with a partial observation (ss, i) and an action-observation history тj Concatenation of the agent-wise action-observation histories denoted as the overall action-observation history т.
We consider value-based policies under the paradigm of centralized training with decentralized execution (CTDE). To this end, we train joint action-value estimator Qje(s, T, u) with access to the overall action-observation history T and the underlying state s. Each agent follows the policy of maximizing an agent-wis agse utility function (T,, u:) which can be executed in parallel without access to the state s. Finally, we denote the joint action-vai value estimator Q₁= to be decentralized into agent-wise utility functions QI..... QN when the following condition is satisfied:
arg max 2er T, u) = [[ arg max @ (TI, u1) uI)...., arg max qN (TN, un)].
(I)
41
uN
Namely, the joint action-valle estimator is decentralizable when there exist agent- wise policies maximizing the estimated action-value.
2.2 RELATED WORK
QMIX. Rashid et al. (2018) showed how decentralization in Equation I is achieved when the joint action-value estimator is restricted as a non-decreasing monotonic function of agent-wise utility functions. Based on this result, they proposed to parameterize the joint action-value estimator mixing network faix for utility functions with parameter 0:
Qee(s, T, u) fai (gi (7i, (7,, u1),...,49 (TN, uN); 0(s))
2

---

Under review as a conference paper at ICLR 2021
Hypernetworks
Weights 0
State s
Seml-Monatonic
History Tı
Mixing Network
Action 14
True action-value Qj
Utility
Concat
Multi Head
Num. agents N
Utility 9i
Mixing Network
Transformed action-value Q
Figure 1: Architecture of QTRAN+
To be specific, QMIX express the mixing network faix as a fully-connecter network with non-neeee non-negative weight 0(s) 2 O. The non-negative weight is state-dependent since it is obtained as output of hypernetworks (Ha et al., 2017) with the underlying state as input.
QTRAN. In contrast to QMIX, QTRAN (Son et al., 2019) does not restrict the joint action-value 2jt to a certain class of functions. Instead, QTRAN newly introduces a decentralizable transformed action-value estimator Qtran expressed linear summation over the utility functions and an additional value estimator:
Qtran (T, u)= = E u/) + V(T),
iEN
where the value estimator (T) act as a bias on the action-value estimate independent from actions. Then for each sample (T, u) collected from replay buffer, QTRAN trains the transformed action- value estimator to approximate the behavior "true" action-value estimator Qje using two loss functions Lape and Lnopt:
Lopt (T) = (Qj=(T, ū) - Qtran (T. u))
Lnopt (T, u) = (Qoup(T (T, u) Qraan (T, u1)
(3)
Qcip((T (T, u) = max{(2e(TT u), Qeran u²);
where iz is the "optimal" action maximizing the utility functions ge(Ti, uz) for E M. Note that QTRAN does not update the true action-value estimator Qj= using Logt and L.
Other CTDE algorithms. Several other value- based methods have been proposed under the CTDE paradigm. Value decomposition network (VDN, Sunehag et al. 2018) is a predecessor of QMIX which factorize the action-value estimator into a linear summation over the utility functions. Due to this structural constraint, VDN often suffers from low expressive power.
Most recently, QPLEX (Wang et al., 2020a) and Weighted QMIX (Rashid et al., 2020b) appeared as new framework for removing the restriction of the joint action-value estimator Q₁² a certain class of functions. To this end, QPLEX decomposes Q5e into a sum of a value function and non-posionn non-positive advantage function. The main role of non-positive advantage function is to enforce the decentralizability constraint of QTRAN. Next, Weighted QMIX use weighted projection from the true action-vale valuee estimator 2₁² to the transformed action- value estimator Qtran that allows more emphasis to be placed on better joint actions. We provide more detailed description of other CTDE algorithms in Appendix B.
3 QTRAN++: IMPROVING TRAINING AND ARCHITECTURE OF QTRAN
In this section, we introduce our framework, coined QTRAN++, to decentralize the joint action-value estimator. Similar to the original QTRAN (Son et al., 2019), our approach is based on two types of action-value estimators with different roles. First, we utilize true action-value estimator, which estimates the true action-value using standard temporal difference learning without any compromise for decentralization. On the other hand, we keep transformed action-value estimator for each agent that is decentralizable by construction; they aim at projecting the true action-valle estimator into a space of functions that can be decentralized into utility functions.
3

---

Under review as a conference paper at ICLR 2021
Crran
Qtran
opt
ot
nopt
op
Qj
Qa
(a) QTRAN
(b) QTRAN+
Figure 2: Learning process of QTRAN and QTRAN++. When Qtean is in overall larger than 2 QTRAN decreases Qtran only for the optimal action ú. On the other hand, our method efficiently learns the transformed action-value estimator through additional constraint for non-optimal actions.
We further demonstrate our main contribution the rest of this section. In Section 3.1, we propose modified training objective of QTRAN, which trains the transformed action-value estimator with enhanced performance. Next, Section 3.2, we introduce new architecture for the action-value estimators, modified from QTRAN; estimator and multi-head monotonic mixing networks for the transformed oetion-value estimators.
semi-monotonic mixing network for the true action-value
3.1 EFFICIENT TRAINING OF ACTION-VALUS ESTIMATORS
The training objective of QTRAN+ is twofold. First, QTRAN+ trains the true action-value estimator for approximating the true action-value with standard temporal difference learning. Next, the transformed action-value estimators attempt to imitate the behavior of the true action-value estimator. Notably, the utility functions decentralize the transformed action-value estimator by construction, and they can "approximately" decentralize the true action-value estimator when the second objective is sufficiently optimized.
First, we use the loss function Lia for updating the true action-value estimator 23 with standard temporal difference learning, defined as follows:
Lea = =QQje(s, T, - (r+ targett (8', "'))))
where (, u) and (8',T','') decision process. Next, ūi' = [u',...., ]
objects collected from consecutive time steps in the Markov the set of actions maximizing the utility functions, i.e, ut = arg max,, (T u() for E N. Finally, 2 is the target network, which periodically updates parameters from 2j=-
Next, we demonstrate the training objectives for promoting similar behavior of the true and the transformed action-value estimators. To this end, for each sample (s, T, u) from the replay buffer, we train one head of the transformed action-value estimator, denoted by Q using two loss functions fooe and Loopt expressed as follows:
(s T) = (Qje(s, T, T. (eran (5, x))²
Cnopt (s, u) (8) u. - Ceran(s, (,, T, u)
if (f, 2 (se(S, ū),
(Qciip u. Qtran (s, u)) if fe (s.
Qs r, u) clipp(tram(s. u), (,, u), (J,(s,T,ū)),
where is "optimal" action maximizing the utility functions qi for E N and the function clip(., &,, (2) bounds its input to be within the interval [[,,4]] 42).
Similar to QTRAN, our QTRAN+ succeeds in decentralizing the action-valus estimator if and only if the training objective is sufficiently minimized. To this end, the two loss functions Copt and Lnopt enforce the following condition:
(s Quran (s, T, u) Quran (s, T. u) Qje(S,T.u)
(6)
Note that the utility functions achieve decentralization of the true action-value estimator by definition when (a) > (d). The first loss Lept enforces (a) = (b) and the second loss Lnope enforces (a) > (d),
(a) > (c), and (c) (d). See Appendix A for formal result on this theoretical guarantee.
4

---

Under review as a conference paper at ICLR 2021
Table 1: Payoff matrix of the one-step non-decentralizable game and reconstructed results of the game. States $1 and are randomly selected with same probability and the agents cannot observe the state. Boldface means optimal and greedy actions from the state-action value function.
A
B
B
B
A
A
2
A
A
3.0
3.0
0.5
1.5
B
0
B
B
1.0
1.0
0.5
1.5
(a) Payoff of matrix game for state $1 and 82
(b) QMIX: (11(s)), 11.(s2)
A
B
A
B
A
B
A
B
A
3.0
2.0
A
1.0
A
3.0
2.5
1.0
B
2.5
1.5
B
0.5
0.5
B
2.0
1.5
B
1.0
0.5
(c) QTRAN+ head Queaa (81), (1ra((2)
(d) QTRAN+ head 2: Quean ($1), Q ($2)
Comparing with the training objective of Qeran QTRAN is similar to our algorithm in a way that it trains the true and the transformed action-value estimators to satisfy Equation 4. However, QTRAN only enforce (a) = (b) and (c) > (d), while our algorithm enforce two additional constraints of (a) (d) and (a) (c). As a result, our algorithm trains the estimators with denser training signal and results in a more efficient training. See Figure 2 for an illustration of such an intuition.
Non-fixed true action-value estimator. Notably, we train both the true action-value estimator Q5= and the transformed valuo estimator Qeran to imitate each other, using loss functions Lops and Zoppt. This is in contrast with QTRAN, which only trains Qtran imitate Q This allows the true action-valus value estimator Qj to additionally minimize the decentralization error, i.e., the gap between maximizing 2 and maximizing agent-wise utility functions qs Such a modification allows the agents to learn better policy for maximizing the true action-values.
3.2 MIXING NETWORK ARCHITECTURES FOR ACTION- VALUE ESTIMATORS
Here, we introduce our choice of architectures for the true and transformed action-value estimators. At a high-level, we construct the estimators using the utility functions 41, ....... IN parameterized as deep recurrent Q-network (DRQN, Hausknecht and Stone 2015). Our main contribution in designing the estimators is twofold: (i) using semi- monitonno mixing network for the true action-value estimator 25t and (ii) using a multi-head monotonic mixing network for the transformed action-value
estimators Q49 for E N. The role of (i) is additionally impose a bias on the utility functions to decentralize the true action-value estimator Qjt. Next, we introduce (ii) to prevent the utility functions from overly relying on the underlying state information that is unavailable during execution.
Semi-monotonic mixing network for Qjt- We express the true action-valus value estimator as mixing network applied decentralized utility functions .......99. IN as follows:
(s t, u.) fmix ....--... IN; /12 (s)) fmix (q..... -......-ee qN; 0g (s)),
where we omit the input dependency of utility functions for convenience of notation, e.g., we let denote (T, u;) for agent i. Furthermore, we express the mixing networks as fully connected networks where the parameters 035" (s), [ are obtained from state-dependent hypernetworkss.
Importantly, we constrain the second parameter to be non non-negative, i.c., 0/2) (s) 2 0. Similar to
QMIX, this guarantees the second network to be monotonic with respect to the utility functions.
Such design choice yields a monotonic action-value functions while implicitly preferring to behave similarly as the monotonic function. Such a semi-monotonic function is easier for (monotonic) transformed action-value to estimate, leading to smaller decentralization error. As a result, the individual agents (decentralizing Qtraa) will effectively maximize the true action- value function.
true action-value estimator which can estimate non-
Multi-head mixing network for Qtran- Now we introduce the multi-head mixing network used for the transformed action-valus estimator. To this end, we express the i-th head of the transformed action-value estimator
( follows:
Q tran (8, T, u.) = + Fmix(q1, (qi,..., 1:-1; qi+1, IN: B (8)) ViEW,
5

---

Under review as a conference paper at ICLR 2021
where Qurm denotes the i-th head of the transformed action-value estimator and we do not denote the input of the utility functions for the convenience of notation, e.g., we used y to denote (i(Ti, us). Furthermore, each of (s) is non-negative parameters obtained from state-dependent hypernetworks.
Following QMIX, we design the multi-head mixing networks to be monotonic and decentralizable using the utility functions. However, unlike QMIX, i.e., Equation 2, we separately design the i-th estimator Q (9) for each agent. This allows the utility function qi to rely less on the underlying state information s, which is not accessible during the execution of the algorithms.
We further elaborate our intuition in Table 1. In the table, we compare QMIX and QTRAN++ for a matrix game where states 81 and 82 are randomly selected with the same probability. This matrix game has optimal actions (A, A) and (B, B3) at state 81 and state 82, respectively. If the agents cannot observe the state in the execution phase, the task is non-decentralizable, and the best policy leads to choosing an action (A, A). As shown in Table 1b, QMIX sometimes learns a non-optimal policy, which is caused by the different state-depent dependennn output weights of mixing networks. In contrast, QTRAN+ in Table Ic and 1d learns an individual utility function for the average reward of states. A more detailed discussion is available Appendix
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We mainly evaluate our method on the Starcraft Multi-Agent Challenge (SMAC) environment (Samvelyan et al., 2019). In this environment, each agent a unit participating combat against enemy units controlled by handcrafted policies. In the environment, agents receive individual local observation containing distance, relative location, health, shield, and unit type of other allied and enemy units within their sight range. The SMAC environment additionally assumes a global state to be available during the training of the agents. To be specific, the global state contains information of all agents participating in the scenario. Appendix D contains additional experimental details. We also provide additional experiments on the multi-domain Gaussian squeeze environment (Son et al., 2019) in Appendix F.
We compare our method with six state-of-the-arr baselines: QMIX (Rashid et al., 2018), QTRAN (Son al., 2019). VDN (Sunchag et al., 2018), QPLEX (Wang et al., 2020a), OW-QMIX (Rashid et al., 2020b), and CW-QMIX (Rashid et al., 2020b). We consider SMAC maps with five different scenarios, including scenarios with difficulty levels of easy, hard, and super hard. All the algorithms are trained using two million steps.' For evaluation, we run 32 test episodes without an exploration factor for every 10 -th time step. The percentage of episodes where the agents defeat all enemy units, i.e., test win rate, is reported as the performance of algorithms. All the results are averaged over five independent runs. We report the median performance with shaded 25-75% confidence intervals.
Rewarding mechanisms. We consider two types of scenarios with distinct mechanisms of reward functions. In the first case, the reward is determined only by the enemy unit's health condition, i.c., units receive reward only when the enemy unit is damaged. We coin this case as the standard scenario to reflect how this setting is widely used by existing works (Rashid et al., 2018; 2020b). Next, we consider the case where each unit is additionally penalized for being damaged, i.e., units are rewarded for having a "selfish" behavior. This case is denoted as the negative scenario since we introduced a negative reward for the penalty.
Ablation studies. We also conduct additional ablation studies to validate the effectiveness of each component introduced in our method. Namely, we consider to verify the effect of (a) multi-head mixing network, (b) semi- monoooic mixing network, (c) loss function modified from QTRAN, and (d) using non-fixed true action-value estimator during training. To consider (a), we replace the multi- head mixing network with single-head mixing network, which we call MiX-QTRAN++ we compare against FC-QTRAN++, which replaces the mixing network used in the true action-vall value estimator with feed-forward network takes the state and the appropriate actions' utilities as input. Next, we analyze (c) through comparing with LB- AB-QTRAN++ which trains the proposed action-value
For (b),
We also report experimental results using more training steps in Appendix E.

---

Under review as a conference paper at ICLR 2021
QTRAN++
QMIX
QTRAN
VON
OW-CMIX
CW-CMIX
OPLEX
1.0
1.0
1.0
0.8
Vin Rete
Rate
0.4
0.4
Win
0.4
0.2
0.2
0.2
5.0x10
1.0x10
1.5x0
2.0x10
5.0x10
1.0x10f 1.5x10" 2.0ki0
0
5.0x10
1.0x10"
1.5x10 20x10
Training Step
Traising Step
Training Step
MMM2
(b) 10m vs vs_
(c) 3s5z
1.0
0.8
0.0
0.8
Win Win
0.8
0.8
Wille
0.6
5.010PP
1.0000
1.5x100
5.0x10P
1.0x10f 1.5x10P
5.0x10
1.0000p
1.5x00 2.0x10
Training Step
Traiinn Step
Training Step
(d) MMM2 (negative)
(e) 10m_0mp vs 11m (negative)
(f) 3s5z (negative)
1,0
Win Rale
0.8
Win Rate
0.4
0.4
0.2
5.0x10f 1.0x10f 1.5x10 20x10
6.0x10
1.0w10 1.5x10 2.0x10 Training Step
Step
(g) ___v____nm_ vm_
3s_vs_52
1.0
1.0
0.8
0.8
Win Pole
0.5
0.4
0.2
0.2
0
5.0x10P
1.0xtdp
1.5x100 2.0xc10
0
5.0x10
1.0x10" 1.5x10" 2.0x.0
Traning Step
Training Step
(i) 5m vs_ 6m (negative)
(i) 3s vs__2___2 (negative)
Figure 3: Median test win rate with 25%-75% percentile, comparing QTRAN+ with baselines. All the results averaged over five independent runs. Each plot corresponds to different scenario names, e.g., 3s5z. Scenarios with additional negative rewarding mechanism are indicated by "(negative)" in the name of scenarios, e.g., 3s5z (negative).
estimators using the loss function of QTRAN. Finally, we validate (d) through comparing with Fix-QTRAN+ which fix the true action-value estimator for optimizing Copt and Lnopt
4.2 RESULTS
Comparisons with baselines. We report the performance of QTRAN+ and the baseline algorithms in Figure 3. Overall, QTRAN+ achieves the highest win rate compared to existing baselines, regardless of the scenario being standard negative. Especially, one observes significant gap between QTRAN+ and the second-best baselines for scenarios such as 10m_vs _a___s_lmmml 11m, 10m_w vs. 11m (negative), and 5m 6m (negative). This is more significant since there is consistent baseline achieving the second place. Furthermore, the original version our algorithm, ie., QTRAN, performs poorly in several tasks, e.g., Figure 3a and Figure 3c. From such results, may conclude
7

---

Under review as a conference paper at ICLR 2021
QTRAN++
Mix-QTRAN++
FC-QTRAN++
LB-OTRAN++
Fix-QTRAN++
1.0
0.8
0.8
0.8
Win Rete
0.8
0.8
0.6
0.4
0.2
Oxx10
1.0xt0
1.5x10
5.0x10
1.0x10f
1.5x10
2.0x10f
5. 50x10
Ox000p
1.5x10 2.0x10
Training Step
Training Step
Training Step
(a) MMM2
(b) 10m_vs_Ilm
(c) 3s5z
1.0
1.0
1.0
V6n Pale
0.6
0.8
0.6
0.4
0.4
Vin
0.4
02
0.2
0.2
Ox010
1.0x10"
1.5x10"
20x10
5,0x10"
1.0x10f 1.5M10
20x10'
5.0x10
1.0x10"
1.5x10"
20x10
Training
Training Stop
Training Step
(d) MMM2 (negative)
(e) 10m 1 11m (negative)
(i) 3s52 (negative)
1.0
1.0
0.9
0.8
Win Rate
0.6
Wi Rate
0.2
0.2
0
5.0x10
1.0x10f 1.5x10 20x10
5.0x10 1.0x10" 1.5x10 2.0x10f
Traning Step
Training Step
(g) 5m_vs_____ _om_vs_6m
(h) 3s_ se_vs_50 vs_5z
0.8
Win Rale
0.8
0.6
0.4
Win
0.4
0-
5.0k10" 1.0x1d' 1.5x10" 20x10" Traning Step
5.0x10
1.0x10" 1.5x10" 2.0xig
Training Stop
5m vs. 6m (negative)
(i) 3s vs 5z (negative)
Figure 4: Median test win rate with 25%-75% percentile, comparing QTRAN++ with its modifica- tions. All the results are averaged over five independent runs. Each plot corresponds to different scenario names, e.g., 3s5z. Scenarios with additional negative rewarding mechanism are indicated by "(negative)) in the name of scenarios, e.g., 3s5z (negative).
that QTRAN+ successfully resolves the issue existing the original QTRAN and achieves state-of- the-art performance compared to baselines for the SMAC environment.
Observations. Intriguingly, Figure 3, one observes an interesting result that negative rewards can improve the test win rate of MARI. algorithms. For example, when comparing Figure 3d and Figure 3i, one may observe how the presence of negative rewards speed up the learning process of algorithms. We conjecture modified environments provide denser reward signal, enabling the agent to learn policies efficiently.
We also illustrate how QTRAN+ successfully solves the most difficult negative scenarios in Figure 5. In such scenarios, as demonstrated in Figure 5b and Figure 5d, existing algorithm such as QMIX may end up learning "locally optimal" policies where the agents avoid receiving negative rewards. However, in Figure 5a and Figure 5c, one observes that QTRAN+ successfully escapes the local optima and chooses to fight the enemies to achieve higher win rate.
away from the enemy units to
8

---

Under review as a conference paper at ICLR 2021
@a) MMM2 (QTRAN++)
(b) MMM2 (QMIX)
(c) 5m vs 6m (QTRAN++)
(d) 5m_ vs 6m (QMIX)
Figure Illustration of the policies learned using QTRAN+ and QMIX negative scenarios. Agents trained by QMIX run away from enemies without fighting (b, d). In contrast, QTRAN++ trains the agents fight the enemies, obtaining higher win rate in overall (a, c).
Ablation study. We show the results for our ablation study in Figure 4. We consider three scenarios show how each element of QTRAN+ is effective for improving its performance. Overall, one can observe how QTRAN++ demonstrates a solid improvement over its ablated versions. In particular, QTRAN+ outperforms them by large margin in at least one of the considered scenarios. For example, QTRAN+ significantly outperforms FC-QTRAN++ in Figure 4a and 4h. It also significantly outperforms Mix- MiX-QTRAN++ LB- +T-ATRA+++ Fix- -TRA+TA++++ in Figure 4a and 4g. Such result validates how the algorithmic components in our algorithm are complementary to each other and crucial for consistently achieving high empirical performance.
5 CONCLUSION
In this paper, present QTRAN++, simple yet effective improvement over QTRAN (Son et al., 2019) for cooperative multi-agent reinforcement learning under the paradigm centralized training with decentralized execution. Our gains mainly come from (i) stabilizing the training objective of QTRAN, (ii) removing the strict role separation between the action-value estimators, and (iii) introducing multi-head mixing network for value transformation. Using the Starcraft Multi-Agent Challenge (SMAC) environment, we empirically show how our method improves QTRAN and achieves Mateeoof-the-at results.
REFERENCES
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arxiv preprint arXiv: arXiv:7509.00 7509.02979. 2015.
M. Yogeswaran, S. G. Ponnambalam, and G. Kanagaraj. Reinforcement learning in swarm-robotics for multi-agent foraging-task domain. In Proceeding of IEEE Symposium on Swarm Intelligence (SIS), pages 15-21, 2013.
Shai Shalav-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv: 1610.03295 2016.

---

Under review as a conference paper at ICLR 2021
Ming Tan. Multi-agent reinforcement learning: Independent cooperative agents. In Proceedings of ICML, pages 330-337, 1993.
Martin Lauer and Martin Riedmiller. An algorithm for distributed reinforcement learning in coopera- tive multi-agent systems. In Proceedings ICML, 2000.
Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning. PloS one, 12(4)xo1122355, 2017.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Proceedings of ICML, 2019.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki. Vinícius Flores Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value-oecompositioon networks for cooperative multi-agent learning based on team reward. In Proceedings of AAMAS, pages 2085-2087, 2018.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shi- mon Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. Proceedings of ICML., 2018.
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: variational exploration. In Proceedings of NeurIPS, 2019.
Multi-agent
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung. Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The star- craft multi-agent challenge. In Proceedings of AAMAS. International Foundation for Autonomous Agents and Multiagent Systems, 2019.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint wwXiv:2003.08839 2020a.
Frans A Oliehoek, Christopher Amato, et al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016.
David Ha, Andrew Dai, and Quoe V. Le. Hypernetorrss, In Proceedings of ICLR, 2017.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjic Zhang. Qplex: Duplex dueling
multi-agent q-learning. arXiv preprint arXiv:2008.01062, 2020a.
Tabish Rashid, Gregory Farquhar, Bei Peng. and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation. arXiv preprint arXiv:2006. 10800, 2020b.
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In Proceedings of AAAI Fall Symposium Series, 2015.
Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of AAAI, 2018.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Abbeel Pieter, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competin environments. In Proceedings of NeurlPS, pages 6379,63990, 2017.
Yali Du, Lei Han, Meng Fang, Liu, Tianhong Dai, and Dacheng Tao. Liir: Learning individual intrinsic reward in multi-agent reinforcement learning. In Proceedings of NeurlPS, 2019.
Shariq Iqhal and Fei Sha. Actor-ateention-criiic for multi-agent reinforcement learning. In Proceedings of ICML, 2019.
Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Multi-agent reinforcement learning with emergent roles. arXiv preprint arXiv:2003.080:3 2020b.
10

---

Under review as a conference paper at ICLR 2021
Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.03939, 2020.
C HolmesParker, M Taylor, Y Zhan, and Tumer. Exploiting structure and agent-centric rewards to promote coordination in large multiagent systems. In Proceedings of Adaptive and Learning Agents Workshop, 2014.

---

Under review as a conference paper at ICLR 2021
A PROOFS
Theorem 1. There exists a set of utility functions her decentralizing the true action-value esti-
mator Q (s, T, u) if and only if there exists function 2 (s, u) satisfying the following conditions:
Qtran (s, T. = (,
(5)
Uraa (s T, Qj,(s,T, u)
iQtran 88, T,
ViEW,
(7)
0q Tsy ut)
ū |arg max u, ss)l
(8)
Proof. We prove first the sufficiency of the theorem by showing that if the conditions hold, then i.
li(Ts; u4) satisfies optimal decentralization arg maxu Qj u.
(,= T, = Qtran (8, T; (
(From (5))
Qtran (s, u.)
(From (7) and (8)
2 ( T, u)
(From (6))
It means that the set of local optimal actions ū maximizes Qst, showing that qi satisfies decentraliz- ability.
= We turn now to the necessity. First, we define Qtran T, 2) = =I ag - ū,))= (s, T, Z). which satisfies condition Equation 5 and Equation 7, where constant a; 0. Since the set of utility functions { NEN decentralizes the true action-value estimator Q5t. utility functions and true action-value estimator satisfy the condition (s, t, Q5r(8, t, u)) Q5= (s. T, u) u # u. So proof for Equation 6 follows from the fact that there exists [a.] small enough such that
(8, Qje(s,
Σ ai(a:(Ti, u;) (.(T,, AJ((S,T, 0.
12

---

Under review as a conference paper at ICLR 2021
B RELATED WORK
Centralized training with decentralized execution (CTDE) has emerged as a popular paradigm under the multi-agent reinforcement learning framework. It assumes the complete state information be fully accessible during training, while individual policies allow decentralization during execution. To train agents under the CTDE paradigm, both policy-based (Foerster et al., 2018; Lowe al., 2017: Du et al., 2019; Iqbal and Sha, 2019; Wang et al., 2020b) and value-based methods (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019; Yang et al., 2020) have been proposed. At a high-level, the policy-based methods rely on the actor-critic framework with independent actors to achieve decentralized execution. On the other hand, the value-based methods attempt to learn a joint action-valle estimator, which can be cleverly decomposed into individual agent-wise utility functions.
For examples of the policy-based methods, COMA (Foerster et al., 2018) trains individual policies with joint critic and solves the credit assignment problem by estimating a counterfactual baseline. MADDPG (Lowe et al., 2017) extends the DDPG (Lillicrap et al., 2015) algorithm to learn individual policies in a centralized manner on both cooperative and competitive games. MAAC (Iqbal and Sha, 2019) includes an attention mechanism in crities to improve scalability. LIR (Du al., 2019) introduces a meta-gradient algorithm to learn individual intrinsic rewards to solve the credit assignment problem. Recently, ROMA (Wang et al., 2020b) proposes a role-oriented framework to learn roles via deep RL with regularizers and role-conditioned policies.
Among the value-based methods, value-decomposition networks (VDN, Sunehag et al. 2018) learns centralized, yet factored joint action-value estimator by representing the joint action-value estimator as a sum of individual agent-wise utility functions. QMIX (Rashid et al., 2018) extends VDN by employing mixing network to express a non-linear monotonic relationship among individual agent-wiss, wise utility functions the joint action-value estimator. Qatten (Yang et al., 2020) introduces multi-head attention mechanism for approximating the decomposition of the joint action-value estimator, which is based on theoretical findings.
QTRAN (Son et al., 2019) has been proposed recently to eliminate the monotonic assumption on the joint action-value estimator in QMIX (Rashid et al., 2018). Instead of directly decomposing the joint action-value estimator into utility functions, QTRAN proposes training objective that enforces the decentralization of the joint action-value estimator into the summation of individual utility functions. In the QTRAN paper, new algorithm called QTRAN-alt is also proposed to more accurately distinguish the transformed action-value for optimal actions from non-optimal actions. However, this QTRAN-alt algorithm has high computational complexity because requires counterfactual action-value estimation when other actions are selected. In addition, Mahajan et al. (2019) experimentally shows that QTRAN-att alt does not work as well as QTRAN-baa in StarCraft Multi- Agent Challenge (Samvelyan et al., 2019) environments. In this paper, we only deal with QTRAN-base modifications, and our proposed method still has the advantage of QTRAN-alt without counterfactual action-valus estimation.
Recently, several other methods have been proposed to solve the limitations of QMIX. QPLEX (Wang et al., 2020a) takes a duplex dueling network architecture to factorize the joint value function. Unlike QTRAN, QPLEX learns using only one joint action-value network and a single loss function. Their experiments on the StarCraft Multi-Agent Challenge (SMAC) environment (Samvelyan et al., 2019) also mainly utilized offline data. Finally, Rashid et al. (2020b) proposed CW-QMIX and OW-QMIX which use weighted projection that allows more emphasis to be placed on better joint actions. However, their methods are susceptible to weight parameter changes and the performance improvements have been demonstrated in limited SMAC environments only. Unlike these methods, we propose novel and performance of the original QTRAN.
modifications that substantially improve the stability and
13

---

Under review as a conference paper at ICLR 2021
C ONE-STEP NON-DECENTRALIZABLL MATRIX GAME
Table 2: Payoff matrix of the one-step non-decentralizaale game and reconstructed results of the game. States 81 and $2 randomly selected with same probability and the agents cannot observe the state. Boldface means optimal and greedy actions from the state-action value function.
A
B
A
B
B
A
A
4
2
A
0
A
2,0
1.5
A
2.0
1.5
B
0
B
2
B
1.5
1.0
1.3
1.0
(a) Payoff matrix game state #1
82
(b) VDN: Q1
B
B
B
A
A
4.0
2.0
A
1.0
1.0
A
3,0
3,0
A
0.5
1.5
B
2.0
0.0
B
1.0
1.0
B
1.0
1.0
0.5
1.5
(c) QMIX case (1+($1)
(s)
(d) QMIX
Qj: (1=
),
(52)
A
B
A
B
A
B
A
B
3.0
2.0
1.0
3.0
2.5
1.0
B
2.5
1.5
B
0.5
0.5
B
2.0
1.5
B
1.0
0.5
(c) QTRAN++ head 1: ( ((eaa(s2)
(1) QTRAN+ head 2: ( (12)
This section presents how multi-head mixing networks perform to existing methods such as VDN and QMIX. The matrix game and learning results are shown in Table 2. this matrix game, states 8; and $2 are randomly selected with the same probability. This matrix game has the optimal action (A, A) at $1, and (B, B) at 82. the agents cannot observe the state, the task is non-decentralizable and the best policy that agents average. Now we show the results of VDN, QMIX, and QTRAN++ through a full exploration. First, Table 2b shows that VDN enables each agent to jointly take the best action only by using its own locally optimal action despite not learning the correct action-values. Table 2c also demonstrates that QMIX learns the best policy. However, as shown in Table 2d, QMIX sometimes learns non-optimal policy, which is caused by the state-dependent output weights of mixing networks. VDN always learns individual utility functions for the average reward of states, but QMIX tends to make the zero mixing network weights for one of the two states to reduce TD-error, and which state to ignore depends on the initial values.
do is to choose action (A, A) that can get the highest rewards
Table 2e and 2f show how multi-heas mixing network solves the biased credit assignment problem with the transformed action-value estimator. Each head of our method assigns the unbiased credit for agent allowing it to learn the best policy. Multi-head mixing networks do not perfectly maintain the representation power of the original mixing network. But as the number of agents increases, the representation power between the two methods becomes the same.
14

---

Under review as a conference paper at ICLR 2021
D EXPERIMENTAL DETAILS
The hyperparameters of training and testing configurations for VDN, QMIX, QTRAN are the same as in the recent GitHub code of SMAC (Samvelyan al., 2019) with StarCraft version SC2.4. 10.. The architecture of all agents' policy networks is a DRQN consisting of two 64-dimensional fully connected layers and 64-dimensional GRU. The mixing networks consist of single hidden layer with 32 hidden widths and ELU activation functions. Hypernetworks consist of two layers with 64 hidden widths and ReLU activation functions. For Weighted QMIX (Rashid et al., 2020b), we additionally use feed-forward network with 3 hidden layers of 128 dim and ReLU non-linearities and set a 0.75. Finally, the hyperparameters of QPLEX are the same in their GitHub code.
All neural networks are trained using the RMSProp optimizer with 0.0005 learning rates. We use e-greedy action selection with decreasing e from 0.05 over 500000 time steps for exploration, following Rashid et al. (2020b). For the discount factor, we set 7 0.99. The replay buffer stores 5000 episodes at most, and the maximum size mini-batch 32. Using a Nvidia Titan Xp graphic card, the training time varies from 8 hours to 24 hours for different scenarios.
In our implementation, all heads share the monotonic mixing network, which takes individual utility functions as input. For head i, the mixing network receives the action-independent value (s(s) as an input instead of the utility function (i ut). Gradients are propagated into all utility functions for training each i-th head. This structure makes it possible to distinguish multi-head values with only one mixing network efficiently, and all transformed action-value estimators share the same optimal action. Furthermore,
use state-independent weights for tasks with heterogeneous agents. For the loss function Laopt, we fixed the value only for the threshold 25e(s, т, i of the clipping function that receives optimal actions as input. To combine our loss functions, obtain the following objective, which is minimized an end-to-end manner to train the true action-valle estimator and the transformed action-value estimator:
L = Lia + Aop +
Aaonn Gnopt
where Aopt Anopt > 0 are hyperparameters controlling the importance of each loss function. We set Aopt 2 and Anopt
15

---

Under review as a conference paper at ICLR 2021
E
EXPERIMENTS USING MORE TRAINING STEPS
QTRAN++
QMIX
QTRAN
VON
OW-OMIX
CW-OMIX
QPLEX
1.0
1.0
0.8
0.8
Win Raie
0.6
Win Rale
0.6
0.4
0.4
0.2
0.2
0
166
2x108
3x10
4x106
0
10
2x108
3x10
4x109
Training Step
Training Step
(a) MMM2
(b) MMM2 (negative)
Figure 6: Median test win rate with 25%-75% percentile, comparing QTRAN+ to baselines using more training steps. All the results are averaged over five independent runs.
In this section, we provide additional experimental results on the SMAC environment, where we train algorithms using more training steps than that of Figure 4. Our goal verify whether if our QTRAN++ maintains state-of-the-ane performance even when the algorithms have converged. To this end, we choose the scenarios where QTRAN+ and the baselines have not converged in Figure iee. MMM2 and MMM2 (negative). Next, we evaluate QTRAN+ and the baselines using four million training steps. The corresponding results are reported in Figure 6. Here, one can observe how QTRAN++ still maintains state-of-the-an performance in both scenarios, regardless of the number of training steps used.
F MULTI-DOMAIN GAUSSIAN SQUEEZE EXPERIMENT
Score
Score
Soore
OTRAN OTRAN OMIX VON
OTRAN++ OTRAN GMIX
OTRAN OTRAN OMIX VON
VON
5x105 Training step
108
5x10
108
5x10 Training step
108
Training step
(a) Partial observability 100%
(b) Partial observability 80%
(c) Partial observability 50%
Figure 7: Experimental result on the multi-domain Gaussian squeeze environment. Partial observabil- ity indicates the amount observation provided for each agent.
In this section, we provide the experimental results for the multi-domain Gaussian squeeze (MGS) environment proposed by Son et al. (2019). The MGS environment specially designed for demon- strating the ability of QTRAN to learn non-montto action-value estimators; it modifies the Gaussian squeeze environment (HolmesParker al., 2014) have multiple agent-wise policies that achieve locally optimal performance. We evaluate QTRAN++ and other baselines under MGS environment with ten agents. Furthermore, we vary the amount of observation provided for the agents to evaluate the performance of algorithms to consider more challenging scenarios. Note that the MGS environment originally assumes the agents have full observation on the environment.
In Figure 7, one can observe that QTRAN+ consistently achieves the best result compared to the baselines. Both QTRAN and QTRAN+ demonstrates the highest performance in the fully observable environment. However, the performance of QTRAN dramatically decreases the observation of agent gets more limited. Hence, one may conclude that QTRAN+ achieves state-of-the -ftte-of-the-art performance across different benchmarks and still retains the main strengths of QTRAN.
16

---

Under review as a conference paper at ICLR 2021
G
ADDITIONAL EXPERIMENTS ON THE SEMI-MONOTONIC MIXING NETWORK
Soore
Score
Soore
OTRAN FTRAN
OTRAN+ FC-GTRAN
OTRAN FC-OTRA
5x10 Training step
10
5x10 Training step
10
5x10 Training step
10
(a) Partial observability 100%
(b) Partial observability 80%
(c) Partial observability 50%
Figure 8: Experimental result on the multi-domain Gaussian squeeze environment. Partial observabil- ity indicates the amount observation provided for each agent.
In this section, provide additional ablation studies validate how our semi-monotonic mixing network provides a consistent improvement even on environments where monotonic action-value estimators fail. To this end, we consider comparing QTRAN+ and FC-QTRAN+ the mult- domain Gaussian squeeze (MGS) environment (Son et al., 2019): complex multi-agent resource allocation problem that requires learning a non-monotonic action-value estimator. Especially, Son et al. (2019) reported that algorithms that relies on monotonic action-value estimator, i.e., VDN and QMIX, to perform much worse than algorithms that use non-monotonic action-valus estimator, iee. QTRAN. In Figure 8, one observes that QTRAN+ performs at least as good FC-QTRAN++ verifying that our semi-monotonic mixing network useful regardless of non-monotonic action- value estimator being required or not.
17


Reviews:
[{"review_id": "oJw_iowQub", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides good improvements that make QTRAN more practical and can be applied to problems other than matrix games. Since QTRAN is a significant improvement for value-based multi-agent reinforcement learning after QMIX, the practical implementation of QTRAN is expected for a long time in the community. However, after the publication of QTRAN, especially in 2020, some other works have explored the question of how to extend QMIX to the full IGM function class. Due to these works (QPLEX is the major concern, given that weighted-QMIX has been compared in the experiments), the expectation of the advanced version of QTRAN is higher than before. I have several concerns regarding several core contributions of QTRAN++.\n\nQTRAN++ relies heavily on the true joint action-value function. (1.1) However, learning joint action-value functions is not an adorable choice in multi-agent problems. (1.2) To ease the training and representation of joint action-value functions, the authors condition $Q_{jt}$ on individual q values and use a semi-monotonic structure. However, it is difficult to tell the contribution of the monotonic part. It has been shown that monotonic functions can not represent some Q-values. Why should this part be included? I expect that I can find the answer from ablation studies, but on two out of three scenarios, FC-QTRAN++ is very similar to QTRAN++. The authors can provide a more serious discussion of this part to make their paper stronger.\n\nAbout the training of $Q_{tran}^{i}$. I have two questions about the training of this value function. (2.1) When training $Q_{tran}^{i}$, whether local utility function of agent $j$ ($q_j$ using the notation from the paper) is updated? (2.2) The training scheme is a midpoint between VDN and QMIX, which is similar to an attention mechanism that has been explored in multi-agent value decomposition settings. The formulation is quite different from previous papers (DOP [Wang et al., 2020] and REFIL [Iqbal et al. 2020]), but based on the results from these previous work, I think the multi-head structure may not improve the performance. Although the authors use a matrix game to illustrate their idea, which I appreciate, I can hardly tell whether this example is specially designed. I was expecting a convincing ablation study on SMAC, but I do not find them sufficient: (1) The authors did not record how many random seeds did they test, and SMAC tasks are typically sensitive to random seeds. (2) The gap between QTRAN and QTRAN++ is not significant. If the authors can provide results with more random seeds on more maps, I will consider revising my rating.\n\nMy last concern is about QPLEX, as cited by the authors. Similar to QTRAN, QPLEX provides full expressivity for the IGM function class. Nevertheless, the implementation of QPLEX seems to be much more lightweight than QTRAN++. Since QPLEX has provided codes that can be freely tested on the SMAC benchmark, I was wondering why the authors cited this paper but did not compare to it. At least, a detailed discussion of the differences can make the contribution of QTRAN++ clearer.\n\n** A minor concern about experiments.\nIt seems that the authors are using an older version of QMIX. In the latest version, QMIX can achieve a win rate pf 80% on MMM2. This fact is unknown for many, because the journal version of QMIX reports the same win rate as in this paper.\n\n[Wang et al., 2020] Wang, Y., Han, B., Wang, T., Dong, H. and Zhang, C., 2020. Off-Policy Multi-Agent Decomposed Policy Gradients. arXiv preprint arXiv:2007.12322.\n\n[Iqbal et al. 2020] Iqbal, S., de Witt, C.A.S., Peng, B., Böhmer, W., Whiteson, S. and Sha, F., 2020. AI-QMIX: Attention and Imagination for Dynamic Multi-Agent Reinforcement Learning. arXiv preprint arXiv:2006.04222.\n\n\n\n##### ======================== UPDATE =========================\n\nThanks for the authors' clarifications.\n\nAfter a careful re-evaluation of the paper, I have many concerns about the performance of baselines on the StarCraft II benchmark tasks. The reported performance is not consistent with those reported in the SMAC benchmark paper (see Figure 4,5,6 in [1]) and QPLEX paper (Figure 5,8,19 in [2]). Moreover, I also evaluate the available GitHub codes of baselines on my own, which is consistent with [1,2].\n\nUsing results in [1,2], QTRAN++ significantly underperforms the baselines on the StarCraft II benchmark tasks. Moreover, the paper claims that it uses the standard StarCraft II benchmark, the latest version of SC2, and the default baseline codes.\n\nDue to these concerns, I tend to lower my rating.\n\n[1] Samvelyan M, Rashid T, de Witt C S, et al. The starcraft multi-agent challenge. arXiv preprint arXiv:1902.04043, 2019.\n\n[2] Jianhao Wang, Zhizhou Ren, Terry Liu, Yu Yang, and Chongjie Zhang. Qplex: Duplex dueling multi-agent Q-learning. ICLR submission. https://openreview.net/forum?id=Rcmk0xxIQV", "extra_scores": {}, "extra_text": {}}, {"review_id": "l1gcpZMfCwK", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary and claims\n\nThis work proposes a MARL (multi-agent reinforcement learning) algorithm.\nIn the MARL setting, multiple agents have to make choices based on independent information to maximize a common objective. An existing algorithm in this space is QTRAN.\nThe authors propose several modifications to QTRAN: changing the architecture, adding two additional constraints to the loss function and also allowing gradients to flow from the QTRAN objective into the \"true\" action-value estimator.\nThe claims of the paper are:\n1) QTRAN++ achieves better performance than QTRAN\n2) The modifications introduced stabilize training compared to QTRAN\n\nIt took me some time to fully understand all of the components that go into QTRAN++ and I find the complexity of the overall algorithm pretty surprising, especially considering that other algorithms in this space are as simple as \"add up all the Q values of the individual agents\".\nI think the proposed changes consist of:\n - Rather than directly training action-value networks, a QMIX-like hypernet approach is used\n - The network that estimates the \"true\" (combined) action-value is implemented through what the authors call a \"semi-monotonic mixing network\", which is the sum of a non-monotonic (regular) hypernet and a monotonic hypernet as used in QMIX. This seems pretty arbitrary. Isn't the original idea behind QTRAN that this would accurately track the true values?\n - In QTRAN the separate network that aggregates the Q values of the individual agents is trained to track the \"true\" action-value. In QTRAN++ this is done through multiple hypernetworks (the authors call these \"heads\").\n - The loss function is modified to impose two additional constraints on the transforming value function.\n - Gradients are now also backpropagated from the \"tracking loss\" into the \"true\" action-value estimator, which makes it somewhat unclear what it is actually representing.\n\n### Relation to prior work\n\nThe paper is positioned sufficiently with respect to prior work. I've noticed that there is a larger section on related work in the appendix. I'm not sure what the purpose of moving the related work into the appendix is, especially if some of the papers mentioned there are not actually related to the work presented in this paper. I think it would be good to try to move as much as possible of that section into the main text, leaving out prior work that is not sufficiently related.\nSome of the additions in QTRAN++ seem very similar to ideas proposed in QMIX, but this is not directly acknowledged as far as I can tell. It would be good to point out which parts of the architecture come from QMIX.\n\n### Are the claims supported?\n\nThe experiments presented in the paper are reasonably thorough and show that\nQTRAN++ consistently outperforms QTRAN on the tasks that were tested. SMAC (StarCraft Multiagent Challenge) is a nontrivial benchmark, so I would agree that claim 1) has been shown sufficiently. But I'm not sure whether the ablation studies are thorough enough to really demonstrate that all of the components of the (rather complex) proposed algorithm are really needed.\n\nThe authors often make claims about improved stability and other properties of the algorithm throughout the paper, but these are not supported by any empirical evidence. If the authors want to claim that QTRAN++ outperforms QTRAN because of a specific mechanism then it would be good to provide some sort of empirical evidence or proof (the proof in appendix A doesn't count since it doesn't make any statements about stability). Therefore I think that claim 2) is currently not well supported and it would be good to either support it better or soften the statements in the paper to make statements in the form of \"we believe that the algorithm has improved stability\".\n\n### Presentation and clarity\n\nThe paper is reasonably clear and understandable. There are some cases where incorrect grammar or word choice made a sentence difficult to understand. For example the choice of \"affluent\" to describe a class of estimators. It would be good to address cases like this to improve the clarity of the paper.\n\n### Conclusions\n\nThe main claim of the paper (that QTRAN++ is an improvement over QTRAN) has been demonstrated sufficiently. But the high complexity of the approach (with several additions to the algorithm feeling somewhat arbitrary) and the fact that the paper \"merely\" presents an upgrade to QTRAN could be potential arguments against accepting it.\n\n\n### *Edit after author comments:*\n\nI have read the author comments and the latest paper revision. The authors have noticeably improved the clarity of the paper in several places and adjusted their claims about the stability of the algorithm, and the improved ablation studies are appreciated. Unfortunately, after thinking it through very carefully and despite the author comments, I have not been able to understand some aspects of the model, for example why gradients from the tracking loss are backpropagated into the value function that is supposed to track the \"true\" action-values. Several parts of the architecture seem to have a complicated dual purpose, which makes it difficult to understand what is going on and why the model is performing better. I suspect that other readers might also encounter similar issues, which makes it difficult for me to raise my rating. I've decided to leave the rating at 6 (marginal accept).", "extra_scores": {}, "extra_text": {}}, {"review_id": "N3N1ASRnPr6", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThis paper presents an improved version of QTRAN [1]. The design is based on new loss function design, as well as new action-value estimator designs. The paper claims superior perfromance gains compared to previous methods on the Starcraft Multi-Agent Challenge (SMAC) environment.\n\n### Strengths\n+ The ideas proposed to improve the previous QTRAN (or might be applied to other MARL algorithms as well) seems novel and general.\n+ The authors perform comprehensive ablation studies for different components they proposed.\n+ The empirical performance on the SMAC benchmark is better and more stable across different runs.\n+ The writing is clear and easy to follow.\n\n### Weaknesses\n- Only one environment is evaluated, which might not be that convincing. It would be good to see more results on different benchmarks.\n- In Figure 3, some results seem to be not converged yet. Since the metric is the win rate, which is bounded, it would be interesting to see given enough training steps, whether all methods can actually converge to similar winning rate, or inherently the proposed scheme can lead to better results.\n\n### Minor issues\nTypo: Page 6, \"... for being “selfish.” This ...\" -> \"... for being “selfish”. This ...\"\n\n\nConsidering all the aspects, I tend to accept the paper in the current stage.\n\n\n### Reference\n1. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. 2019.\n\n----------------------------------------------------------------------\n\n**Updates**: After reading the other reviews and the rebuttal, I still maintain my current score. The additional experiments on the converged results are good to me. As I'm not very familiar with the performance in MARL literatures, I have decreased my confidence from 3 to 2 to reflect some of the concerns of other reviewers involving the performance for the baselines.", "extra_scores": {}, "extra_text": {}}, {"review_id": "AI_A8kgknFy", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThis paper addresses the domain of cooperative multiagent learning with centralised learning and decentralised execution. Specifically, it improves on the QTRAN algorithm, a theoretically justified algorithm which previously had not produced strong learning performance. With these improvements, QTRAN++ outperforms baselines on the SMAC environments.\n\nI recommend accepting this paper. It delivers strong performance on a popular benchmark for complex, cooperative multiagent learning (SMAC). While the algorithmic contribution is incremental, it still delivers insight into how to improve the empirical performance of a theoretically interesting and well-justified algorithm.\n\n ## Positives\n\nThe problem addressed - cooperative multiagent environments with CTDE - is a widely studied and important one. It is well set up in the paper, including discussion of related algorithms.\n\nThe base algorithm - QTRAN - should theoretically perform well in a wider variety of environments than other algorithms for these problems, so improving its performance is particularly valuable. The improvements made to the algorithm are clear and well motivated; section 3.1 in particular explains clearly the difference the modified loss is intended to make.\n\nThe empirical studies in the paper are strong. They show that QTRAN++ outperforms several baselines in data efficiency and final performance across a variety of domains. Further, a comprehensive ablation study shows that each of the improvements made to QTRAN is independently important (in at least some domains).\n\n## Negatives\n\nThe algorithmic contribution of the paper is relatively minor, since it provides fairly simple modifications to an existing algorithm.\n\nExperimentally, it would be good to see experiments on similar domains to those addressed in the original QTRAN paper, which are designed to probe the advantages QTRAN has over related algorithms. This would demonstrate that QTRAN++ retains the benefits of QTRAN in non-monotonic factorisable environments.\n\n## Sources of reviewer uncertainty\n\nI am not knowledgeable enough in this domain to be certain of the coverage of the baselines and domains in the paper. Since the empirical performance of the algorithm is central to the paper, this is important.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes practical improvements to theoretically well founded QTRAN, which is a state-of-the-art technique of cooperative multi-agent reinforcement learning.  The improvements include new designs of loss function and action-value estimator, which might be widely applicable beyond QTRAN.  However, it is not obvious if the proposed improvements actually improves the performance of QTRAN, and experimental evaluation is essential to this work.  After the discussion, there remain some major concerns about the experimental results.  In particular, the performance of baselines in the experiments is not consistent with those reported in the prior work.
Decision: Reject