Multiagent Q-learning with Sub-Team Coordination
Authors: Wenhan Huang
Kai Li
Kun Shao
Tianze Zhou
Matthew E. Taylor
Jun Luo
Dongge Wang
Hangyu Mao
Jianye HAO
Jun Wang
Xiaotie Deng
Venue: NeurIPS 2022 Accept (2022)
Keywords: Cooperative multi-agent reinforcement learning
Centralized training with decentralized execution
Value Factorization
Sub-team coordination

Abstract: In many real-world cooperative multiagent reinforcement learning (MARL) tasks, teams of agents can rehearse together before deployment, but then communication constraints may force individual agents to execute independently when deployed. Centralized training and decentralized execution (CTDE) is increasingly popular in recent years, focusing mainly on this setting. In the value-based MARL branch, credit assignment mechanism is typically used to factorize the team reward into each individual’s reward — individual-global-max (IGM) is a condition on the factorization ensuring that agents’ action choices coincide with team’s optimal joint action. However, current architectures fail to consider local coordination within sub-teams that should be exploited for more effective factorization, leading to faster learning. We propose a novel value factorization framework, called multiagent Q-learning with sub-team coordination (QSCAN), to flexibly represent sub-team coordination while honoring the IGM condition. QSCAN encompasses the full spectrum of sub-team coordination according to sub-team size, ranging from the monotonic value function class to the entire IGM function class, with familiar methods such as QMIX and QPLEX located at the respective extremes of the spectrum. Experimental results show that QSCAN’s performance dominates state-of-the-art methods in matrix games, predator-prey tasks, the Switch challenge in MA-Gym. Additionally, QSCAN achieves comparable performances to those methods in a selection of StarCraft II micro-management tasks.

Multiagent Q-learning with Sub-Team Coordination
Wenhan Huang'; Kai Li", Kun Shao', Tianze Zhou', Matthew E. Taylor*, Jun Luo?, Dongge Wang', Hangyu Mao², Jianye Hao Jun Wang", Xiaotie Deng
Shanghai Jiao Tong University, Huawei Noah's Ark Lab, 'Beijing Institute of Technology University of Alberta, AAlberta Machine Intelligence Institute (Amii), "EPFL
'Tiajinn University, UUniversity College London, "Peking University jowdarkesjtu. edu cn, {likai210, shaokun2, luo1, maohangyyu))hhuaweet com simsimiztze126. com, matthew. e. taylor ca. dongge wang@epf1.co jianye hao@tju. edu. en, jun. wang@cs ucl ac. uk, xiaotie@pku, edu. cn
wongge.wang@epf..
Abstract
In many real-world cooperative multiagent reinforcement learning (MARL) tasks, teams agents can rehearse together before deployment, but then communication constraints may force individual agents to execute independently when deployed. Centralized training and decentralized execution (CTDE) is increasingly popular in recent years, focusing mainly this setting. In the value-based MARL branch, credit assignment mechanism is typically used to factorize the team reward into htteiidduugggobal--ee
each individual's reward factorization ensuring that agents' action choices coincide with team's 's optimal joint action. However, current architectures fail consider local coordination within sub-teams that should be exploited for more effective factorization, leading to faster learning. We propose novel value factorization framework, called multiagent Q-learming with sub-team coordination (QSCAN), to flexibly represent sub-team coordination while honoring the IGM condition. QSCAN encompasses the full spectrum of sub-team coordination according to sub-team size, ranging from the monotonic value function class to the entire IGM function class, with familiar methods such as QMIX and QPLEX located at the respective extremes of the spectrum. Experimental results show that QSCAN's performance dominates state- of-the-art methods in matrix games, predator-pre -rreyy tasks, the Switch challenge in MA-Gym. Additionally, QSCAN achieves comparable performances to those methods in selection of StarCraft Il micro-management tasks.
(IGM) is condition
the
1 Introduction
For many complex real-world tasks, such as robot swarms [D and autonomous vehicles 2 the coordination of agent teams is critical. Cooperative multiagent reinforcement learning (MARL) a common setting where teams of agents, potentially using deep learning, can learn to solve a common task. This paper considers the centralized training with decentralized execution (CTDE) 3 framework, where a team of agents can train together with communication but execute fully decentralized during deployment (e.g., because of high communication costs, the risk of being detected).
Value-based methods in the CTDE paradigm have become increasingly popular more capable methods have been proposed. These methods typically factorize the global action-value function
"This work was done when Wenhan Huang was an intern Huawei Noah's Ark Lab. CCoreeponndig author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

into agents' individual action-value functions to deal with the exponential growth of the joint-action space the number agents. If the solution the problem can be identified so that the individual- global-max (IGM) property [4] can be satisfied, the problem can be simplified - in this case, an individuals' best (greedy) action corresponds to the best action for the team. Rather than having to reason about the full optimal joint action, the IGM condition allows individual agents to maximize their own individual action values.
Among these value-based methods, various factorization architectures satisfying the IGM condition have been proposed. Sunehag et al. [5] propose linear factorization of the global action-valus function named VDN. VDN treats the global action-value function as the sum of each agent's 's action value. QMIX 66] extends this factorization to a monotonic function so that QMIX can represent some non non-linear global action-value functions. Qatten [77] decomposes the global action-value function with a multi-head attention approach based on the Taylor-expansion optimal joint strategy, which finds more precise monotonic functions than QMIX. However, these structures cannot represent non-monotonic global action-value functions, which might cause the relative overgeneralization 8 problem: The reward of one agent's cooperative action might be underestimated due to other agents' non-cooperative behavior.
the point of the
To extract coordination information in tasks requiring non- monotonic global action-valus value functions, Deep Coordination Graph (DCG) 8 factorizes the global action-valus function into pairwise action-value functions and converts the original problem to co- ordination polymatrix game [00] leading to violation of the IGM condition. Due to the violation, DCG requires several rounds of inner-agent message passing at execution time. Wang et al. [I]] propose another approach, QPLEX, which exploits the advantage- based IGM condition and achieves complete IGM function class. However, this method treats different joint actions as totally differ- ent atomic items, which may lead to sample inefficiency and poor generalization.
predator
predator
predator
prey
Figure 1: Sub-team coordina- tion. In this simple predator- prey game, three predators (red circles) try to catch prey (green circle). At this moment, only the sub-team of predators and 2 matters because preda- tor 3 is too far away.
In many settings, some agents must coordinate for success, but full coordination of all agents is not required. Instead, the task can be divided into several sub-tasks, and each sub-task requires exactly one sub-team of agents [12] As Fig [ shows, the sub-team of predator 1 and predator 2 is enough to catch the prey. The sub-team structure has been widely used for robotic tasks and unmanned aerial vehicle tasks with communication [33. Yang et al. (14) also show that the coordination among sub-teams enough for solving many complicated cooperative multiagent tasks under the CTDE paradigm.
Inspired by the sub-team organization, we study the sub-team factorization of the global action value and propose hierarchical structure, called multiagent Q-learning with Sub-team CoordinAtioN (QSCAN), to strike a tradeoff between the representation capability of coordination patterns and the complexity of function classes. Our model generalizes the monotonic and the IGM structures via the sub-team factorization of the global action value. QSCAN guarantees the IGM condition and flexibly handles sub-team formations. Two approaches, QPAIR and QSCAN, are proposed for our hierarchical structure QSCAN. We empirically evaluate these methods in several coordination tasks, including a matrix game, predator-prey tasks [15), and the Switch challenge [16). Comparing our approaches with QMIX and QPLEX, we illustrate the sub-team coordination pattern improves the results in these tasks. QSCAN significantly outperforms the two baselines in these three settings, while QP (PAIR achieves comparable performance. with these baselines in some widely used StarCraft Multi-Agent Challenge (SMAC) scenarios [17]. These results show that our method can not only significantly benefit CTDE learning, but also suggest way forward for more flexible sub-team coordination and learning in multiple settings beyond the CTDE.
addition, we show that our approaches are also comparable
2

---

2 Background
Dec-POMDP We characterize fully cooperative multiagent leaning task decentralized partially observable Markov decision process DDec-POMDP) with a tuple M = -W,S, f2, O r, is set of agents and S is consisting of environment states. At each time step, each agent i E N chooses an action 0; E A,, forming a joinnt action E A = of all agents. This leads to a transition from the current state the next one s' governed by transition function ([s'|s,e a). Due to the partial observability, . = x Oi is the joint observation of agent O a') d') is the conditional probability of
[8] .V= .1.2...... .n)
set, where 22 is the partial observation joint observations given the current state s and the previous joint action a'. r SS x A - I denotes the global reward function. The objective of the task is to maximize the total discounted reward 22 ("r(st, az) where E [0, 1) is discount factor, S and ag are the state and the joint action at time step respectively.
Multiagent deep Q-learning. For a given joint policy π of agents, let the global action-value function (", a,) > "rr===>""(((_... 7 dt+4]] denotes the expected discounted reward starting at state 8, with joint action Og When π is optimal, the global cction-value function satisfies the Bellman opti- mality equation. Due to the partial observation setting, would use the joint observation history T in place of the state 8. Multiagent deep Q-learning represents the global action-value function Qtor with a deep neural network parameterized by O. In centralized algorithms, the parameters are obtained by minimizing the expected TD error L = E (r + maxa Qtor aa' Qtor (T, a; 0))² where
0- is the parameter of a target network.
Value-based methods in the CTDE paradigm. A typical dilemma cooperative multiagent learning is that each agent should act independently based on its observation, but global information needed to make good decision for teamwork. CTDE B is the popular framework these years to remedy this challenge. In most value-based methods in the CTDE paradigm, the individual action-value functions for all players are trained in a centralized way, and an agent only relies on its trained individual function and its partial observation to choose actions. Due to the partial observation, 4 suggests that the optimal joint strategy should be executed as each agent plays its own optimal strategy concurrently. Formally, the condition of such action-value functions can be described as follows:
Definition (Indivduual-Global-Maa (IGM) 4) For global action-value function Qiot : : x A + and a series of individual action-valie functions [Qi] with Qi : Te A, + R, where T E T is the joint history and T E Ti is agent i's individual history. if E TT
arg max
arg max Qu(Tn: an)) on E...
E arg max (T,
a1EA1
aeA
then we say that [Q:]=_ satisfy the IGM condition for Qiot-
Once framework satisfies the IGM condition, the global action value be maximized efficiently via individually optimal choices of all the agents.
Self-attention mechanism. The self-attention mechanism [19 is widely used to extract the relation- ship between different positions of an input sequence in the natural language processing community, It can efficiently relate its different inputs. In the MARL community, the self-attention mechanism is used to learn the relationship between a team of agents. Jiang and Lu (20] employ the self-attention mechanism to learn the communication among agents. Li et al. (21) use the self-attention mechanism to determine the implicit coordination graph structure of agents. The formula of the self self-attention mechanism be written as: Attention Q, IC, V) softmax QVTT), where Q. AC, V denote the query vectors, the key vectors, and the value vectors, respectively. The weight of each value V/ to the i-th output is computed by the compatibility 2rKj, dot product of i-th query vector and j-th key vector. The softmax activation function translates the dot product into a measurement of the attention.
3 Sub-Team Coordination Patterns
In a large multiagent system, the task usually decomposed as several sub-tasks so that agents can accomplish each sub-task separately, For each sub-task, sub-team is formed with several agents.
3

---

Besides, one agent could simultaneously belong to different sub-teams. This sub-team organization is general and can implicitly characterize most coordination patterns among agents [2] In this paper. we exploit sub-team coordination patterns in value factorization frameworks. To this end, we would first explore sub-team coordination patterns this section and then discuss the concrete architectures in the next one. The organization of this section is as follows. First, we propose a factorization of global action value based on the sub-team organization, named sub-team factorization. Then, we consider two typical coordination patterns, individuals as sub-teams and the grand team, used in previous works. Finally, we present a general framework characterizing coordination patterns with different sub-team sizes. For simplicity, we only consider the fully observable settings in this section.
Sub-team factorization. With the sub-team organization, each sub-team is assigned to a sub-task. Ideally, the sub-tasks are disjoint so that the agents beyond a sub-team would not affect the outcome of this sub-team, Mathematically, each sub-team ST C A has an action- function QST(s, asT) aQ for the performance of its sub-task and we have Vj & ST, eQ, = O. aST is the joint action of agents in ST. The global action value Qiot can be treated as an evaluation of the set of sub-teams' action values {QST}. Since the sub-tasks are disjoint, Qiot should be monotonic with each QST. that is 8Quet 2 O.
öQST
Individuals. A typical solution is to treat each agent as a sub-team {i} so that each agent is assigned with a distinct sub-task. In this scenario, we can assume ( = Qi and &Qua state 8, the function class can be rewritten as Qtoe(s, a) f (s, a1)................... ..... Qs, am), s), where f a monotonic mixing function. It is identical to the monotonic value factorization used by QMIX [22]. As in [4], this formulation immediately satisfies the IGM condition. However, this monotonic factorization may limit the representation capacity of the framework and lead to incorrect solutions
aQ, 2 0. For given
during the training process 4 15
One may consider encoding more coordination patterns into the same value factorization framework but with non- monnoonoo function f. However, without any constraints for the input of f. we prove in Proposition [ (detailed proof in Appendix B.4) that the considered function space satisfying the IGM condition should be equivalent to the space in which the monotonic factorization can represent. In this proposition, we assume the input of f can take arbitrary values because the framework should satisfy the IGM condition consistently during the course of training.
Proposition 1 Consider a fixed mixing function R x S where Qiot and [Qi]== satisfy the IGM condition consistently for any function Qi(s,)) which contains a unique maximum point, then f should satisfy Vi Vivz; E I. 0f(2 (n.8)
R. If (fttel)))= f(Q Qn.s),
> 0.
82g
The grand team. Only considering the grand team {1.... } is another typical solution. In this factorization, we can assume Q(1.....) = Qtot. With the individual action values (Qi), the global action value can be written as (s, f(Q1...., Qni 8, a). This kind of value factorization may not always satisfy the IGM condition.
QPLEX [1 instance of this function class, employs the duplex dueling structure and transfers the IGM condi- tion to advantage values A(s, a) = Q((s, Vs),, where V(x)= = max, Q(s, a). The global advantage value Azot is factorized as (s a) Em (s aa) (s, where [A,(s, ))} is an importance weight. The joint action a here is considered atomic. As pointed in (II) the IGM function space is equivalent to the space that QPLEX can represent. However, the indivisibility of the joint ac- tion a in the grand team may prevent the mixing network from reusing the knowledge from previous coordination patterns, leading to poor generalization. Specifically, the importance weights using atomic joint actions in QPLEX may not perform well in credit assignment when the correlations among agents become compli- cated, As the predator-prey results shown in [15] and [II], QPLEX requires additional tuning of the exploration rate to extract predators' coordination patterns.
OPLIE-OCCAAL
CECANG
(IMM
dECAR)
Figure 2: Coordination hierarchy. We identify and prove general representa- tion of relation among sub-team coordi- nation function classes, from the mono- tonic function class to the IGM function class.

---

Rcae (T.
Mixing
Network
(cxett.
Coordination Mixing
Aval (F. a)
Voor (t)
Vi A (r.a)) . Ar(.a)
Momolonicc function
Weighted Sum
@: (r)
ault,a,)
Sub-Team Coordination Module
2
w 0,b
Transformation
MLPP
(.(T,,)))
Qn(ta
((T..))
a.aa)... (n..a))
Agent 1
Agent
(og, a2)
(G oq)
Figure 3: The base architecture of QSCAN. (a) Coordination mixing sub-network structure. (b) The overall QSCAN architecture. (c) Transformation module.
k-member sub-teams. To provide general view of the sub-team organization and characterize coordination patterns more systematically, we consider the organization with k-member sub-teams, where & E 21, n.. We have been pursuing a value factorization framework QSCAN,, which could characterize the coordination patterns within the k-member sub-teams. To this end, we would first analyze the properties of QSCAN, and then discuss the concrete architecture in the next section.
QSCAN; should characterize the coordination patterns beyond k-member sub-teams, and thus we would accordingly establish a hierarchical structure of these QSCAN classes. To this end, we first show the relation between QSCAN; and QSCAN&-1 in some special team reward functions. For several types of team reward, including c-degree polynomial functions, c-clause functions, and hypergraphical functions 223, coordination patterns within (k- - 1)-member sub-teams can be transferred to coordination patterns within k-member sub-teams (see Appendix B.2). More generally, when the sub-team size is typically small as practice (k: S 5), according to Hall's marriage theorem [24), QSCANs-1
be shown special case of QSCAN, with the only assumption JQ 2 0 (see Appendix B.3). Although, for general cases, such relationship might no longer "QsT'
when & 2 (except for n, when the sub-team is exact the grand team), the hierarchical structure of coordination patterns inspires us on the design of our value factorization method. As the special team reward functions above, intuitively, linear multiplicative factorization based on some sub-team features can ensure the hierarchical property. We pursue that our QSCAN establishes a hierarchical structure for all sub-team sizes, named coordination hierarchy, shown in Fig 2
Multiagent Q-learning with Sub-team Coordination
In this section, we present the concrete architecture of QSCAN, which is a value factorization framework exploiting sub-team coordination patterns while honoring the IGM condition. We first describe the base architecture, which employs duplex dueling structures [I]] to guarantee the IGM condition. We then discuss the crucial component of the QSCAN, sub-team coordination module, which characterizes the organization and coordination patterns of the team. A series of function classes could be derived from the QSCAN framework. Finally, we discuss the coordination module in detail and present two practical architectures for QSCAN.
Base architecture. The overall architecture of our model is shown in Fig The duplex dueling struc- ture factorizes each agent i's action-value function into its individual value and an advantage function The global action-valus function is fac- torized similarly as Qeor (T, a) Vine(T) + Aror (T, .), where the global value V(T) = = Vi(T). The global advantage function is represented Acoe (T, a) f([A(T, a) ()(, a))le1ss)) where
Vi(T) max.; Qi(T, af), Ar(T, a;) = Q,(T,
5

---

non-negative weights À is generated through a Sub-team Coordination module and f is monotonic of function with respect to each S (i.e., Sm; 2 0), and maintains a maximum value 0 (f,,,,,)) s) = 0). It can be shown that this structure satisfies the IGM condition (see Proposition 2 in Appendix B.5).
Following previous works 6 II 25],
we would use the global state as the centralized information,
if applicable, or the joint history T.
Sub-team coordination module and QSCAN framework. We analyze the design of the theoretical coordination module which characterizes coordination patterns within k-member sub-teams and then propose the QSCAN framework. Intuitively, the team reward can be credited to each sub-team and then to each individual. Consider sub-team ST containing k agents and aST is ST"s joint action. The contribution ST be assigned to each member with an importance weight g evaluating i's contribution in ST. Therefore, we could approximately factorize global advantage into each agent's s individual advantage
Ator (T, a) 2
Σ
Σ LIEST
(0e ST (T((T)))) Ai(T,
ST:STCN, ST
Σ
9. ST (T, (T,
Al(T,a)).
Based on this factorization, we propose QSCAN,
Definition 2 (QSCAN+) QSCAN& a branch of QSCAN which concerns coordination among sub- leams containing only k agents. Specifically, QSCAN& adopts the following weights yield from the sub-team coordination module >4 a) = (EsTreSTEN g." (T asr)). where h is (
non-negative activation function.
7r(T, a) corresponds the total contribution of agent to all sub-teams containing k members based on joint history т. Since the advantage as the disparity from the optimal action should keep non- positive, & is used ensure positivity of each X. According to QSCAN, we could obtain QSCAN, by the analytic continuation, in which g. does not take actions as input. With the continuation, Fig illlustraas that QMIX and QPLEX locate at the respective extremes of the spectrum for QSCAN framework. The detailed explanation and proofs for these propositions are in Appendix B.7 and B.8.
After demonstrating the QSCAN framework, we present two different coordination architectures: QPAIR is based on the natural enumeration of sub-teams, and QSCAN is based on the self attentttni mechanism.
Pairwise coordination. We first present a pair- wise coordination module of QPAIR by enumer- ating all sub-teams with size 2. The module uses multi-layer perceptron (MLP) to calculate the pairwise coordination coefficients (Fig 4.). This structure a QSCAN, level the co- ordination hierarchy. The coefficient of each A, a:) based on pairs of agent's 's action as well as the global state (s, i, 0z, az). Then >a(T,a) a) = EE= | MLP(s, i, 0z (t. practice, we use absolute value function as h here.
Ade Aerichhmm
Verrs
MU
Food
Faroard
In
(1.90)
Figure 4: Coordination modules. (a) Pairwise co- ordination module. (b) Self- attention module.
The simple extension of QPAIR for larger sub- team size & increases the computation cost ex- ponentially due to the enumeration of O(n*) sub-teams. However, such exponential enumeration might not be required because identifying the optimal sub-team coordination patterns is enough for the task. That is, the problem becomes searching the optimal patterns, which can be solved via data driven methods. This kind of searching- ootiimal problems have been approximately solved by neural

---

network structures. For example, some combinatorial optimization problems like traveling salesman problem or Boolean satisfiability have been approximated by some neural network structures [26] 27
Self-attention, Inspired by the structure of Transformer [19], we may use self-attention mechanisms to characterize correlations among agents hierarchically. The self-attention mechanism has been shown to be universal approximators for continuous permutation equivariant functions (28), where the importance weights g.ah and A, belong to. Meanwhile, the self-attention mechanism can aggregates agent-action information effectively and flexibly. 4p illustrates the sketch of our module of QSCAN. The module takes all agent- action pairs [i,, ) and the global state as input and produces series of non-negative weights [A:]i-1
Let us explain how the agent-action information is aggregated through the self-attention mechanism. The embedded input vector of agent-action pairs [(i, az)]]== can be viewed as the individual agent- action information vector the action information vector of 1-member sub-teams. When the action information vector of 1-member sub-teams passes an attention layer, the pairwise weights 2 will relate the i-th and j-th positions of the vector. Notice that the action information of 21-member sub-team can be obtained by aggregating the action information of the first members' sub-team and that of the last members' sub-team. Therefore, the output of this attention layer can be viewed the action information vector of 2l-member sub-teams. Without considering the residue links in Fig, 4 the output of the m-th attention layer compresses the action information of me--mebber sub-teams. The residue links will mix up the action information of sub-teams with different sizes, and it provides the network more flexibility learn different forms of the coordination patterns.
5 Empirical Results
We compare QPAIR and QSCAN with state-of-the-art MARL approaches, QMIX and QPLEX, in various coordination tasks, including matrix games, predator-prey challenges (8), the Switch task (16),
and the StarCraft Multi-Agent Challenge (SMAC) [17]. For fair comparison, the implementation of QSCAN uses one attention layer, that is, with sub-teams of size 2. The learning curves are plotted with smooth factor of 0.6 except the last point. The implementation detail and environment detail are in Appendix C. In the matrix game, the agents need explicit coordination to obtain the highest payoff. The predator-prey tasks are complicated scenarios with immediate coordination rewards. In these tasks, the agents need to learn spatial-tent nettial-emppoal local coordination. The Switch task is more complicated coordination task due to the sparse and long-term rewards. In this task, the sub-team coordination in the beginning steps influences intensely over the final rewards. The SMAC environment is common-used for evaluating the MARL approaches. We evaluate our approaches in selection of SMAC scenarios. Additional empirical results including ablation study for the number of attention layers of QSCAN and a super-hard scenario 27m_vs. v0__v___0mmmmm are in Appendix E.
5.1 Matrix Game
Table shows the payoff's of 3-player with 2-action matrix game. Fig 5a illustrates the em- pirical results for QMIX, QPLEX, QPAIR and QSCAN. Due the relative overgeneraliizttion, QMIX fails to find the optimal solution. QPLEX suffers another problem about the poor general- ization. QSCAN and QPAIR can find the optimal solution more quickly because the pairwise co- ordination patterns provide more suitable gener- alization this task. In the exploration period, the pairwise coordination would allow the algo- rithm to explore the optimal solution more easily. Moreover, for all different random seeds in our experiments, QSCAN always finds the optimal solution rapidly in this matrix game.
Table 1: Payoff's of a 3-player matrix game. Each player E (1,2,3) has two actions {A, B). The complete payoff matrix is split into two sub- matrices according to player 1's action Ø1.
When 01 =A
When a
03
A
B
B
02
022
A
0
0
A
17
20
23
12
B
17
7

---

200w
0800
(a) Matrix game.
(b) Predator-Pre (6 versus 6).
(c) Switch4 challenge.
Figure 5: Learning curves of QPAIR, QSCAN, QPLEX, and QMIX in three different tasks. For the timestamp I', k and mil are short for is 'kilo' and 'million', respectively. The average reward with 95% confidence intervals is shown. Best viewed color.
5.2 Predator-Prey
We compare our approaches QPAIR, QSCAN with QMIX and QPLEX in partially-observable cooper- ative Predator-Prey tasks created by [8]. We evaluate algorithms in the scenario with predators against prey.
The results are shown in Fig 5b QMIX fails to learn a positive reward due to the relative over- generalization caused by the miscoordination of "capture" actions. QPLEX can find some correct coordination among agents. For QPAIR, it performs better than QPLEX because the correct coordi- nation of "capture" actions only needs pairwise coordination, which QPAIR is forced learn. For QSCAN, outperforms all these approaches due to its adaptive balance of the pairwise coordination and each individual's local information.
Fig 6a illustrates the QSCAN's self-attention weights for continuous steps over the complete process that predator 2 and predator 5 capture prey. The figures the left side demonstrate the state, where a predator is described as a rectangle with agent id on it, and the preys are denoted as the yellow rectangles. The figures the right side show the corresponding self-attention heat maps. For each map, a rectangle lying in row and column j represents the agent i's attention for agent j. Brighter the color is, larger the weight is, and more attention is attracted. In the first step of the capture process, all predators pay attention to predator 2 who attempts to capture the prey in row and column 3. In the second step, predator 2 pays attention to predator 5 who responses, and they successfully coordinate to capture the prey in row 4 and column 2. When current capture is completed, predator 2 and predator finish their jobs and predator will organize the next capture process.
5.3 Switch4 in MA-Gym
Switch4 MA-Gym [16] is a partially observable task that 4 agents need to reach their corresponding home by passing through the one-agent wide narrow corridor as 7
As Fig shosss our approach QSCAN outperforms others in this task while QPLEX performs worst. QMIX and QPAIR achieve comparable results, while our approach achieves better performance during the training phase.
Figure 7: The starting state of Switch4. 4 agents need to reach their corresponding destinations. Each colored block (yel- low, blue, green, or red) with a number denotes agent. Black blocks are bar- riers. Blocks with colored boundaries denote the destinations for correspond- ing colored agents.
We show the self-attention heat map QSCAN in the first 3 steps of optimal and sub-optimal solutions respectively. Fig (6b] shows the heat map for both optimal and sub- optimal solutions. The left side of Fig [b) is the heat map of the optimal solution. In the first step. all agents pay attention to agent who will lead the first passing. In the mean time, agent pays attention to agent because agent should not move down to cause traffic jam. In the second step, agent 2 pays attention to agent ( who needs to follow agent 2 to pass the corridor. In the third step, each agent follows the coordination organized the first two steps. The right side of Fig is is the heat map of the optimal solution. Similarly, in the first step, all agents pay

---

attention weightsstep:
6.4
0.4
CIV statelstep:
attention weightistep. 2)
6.
CIW Statelstep: 3)
attontion weight/step. 5)
63
(a) First 3 steps in Predator-Prey Prey (6 versus 6).
Optimal solutSon
Sub-optianal solution
dubo plaiie
in
Aade wylkua
(b) First steps in different solutions for Switch4.
Figure 6: States of the games and the corresponding attention heat-maps. In each of 3 columns of pictures, the left side shows the states and the right side shows the attention. (a) In these states, the predators are numbered from 0 to and the preys are yellow rectangles. (b) We demonstrate two solutions. See Fig for more details of the states.
attention to agent 2, while agent 2 pays attention to agent and asks agent to pass the corridor first. In the second step, all agents pay attention to both agent 2 and agent 3 that they will organize the passing. In the third step, agent 3 additionally pays attention to agent 1, who needs to follow agent to pass the corridor.
5.4 The StarCraft Multi-Agent Challenge
The StarCraft Multi-Agent Challenge (SMAC) Table 2: Median of the test win rates in the SMAC. [17] is a widely-used benchmark for cooperative MARL. We evaluate our approaches a wide- range of SMAC scenarios, including homoge- neous (e.g., 5m_vs _s_______m__ (3_)mm) and heterogeneous (e.g., 3s5z) agents. Furthermore, we compare our ap- proaches with state-of-ther
QSCAN
100
100
100
100
2d32
100
baselines: QMIX and QPLEX. The empirical results are shown Table and the corresponding figures are presented in Appendix E.4.
3x5x
98
97
97
94
Ic3sSz
88
97
97
94
20.s5_64gg
62
86
88
42
76
77
72
69

---

The results shows that QSCAN and QPAIR are superior to QMIX in most scenarios. QSCAN performs almost well except for for2c__ 2c_vs 64zg. Its win rate is about 20 percents less than that of QPAIR or QPLEX. 2c_ ___vs__44 vs_64zg needs the coordination of two colossi, where the pairwise coordination characterized by QPAIR coincides with the grand team coordination characterized by QPLEX. As QMIX does not perform well in this scenario, we conjecture that this scenario requires the pairwise coordination patterns rather than individual patterns, One possible reason for the performance of QSCAN could be that the individual patterns through the residue links prevent QSCAN to achieve better performance in this scenario. QPAIR achieves the best performance some scenarios and no more than 2 percents less than the best one's win rate in other selected scenarios. Overall, our approaches achieve comparable performances with the SOTA baseline QPLEX which uses the joint action in the mixing function.
Conclusions and Future Work
We propose QSCAN, novel value-based multiagent reinforcement learning framework that can characterize coordination patterns within sub-teams hierarchically and guarantee the IGM condi- tion. QSCAN provides value factorization architectures with an expressive mixing network for the centralized end-to-end training and learms a series of individual action-value functions for decen- tralized execution. We establish coordination hierarchy based on QSCAN after analyzing the sub-team factorization, and present two efficient implementations based on pairwise coordination and self-attention mechanisms. Empirically, we show that our methods achieve better or comparable performance with baselines in several benchmarks.
While our value-based architecture employs the duplex dueling structure for the sub-team factorization, we believe that the sub-team factorization benefits other architectures, e.g., policy-based methods like COMA 225 As our discussion about sub-teams Sec. designing more effective hierarchical structures for sub-team organization beyond the factorization on advantages, remains a challenge. From theoretical side, it remains a problem about whether general coordination hierarchy exists
according the sub-team factorization aQ 0 when sub-team size k 2
2 One more thing,
aQst
there has not been universal organizational paradigm suitable for most tasks 22
30 Besides the
sub-team organization, exploring other organizational paradigms in cooperative MARL is promising.
In the future, we will continue exploring organizational paradigms in large multiagent systems.
Acknowledgments and Disclosure of Funding
We would like to thank Weixun Wang, Xiaotian Hao, and Zipeng Dai for various helpful discussions. This work is partially supported by the National Natural Science Foundation of China (Grant No. 62172012). Part of this work has taken place in the Intelligent Robot Learning (IRL) Lab the University of Alberta, which is supported in part by research grants from the Alberta Machine intelligence Institute; Canada CIFAR AI Chair, Amii; CIFAR; Compute Canada; and NSERC.
References
[1] Maximilian Hüttenrauch, Adrian Šošić, and Gerhard Neumann. Guided deep reinforcement learning for swarm systems. arXiv preprint 17Xiv:1709.00011, 2017.
[2] Yongcan Cao, Wenwu Yu, Wei Ren, and Guanrong Chen. An overview of recent progress in the study of distributed multi-agent coordination. IEEE Transactions on Industrial informatics, (1):427-338, 2012.
[3] Frans A Oliehock, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate q-vulue functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289-353, 2008.
[4] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. arXiv preprint arXiv: 1905: 005000508, 2019.
[5] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinieius Zambaldi, Max Jaderberg, Mare Lanetot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-
10

---

decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv: 2017.
[6] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint arXiv: 1845. 2018.
[7] Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning, arXiv preprint arXiv:2002.0398 03939, 2020.
[8] Wendelin Böhmer, Vitaly Kurin, and Shimon Whiteson. Deep coordination graphs. In Interna- tional Conference on Machine Learning, pages 980-991. PMLR, 2020.
[9] Liviu Panait, Sean Luke, and R Paul Wiegand. Biasing coevolutionary search for optimal multiagent behaviors. IEEE Transactions on Evolutionary Computation, 10(6):629-645 645, 2006.
[10] Yang Cai and Constantinos Daskalakis. On minmax theorems for multiplayer games. In Proceedings of the twenty-second anmual ACM-SIAM scmpSSAAsymmoosu on Discrete algorithms, pages 217-234. SIAM, 2011.
[11] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008. 01062, 2020.
[12] Bryan Horling and Victor Lesser. A survey of multi-agent organizational paradigms. Knowledge Engineering Review, 1944)281133,, 2004.
[13] Milind Tambe. Towards flexible teamwork. Journal of artificial intelligence research, 7:83-124, 1997.
[14] Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi-agent reinforcement learning. In International Conference on Machine Learning, pages 5571-5580. PMLR, 2018
[15] Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation. arXiv preprint arXiv:2006. 10800, 2020.
[16] Anurag Koul. ma-gym: Collection of multi-agent environments based on openai gym. https
//github. cottibb.comm/eeee wwt./com/ooooe wowo://kuuaauurg/mmaa. gym 2019.
[17) Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. Rudner, Chia-Man Hung. Philip H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. In arXiv: 04043 [cs, stat], February 2019. URL htttp://arxiv.org////
04043
arXiv: 1902.04043.
[18] Frans Oliehoek, Christopher Amato, et al. A concise introduction to decentralized POMDPs. volume 1. Springer, 2016.
[19] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pages 1000060100 2017.
[20] Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent coopera- tion. Advances in Neural Information Processing Systems, 31:7244-7264, 2018
[21] Sheng Li, Jayesh K Gupta, Peter Morales, Ross Allen, and Mykel J Kochenderfer. Deep implicit coordination graphs for multi-agent reinforcement learning. arXiv preprint arXiv:2006.11 11438, 2020.
[22] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint arXiv:2003.08839 2020.
[23] Christos Papadimitriou and Tim Roughgarden. Computing correlated equilibria in multi- player games. Journal of the ACM (JACM), 55(3):1-29, 2008.

---

[24] Philip Hall. On representatives of subsets. Classic Papers in Combinatorics, pages 58 58-62, 1987.
[25] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[26] Marcelo Prates, Pedro HC Avelar, Henrique Lemos, Luis Lamb, and Moshe Y Vardi. Learning to solve np-complete problems: A graph neural network for decision tsp. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4731-4738, 2019.
[27] Vitaly Kurin, Saad Godil, Shimon Whiteson, and Bryan Catanzaro. Can q-learning with graph networks learn generalizable branching heuristic for sat solver? Advances in Neural Information Processing Systems, 1199608-66211 2020.
(28) Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J Reddi, and Sanjiy Kumar. Are transformers universal approximators of sequence-to-sequeen functions? In Proceedings of the 8th International Conference on Learning Representations, 2020.
[29] Toru Ishida and M. Yokoo. Organization self-design of distributed production systems. JEEE Trans. Knowl. Data Eng., 4:123-134, 1992.
[30] Pierre Romelaer. Organization: A diagnosis method. 2002.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes] See Sec. 6.
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix B.
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material as a URL)? [No] We provide the detailed structure of our approaches in Sec. 4 and the benchmarks are open-sourced. We will publish our code after acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix C.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you using existing assets (e.g., code, data, models) curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] The codes we used are open-sourced and have been published in Github.
(c) Did you include any new assets either in the supplemental material or as URL.? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
12

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "Zymlq9s53D", "reviewer": "Reviewer_wAct", "summary": "In this paper, the authors proposed a method called QSCAN to perform value-based decomposition on a subset of agents in multi-agent reinforcement learning. For the given task, the idea is to arrive at weights by which Q-functions are scaled to get an idea of sub-team coordination. The proposed technique is tested on different environments and obtained good performance when compared with different existing methods. ", "questions": "Please refer to my points in the weakness of the paper. ", "limitations": "There are no limitations addressed in the proposed approach. There are some typos in the paper which the authors need to correct. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is structured well and written well. The paper deals with an interesting problem that I believe no one is dealt with before. The problem is of good importance in industries where there will be multiple tasks and only a subset of agents participate in a single task. However, below are some of the points which I think may improve the paper. \n\n1. How the weights $\\lambda_i$ are learned over time. Do authors use the regular experience replay buffer for this case? I suggest incorporating more details here. \n\n2. What about the scalability of the proposed approach? I see the computational complexity is so enormous (especially when the number of combinations to try out is huge). \n\n3. I am confused on the execution part other proposed approach. Are weights calculated for every time step? \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RMc66WOekwM", "reviewer": "Reviewer_3a1Z", "summary": "This paper proposes QSCAN, which considers local coordination within a subset of the agents and dynamically solves each task by sub-team coordination. In order to incorporate sub-team coordination, QSCAN extends the monotonic mixing network in QPLEX and achieves the full representational capacity of IGM. The experimental results showed that QSCAN can outperform QPLEX and QMIX in a number of benchmarks, especially Predator-Prey and Switch4.\n", "questions": "1. What is the main benefit of using (1) sub-team coordination in general and (2) QSCAN in particular?\n2. What is the main difference between QSCAN and VAST ?\n3. How does QSCAN overcome the relative overgeneralization mentioned as an issue for QMIX?\n4. Right before Proposition 1, what is the meaning of “satisfy the IGM Condition consistently during the course of training”?\n5. Can we really say QSCAN significantly outperforms the benchmarks in the matrix game? Also, what do you mean by poor generalization for QPLEX in 5.1? QPAIR seems to have the same performance as QPLEX in the Matrix Game, around a reward value of 22. Thus, I think it would be misleading to say pairwise coordination patterns provide better generalization. Also, I am not convinced that the reward difference between 22 (QPAIR/QPLEX) and 23 (QSCAN) is significant. Is there a crucial point worth mentioning for this gap?\n6. In the matrix game, what is the k used for QSCAN? Why is it able to perform (slightly) better than QPLEX/QPAIR? If the grouping is 2 sub-teams with 2 agents and 1 agent for QSCAN, are there some groups (e.g. player 1-2, 3) that can do well while other groupings don’t? \n", "limitations": "The authors outline the limitations of the work in the Conclusion (Section 6). I agree that going beyond advantage factorization is one of the main limitations in this work because QSCAN as it stands is somewhat limited by IGM and QPLEX. \n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The authors provide a good motivation for coordinating sub-teams, which can be beneficial for tasks where agents should divide into groups and focus on solving each sub-task. \n- The paper provides a nice sub-team generalization of some MARL algorithms where  QPLEX can be interpreted as QSCAN_n and QMIX as QSCAN_0. They provide a thorough theoretical analysis for various properties of QPLEX and QMIX such as the equivalence between QMIX and IGM within individual sub-team assignments (Proposition 1). \n- The experimental results for Predator Prey and Switch4 are interesting. The environment states and attention maps given in Figure 6 provide interesting insight into the potential reasons for choosing a particular action. This can be viewed as one evidence for communication among agents via actions\n\n\nWeaknesses:\n- There is a lack of adequate comparisons with relevant benchmarks in sub-team coordination, especially VAST[18 (supp.)], which also proposes variable sub-teams beyond pairwise coordination and retains the IGM consistency. Sub-team coordination itself is not a novel concept, and I believe it is necessary to emphasize the differences between QSCAN, VAST [18], and other sub-team formulation algorithms (if any), both qualitatively and quantitatively as benchmarks.  VAST [18] is also working in cooperative MARL so it could be misleading to categorize them in the same group as the ones using coalitions and Shapley values.\n- Related to the first point, the paper has a brief discussion of GNN-based approaches which are a different approach with similar motivations. The authors write in the supplementary material A that with GNNs, “it is difficult for them to extract hierarchical coordination patterns.” However, this claim was not verified in the experiments. \n- With the lack of benchmarks against similar sub-team formulation algorithms and GNN-based ones, it is hard to agree with the statement that QSCAN “significantly outperforms” other benchmarks in the predator-prey and Switch4 tasks. Moreover, QSCAN only performs on-par with QPLEX on the SMAC tasks, and the improvement over QPLEX in the matrix game also appears marginal. So overall, the experimental results are not entirely convincing.\n- There is mention of “relative overgeneralization”, “credit assignment”, “faster learning” scattered around the paper, but it is not clear what the main problem is that QSCAN aims to solve. I understand the intuitive motivation for sub-team formulation, but there is no characterization of the types of tasks in which the proposed algorithm is beneficial.\n- Related to the previous point, the illustrative example given in Figure 1 is confusing since it only shows an example of when a sub-team is sufficient for solving the task. On the other hand, the experimental results seem to suggest that the main point of using sub-team formulation is to have a more flexible algorithm which can solve tasks which QPLEX/QMIX cannot, i.e. tasks in which sub-team formulation is necessary. Again, the main argument of the paper is ambiguous, especially with respect to why the extension of QPLEX is specifically the better method.\n- VAST [1] shows that sub-team formulation can be useful for when the number of agents is sufficiently large. This is not necessarily a weakness of QSCAN but again, the main benefit of using QSCAN is not entirely clear. The benefit of using sub-team coordination (e.g. scalability to # of agents, fast convergence, etc.) and the benefit of using QSCAN in particular has to be properly addressed.\n- The algorithm is derived from a fairly simple extension of QPLEX, where the architectural figures look almost identical. \n- (Minor) Figure 6 is ineligible without zooming in.\n\nEDIT: Much of my initial concerns in the review were clarified by the responses by the authors. The authors have shown that the flexible sub-team coordination is a novel problem which was not tackled in previous work. Furthermore, they have added additional experiments to show that the performance is comparable to VAST and scalable to a larger number of agents. To reflect this, I’ve raised my overall score (3→5) as well as the soundness (2→3) and contribution scores (2→3). \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NXY8ZlxtkNp", "reviewer": "Reviewer_MzdJ", "summary": "The paper proposes a hierarchical approach to value function factorization called QSCAN, where the multi-agent team is divided into sub-teams of fixed size, which solve different sub-tasks, in order to exploit local coordination. The architecture of QSCAN is based on QPLEX, where a sub-team coordination module determines the contribution weights of each agent for all potential sub-teams. The paper presents two instantiations of QSCAN called QPAIR, enumerating all sub-teams of size 2, and QSCAN, using self-attention on the agents themselves, which are evaluated against QMIX and QPLEX in various benchmark domains.", "questions": "- Line 225: Is the importance weight $g^{\\textit{ST}}_i$ assumed to be non-negative?\n- How many training runs were conducted per task? I could not find any information in the paper or appendix.", "limitations": "The provided checklist refers to Section 6, but I did not find any explicit limitations. There is a statement about a challenge of designing more effective hierarchical structures but the actual limitation is unclear to me, given that the results are mostly in favor of QPAIR/QSCAN anyway.\n\nPotential negative societal impact is not discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- **Originality**: A similar idea about sub-team based hierarchical value factorization was published in NeurIPS 2021 [1]. However, the introduced coordination hierarchy with QMIX and QPLEX as special cases is interesting and novel in my view.\n- **Quality**: The paper seems technically sound and the performance of QSCAN is supported in the experiments. The number of agents in all benchmarks is rather small, which would make an evaluation with $k > 2$ teams feasible. Regarding the claim about focusing on local coordination, I recommend to evaluate in SMAC scenarios with > 10 agents like ``bane_vs_bane`` rather than ``2c_vs_64zg``, where the single sub-team is obvious.\n- **Clarity**: The approach is well explained. I did not check the attention heat-maps in Figure 6 because the plots are too small, making it very hard to follow the descriptions in the text.\n- **Significance**: QPAIR and QSCAN seem to clearly outperform QMIX and QPLEX in most settings. However, I could not find any information about the number of training runs. The error/deviation bars in Table 2 regarding the SMAC results are missing. Thus, I cannot assess the significance of the results.\n\nMinor comments:\n- Line 72: QPAIR is mentioned for the first time without further description, which is confusing. The QSCAN variant should be named differently from its actual framework as it is not obvious that QSCAN is used in different contexts in the same parapgraph.\n- Line 91: $r: \\mathcal{S} \\times \\mathcal{A} \\rightarrow \\mathbb{R}$ denotes __the__ global reward function\n- The text and labels on all figures are very small and hard to read when printed (the white texts in Figure 4 are completely unreadable)\n- Line 283: „several state-of-the-art MARL approaches“ are just two approaches: QMIX and QPLEX\n\n[1] Phan et al., \"VAST: Value Function Factorization with Variable Agent Sub-Teams\", NeurIPS 2021", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1XGhR5XgNZf", "reviewer": "Reviewer_f854", "summary": "Authors present QSCAN, an attention-based, sub-team-level organization, task allocation, and credit assignment algorithm to outperform certain special cases of MARL algorithms that fall under the sub-team organization, namely, QMIX (k=1) and QPLEX (k=n). QSCAN is evaluated in three environments against other factorization methods and shows better performance consistently.", "questions": "Q1: How important is the addition of self-attention? It would be nice if the authors could share some ablated evaluation results, which are probably available if they begun initially without attention.\n\nQ2: What is the effect of altering k but for the same evaluation setting? What is the (expected) effect of forming different-size sub-teams?", "limitations": "Please have a look into the effects of different k as well as the effects of attention. Other than that, the limitations of QSCAN have already been clearly stated in Section 6.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "QSCAN ties in well with the literature, and the evaluation is carried out under varied settings.\n\nThe paper's weakness lies in its lack of proof for the initial claim of more effective factorization and faster learning. Sub-team organization and attention are additions to the computational complexity, so the number of steps in Fig. 5 cannot vouch for \"faster\" learning. For the SMAC evaluation, even the number of steps data is omitted for reasons unknown. On a separate note, some explanation on role assignment line of MARL research (e.g., RODE) could make the analyses more comprehensive, as role assignments (with no explicit team formation) and sub-team formation (with no explicit role definition) could be positioned interestingly as duals.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept