Automatic Grouping for Efficient Cooperative Multi-Agent Reinforcement Learning
Authors: Yifan Zang
Jinmin He
Kai Li
Haobo Fu
QIANG FU
Junliang Xing
Jian Cheng
Venue: NeurIPS 2023 poster (2023)
Keywords: MARL
Cooperative Multi-Agent Reinforcement Learning
Coordination and Cooperation
Automatic Grouping
Group-Wise Learning
TL;DR: This paper presents GoMARL, a domain-agnostic method that learns automatic group division for efficient cooperation by promoting intra- and inter-group coordination.

Abstract: Grouping is ubiquitous in natural systems and is essential for promoting efficiency in team coordination. This paper proposes a novel formulation of Group-oriented Multi-Agent Reinforcement Learning (GoMARL), which learns automatic grouping without domain knowledge for efficient cooperation. In contrast to existing approaches that attempt to directly learn the complex relationship between the joint action-values and individual utilities, we empower subgroups as a bridge to model the connection between small sets of agents and encourage cooperation among them, thereby improving the learning efficiency of the whole team. In particular, we factorize the joint action-values as a combination of group-wise values, which guide agents to improve their policies in a fine-grained fashion. We present an automatic grouping mechanism to generate dynamic groups and group action-values. We further introduce a hierarchical control for policy learning that drives the agents in the same group to specialize in similar policies and possess diverse strategies for various groups. Experiments on the StarCraft II micromanagement tasks and Google Research Football scenarios verify our method's effectiveness. Extensive component studies show how grouping works and enhances performance.

Automatic Grouping for Efficient Cooperative Multi-Agent Reinforcement Learning
Yifan Zang Jinmin Hel Kai Li Haobo Fu², Qiang Fu², Junliang Xing", Jian Cheng" 12-
Institute of Automation, Chinese Academy of Sciences
Schooo of Artificial Intelligence, University of Chinese Academy of Sciences
Tencent Al Lab, "Tsinghua University
{zangy fan0019 hej immin2021 kai. .11)@ia.ac ac. cn, (haobofu, Leenfu com, jixing@ts inghua. edu. en, cheng@nlpr iia. iaaaa.ac...o ac.
Abstract
Grouping is ubiquitous in natural systems and is essential for promoting efficiency in team coordination. This paper proposes a novel formulation of Group-oriented Multi-Agent Reinforcement Learning (GoMARL), which learns automatic group- ing without domain knowledge for efficient cooperation. contrast to existing approaches that attempt to directly learn the complex relationship between the joint action-values and individual utilities, we empower subgroups as a bridge to model the connection between small sets of agents and encourage cooperation among them, thereby improving the learning efficiency the whole team. In particular, we factorize the joint action-values as a combination of group-wise values, which guide agents to improve their policies in a fine-grained fashion. We present an au- tomatic grouping mechanism to generate dynamic groups and group action-values. We further introduce hierarchical control for policy learning that drives the agents in the same group specialize in similar policies and possess diverse strategies for various groups. Experiments on the StarCraft II micromanagement tasks and Google Research Football scenarios verify our method's 's effectiveness. Extensive component studies show how grouping works and enhances performance.
Introduction
Cooperative multi-agent reinforcement learning (MARL) aims to coordinate multiple agents' actions through shared team rewards and has become helpful tool for solving multi-agent problems [47] [44) Learning centralized policies to address this problem conditions on the full state, which is usually unavailable during execution due partial observability or communication constraints 3 An alternative paradigm to learn decentralized policies based on local obser- vations 36 However, simultaneous exploration suffers from non-stationarity that may cause
unstable learning 7 [50] The centralized training with decentralized execution (CTDE) inherits the advantages of these two paradigms and learns decentralized policies in centralized fashion [23 15]
Value Function Factorization (VFF) under the CTDE paradigm a popular approach to MARL, where centralized value function is leamedd from global rewards and factorized into local values to train decentralized policies. In recent years, value factorization methods have been widely proposed. Existing methods usually adopt the flat VFF scheme (24). which directly estimates the joint action- value from local utilities. These methods achieve remarkable performances from various perspectives, such as enhancing the expressiveness of the mixing network [27
33 38 37) and encouraging
exploration 21 19. However, learning efficient cooperation by directly estimating the joint action- values from individual utilities is exceptionally difficult. Phan al. 24 illustrate that the flat VFF
"Conessonding authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

scheme leads to performance bottleneck, where it gets challenging to provide sufficiently informative training signals for each agent. Although the state information provides complete knowledge, is burdensome for agents to extract effective guidance that facilitates cooperative policy learning.
Research in natural systems K 4 and multi-agent systems 24 has validated grouping as a means to promote efficient collaboration. Dividing the team into smaller sets allows for fine-grained learning and opens up further opportunities to integrate informative group-wise learning signals. However, formulating a general division criterion without domain knowledge is still matter of interest to the community. Most previous works are proposed for well-structured tasks and typically predefine specific responsibilities or forms of task decomposition [8, 224). They require apriori settings that are potentially unavailable in practice and discourage methods' transferring to diverse environments.
This paper proposes a novel formulation of Group-oriented MARL (GoMARL), which learns au- tomatic grouping without domain knowledge for efficient cooperation. GoMARL holds dual- hierarchical value factorization and learns dynamic groups with a "select-and-kick-one Concretely, GoMARL continuously selects agents unsuitable for their current groups based on the learning weights of the decomposition from the group-wise value to local utilities and kicks them out to reorganize the group division. Furthermore, GoMARL transforms various informative training signals, including individual group-related information, group state, and global state, into network weights, which extracts effective guidance for policy improvement and enables flexible adaptation to the dynamic changes in the number of subgroups and the number of agents per group.
scheme.
The advantages of GoMARL lie in two aspects. Firstly, group information offers richer knowledge for group-wise value factorization and provides more direct guidance for efficient learning. Our value more focused and compact input representation. Unlike existing methods that learn value factorization with no information or only with the global state which is complete but hard to extract efficient guidance, GoMARL proposes a fine-grained learning scheme to integrate information from the group perspective into the policy gradient, promoting intra-group coordination with the group state and facilitating inter-group cooperation with the global state. Secondly, GoMARL extracts individual group group-related related information to provide informative signals for policy diversity. The agents condition their behaviors on their individual group-related information embedded by shared encoder, which is trained following specialization guidance. i.e., imposing similarity within group and diversity among groups. In this way, GoMARL synergizes subgroups with policy styles, proposing a seaameter-shringg mechanism for specialized policies.
factorization with dual hierarchy is based
We test our method a challenging set of StarCraft II micromanagement tasks 29 and Google Research Football scenarios [16). GoMARL achieves superior performance with greater efficiency compared with notable baseline methods. We also conduct detailed component analyses and ablation studies to give insights into how each module works and enhances learning efficiency.
2 Related Work
Value function factorization under the CTDE 23 paradigm is popular approach MARL. Most existing methods learn flat value factorization by treating agents as independent factors and directly estimating the joint action-value from local utilities. The earlier work, VDN 35). learns a linear
decomposition into a sum of local utilities used for greedy action selection. QMIX 27 learns a non-linear mixing network with the global state and enlarges the functions the mixing network can represent, but it still faces the monotonicity constraint. QTRAN (33 further improves the expressivity by proposing the addviduall-Global-e the global action-value. Subsequent works follow the IGM principle and further boost performance by encouraging exploration [21 5) enhancing the expressiveness of the mixing network (38) preventing
(IGM) principle between individual utilities and
sub-optimal convergences [26] 37), and integrating functional modules 42 46
45
51
34.44 49
This paper explores automatic group division for multi-agent teams realize group-wise learning. Early related works [30] 28 20 [8] predefine specific responsibilities to each agent based on goal, visibility, capability, or by search. Some other efforts achieve implicit grouping by task allocation [7 [0). These methods only address tasks with clear structure and require domain
[7
knowledge or apriori settings. Another class of approaches focuses on individuality [41 19 [4] or
role learning 33 40 Among them, ROMA (39 learns dynamic roles that depend on the context
agents observe. RODE 40 decomposes the joint action spaces and integrates the action effects into
the role policies to boost learning. A similar work, VAST (24]. also studies the impact of subgroups
2

---

on value factorization but still requires a priori of group number. Our method, GoMARL. does not rely on domain knowledge and gradually adjusts the group division according the learned factorization weights. Based on the dynamic group division, GoMARI. models several group-related signals to learn specialized policies and promote efficient team cooperation.
3 Preliminaries
Dec-POMDP.
This paper focuses on cooperative tasks with n agents A = {a),..., an as a Dec- POMDP 22 defined by tuple G (S U, Prr, Z, O, n, 7). The environment has global state s S S. Each agent a chooses an action 2n from its action space Ue at timestep t and forms a joint action u E (U1 X.. x Un) = U that induces a transition according to the state transition distribution P(s uz) S x U S + [0, 1]. r(s, u) S + IR is the reward function yielding a shared the discount factor. We consider partially observable scenarios in which agent a acquires its local observation 2t E % drawn from ((s,,a) S x A - Z and has an action- observation history 7 E T== x Z)* on which conditions a policy n (u* (r"): T TxU + [0, 1].
reward, and y E [0, 1)
Value Function Factorization and the IGM Principle. We consider cooperative MARL with the CTDE paradigm, which has been a significant focus in recent efforts 22 35
27
12
21. A
majority of methods achieve CTDE through flat value function factorization 24 i.e. factoring
oetion-value functions into combinations of per-agent utilities. The individual utility only depends on
the local history of actions and observations, allowing agents to maximize their local utility functions independently under the Individual Individual-Global- Global-Maa (IGM) Principle (33]: arg max, Q7 (T, u)
(arg max,, (T',), w²) w²),... .')..... ..... arg max," @""r", }² Among these attempts, the representative
deep MARL approach QMIX (7 improves the simple summation of individual utilities (35) by
introducing more expressive factorization: Qiot = f(Q' ; (Q)..... Q""(", (r",,"" u": 0
where f denotes the parameters of the monotonic mixing function generated by hypemetwork 66.
4 Group-oriented Multi-Agent Reinforcement Learning
Definition 1. (Individual and Group) Given a cooperative task with n agents A = {a), ..., ) we have a set of groups 9 = {91, ..., Jm ). 1 S m n. Each group 9j contains n, (1 < nj n) different agents, 9j- = {a,,...... a,,, } S A, where Uو = A gy 7 gle = Ø for j. k E (1,2,... m] and * k. The superscript describes the variable owner, e.g., wi: is the action of the i-th agent aj, in group Øj- We denote joint quantities in bold and joint quantities over agents other than a given agent a with the superscript -a, e.g., u i: is the joint action of agents in group Jj other than agent aj,-
GoMARL decomposes the global action- Q into group- wise values Q" and trains agents by groups in a fine-grained manner. F iilustraaes the overview of the learning frame- work. It consists of an automatic grouping module, special- ized agent networks generating local utilities and and mixing network among groups. In Section 4.1] we introduce the au- tomatic grouping module. It progressively divides the team into dynamic groups as training proceeds. Based on the dy- namic group division, we propose specialized agent networks that achieve similarity within each group and diversity among groups to generate local utilities Q' in Section 4.2 Section 4.3 presents the overall training framework to detail the estimation of the group-wise values and the global action-values.
Qzot
Mixing Network among Groups
Q grrupp
Q 92 group
Q
Oml group
m dynamic groups
Automatic Grouping Module
('rr)) (?.))
("((",)
Figure 1: Overview of GoMARL.
4.1 Automatic Grouping Mechanism
The automatic grouping mechanism aims to learn a mapping relationship fg A + G. The key idea is to divide the team into dynamic groups in an end-to-end fashion by maximizing the expected global return Q (s,, u,) = E., 11o...l 1:00 - - & uy; G]. Value function factorization approaches represent the joint action-value as an aggregation of individual utilities, i.e., a weighted sum of Q' and biases. We follow this setting and represent the group-wist value Q" as an aggregation
of the individual value Q'. Intuitively, illustrated in the right side of Figure 2 if the learned mixing
3

---

Initial grouping 9 [m: "i.""
If vecter w! - 0. 0' shows a little contribation to Q
concal w| accord nng to grouping 5
0x0
0x
0x0
k hidden values
send to correspooding wi
Group Adjustment Operator
I
local Q-values
W2
(a.) A grouping [1 "..4 "111
Individual W Generator
The grouping updates every timestipp.. More than one agent can be selected at one time, and the selected agents are ussigned to the next group,
ML.P
GRU
g" = [51.82) = ((R-.ga) [a, 4.1]. 9 [m.
[[--- / (3, a. (a, m.).
(3:44-17 M;-1)* MLP
Actor
Actor
Actor
Until a stable (0x, Bmn aequired.
Figure 2: The left side illustrates the schematic diagram of the grouping mechanism. The group adjustment operator concatenates the weights wi of agents in the same group to form the group-wise weights w The right side gives an example to show how grouping changes during training.
weight of Q' is small enough, then Q' contributes little to its group-wise value Q". In other words, when an agent 0, takes action wh but does not affect its group-wise value Qo: indicates that agent aj, does not belong to group 9j anymore and needs group adjustment.
We factorize the group value Qa into individual utilities Q' with group-wise mixing weights Wis Le., Qa) = f(Q' (r",,,")'..... "....."Quh Q (r"n;, w²ns) ); w(r). where w denotes group gj's mixing parameters generated by individual Wi generators füg = history information r' to a k-dimensional weight vector. A regularization on this W1 generator fw, drives the automatic grouping mechanism. Specifically, GoMARL "selects and kicks out" agents whose individual utilities hold small mixing weights and contribute a little to their group-wise Q values; thus, a sparsity regularization on W1 implemented select those agents with small group- value mixing weights. The W1 generators fú, are trained by minimizing the following loss function:
ud)),....
-
that map each agent a,'s
bg (0w,) = B. E ((I (7' ( (2,) lls)),
where B is the replay buffer, and || ||i, stands for the l no-rr penalty.
Figure Dillustrates the schematic diagram of the grouping mechanism. Initially, all the agents belong to the same group, and the grouping G is adjusted as training proceeds. The grouping shifts every e timesteps, and each selected agent assigned to the following group (a new one for agents in the last group) until it properly contributes to where it belongs. When adjusting groups, the of all agents are sent to group adjustment operator Og, According to the adjusted grouping G.
Og: (w!,-.. ... wT] g [wi':-. .... w(m) concatenates the w of agents in the same group to form a set of group wise w to generate group action-value. This implementation flexibly adapts to the grouping dynamics since each w is tied to agent a, by the individual Wi generator füy. No matter which group 9j agent a; belongs to, Q' can engage in the estimation of Q5: through w E w
In practice, the weights' shrinkage to zero is infeasible. It is also challenging to determine fixed threshold for groups of various sizes in diverse environments. We empirically utilize seventy percent of each group's 's average weight to assess whether an agent fits its current group. Extensive experiments and component studies in Section confirm the universality of this configuration.
4.2 Specialized Agent Network for Decentralized Execution
Group status describes the cooperation among agents within each group and is essential to local utility generation. Integrating group-wise information e into the decision- mrdisinnmmainng process enables consideration of cooperative behaviors. In this section, introduce GoMARL's agent network to generate local action-value functions with individual group-related information e embedded.
As shown in Figure we construct group-related info encoder Sa (:; 0.) to embed agents' hidden states. To achieve a group-related view, we train the encoder network as an extractor, where the extracted agent info of agents from the same group should be similar. To avoid all agents' et

---

collapsing to be alike, the regularizer also encourages diversity between agents from different groups. Formally, we minimize the following mmmilarity-iveersi objective to train the encoder fa:
LSD (0c) = Es
Σ
[i. cosine (f.
fc(h);
ig
(2)
where [(i,)) = -1,
a4, aj E 9k.
E 9h, Oj yt, k +l.
The encoder trained by the SD-loss extracts agent info e' that is recognizable to agents' groups. This group-related information then fed into a decoder fa (+; 0u) to generate the pa- rameters of the agent network's 's upper MLP. In this way, each agent ag conditions its behav- iors on e with group-related information em- bedded, promoting potential cooperation within each group during decentralized execution. The decoder hypernetwork fa is trained by the TD- loss introduced in Section 4.3
Q"(r", u², e
TD Loss
SD Loss
MLP
Group- related Info
Agent Info of of
Group- related Info Encoder
PA
Decoder
GRU
kg
AE-1
Thila Flow
MLP
Gradient Produce
Actor
..
Actor"
(xe) uf-1)
Figure 3: The specialized agent network.
The proposed actor network has two merits. Firstly, it enables diversified policies while sharing all the parameters. Policy decentralization with shared parameters widely utilized to improve scalability and learning efficiency. However, agents tend to behave similarly when sharing parameters, preventing effective exploration and complex cooperative policies. also undesirable to entirely forgo shared parameters in pursuit of diversity since proper sharing accelerates learning. Our method hybridizes the efficiency of parameter-sharing and the policy diversity needed for complex collaboration. Secondly, the decoder hypernetwork fa integrates the extracted agent info into the policy gradients, providing informative group-related information to enrich local utilities and promote intra-group cooperation. Concretely, the partial derivative for updating the parameters 8p. of the GRU and the bottom MLP is:
aQtot aQtat aQ aQtat aQ Oog a0, aQ 00, aQ Oug, 00,
aQuot dog
fa(c")
aQu 00,
where U) is the representation after the GRU illustrated in the left side of Figure 3 Eqn. 3 shows that e" is deeply involved in the policy updating of agent a, providing richer group knowledge to generate local utilities and facilitating group-related guidance for cooperative decentralized execution.
4.3 Overall Learning Framework
We next introduce GoMARL's overall learning framework. It contains two mixing networks that generate the group-wise Q" and the global Qtat, respectively. As for the group mixing net- work, although w enables weighted mixing of local utilities of agents in group g. the naive mixture of >'e,, w"Q" + b lacks the guidance of group status reflect the group action-value in a specific group-wise state. Therefore, we build the group mixing network with a two-layer structure that also embeds the group- group-wise state into the weights w to generate Q". particular, group 9, 's group-wise state gdt is a fusion of the agent info et for all a; 9j- To cohesively summarize the group state based on the agent info of all agents in group 9j. we apply a pooling operation [25] over each dimension of the agent info e' to generate the group-wise state sh; describing the current group status. The pooling operation also ensures adaptability to the dynamic group size (i.e.. the number of agents per group). We build a group-wise W2 generator fw)(s)) to map the fused group state into w Similar to the group-related info encoder fe, the W2 generator integrates the group state 3 into the policy gradient:
2
Mixing
NLP
Greeyin Group Afjustment Operatnr
M; Generator
Group-wise My Generator
MLP
Actort
Agent lnfe
Pholing Qpanation
Croup State
( (R
Bronder
o), my
8
Figure 4: Overall learning framework of GoMARL.
aq aQ āug
Ovg (59). aq" providing group status information for Q9 generation that facili-
eQ= dog aQ""
tates efficient intra-grouy group cooperation. UQ is the representation after wi marked in Figure 4 The
5

---

two-layer mixing structure of the group mixing network estimates the group action-value Q9 by Wi which decides the group division and W2 which carries group status information.
GoMARI estimates the global action-value Q by mixing all the Qo in a similar fashion. Concretely, the two layers of the total mixing network generated by two hypernetworks, respectively taking group states 8 and the full state as inputs. Likewise, 8 and s are deeply involved in the gradients, promoting inter-group cooperation. The architecture of the total mixing network is akin the group for brevity. The TD-loss of the estimated Qiot is:
mixing network and is omitted in Figure 4
(DD EB (r+ (r+ 7 max Qhat (5', - Qtot (s. u)
(4)
ur
where Qtat is a target network with periodic updates. The overall learning objective of GoMARL is: [( (0) (Ow.) + AsDCsD(Oe),
(5)
where 0 = (0,, lz. 0d. Bw). .. Ow denotes the parameters of hyperetworrs producing all the mixing weights and biases. Ay and ÀSD two scaling factors.
Although containing two mixing networks, the total mixing-net size of GoMARL is smaller than the
commonly used single monotonic mixing network [27], as verified in Section 5.1 This is mainly attributed to the input dimension reduction. The monotonic mixing network takes the glohal state s as imput of all the hypernetworks. In contrast, group-related info e, group state 89, and state s is only used in one hypernetwork). This parameter reduction offsets the increase of an extra mixing network. Compared with flat value factorization methods learning only with the global state, our method allows fine- ane-grained learning guided by more direct signals embedded in the policy gradients, facilitating intra- and inter- inter-group group cooperation.
take specific group information (i.e., individual
5 Experiments
Baselines. We compare GoMARL with prominent baselines to verify its effectiveness and efficiency. Hu et al. [8] fairly compared existing MARL methods without code-level optimizations and reported that QMIX (27) and QPLEX (38] are the top two value factorization methods. The authors also finetuned QMIX (denoted as Ft-QMIX in our paper), which attains higher win rates than the vanilla QMIX. VAST 24 learns value factorization for sub-teams based on apriori setting on group number. Therefore, we compare GoMARL with Ft-QMIX, QPLEX, and VAST show its performance as a value factorization method. The baselines also include role-based methods ROMA [39] and the representative credit assignment method RITT []. The latter combines effective modules of noticeable methods, and comparing them can further illustrate the superiority of GoMARL.
Experimental setup. All the methods are trained with 8 parallel runners for 10M steps and evaluated every 10K steps with 32 episodes. We report the 1st, median, and 3rd quartile win rates across 5 random seeds. Please refer to Appendix [ for experimental setup details.
5.1 Performance on SMAC
Methods are evaluated on six challenging S.MAC maps. The chosen maps involve homogeneous and heterogeneous teams with asymmetric battles, allowing holistic study of all methods.
Overall performance. The performance comparison of all the methods in the six Hard and Super Hard SMAC maps is shown in Figure 5 As the results show, each baseline method only achieves satisfactory performance on some of the tasks with specific properties they specialize in; e.g., RIT performs well on MMM2 but converges much slower in other tasks; QPLEX's 's leaning is not efficient in 8m_ ___ and corridor. Ft-QMIX significantly outperforms the vanilla QMIX and has more efficient learning than other baselines. GoMARL has similar performance as Ft-QMIX in 8m_ _n_ve_9m 9m and corridor. However, the superiority of GoMARL can be clearly validated in all the other maps.
Parameter size for value mixing. Many methods use larger mixing networks with stronger expres- siveness and fitting abilities to obtain superior performance. However, they often fail when compared to baselines that use a mixing net of the same size [ As shown in Table our dual-hierarchical mix- ing architecture has fewer parameters when there are large number of agents (when n > 5). VAST is not compared since it utilizes a linear summation of local utilitics. The results of GoMARL are the

---

QMIX
PL-QAIIX
OPLEX
ROMA
RITT
VAST
GoMARL
2
Teo Win Itte (%)
Tie Win Raate (%)
The
Test Win Itte (1)
Text Win Rtate (%)
EXT(P)
Figure Comparison of GoMARL against baseline algorithms on six SMAC maps.
average of the five runs since each run may Iearn slightly different grouping with various group numbers. GoMARL outperforms other methods despite using fewer mixing parameters, highlighting its inherent superiority over methods relying on stronger mixing networks.
Table 1: Size comparison of all methods' mixing network(s)
Maps
(Ft-)QMIX
QPLEX
ROMA
RITT
GoMARL
3s_vs_5z
21.601K
72.482K
13.281K
37.986K
26.530K
5m vs 6m
31.521K
107.574K
25.377K
51.362K
31.554K
8m vs 9m
53.313K
197.460K
63.393K
93.986K
51.427K
corridor
68.929K
303.808K
81.537K
122.882K
53.859K
MMM2
84.929K
342.248K
134.401K
177.282K
74.244K
3557_v8_3sd 3863
63.105K
243.156K
81.345K
118.466K
61.028K
Component analysis and ablation study. We next conduct detailed component studies to analyze how GoMARL improves efficiency and enhances performance. GoMARL contains three key compo- nents: (1) an automatic grouping mechanism that progressively divides the team into proper groups; (2) specialized agent networks that generate diversified policies while sharing all the parameters; and (3) sufficiently informative signals integrated into the gradients to promote efficient cooperation. We respectively study each module on three Super Hard maps to show how they influence performance.
(1.1) Ablation study of the grouping mechanism. We validate our grouping mechanism by compar- ing GoMARL with other intuitive alternatives. All methods utilize the same architecture GoMARL to reflect how grouping itself influences performance. As shown in the top row of Figure O setting all agents as a group in corridor converges faster but has significant variances. may be due to the hard exploration of efficient cooperation without grouping guidance. Another two intuitive groupings, each agent a group and an equal division into two groups {{a1, 02 02,43), as). {ag. a5, ds) have minor variance. However, their learning efficiency is affected since inappropriate groupings fail to promote cooperative behaviors. MMM2 contains heterogeneous agents; thus, a natural grouping is to keep the agents of the same type in one group. This natural grouping outperforms most alternatives but is inferior to our dynamic grouping adjustment. 3s5z_ve vs_ _s_666 is another scenario with heterogeneous agents, and the natural grouping of setting homogeneous agents nearly the same as GoMARL because our mechanism learns the grouping precisely according to the agents' type in this scenario. We also analyze this map with grouping containing three groups (fai, a., as ) {az, a6 a7 { (as, og) }} (1s2z, 1s2z, 1slz); however, this balanced grouping fails to form effective collaboration. We see from these results that appropriate grouping facilitates efficient cooperation and accelerates learning. Our grouping mechanism automatically learns adaptive grouping in different tasks and assists GoMARL with superior performance and efficiency.
group is also studied. It performs
(1.2) Learned grouping analysis. To demonstrate whether the learned grouping makes sense, we further visualize the trained strategy in corridor battle, illustrated in Figure 7 Six allied Zealots
7

---

Cumpoornt study on corridor for BM stepa
Compunent study on XDMM2 for BM steps
Compaunent study un Jaka, vo
TIst Wio Rate
pas
(12
That Win Hate
proe (11
Srack
Test Wit
Abiation on corridor for BM steyo
Abloisee on MMM2 for SMI steps
Ablation on Joba va SoBe
116
Test Win Bate (%)
Test Win Haro (HL
Test Wit Itar
Steps (XII))
Stepo (xI0P)
Steps (x327)
Figure 6: Ablations of the grouping mechanism (top row) and the group-related signals (bottom).
Agont
18 Enemies
Agent 1&2 died,
Agent 3:6
Agent 3-5 survived
Agent 1
Eaemies
Agest
Agent 3-6 fight as group kill survis is ing unemits.
Agent 3-6
Agent 3-0
Figure 7: The learnt groups (2),,, as, as, 0s) }} is explicable to the corridor combat situation.
fight twenty-four Zerglings on this Super Hard map. The massive disparity in unit numbers between the two sides implies that the whole team must refrain from launching an attack together. The only winning strategy is sacrifice a small number of agents who leave the team and attract the attention of most enemies. Taking this opportunity, our large force eliminates the rest of the enemies. The surviving agents then use the same tactic to keep attracting several enemies and kill them together. In this battle, Agent 1 and 2 sacrifice themselves to attract most enemies and bring enough time for the team to eliminate the remaining enemies. Other agents fight as subgroup and successfully kill all the surviving enemies. GoMARL learns a double-group group setting in this battle, where Agent and 2 are in the same group while the others are set in another group. This grouping explicable in light of the combat situation, and this reasonable grouping guidance contributes to our superior performance.
(2) Component study of the specialized agents. We transplant our specialized agent network (SAN) into other basclines to verify module effectiveness. ROMA is not included since its actors are produced by its learned roles, and the replacement will invalidate the method. As shown in Figure X all methods are improved when equipped with SAN. However, even with SAN, all baselines fail to surpass GoMARL. As shown in the learning curves in Appendix efficiency in the early stage of training, the learned grouping will significantly accelerate learning in the later stage. Therefore, the automatic grouping module and SAN are both crucial to GoMARI.
Component Spectaliand Agont
BITT. SAN
VAST
GUSTAZE
Figure 8: Method performance with and without SAN in 3s5z_ ___r___________ ___66666 after training 8M steps.
C although dynamic grouping may reduce learning
(3) Ablation study of the informative group-related signals. Our method models individual group- related information and group state s to integrate them into the gradients. The former adjusts local utilities for cooperative behaviors and realizes policy diversity, while the latter fosters efficient group cooperation. We ablate them respectively to validate their effectiveness. Concretely, we replace the group state ' with the full state s and remove the group-related information e to degenerate the specialized agent networks into vanilla agent networks. As shown in the bottom row of Figure O both signals greatly improve learning efficiency. On the one hand, the agent info e carrying group-related information is trained to encourage inter-group diversity, enabling extensive cooperative exploration and learning acceleration. On the other hand, the fused a summarizes informative group-related status that facilitates efficient group-wise coordination. Although the global state provides complete
8

---

FL-QMIX
RITT
academy paso and aBoutt with Keeper
aradhemy 3. Aeeruer
aradhmmy poemderatiark
Tol Score Rewand (16)
Tor Score Rewarr (56)
5 9 3 9:2
Tol Rewarr
Stagu (x:1
Stops Es (:P)
Steps (Nitf))
Figure 9: Performance comparison with baselines in three Google Research Football scenarios.
SOORL OWNER
ScORE
Oppanent playet
Agest
Opprecse playir
Anced sheods
Opponent
Agerr
player
AgcuE
Agest 2 securd Sisct
Passel Nacel
Agost
The licepet sEvIS lie Teal
and sccrrs)
Figure 10: The learned groups {1a,, a2), {az, a4}} } is explicable to counterattack situation.
knowledge, it is burdensome for agents extract effective guidance that promotes policy learning. Therefore, utilizing our specific group-wise signals and 8 more efficient than the global state s.
5.2 Performance on Google Research Football
We also test GoMARL on three challenging Google Research Football (GRF) offensive scenarios. Agents in GRF need to coordinate timing and positions for organizing offense to seize fleeting opportunities, and only scoring leads to rewards. Therefore, the GRF tasks are more complicated than the SMAC battles, and the comparison in GRF is a secondary proof of our method's effectiveness.
Performance. Figure 9 shows the performance of all methods in GRF. GoMARL, Ft-QMIX, and QPLEX all perform well in pass ___dd__d_____ shoot shoot_withh tceekeeerr. However, the advantages of our method become increasingly evident in the other two environments requiring Only GoMARL achieves over 50% of the score reward in all scenarios. The superior performance with significant efficiency in the second testbed further demonstrates the transferability of our method.
coordination.
Visualization. The trained strategy counterattack, easy is visualized to validate the learned group division makes sense. GoMARL learned reasonable grouping g {{a1. J. {ag, a1) for this complex goal. in which the first group brought the ball into the penalty area through smooth coordination, while the second group created two shoots and the final goal through skillful cooperation, as shown in Figure I0] Appendix provides a detailed discussion of this visualization.
Conclusion and Future Work
Grouping is essential to the efficient cooperation of multi-agent systems. Instead of utilizing apriori knowledge, this paper proposes an automatic grouping mechanism that gradually learns reasonable grouping as training proceeds. Based on the dynamic group division, we further model informative group-related signals to achieve fine-graine ame-grrined value factorization and encourage policy specialization, promoting efficient intra- and gour- coordination. With these novelties, our method GoMARL achieves impressive performance with high efficiency on the SMAC and GRF benchmarks.
The automatic grouping mechanism of GoMARL is currently grounded in the value decomposition process, rendering it inapplicable policy- based methods. Our future research will delve into automatic grouping based on policy gradients to enhance the learning efficiency of the policy- based methods. Furthermore, the current version GoMARL only designed for one-stage and one-level grouping. and it possesses the potential to achieve finer granularity in group division. We will investigate multi-stage and multi-level grouping (i.e., wroup-within-a-grone in the future to pursue better performance and higher learning efficiency in more complex environments.

---

7 Acknowledgement
This work is supported in part by the National Key Research and Development Program of China under Grant No. 2020AAA01034011 in part by the Natural Science Foundation of China under Grant 62076238, Grant 62222606, and Grant 61902402; and part by the China Computer Federation (CCF))Tennent Open Fund.
References
[1] Noelie Bonjean, Wafa Mefteh, Marie-Pierre Gleizes, Christine Maurel, and Frédéric Migeon. Adelfe 2.0. Handbook on Agent-Oriented Oriented Design Processes, pages 19-63, 2014.
[2] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, pages 2974-2982, 2018.
[3] Sven Gronauer and Klaus Diepold. Multi-agent deep reinforcement learning: A survey. Artificial Intelligence Review, pages 1-49, 2021.
[4] Jayesh K. Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multi-Agent Systems, pages 66 23,83,201 2017.
[5] Tarun Gupta, Anuj Mahajan, Bei Peng, Wendelin Böhmer, and Shimon Whiteson. Uneven: Universal value exploration for multi-agent reinforcement learning. In International Conference on Machine Learning, pages 3930-3941. PMLR, 2021.
[6] David Ha, Andrew Dai, and Quoc V. Le. Hypernetorrss. arXiv preprint 1rXvv:1609.09106. 2016.
[7] Pablo Hernandez-Leal, Bilal Kartal, and Matthew E. Taylor. A survey and critique of multiagent deep reinforcement learning. In International Conference on Autonomous Agents and Multi- Agent Systems, volume 33, pages 750-797, 2019.
[8] Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, and Shih-wei Liao. Riit: Rethinking the importance of implementation tricks in multi-agent reinforcement learning. arXiv preprint arXiv:2102.03479, 2021.
[9] Maximilian Hütteenrach, Adrian Šošic, and Gerhard Neumann. Guided deep reinforcement learning for swarm systems. arXiv preprint urXiv:1709.06011 2017.
[10] Shariq Iqbal, Robby Costales, and Fei Sha. Alma: Hierarchical learning for composite multi- agent tasks. In Advances in Neural Information Processing Systems, volume 35, pages 7155- 7 7166, 2022.
[11] Shariq Iqbal, Christian A. Schroeder de Witt, Bei Peng. Wendelin Böhmer, Shimon Whiteson, and Fei Sha. Randomized entity-wise factorization for multi-agent reinforcement learning. In International Conference on Machine Learning, pages 4596-4606. PMLR, 2021.
[12] Shariq lqbal and Fei Sha. Actor-attention-critic for multi-agent reinforcement learning. In International Conference on Machine Learning, pages 2961-2970. PMLR, 2019.
[13] Raphaël Jeanson, Penelope F. Kukuk, and Jennifer H. Fewell. Emergence of division of labour in halictine bees: Contributions of social interactions and behavioural variance. Animal Behaviour. 70(5): 183))1183-1193, 2005.
[14] Jiechuan Jiang and Zongqing Lu. The emergence of individuality. In International Conference on Machine Learning, pages 4992-5001. PMLR, 2021.
[15] Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing, 190:82-94, 2016.
10

---

[16] Karol Kurach, Anton Raichuk, Piotr Starczyyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, and Sylvain Gelly, Google research football: A novel reinforcement learning environment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4501-4510, 2020.
[17] Youngwoon Lee, Jingyun Yang, and Joseph J. Lim. Learning to coordinate manipulation skills via skill behavior diversification. In International Conference Learning Representations, 2020.
[18) Kemas M Lhaksmana, Yohei Murakami, and Toru Ishida. Role-based modeling for designing agent behavior self-organizing multi-agent systems. International Journal of Software Engineering and Knowledge Engineering, 28(01):79-96, 2018.
[19] Chenghao Li, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang. Celebrating diversity in shared multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, volume 34, pages 3991-4002, 2021.
[20] Kathryn Macarthur, Ruben Stranders, Sarvapali Ramchurn, and Nicholas Jennings. A distributed anytime algorithm for dynamic task allocation in multi-agent systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, pages 701-706, 2011.
[21] Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. In Advances in Neural Information Processing Systems, volume 32,
pages 761 7-11-7 2019.
[22] Frans A. Olichoek and Christopher Amato. A Concise Introduction to Decentralized POMDPs, POMPH volume 1. Springer, 2016.
[23] Frans A. Olichoek, Matthijs T.J. Spaan, and Nikos Vlassis. Optimal and approximate q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289-353, 2008.
[24] Thomy Phan, Fabian Ritz, Lenz Belzner, Philipp Altmann, Thomas Gabor, and Claudia Linnhoff- Popien. Vast: Value function factorization with variable agent sub-teams. Advances in Neural Information Processing Systems, volume 34, pages 24018 -142222, 2021.
[25] Marc' Aurelio Ranzato, Y-Lan Boureau, and Yann LeCun. Sparse feature learning for deep belief networks. In Advances in Neural Information Processing Systems, volume 20, pages 1 1185-1199 1192, 2007.
[26] Tabish Rashid, Gregory Farquhar, Bei Peng. and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pages 10199-10210, 2020.
[27] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt. Gregory Farguhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pages 4295 4304. PMLR, 2018
(28) Stuart J. Russell and Andrew L.. Zimdars. Q-decomposition for reinforcement learning agents. In International Conference on Machine Learning, pages 656-663. PMLR, 2003.
[29] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. International Conference on Autonomous Agents and Multi-Agent Systems, pages 2186-2188, 2019.
[30] Jeff Schneider, Weng- Keen Wong, Andrew Moore, and Martin Riedmiller. Distributed value functions. In International Conference on Machine Learning, pages 371-378, 1999.
[31] Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip H.S. Tom; Mingfei Sun, and Shimon Whiteson, Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533 2020.

---

[32] Tianmin Shu and Yuandong Tian. M3rl: Mind-awart multi- management reinforcement learning. In International Conference on Learning Representations. 2019.
[33] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, pages 5887-5896. PMLR, 2019.
[34] Jianyu Su, Stephen Adams, and Peter Beling, Value-decomposition multi-agent actor-critics, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11352-11360, 2021.
[35] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. deloppoocoooooitionn networks for cooperative multi-agent learning based on team reward. In International Conference on Autonomous Agents and Multi-Agent Systems, pages 2087-2087, 2018.
[36] Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Interna- lional Conference on Machine Learning, pages 330-337, 1993.
[37] Lipeng Wan, Zeyang Liu, Xingyu Chen, Xuguang Lan, and Nanning Zheng. Greedy based value representation for optimal coordination in multi-agent reinforcement learning. In International Conference on Machine Learning, pages 22512-22535 PMLR, 2022.
[38] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning International Conference on Learning Representations, 2021.
[39] Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: Multi-agent reinforce- ment learning with emergent roles. In International Conference Machine Learning, pages 9886-9886. PMLR, 2020.
[40] Tonghan Wang. Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles to decompose multi-agent tasks. In International Conference on Learning Representations, 2021.
[41] Tonghan Wang, Jianhao Wang, Yi Wu, and Chongjie Zhang. Influence-based multi-agent exploration. In International Conference on Learning Representations, 2019.
[42] Chao Wen, Xinghu Yao, Yuhui Wang, and Xiaoyang Tan. Smix (A): Enhancing centralized value functions for cooperative multi-agent reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 7301-7308 2020.
[43] G. Wittemyer and Wayne M. Getz. Hierarchical dominance structure and social organization in african elephants, loxodonta africana. Animal Behaviour, (1(4):671-681) 2007.
[44] Zhao Xu, Yang Lyu, Quan Pan, Jinwen Hu, Chunhui Zhao, and Shuai Liu. Multi-vehicle flocking control with deep deterministic policy gradient method. In International Conference on Control and Automation, pages 306-311. 1. IEEE, 2018.
[45] Yaodong Yang. Jianye Hao, Guangyong Chen, Hongyao Tang, Yingfeng Chen, Yujing Hu, Changjic Fan, and Zhongyu Wei. Q-value path decomposition for deep multiagent reinforcmment learning. In International Conference on Machine Learning, pages 10706-10715. PMLR, 2020.
[46] Yaodong Yang. Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang, Qatten: general framework for cooperative multiagent reinforcement learning, arXiv preprint arXiv:2002.03933 2020.
[47] Dayong Ye, Minjie Zhang, and Yun Yang. A multi-agent framework for packet routing in wireless sensor networks. Sensors, 15(5::1002 2015.
[48] Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances Neural Information Processing Systems, volume 35, pages 24611-24622 2022.
12

---

[49] Yifan Zang, Jinmin He, Kai Li, Haobo Fu, Qiang Fu, and Junliang Xing. Sequential cooperative aultt-agent reinforcement learning. International Conference on Autonomous Agents and Multi-Agent Systems, pages 485-493, 2023.
[50] Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview theories and algorithms. Handbook of Reinforcement Learning and Control, pages 321-384 2021.
[51] Meng Zhou, Ziyu Liu, Pengwei Sui, Yixuan Li, and Yuk Ying Chung. Learning implicit credit assignment for cooperative multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pages 11853-11864 2020.
13

---

A The Insight of the Group Shifting Scheme
The proposed automatic grouping mechanism dynamically adjusts the group division the training proceeds. In the beginning, all agents belong to one group. As introduced Section 4.1 we examine the learned wi of each agent every timesteps to check if the grouping needs adjustment. If there are agents who contribute little to their group, it indicates that these agents do not belong to their current group. All these selected agents are assigned to the following group until they appropriately contribute to where they belong. A new group is built for selected agents in the last group.
The proposed scheme ensures that agents who belong to the same group will not be misassigned to different groups after training. For the example in the right side of Figure | agent a; is first selected after 01 timesteps' training, and the initial grouping 9 {a1. dg.
aa,) changes to g' = ffaa...................... ... a:-1, a++1, ... an J. {a;} }] Subsequently. after another training period, at timestep 02 two agents Oj and ag in the first group selected simultaneously to be moved out of the first group. At this point, they are automatically placed in the second group, i.e., the group to which agent o; belongs, and g" ((A-i..k)) {ai, ajj ax) }- Instead of placing aj and ag brand new group, our group shifting scheme ensures that they have the opportunity to train with ag together in one group, determining if ag aj, and Og (or two of them) are supposed to be in group.
Later, agent aj in the first group and agent a, ag in the second group are chosen at timestep ag c. The selection of a; and au indicates that agent 0; cannot cooperate well with them. Therefore, a; and ag are set in a new group (they should not return to the first group since they have been proved inappropriate for the first group). while aj stays alone in the second group. Agent a, from the first group is assigned automatically to the second group {ay }. If aj is supposed to be with a, and ag. will be selected afterward. Because aj fails to form efficient cooperation with {a: ) and if aj also cannot cooperate well with aj, then a₁ will be selected later on and set in the third group {ai, 4x:).
The group shifting scheme of GoMARL assigns the selected agents to their following group and ensures that each group is trained for a fixed period of cooperative attempts. As analyzed above, each agent can be grouped with appropriate agents that can efficiently work together after training.
B Experiment Details
B.1 Detailed Experimental Setup
We compare all the methods in six Hard and Super Hard StarCraft II micromanagement tasks (SMAC) 229 and three challenging Google Research Football (GRF) [16] scenarios. When resources are available, parallel training can facilitate faster convergence of methods [8] and is very common in RL and MARL communities 33 45 1. GoMARL and all the baseline methods in our paper are trained with 8 parallel runners for 10M steps in both testheds. Therefore, the timestep is not comparable to some papers that conduct experiments with only one "episode" runner. We use one NVIDIA Titan V GPU for training. We evaluate each method every 10K steps with 32 episodes and report the 1st, median, and 3rd quartile win rates across 5 random seeds. The detailed setting of GoMARL's hyperparameters is shown in our source code [
Many algorithmss introduce implementation tricks when they are built. These code-level optimizations were studied in depth in [3 48, g and were shown to have a significant impact on algorithm performance. To ensure fair comparisons, our experiments proposed to compare algorithms fairly. Please refer to PyMARL2''S open-source implementation for further training details and fair comparison settings. In addition, some methods implement specific parameter tuning in diverse scenarios, and is unfair for comparison. Therefore, we fixed parameters of all methods for all scenarios in our experiments.
based on the Py g framework
Lastly, adding vanilla-QMIX and QMIX. After all, GoMARL is already better than the finetuned version of QMIX (denoted as Ft-QMIX), and there is no need to compare it with vanilla-QMan MMIll--QIMI to show our superiority. We added vanilla-QMIX 's learning curves in Figure 5 only to avoid potential misunderstanding caused by the performance difference between Ft-QMIX in our paper and that of vanilla QMIX in other papers.
our experiments not intended to unfairly compare GoMARL
httt::://gttuub.oommyyyt/ss/////
14

---

B.2 Detailed Information about SMAC Tasks
In each SMAC micromanagement problem, a group of units controlled by decentralized agents cooperates to defeat the enemy agent system controlled by handcrafted heuristics. Each agent's partial observation comprises the attributes (such as health, location, on, unit_type) of all units shown up in its view range. The global state information includes all agents' positions and health, and allied units' last actions and cooldown, which is only available to agents during centralized training. The agents' discrete action space consists of attack [enemy_id]] move [direction], stop, and no-ov no-op for the dead agents only, A particular unit, Medivac, has no action attack [enemy_id] but has action heal [enemy_ioe __i]] Agents can only attack enemies within their shooting range. Proper micromanagement requires agents to maximize the damage to the enemies and take as little damage as possible in combat, so they need to cooperate with each other or even sacrifice themselves ves. We follow the default setup of SMAC in our experiments, and more settings, including rewards and observation information, can be acquired from the original paper [29] or open-source implementation.
agent
Based on the performance of baseline algorithms, the tasks in SMAC are broadly grouped into three categories: Easy, Hard, and Super Hard. The key winning some Hard or Super Hard battles is mastering specific micro techniques, such as focusing fire, kiting, avoiding overkill, et cetera. The battles can be symmetric or asymmetric, and the group of agents can be homogeneous or heterogeneous. Here we provide some characteristics of the scenarios to help gain insights into the good or poor performance of the methods:
3s_ ___52i_ is Hard asymmetric battle between two homogeneous teams. The allied Stalkers have to master the kiting technique and disperse in the area to kill the Zealots that chase them one after another. This map faces the delayed reward problem; however, it is not very strict about micro-cooperation between agents because of agents' scattering.
The asymmetric scenarios 5m_vs vs_ 6m and 8m_vs_ 9m are two Hard maps offering a substan- tial challenge. The allied agents must learn to focus fire without overkill and to correctly position themselves with considerable precision to overcome the enemy team with more agents. The 5m_vs _a__v_______ 66 battle is relatively more difficult than 8m_vs_9m 9m due to fewer agents, the enemy force with one more agent have a greater advantage.
corridor is a Super Hard map that needs extensive exploration. Six allied Zealots fight twenty-four Zerglings on this task. The massive disparity unit numbers between the two sides implies that the whole team cannot launch an attack together. The only winning strategy is to sacrifice a small number of agents who leave the team and attract the attention of most enemies, and the large force eliminates the rest of the enemies. The surviving agents then keep repeating the same tactic till the whole team kills all the enemies.
MMM2 is representative Super Hard asymmetric battle between two heterogeneous teams with three kinds of units. One Medivac, two Marauders, and seven Marines have to battle against team with one more Marine. Marauder has greater attack damage and health than Marine but with a longer cooldown. Medivac has no damage but can heal other agents.
_s5z_va_3stt is a Super Hard map that requires breaking the bottleneck of exploration, where three Stalkers and five Zealots battle against three Stalkers and six Zealots. This map requires more cooperation between agents and is the most challenging task in SMAC.
B.3 Detailed Information about GRF Academy Scenarios
Google Research Football (GRF) Academy includes several scenarios which can be commonly found in football games. Agents need to coordinate timing and positions for organizing offense to seize flecting opportunities, and only scoring leads to rewards. Each agent's partial observation contains the absolute positions and moving direction of the ego-agent, relative positions and moving directions of other agents, and the ball. The global state information includes the absolute positions and directions of all agents and the ball. Agents have discrete action space of 19, including moving in eight directions, three kinds of ball pass, two kinds of dribble, two kinds of sprint, shot, sliding, stop-moving and do-nothing. Proper cooperation requires agents to pass and shoot effectively with good direction at proper timing, thus making GRF scenarios much more complicated than SMAC's. Other details can be acquired from the original paper [16] or the open-source implementation.
15

---

Ablation Specialised Agunt Network
Ablation of Specialized Agunst Network
Ablation of Specialized Agent Notwork
Win Rate (%)
PugldRoat 5L8
Rote: (75)
GREES yed sax
Steps (x()F)
Stops Ablation of the Speeislised Aguun Network
Stepe
Ablution of the Specialized Agont Network
Win Rate (95)
Stape (x
Steps (x(1P)
Figure All: Method improvement with the proposed SAN in the Super Hard map 3s5z_vs_3s6e
Here we provide an introduction of the three scenarios we utilized to help gain insights into the good or poor performance of all the methods in our paper:
academy_paan Saapiyypps____dde aad_aandd____aeeeee wi th_ keeper includes two of our players trying to score from the edge of the box. One of our agent is on the side with the ball and next to a defender.
The other agent is at the center, unmarked, and facing the opponent keeper.
contains three of our agents and two opponent players (a defender and a keeper). Our agents try to score from the edge of the penalty area. One stands in the middle, while the others are located on both sides of the area. Initially, the agent at the center keeps the ball and directly faces the defender.
academy_3.
3_v_______________ 1_____wii
academy_ counterattack. easy contains four of our agents and two opponent players (a defender and keeper). Agents are initialized far from the penalty area and stand evenly in an are centered on the goal. The second agent from the top initially keeps the ball and has to pass it to a teammate at the appropriate time to avoid interception.
C Performance of Methods with and without the Specialized Agent Network
The proposed specialized agent network (SAN) is highly transferable. To further validate its effec- tiveness, perform the specialized agent network on other baseline methods in Section 5. to see they can perform better. ROMA is not included in this study since ROMA's agent networks are produced by its learned roles, and the replacement will invalidate the main idea of ROMA. As shown in Figure our specialized agent networks significantly improve methods' performance and learning efficiency the Super Hard SMAC task 3s5z_ vs 3s6z. Compared to agent networks utilizing the vanilla parameter-sharing mechanism that limits policy diversity, our specialized agent network adjusts the local utility with the individual group group-related related information and enables various styles of behaviors related to the group status, encouraging extensive exploration and accelerating learning. The complete learning curves are shown in Figure Al1
As Figure All illustrates, the performance of all methods is further improved when equipped with our specialized agent network. Specifically, the learning efficiency of Ft-QMIX is boosted. The variance of RIIT is markedly reduced, and the win rate is increased by about 10%. The improvement of VAST is the most obvious; both learning efficiency and win rate are enhanced, and the variance is very clearly reduced. QPLEX's improvement is not very obvious; however, it obtains a slightly higher learning speed and achieves the highest win rate among all the baseline methods.
Most importantly, even equipped with our specialized agent network, all baseline algorithms fail to surpass GoMARI in terms of learning efficiency and the final win rate. GoMARL with vanilla agent networks looks much inferior to GoMARL with the specialized agent network, so it is worth questioning whether the dynamic grouping module is ineffective. The dynamic group learning may
16

---

STEP: 9
SCORE: BALL O INNEX
STEP: 42
SCORE: O BALL OWNED:
PLAYER SPENT DEBRLA DNECIION ACTIN
PLATER
Spinn
DEOBLE DIRETTON ACTIH
Passes
Agent
Opponent player
Agent
A gent
Opponent
player
Agent
Keeper
Paisses to Agent
Agent
The keeper prevent
Agent sheoting.
(a)
(b)
STEP: 56
SCORE: O
STEP: 65
SCORE: O BALL OWNED:
PLATER sPint DHIBLE DHEERON ACTINN
BALL OWNED
PLATER
SPann DROBLE DIRETTON ACTÓN
Opponent player
Keeper
Agent shoots
Agent 3
Agent'
Keeper
Agent 4 gives second shoot and scores.
Agent
Agent
The keeper the ball
(c)
(d)
Figure A12: A visualization of learned policies of academy. arrows show the motion of the ball, and the red arrow illustrates the scoring shoot.
ccaaemy_ccunterateete wounterattack. _esyc. The yellow
reduce efficiency to certain extent in the early stage of training compared to QPLEX, however, in the middle and late training stages, the learned grouping will have significant effect when equipped with the proposed specialized agent network. Therefore, the two modules of GoMARL, the automatic grouping module and the specialized agent network, are both crucial the performance of GoMARL.
D Visualizations of the Learned Policies in GRF
The trained strategy in Google Research Football is visualized to validate if the learned grouping makes sense. Section 5.2 we visualize a match
conttertttccttttaaccc.k.. easy with more complex strategy to prove the rationality of the grouping GoMARI. learned. Here, we provide detailed discussion of the visualization in Figure A12 zoom-in version Figure I0 for clarity).
As shown in Figure A12 a). Agent 2 holds the ball at the beginning and faces an opponent rushing toward him. Agent passes the ball to Agent to prevent the ball from being stolen. Subsequently, in (b), Agent carries the ball and tries to break through. However, the opponent goalkeeper blocks his attacking route, and Agent 1 continues passing after short carry. Agent 3 makes a good run and catches the ball smoothly but shoots quickly in Figure A12(c) since the opponent is close. The goalkeeper easily saves this hasty attack. Agent in (d), who learns excellent coordination with Agent 3, stops the ball and immediately adds another shot to create the goal.
GoMARL's automatic grouping module learns reasonable grouping G { {{ai, a2), {as, a4} } for this complex goal, in which the first group successfully brings the ball into the penalty area through smooth coordination, while the second group creates the final goal through skillful cooperation.
17


Reviews:
[{"review_id": "eUD61o9s83", "reviewer": "Reviewer_p19m", "summary": "This work introduces group-oriented MARL (GoMARL). GoMARL leverages a grouping mechanism that automatically creates sub-teams in a cooperative multi-agent team in an end-to-end fashion by maximizing the expected global return. \n\nIn my opinion this approach is a nice, balanced solution for CTDE frameworks where the issues of fully centralized and decentralized models are greatly alleviated by instead looking at the coordination problem at a sub-team level, rather than individual agents, or one giant connected team. A major contribution and perhaps the most interesting part of the approach is the automatic grouping which does not require domain knowledge or a priori determined group number. The paper is very well-written and is easy to follow and understand. Extensive evaluations and ablation studies are carried out which confirms the applicability and efficiency of the approach.\n\nI only have a few minor comments and questions:\n\n- Since automatic grouping is being tackled, can this approach be directly applied to cooperative heterogeneous teams where each agent has a class-specific action-space (or observation-space) and objective. In such scenarios, despite having local separate objectives, classes of agents must collaborate towards a greater global objective/mission. Can this approach be reliable enough to include such class-dependent grouping integrated, for instance when specific classes/types of agents must be grouped for the whole thing to work?\n\n- The process of end-to-end learning for automatically clustering agents can be highly complex and computationally expensive. I can see that with increasing number of agents, the possible combinations of sub-groups could grow exponentially, making the task of finding the optimal group indexes a significant challenge. Can authors comment on this please?\n\n- Can inter-agent communication be leveraged here, for instance as a part of input/observation when needed? Would it require a similar locality and grouping?\n\n- Limitations of the proposed approach are never discussed in the paper. Please add a separate subsection at the end to cover potential known limitations of the method.\n", "questions": "See above", "limitations": "No. Limitations are not discussed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "See above", "weaknesses": "See above", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive feedback and acknowledgment of our efforts. Following are our responses to all your concerns.\n\n**Q1: Applicability of automatic grouping to heterogeneous teams.**\n\nA1: GoMARL can be directly applied to cooperative heterogeneous teams. This paper focuses on fully cooperative tasks as Dec-POMDP (Lines 92-99), and this setting has no local separate objectives. Classes (groups) of agents trained by GoMARL collaborate towards the global objective (global reward for the team) since our grouping mechanism and the individual/group Q-values are learned by maximizing the expected global return.\n\nThe two super hard scenarios showcased in Fig.4, 3s5z_vs_3s6z and MMM2 (involving agents with distinct action spaces), are heterogeneous settings where GoMARL outperforms baseline methods significantly. The efficacy of our grouping mechanism lies in its ability to adapt to varying task contexts. For example, agents are grouped according to their class/type in 3s5z_vs_3s6z (i.e., $G = \\\\{\\\\{3s\\\\},\\\\{5z\\\\}\\\\}$), while in MMM2, GoMARL clusters agents based on task context (Fig.5 and Lines 297-306). Notably, GoMARL automatically learns the grouping, but it is also reliable when facing predefined grouping requirements (like your request that specific classes/types of agents must be grouped). This can be achieved by initializing agents that must be grouped within the same subgroup and fixing this subgroup throughout training.\n\n**Q2: Increasing agent numbers results in exponentially growing sub-group combinations, posing a challenge in finding optimal group indexes.**\n\nA2: Our automatic grouping mechanism diverges from conventional search algorithms. Rather than seeking optimal indexes across all sub-group combinations, GoMARL employs the \"*select and kick-out*\" scheme (as illustrated in Lines 146-147 and Fig.1.right) based on the proposed GGM principle. With each new agent $a_i$, GoMARL determines whether it belongs to the current group $g_j$ at every timestep. This extra assessment is essentially an additional evaluation of $w_1^i$. Therefore, GoMARL's computational cost scales linearly with the number of agents, ensuring significant learning efficiency.\n\n**Q3: Integrate inter-agent communication into GoMARL.**\n\nA3: This paper focuses on cooperative Dec-POMDP without communication mechanisms. However, GoMARL is an adaptable and versatile framework for cooperative multi-agent learning, so that it can be easily and quickly migrated to tasks with communication settings. This can be achieved by incorporating communication information as part of the input/observation or other customized communication channels.\n\nGoMARL is a generalized framework that does not have any restrictions on communication. Whether agents require similar locality and grouping only relies on your communication settings and the communication mechanism (*e.g.,* environmental constraints, communication objectives, *etc.*). We believe that GoMARL can serve as a very strong foundational framework for researchers in the field of multi-agent communication.\n\n**Q4: Discussion on limitations.**\n\nA4: Thank you for suggesting including a separate subsection of limitations. Here we present our discussion on limitations and future plans: \"The automatic grouping mechanism of GoMARL is currently grounded in the value decomposition process, rendering it inapplicable to policy-based methods. Our future research will delve into automatic grouping based on policy gradients to enhance the learning efficiency of policy-based methods. Furthermore, the current version of GoMARL is only designed for single-level grouping, and it possesses the potential to achieve *finer granularity in group division*. We will investigate multi-level grouping (*i.e.,* group-within-a-group) in the future to achieve nested group division, pursuing better performance and higher learning efficiency in more complex environments.\"\n\n**We would like to express our gratitude once again for your insightful feedback. We hope that our response has addressed your concerns. If there are any remaining uncertainties, we welcome further discussion to ensure clarity and satisfaction.**"}, {"review_id": "x6LOvx1QN3", "reviewer": "Reviewer_y5k7", "summary": "This paper considers fully cooperative MARL in the CTDE setting. Specifically, this work focuses on the notion of groups and proposes three components to improve learning performance: \n- specialized agent network (SAN): relies on parameter-sharing across agents, but the shared policy is group conditioned and regularized such that it is similar for agents in a same group and different across groups. \n- three stage value mixing: (1) w_1, from agent-level to group-level with agent information. (2) w_2, from group-level to group-level, but incorporating group information. (3) from group-level to total-level (mixing weights and the information used is unclear).\n- automatic and dynamic group formation during learning. \n\nThe resulting method GoMARL is compared against several baselines on two environments and several challenging tasks. It is shown to outperform the baselines. Authors also propose ablation to investigate the importance of SAN, incorporating different level of information in the value mixing, and automatic group formation. Finally, the learned groups and strategies are qualitatively analyzed and discussed by looking at how agents behave in selected scenarios. \n", "questions": "- Figure 1.: shouldn't G''' be {{A}, {a_i, a_l}, {a_j, a_k}} since a_i wasn't selected but a_j was?\n- I found using $a$ as agent index a bit confusing, as usually $a$ is for action and agent index is $i$. \n- could it be that no stable grouping is achieved? Like varying team strategy or any other reason?\n- equation (4): does the h correspond to the same sample, i.e. the same situation but observed by different agents, or does it also correspond to different samples of the replay buffer?\n\n### Summary\nI think this is a very good paper, yet I would like to have a better understanding of it before increasing my score. Therefore, I encourage authors to address my concerns (see. Weaknesses). \n\n### references \n[1] Yu, Chao, et al. \"The surprising effectiveness of ppo in cooperative multi-agent games.\" Advances in Neural Information Processing Systems 35 (2022): 24611-24624.\n\n\n### EDIT\nI have updated my score from 6 to 7 post-rebuttal", "limitations": "n/a", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Improving performance of cooperative MARL in the CTDE setting is of significant importance for the research community. \n- Relying on group for cooperation and coordination is interesting and well motivated. \n- The paper is well written and does a good job at explaining the method despite its complexity. \n- Empirical evaluation is thorough and highlight significant and reliable performance improvements. \n- The method relies on less network parameters than the baselines. \n- SAN improvement is transferable to other methods. ", "weaknesses": "- Clarity: I have trouble understanding several points related to the GGM principle. \n    - l.176: I cannot understand how \"sacrificing self-interest for greater team benefit\" could occur in the fully cooperative setting (shared reward) since individual agent's reward and team reward are exactly the same. If an agent killing itself increases the team reward, it would thus be incentivized to do so by its own individual reward. Therefore, I do not understand why consistency would be lost inside the group:\n        - Definition 2: I do not understand why individual-group consistency is lost.\n        - l.140: I do not understand why one would enforce group inconsistency by kicking out consistent agents out of the group.\n        \n- Limitation: I think that the assumption that agents in a group must have similar policies while being different to agents in another group could be overly restrictive. Imagine the setting of two groups each building half of a bridge: both groups need varied roles (resource gatherer, architect and builder for instance). My understanding is that grouping in that case is still relevant for credit assignment but impedes policy specialization inside each group. \n\n- Clarity: Section 4.3. The method is quite complex and has lots of moving part, so I believe it would be very valuable to summarize it by explicitly expressing Q_tot in terms of individual Q and highlighting the different mixing levels, weights and the information used (agent's, group's, total, etc.). For instance, on what are the total mixing weights conditioned?\n\n- Comparison: I think it is important to report GoMARL performance against MAPPO's given that [1] considers the same environments. Also, MAPPO's centralized value function could leverage GoMARL to see if it improves credit assignment and performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thorough reading and insightful comments. Following are our responses to all your concerns.\n\n**Q1: Ambiguity of the GGM principle.**\n\nA1: We apologize for the unclarity. We agree with your view on consistency under fully cooperative settings, and we also follow the consistency principle when implementing GoMARL (Eq.3). However, in the initial training phase or before convergence, the still-learning $Q_i$ may not yet be able to guide the agent to take a series of actions to maximize team benefits. As in your example, the action that maximizes the eventual $Q_i$ is \"suicide\", but it may not be before training ends; Hence, the agent needs to \"sacrifice\" its *current benefits during training*, and opts for actions with smaller *present $Q_i$*, to explore greater team gains (to learn suicide). Our viewpoint is to integrate group-wise information into the gradient (Eq.3) to help learn the optimal action (suicide in your example) and enhance collaboration within groups. On the other hand, if an agent $a_i$ perceives no necessity to \"sacrifice\", i.e., $Q_i$ has consistency with $Q_{group}$, then $a_i$ can leave this group. We are sorry for not clearly expressing our thoughts. We have revised our expression in this part and hope our response can solve your doubts. \n\n**Q2: Example of building two separate half bridges.**\n\nA2: GoMARL can handle this example. In specific, by pursuing policy similarity within groups and policy diversity between groups, we can attain a finer-grained grouping of 6 groups (the resource gatherer group $g_{rg}^1$, architect group $g_{a}^1$, and builder group $g_{b}^1$ in half-bridge 1; resource gatherer group $g_{rg}^2$, architect group $g_{a}^2$, and builder group $g_{b}^2$ in half-bridge 2). Even if there is only one architect, GoMARL can still learn a separate group for the architect. With the above reasonable grouping, the intra/inter-group coordination will significantly improve the overall team efficiency. Therefore, our SD-regularizer showcases relatively broad applicability.\n\n**Q3: Details of mixing levels, weights, and the information used.**\n\nA3: Thank you for this valuable suggestion. We hope the detailed explanation of the mixing weights and the information utilized can facilitate a better understanding of GoMARL.\n\n(1) First mixing level:\n\n$w_1^i=f^i_{w_1}(h^i;\\theta^i_{w_1})\\in\\mathbb{R}^{1\\times k}$, where $k$ is the embedding size; and the group-wise $W_1^g\\in\\mathbb{R}^{n_g\\times k}$ is the stack of $w_1^i$ for all $n_g$ agents $a_i$ in group $g$.\n\n$w_2^g=f^g_{w_2}(s^g;\\theta^g_{w_2})\\in\\mathbb{R}^{k\\times 1}$, where $s_g$ is the group-wise state of group $g$ (details in Lines 223-227).\n\nInsights of $w_1$ and $w_2$: The automatic grouping relies on $w_1$. Stacking individual $w_1$ of agents within the same group ensures the dynamic changes in group size (agent number per group). $w_2$ incorporates group-wise information. GoMARL generates $Q^g_{group}$ by mixing $Q^i$ through $w_1$ and $w_2$.\n\n(2) Second mixing level:\n\n$w_3=f_{w_3}(s^g;\\theta_{w_3})\\in \\mathbb{R}^{1\\times k}$; and $W_3 \\in \\mathbb{R}^{m \\times k}$ is the stack of $w_3$ for all $m$ groups.\n\n$w_4=f_{w_4}(s;\\theta_{w_4})\\in \\mathbb{R}^{k\\times 1}$.\n\nInsights of $w_3$ and $w_4$: The second mixing level has a similar architecture as the first level. $w_3$ conditions on the group-wise information and enables dynamic changes in the group number, while $w_4$ incorporates global state information $s$ and generates $Q^{tot}$.\n\nAlthough containing two mixing levels, GoMARL relies on fewer mixing parameters than the baselines (Appendix D).\n\n**Q4: Comparison with MAPPO.**\n\nThe requested comparison is shown in **the PDF of the global Rebuttal Comment**. MAPPO was not included in our initial paper because it is not a value factorization method but a policy-based method. Notably, MAPPO is published in Track on Datasets and Benchmarks. It is a valuable *engineering* work that thoroughly studies five implementation factors (tricks) of PPO in the multi-agent setting. Among them, adjusting \"input representation to value function\" involves *changes to the original SMAC environment*, which causes *unfair comparison* when compared with other methods. Thus, we compare GoMARL with MAPPO-Environment-Provided (i.e., MAPPO-EP in the original paper). GoMARL (with no trick) outperforms MAPPO (with four tricks) in all six SMAC maps.\n\nGoMARL is a value decomposition method where agents select their actions based on $Q_i$; however, the policy-based method MAPPO does not use $Q_i$ for the value estimation. GoMARL is currently grounded in the value decomposition process, rendering it unhelpful to policy-based methods. Following your valuable suggestion, our future research will delve into automatic grouping based on policy gradients to enhance the learning efficiency of policy-based methods.\n\n**Q5: Response to the \"Questions\" list.**\n\nA5: Following are responses to the \"Questions\" list. Due to rebuttal length limitations, we do not repeat your questions here.\n\n1. Highlights with purple in Fig.1 denote the agents that *have adjusted groups*. $a_l, a_i, a_k$ adjusted their groups at the third iteration, and $G'''=\\\\{ \\\\{ A \\\\},\\\\{ a_j, a_l \\\\},\\\\{ a_i, a_k \\\\}\\\\}$ is the new grouping after adjustment. Please see the detailed explanation in Appendix B.\n\n2. We will follow your advice to change the agent index from $a$ to $i$.\n\n3. GoMARL gradually learns a stable grouping during training, with no instability observed in experiments; Each task can be solved with multiple strategies, and GoMARL may learn different groupings based on the map context.\n\n4. $h$ in Eqn.4 corresponds to the same situation observed by different agents.\n\n**Once again, we appreciate your thorough review and helpful comments. If we address your concerns, we kindly ask for your consideration in increasing the score, as you mentioned. Should any questions remain unresolved, we welcome further discussions in the next phase.**"}, {"review_id": "sRcjkjOvDc", "reviewer": "Reviewer_wS3D", "summary": "This paper proposes an automatic grouping mechanism and a GGM principle to build a value function factorization. And the authors model informative group-related signals to encourage specialization in policies and inter-group coordination. The authors evaluate the performance of all methods on the SMAC and GRF benchmarks.", "questions": "Please answer the questions in the weaknesses.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The idea of grouping agents is sound.\n2. The paper is well-written and easy to follow. ", "weaknesses": "1. As the authors claim that the method is capable of the correct selection of non-optimal (sacrificial) actions, does this imply that the value decomposition enables the representation of various types of payoff matrices and facilitates decentralized execution? If this is the case, it would be worthwhile to discuss how this method can potentially address the issue of relative overgeneralization. Additionally, it might be beneficial for the authors to test their approach on a simple matrix game, such as\n\n   |      | a_1 | a_2 |a_3|\n   |---------|---------|---------|---------|\n   | a_1 | 3 | -7 |   -5|\n   | a_2 | -7 | 0 |   0|\n   | a_3 | -5 | 0 |   2|\n\n2. The absence of a theoretical analysis regarding the representation capacity of the proposed value decomposition weakens the contribution of the paper. It would greatly enhance the strength of the work if the authors could provide a thorough theoretical analysis to support and validate the effectiveness of their value decomposition approach.\n3. What is the difference between e^i and role in other role-based methods? \n4. The performance of ROMA and RODE is very poor. Could the author give a further explanation why role-based methods are not suitable for these tasks? \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the insightful comments and your efforts in reviewing our paper. Following are our responses to all your concerns.\n\n**Q1: Value Decomposition Representation.**\n\nA1: GoMARL is not a method that aims to resolve the representation constraints on value mixing. Instead, it is a value factorization framework designed to facilitate efficient cooperation by learning group divisions.\n\nThe term \"sacrifice\" does not imply that we enhance the expressiveness of the mixing network. Before convergence, the still-learning $Q_i$ might be unable to guide agents to take actions that maximize team benefits. Hence, agents need to \"sacrifice\" their *current benefits*, and opt for actions with smaller *present $Q_i$*, to explore greater team gains and learn the optimal action. We hope the above statement can clarify any misunderstanding regarding the term \"sacrifice.\"\n\nAlthough GoMARL does not focus on expressiveness, it is a general group-wise framework for value factorization. Existing methods to overcome representation constraints can be easily integrated with GoMARL. Notably, GoMARL will not influence the inherent expression capacity of the base mixing scheme. By adopting QPLEX's advantage mixing, GoMARL effectively learns optimal actions in the provided matrix:\n\n   |  | $u_1$    | $u_2$ | $u_3$ |\n   | ----- | -------- | ----- | ----- |\n   | $u_1$ | **3.03** | -6.29 | -4.38 |\n   | $u_2$ | -6.49    | 0.07  | 0.08  |\n   | $u_3$ | -4.00    | 0.11  | 2.12  |\n\nThis also clarifies the absence of a theoretical analysis regarding representation capacity. GoMARL, like RODE and ROMA, does not focus on theoretical analyses of representation. Instead, GoMARL is oriented towards further studying the relationship between individuals and the team. Extensive component studies show how grouping works in GoMARL, which are also very convincing (highly recognized by all the other reviewers).\n\n**Q2: Comparison between $e_i$ and role.**\n\nA2: Thank you for this insightful question. We hope this response can assist you in better grasping GoMARL's motivation and distinct advantages. We first emphasize that $e_i$ does not represent the learned group; instead, it is the group-related information that summarizes the agent's *local* history *from the group perspective* (Line 188).\n\n* Compared with role in RODE\n\nThe role in RODE is essentially a cluster of action representation, with the **number of roles being predefined**. RODE learns action representations through **supervised learning** and clusters them into $k$ (a hyperparameter) roles. The role representations are derived by averaging the action representations within each role. RODE determines each agent's role by selecting the role representation that maximizes the local $Q_i$ value.\n\nGoMARL and RODE are both thoughtfully designed methods. However, it's important to note that $e_i$ diverges from the role in RODE. $e_i$ is motivated by novel insights, providing informative knowledge for fine-grained value decomposition. Unlike RODE, $e_i$ does **not rely on apriori knowledge** and is trained in an **end-to-end** manner.\n\n* Compared with role in ROMA\n\nCommon Ground: The only similarity between $e_i$ and role $\\rho$ in ROMA is their generation through an encoder-decoder architecture—a common practice for information encoding. Beyond this architectural similarity, $e_i$ and $\\rho$ are entirely different.\n\nDifferences: The grouping $G$ (not $e_i$) of GoMARL is driven by $Q_{tot}$, and the group division is achieved during the value decomposition process. $e_i$ is the group-wise information integrated into policy gradients to promote intra-group coordination. On the other hand, $\\rho$ in ROMA is an embedding of the encoded local observation. Unlike $e_i$, it is not aimed at promoting intra-role coordination. In addition, different from the specialized regularizer in ROMA, our SD-regularizer is set to extract  *group-related* information from the *individual* history. In summary, $e_i$ and $\\rho$ have entirely distinct motivations, insights, and purposes.\n\n**Q3: Performance of RODE and ROMA.**\n\nA3: **RODE**'s role selection highly relies on the environment. Factors such as the predefined role numbers $k$, supervised learning iterations, and the way to generate role representation, are acutely attuned to the particular environment. This environmental sensitivity may render agents incapable of executing crucial actions, owing to unreasonable action space divisions. That is why RODE performs well in SMAC but fails in GRF. In addition, RODE uses several rules based on prior knowledge[1], which leads to an unfair comparison. Thus under our fair comparison (detailed in Appendix C.1), RODE's performance may sometimes be unstable.\n\n**ROMA** learns each agent a role, with all regularizers acting directly on the role. It *implicitly* guides policy learning through three attributes (dynamic, identifiable, and specialized). However, ROMA lacks explicit mechanisms that directly stimulate cooperative behaviors. In contrast, GoMARL learns multiple modules based on the groups, explicitly facilitating efficient intra/inter-group cooperation. As a result, GoMARL is more efficient than ROMA.\n\nThe performances of RODE and ROMA in our paper are one of the best among public papers, better than their performances in papers from the same research team [2,3] and other teams [1,4,5,6], with no unfair comparisons.\n\n[1] LDSA: Learning dynamic subtask assignment in cooperative multi-agent reinforcement learning. NeurIPS, 2022.\n\n[2-6] are NeurIPS and ICML papers listed in Appendix Line 95. (omitted here due to rebuttal length restriction)\n\n**We hope our response can address your concerns, clarifying our main contribution (which does not center on solving the representation problem). Your *re-evaluation* of our paper is critical to us, and we genuinely appreciate your consideration. If you have any new concerns, we would be glad to further discuss them to ensure clarity in the next phase.**"}, {"review_id": "8GuD4Nua3r", "reviewer": "Reviewer_Dgdo", "summary": "This paper claims that grouping is beneficial for the efficiency of team coordination. To verify this claim, the authors propose a new learning framework called GoMARL, which achieves efficient cooperation by learning automatic grouping without domain knowledge. The joint action-values are factorized in a group-oriented way, which makes this work different from other related works. The group is generated dynamically and automatically. Meanwhile, a policy learning strategy is proposed to utilize the proposed grouping mechanism. The aim of this policy is to let agents in a same group specialize in similar policies, and agents in different groups have diverse strategies. The proposed solution is evaluated in StarCraft II micromanagement tasks and Google Research Football. ", "questions": "Grouping is common in reality, but it also restricts the whole policy space into a sub-space where the requirements of grouping holds. Will this restriction influence the overall outcome in some cases? ", "limitations": "The authors do not appear to discuss the limitations of their work. I suggest that the authors discuss more scenarios where grouping is beneficial and where the contribution of grouping is limited.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The overall idea of introducing grouping into multi-agent value function factorization is quite novel. I’m also happy to see that it is executed well in many challenging scenarios. \n\nThe authors also provide detailed case studies of the learned groups and their performance in both SMAC and Google Research Football. I find these materials very helpful to show that grouping is helpful in multi-agent cooperation tasks.  \n\nThe experiments are also very detailed, showing that GoMARL performs well on multiple domains. I highly appreciate the authors’ efforts in designing ablation studies on each component of their proposed solution, which makes it very intuitive to verify the contribution of each component. \n", "weaknesses": "Although the overall idea of grouping is intuitive, it is quite challenging to grasp the detailed implementation procedures. For instance, in Line 139: “if agent $a_{j_{i}}$ in group $g_j$ establishes consistency with all the other agents $a_{-j_{i}}$ in group $g_j$, …, then $a_{j_i}$ should not be in group $g_j$ anymore”, this statement represents the standard of grouping, but it is quite abstract. Is it supported by natural systems on grouping or is it a compromise due to the restriction of GMM principle?\n\nSome statements can be rephrased in a more straightforward manner. For instance, in Line 270: “GoMARL maintains two mixing networks, and if it has more parameters for value mixing is a natural question”, this representation is quite ambiguous. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the insightful review and helpful comments. Following are our responses to all your questions.\n\n**Q1: Ambiguity in Line 139.**\n\nA1: We are sorry for any confusion caused. The automatic grouping mechanism we propose in this paper is guided by the GGM principle, rather than being derived from natural systems. This insight motivates our \"select-and-kick-out\" grouping mechanism design. The detailed implementation procedures are illustrated in Lines 142-147 and the clear example in Fig.1(right). We highly recommend the reviewer refresh these parts, as they provide comprehensive insights into the implementation details. Appendix B (the insight of the group shifting scheme) is also beneficial to understand the grouping mechanism details. We have revised the statement in Line 139 to eliminate the ambiguity. Thank you very much for pointing out this unclear expression.\n\n**Q2: A statement should be rephrased in a more straightforward manner.**\n\nA2: We sincerely appreciate this helpful comment. We revised the statement you pointed out in Line 270: \"GoMARL maintains two mixing networks but employs fewer mixing parameters compared to the baseline methods, particularly in scenarios with a large number of agents. GoMARL outperforms other methods despite using fewer mixing parameters, highlighting its inherent superiority over methods relying on stronger networks\". Additionally, we thoroughly checked our paper and rephrased essential statements in a clear and straightforward manner to eliminate any ambiguity.\n\n**Q3: Whether GoMARL restricts the whole policy space into a sub-space?**\n\nA3: While GoMARL involves group division in the multi-agent system, all agents, regardless of their group, still explore and learn within the entire policy space.\n\nWe want to emphasize that GoMARL is a value decomposition method under the CTDE framework. We learn group division only as a reasonable way to utilize the group-wise information to estimate fine-grained value functions and facilitate cooperation among agents. Hence, there are neither restrictions on the overall policy space nor requirements that agents can only explore in the group-wise policy spaces.\n\nThe automatic grouping mechanism based on the GGM principle also has no restriction on the policy space. As for the joint action space of the multi-agent system, each agent's action selection is driven and learned by $Q_{tot}$ in GoMARL. As for the agent's individual action space, no limitations related to the agent's group are imposed; agents continue to learn and make decisions within their complete action spaces.\n\nTherefore, GoMARL's grouping mechanism does not restrict the entire policy space into sub-spaces and does not influence the overall outcome.\n\n**Q4: Discussion on limitations.**\n\nA4: Thank you for suggesting including a separate subsection of limitations. It is very constructive to further improve the quality of our paper.\n\nHere we present our discussion on limitations and future plans: \"The automatic grouping mechanism of GoMARL is currently grounded in the value decomposition process, rendering it inapplicable to policy-based methods. Our future research will delve into automatic grouping based on policy gradients to enhance the learning efficiency of policy-based methods. Furthermore, the current version of GoMARL is only designed for single-level grouping, and it possesses the potential to achieve *finer granularity in group division*. We will investigate multi-level grouping (*i.e.,* group-within-a-group) in the future to achieve nested group division, pursuing better performance and higher learning efficiency in more complex environments.\"\n\n**Thank you once again for your insightful review and constructive comments. We hope our response can address all your concerns. It would be greatly appreciated if you would re-evaluate our paper based on our responses. If there are any remaining uncertainties, we welcome new comments to ensure clarity and satisfaction during the discussion phase.**"}]

Author rebuttal:
To AC and all the reviewers:

We would like to express our sincere gratitude to AC and all the reviewers for their great efforts in evaluating our paper. We sincerely appreciate the valuable insights and suggestions provided by each of you. We have carefully addressed all the questions and concerns raised in the reviews through our rebuttal comments.

The attached PDF to this global rebuttal comment contains additional experimental results requested by Reviewer y5k7.

If there are any remaining queries or uncertainties after reviewing our responses, we welcome further discussions during the upcoming phase. Your continued engagement is highly valued and appreciated. Thank you once again for your time, expertise, and contribution to our paper.

Meta-review: This paper studies subgrouping of agents for value decomposition-based approaches to cooperative MARL. All reviewers appreciated that the method was novel, clearly explained, and the experimental evaluations were appropriate.
Decision: Accept (poster)