Title: Automatic Grouping for Efficient Cooperative Multi-Agent Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: MARL

Cooperative Multi-Agent Reinforcement Learning

Coordination and Cooperation

Automatic Grouping

Group-Wise Learning
Abstract: Grouping is ubiquitous in natural systems and is essential for promoting efficiency in team coordination. This paper proposes a novel formulation of Group-oriented Multi-Agent Reinforcement Learning (GoMARL), which learns automatic grouping without domain knowledge for efficient cooperation. In contrast to existing approaches that attempt to directly learn the complex relationship between the joint action-values and individual utilities, we empower subgroups as a bridge to model the connection between small sets of agents and encourage cooperation among them, thereby improving the learning efficiency of the whole team. In particular, we factorize the joint action-values as a combination of group-wise values, which guide agents to improve their policies in a fine-grained fashion. We present an automatic grouping mechanism to generate dynamic groups and group action-values. We further introduce a hierarchical control for policy learning that drives the agents in the same group to specialize in similar policies and possess diverse strategies for various groups. Experiments on the StarCraft II micromanagement tasks and Google Research Football scenarios verify our method's effectiveness. Extensive component studies show how grouping works and enhances performance.

Automatic Grouping for Efficient Cooperative Multi-Agent Reinforcement Learning

Abstract

Grouping is ubiquitous in natural systems and is essential for promoting efficiency in team coordination. This paper proposes a novel formulation of Group-oriented Multi-Agent Reinforcement Learning (GoMARL), which learns automatic grouping without domain knowledge for efficient cooperation. In contrast to existing approaches that attempt to directly learn the complex relationship between the joint action-values and individual utilities, we empower subgroups as a bridge to model the connection between small sets of agents and encourage cooperation among them, thereby improving the learning efficiency of the whole team. In particular, we factorize the joint action-values as a combination of group-wise values, which guide agents to improve their policies in a fine-grained fashion. We present an automatic grouping mechanism to generate dynamic groups and group action-values. We further introduce a hierarchical control for policy learning that drives the agents in the same group to specialize in similar policies and possess diverse strategies for various groups. Experiments on the StarCraft II micromanagement tasks and Google Research Football scenarios verify our method's effectiveness. Extensive component studies show how grouping works and enhances performance.

1 Introduction

Cooperative multi-agent reinforcement learning (MARL) aims to coordinate multiple agents' actions through shared team rewards and has become a helpful tool for solving multi-agent decision-making problems [47, 9, 44]. Learning centralized policies to address this problem conditions on the full state, which is usually unavailable during execution due to partial observability or communication constraints [4, 3]. An alternative paradigm is to learn decentralized policies based on local observations [36, 31]. However, simultaneous exploration suffers from non-stationarity that may cause unstable learning [7, 50]. The centralized training with decentralized execution (CTDE) inherits the advantages of these two paradigms and learns decentralized policies in a centralized fashion [23, 15].

Value Function Factorization (VFF) under the CTDE paradigm is a popular approach to MARL, where a centralized value function is learned from global rewards and factorized into local values to train decentralized policies. In recent years, value factorization methods have been widely proposed. Existing methods usually adopt the flat VFF scheme [24], which directly estimates the joint actionvalue from local utilities. These methods achieve remarkable performances from various perspectives, such as enhancing the expressiveness of the mixing network [27, 33, 38, 37] and encouraging exploration [21, 19]. However, learning efficient cooperation by directly estimating the joint actionvalues from individual utilities is exceptionally difficult. Phan et al.[24] illustrate that the flat VFF

scheme leads to a performance bottleneck, where it gets challenging to provide sufficiently informative training signals for each agent. Although the state information provides complete knowledge, it is burdensome for agents to extract effective guidance that facilitates cooperative policy learning.

Research in natural systems [13, 43] and multi-agent systems [24] has validated grouping as a means to promote efficient collaboration. Dividing the team into smaller sets allows for fine-grained learning and opens up further opportunities to integrate informative group-wise learning signals. However, formulating a general division criterion without domain knowledge is still a matter of interest to the community. Most previous works are proposed for well-structured tasks and typically predefine specific responsibilities or forms of task decomposition [1, 18, 24]. They require apriori settings that are potentially unavailable in practice and discourage methods' transferring to diverse environments.

This paper proposes a novel formulation of Group-oriented MARL (GoMARL), which learns automatic grouping without domain knowledge for efficient cooperation. GoMARL holds a dualhierarchical value factorization and learns dynamic groups with a 'select-and-kick-out' scheme. Concretely, GoMARL continuously selects agents unsuitable for their current groups based on the learning weights of the decomposition from the group-wise value to local utilities and kicks them out to reorganize the group division. Furthermore, GoMARL transforms various informative training signals, including individual group-related information, group state, and global state, into network weights, which extracts effective guidance for policy improvement and enables flexible adaptation to the dynamic changes in the number of subgroups and the number of agents per group.

The advantages of GoMARL lie in two aspects. Firstly, group information offers richer knowledge for group-wise value factorization and provides more direct guidance for efficient learning. Our value factorization with a dual hierarchy is based on a more focused and compact input representation. Unlike existing methods that learn value factorization with no information or only with the global state which is complete but hard to extract efficient guidance, GoMARL proposes a fine-grained learning scheme to integrate information from the group perspective into the policy gradient, promoting intra-group coordination with the group state and facilitating inter-group cooperation with the global state. Secondly, GoMARL extracts individual group-related information to provide informative signals for policy diversity. The agents condition their behaviors on their individual group-related information embedded by a shared encoder, which is trained following specialization guidance, i.e. , imposing similarity within a group and diversity among groups. In this way, GoMARL synergizes subgroups with policy styles, proposing a parameter-sharing mechanism for specialized policies.

We test our method on a challenging set of StarCraft II micromanagement tasks [29] and Google Research Football scenarios [16]. GoMARL achieves superior performance with greater efficiency compared with notable baseline methods. We also conduct detailed component analyses and ablation studies to give insights into how each module works and enhances learning efficiency.

2 Related Work

Value function factorization under the CTDE [23] paradigm is a popular approach to MARL. Most existing methods learn flat value factorization by treating agents as independent factors and directly estimating the joint action-value from local utilities. The earlier work, VDN [35], learns a linear decomposition into a sum of local utilities used for greedy action selection. QMIX [27] learns a non-linear mixing network with the global state and enlarges the functions the mixing network can represent, but it still faces the monotonicity constraint. QTRAN [33] further improves the expressivity by proposing the Individual-Global-Max (IGM) principle between individual utilities and the global action-value. Subsequent works follow the IGM principle and further boost performance by encouraging exploration [21, 5], enhancing the expressiveness of the mixing network [38], preventing sub-optimal convergences [26, 37], and integrating functional modules [42, 46, 45, 51, 34, 49].

This paper explores automatic group division for multi-agent teams to realize group-wise learning. Early related works [30, 28, 20, 18] predefine specific responsibilities to each agent based on goal, visibility, capability, or by search. Some other efforts achieve implicit grouping by task allocation [32, 17, 10]. These methods only address tasks with a clear structure and require domain knowledge or apriori settings. Another class of approaches focuses on individuality [41, 19, 14] or role learning [39, 40]. Among them, ROMA [39] learns dynamic roles that depend on the context agents observe. RODE [40] decomposes the joint action spaces and integrates the action effects into the role policies to boost learning. A similar work, V AST [24], also studies the impact of subgroups

on value factorization but still requires a priori of group number. Our method, GoMARL, does not rely on domain knowledge and gradually adjusts the group division according to the learned factorization weights. Based on the dynamic group division, GoMARL models several group-related signals to learn specialized policies and promote efficient team cooperation.

3 Preliminaries

Dec-POMDP. This paper focuses on cooperative tasks with n agents A = { a 1 , ..., a n } as a DecPOMDP [22] defined by a tuple G = 〈 S, U, P, r, Z, O, n, γ 〉 . The environment has a global state s ∈ S . Each agent a chooses an action u a t from its action space U a at timestep t and forms a joint action u t ∈ ( U 1 × ... × U n ) ≡ U n that induces a transition according to the state transition distribution P ( s t +1 | s t , u t ) : S × U n × S → [0 , 1] . r ( s, u ) : S × U n → R is the reward function yielding a shared reward, and γ ∈ [0 , 1) is the discount factor. We consider partially observable scenarios in which agent a acquires its local observation z a ∈ Z drawn from O ( s t , a ) : S ×A→ Z and has an actionobservation history τ a ∈ T ≡ ( U × Z ) ∗ on which it conditions a policy π a ( u a | τ a ): T × U → [0 , 1] .

Value Function Factorization and the IGM Principle. We consider cooperative MARL with the CTDE paradigm, which has been a significant focus in recent efforts [2, 35, 27, 12, 21]. A majority of methods achieve CTDE through flat value function factorization [24], i.e. , factoring action-value functions into combinations of per-agent utilities. The individual utility only depends on the local history of actions and observations, allowing agents to maximize their local utility functions independently under the Individual-Global-Max (IGM) Principle [33]: arg max u Q tot ( τ , u ) = { arg max u 1 Q 1 ( τ 1 , u 1 ) , · · · , arg max u n Q n ( τ n , u n ) } . Among these attempts, the representative deep MARL approach QMIX [27] improves the simple summation of individual utilities [35] by introducing a more expressive factorization: Q tot = f ( Q 1 ( τ 1 , u 1 ; θ Q ) , · · · , Q n ( τ n , u n ; θ Q ); θ f ) , where θ f denotes the parameters of the monotonic mixing function generated by a hypernetwork [6].

4 Group-oriented Multi-Agent Reinforcement Learning

glyph[negationslash]

GoMARLdecomposes the global action-value Q tot into groupwise values Q g and trains agents by groups in a fine-grained manner. Figure 1 illustrates the overview of the learning framework. It consists of an automatic grouping module, specialized agent networks generating local utilities Q i , and a mixing network among groups. In Section 4.1, we introduce the automatic grouping module. It progressively divides the team into dynamic groups as training proceeds. Based on the dynamic group division, we propose specialized agent networks that achieve similarity within each group and diversity among groups to generate local utilities Q i in Section 4.2. Section 4.3 presents the overall training framework to detail the estimation of the group-wise action-values and the global action-values.

4.1 Automatic Grouping Mechanism

The automatic grouping mechanism aims to learn a mapping relationship f g : A ↦→ G . The key idea is to divide the team into dynamic groups in an end-to-end fashion by maximizing the expected global return Q tot G ( s t , u t ) = E s t +1: ∞ , u t +1: ∞ [∑ ∞ k =0 γ k r t + k | s t , u t ; G ] . Value function factorization approaches represent the joint action-value as an aggregation of individual utilities, i.e. , a weighted sum of Q i and biases. We follow this setting and represent the group-wise value Q g as an aggregation of the individual value Q i . Intuitively, as illustrated in the right side of Figure 2, if the learned mixing

Definition 1. (Individual and Group) Given a cooperative task with n agents A = { a 1 , ..., a n } , we have a set of groups G = { g 1 , ..., g m } , 1 ≤ m ≤ n . Each group g j contains n j (1 ≤ n j ≤ n ) different agents, g j = { a j 1 , ..., a j n j } ⊆ A , where ⋃ j g j = A , g j ∩ g k = ∅ for j, k ∈ { 1 , 2 , . . . , m } and j = k . The superscript describes the variable owner, e.g. , u j i is the action of the i -th agent a j i in group g j . We denote joint quantities in bold and joint quantities over agents other than a given agent a with the superscript -a , e.g. , u -j i is the joint action of agents in group g j other than agent a j i .

⋯

Automatic	Grouping	Module

𝑄

!

𝜏

!

,·

𝑄

"

𝜏

"

,·

𝑄

#

𝜏

#

,·

𝑄

$%&'(

𝒈

𝟏

⋯

𝑄

$%&'(

𝒈

𝟐

𝑄

$%&'(

𝒈

𝒎

𝒎dynamic	groups

𝑄

*+*

Mixing	Network	among	Groups

…

⋯

concat

w

!

"

accord ing to grouping

𝒢

Actor

!

Actor

"

Actor

#

⋯

𝐰

𝟏

𝒈

𝟏

MLP

ℎ

$

%

𝑄

%

𝜏

%

, 𝑢

%

(𝑧

$

%

, 𝑢

$&!

%

)

ℎ

$&!

%

𝐰

𝟏

𝒈

𝒎

⋯⋯

send to corresponding

𝐰

𝟏

𝒈

⋯⋯

𝐰

𝟏

𝒈

𝟏

𝐰

𝟏

𝒈

𝒎

Group	Adjustment	Operator

GRU

MLP

ℎ

%

!

ℎ

%

&

ℎ

%

'

⋯

w

!

!

Individual

w

!

Generator

⋯

w

!

"

w

!

#

If vector

w

!

"

→ 0

,

𝑄

!

shows a little contribution to

𝑄

(

!

:

⋯

A new grouping

𝒢

)

=

𝑎

!

, ⋯ , 𝑎

"*!

, 𝑎

"+!

, ⋯ 𝑎

'

, 𝒂

𝒊

𝑸

𝒊

𝑄

!

𝑄

'

⋯

𝟎

𝟎

𝟎

𝑛 local Q-values

𝑘	hidden	values

0×𝑄

"

+

⋯

0×𝑄

"

+

⋯

1

2

𝑘

Initial grouping

𝒢 =

𝑔

!

= 𝑎

!

, ⋯ , 𝑎

"

, ⋯ 𝑎

'

.

The grouping updates every

𝑐

timesteps.

More than one agent can be selected at one time, and

the selected agents are assigned to the next group,

e.g.

:

𝒢

))

= 𝑔

!

, 𝑔

&

=

𝒜

*",𝒋,𝒌

,

𝑎

"

, 𝒂

𝒋

, 𝒂

𝒌

.

𝒢

)))

= 𝑔

!

, 𝑔

&

, 𝑔

0

=

𝒜

*",1,2,𝒍

,

𝑎

1

, 𝒂

𝒍

, 𝒂

𝒊

, 𝒂

𝒌

.

Until a stable

𝒢 @ = 𝑔

!

, ⋯ , 𝑔

4

is acquired.

0×𝑄

"

+

⋯

…

weight of Q i is small enough, then Q i contributes a little to its group-wise value Q g . In other words, when an agent a j i takes action u j i but does not affect its group-wise value Q g j , it indicates that agent a j i does not belong to group g j anymore and needs group adjustment.

We factorize the group value Q g into individual utilities Q i with group-wise mixing weights w 1 , i.e. , Q g j = f ( Q j 1 ( τ j 1 , u j 1 ) , · · · , Q j n j ( τ j n j , u j n j ); w g j 1 ) , where w g j 1 denotes group g j 's mixing parameters generated by individual w 1 generators f i w 1 ( · ; θ i w 1 ) : τ i → w i 1 that map each agent a i 's history information τ i to a k -dimensional weight vector. A regularization on this w 1 generator f i w 1 drives the automatic grouping mechanism. Specifically, GoMARL 'selects and kicks out' agents whose individual utilities hold small mixing weights and contribute a little to their group-wise Q values; thus, a sparsity regularization on w 1 is implemented to select those agents with small groupvalue mixing weights. The w 1 generators f i w 1 are trained by minimizing the following loss function:

where B is the replay buffer, and ‖ · ‖ l 1 stands for the l 1 -norm penalty.

Figure 2 illustrates the schematic diagram of the grouping mechanism. Initially, all the agents belong to the same group, and the grouping G is adjusted as training proceeds. The grouping shifts every c timesteps, and each selected agent is assigned to the following group (a new one for agents in the last group) until it properly contributes to where it belongs. When adjusting groups, the w 1 of all agents are sent to a group adjustment operator O g . According to the adjusted grouping G ,

O g : { w 1 1 , · · · , w n 1 } G -→ { w g 1 1 , · · · , w g m 1 } concatenates the w i 1 of agents in the same group to form a set of group-wise w g 1 to generate group action-value. This implementation flexibly adapts to the grouping dynamics since each w i 1 is tied to agent a i by the individual w 1 generator f i w 1 . No matter which group g j agent a i belongs to, Q i can engage in the estimation of Q g j through w i 1 ∈ w g j 1 .

In practice, the weights' shrinkage to zero is infeasible. It is also challenging to determine a fixed threshold for groups of various sizes in diverse environments. We empirically utilize seventy percent of each group's average weight to assess whether an agent fits its current group. Extensive experiments and component studies in Section 5 confirm the universality of this configuration.

4.2 Specialized Agent Network for Decentralized Execution

Group status describes the cooperation among agents within each group and is essential to local utility generation. Integrating group-wise information e into the decision-making process enables consideration of cooperative behaviors. In this section, we introduce GoMARL's agent network to generate local action-value functions with individual group-related information e embedded.

As shown in Figure 3, we construct a group-related info encoder f e ( · ; θ e ) to embed agents' hidden states. To achieve a group-related view, we train the encoder network as an extractor, where the extracted agent info e of agents from the same group should be similar. To avoid all agents' e i

collapsing to be alike, the regularizer also encourages diversity between agents from different groups. Formally, we minimize the following similarity-diversity objective to train the encoder f e :

glyph[negationslash]

The encoder trained by the SD-loss extracts agent info e i that is recognizable to agents' groups. This group-related information is then fed into a decoder f d ( · ; θ d ) to generate the parameters of the agent network's upper MLP. In this way, each agent a i conditions its behaviors on e i with group-related information embedded, promoting potential cooperation within each group during decentralized execution. The decoder hypernetwork f d is trained by the TDloss introduced in Section 4.3.

glyph[negationslash]

ℎ

!

"

Actor

"

Actor

#

⋯

ℎ

!

#

MLP

GRU

MLP

ℎ

!

$

𝑄

!

𝜏

!

, 𝑢

!

; 𝑒

!

(𝑧

!

$

, 𝑢

!%"

$

)

ℎ

!%"

$

𝑒

!

"

,

⋯

,

𝑒

!

#

…

…

SD	Loss

TD	Loss

𝑣

&

Data Flow

Gradient Flow

Produce

Group-

related	Info

Encoder

Group-

related	Info

Decoder

The proposed actor network has two merits. Firstly, it enables diversified policies while sharing all the parameters. Policy decentralization with shared parameters is widely utilized to improve scalability and learning efficiency. However, agents tend to behave similarly when sharing parameters, preventing effective exploration and complex cooperative policies. It is also undesirable to entirely forgo shared parameters in pursuit of diversity since proper sharing accelerates learning. Our method hybridizes the efficiency of parameter-sharing and the policy diversity needed for complex collaboration. Secondly, the decoder hypernetwork f d integrates the extracted agent info e into the policy gradients, providing informative group-related information to enrich local utilities and promote intra-group cooperation. Concretely, the partial derivative for updating the parameters θ h of the GRU and the bottom MLP is:

where v h is the representation after the GRU illustrated in the left side of Figure 3. Eqn.(3) shows that e a is deeply involved in the policy updating of agent a , providing richer group knowledge to generate local utilities and facilitating group-related guidance for cooperative decentralized execution.

4.3 Overall Learning Framework

We next introduce GoMARL's overall learning framework. It contains two mixing networks that generate the group-wise Q g and the global Q tot , respectively. As for the group mixing network, although w g 1 enables a weighted mixing of local utilities of agents in group g , the naive mixture of ∑ a ∈ g w a 1 Q a + b lacks the guidance of group status to reflect the group action-value in a specific group-wise state . Therefore, we build the group mixing network with a two-layer structure that also embeds the group-wise state

MLP

GRU

MLP

ℎ

!"#

$

ℎ

!

$

𝐰

𝟏

Generator

Group-

related	Info

Encoder

Pooling

Operation

𝑒

!

#

,

𝑒

!

&

,

⋯

,

𝑒

!

'

𝑠

!

(

!

,

⋯

,

𝑠

!

(

"

Group State

Group	Adjustment

Operator

𝑄

$

𝜏

$

, 𝑢

$

; 𝑒

$

𝐰

𝟏

𝒈

𝒎

⋯

𝐰

𝟐

𝒈

𝒎

𝐰

𝟏

𝒈

𝟏

𝐰

𝟐

𝒈

𝟏

⋯

𝑄

!"#$%

𝒈

𝟏

𝑄

!"#$%

𝒈

𝒎

⋯

Total

Mixing

𝑄

$%$

Group-wise

𝐰

𝟐

Generator

Grouping	Index	𝒢

𝑄

#

𝜏

#

, 𝑢

#

; 𝑒

#

𝑄

'

𝜏

'

, 𝑢

'

; 𝑒

'

𝐀𝐜𝐭𝐨𝐫

𝒂

(𝑧

!

$

, 𝑢

!"#

$

)

𝑣

+

into the weights w g 2 to generate Q g . In particular, group g j 's group-wise state s g j is a fusion of the agent info e i for all a i ∈ g j . To cohesively summarize the group state based on the agent info of all agents in group g j , we apply a pooling operation [25] over each dimension of the agent info e i to generate the group-wise state s g j describing the current group status. The pooling operation also ensures adaptability to the dynamic group size ( i.e. , the number of agents per group). We build a group-wise w 2 generator f w 2 ( s g ) to map the fused group state into w g 2 . Similar to the group-related info encoder f e , the w 2 generator integrates the group state s g into the policy gradient: ∂Q g ∂Q a = ∂Q g ∂v a Q ∂v a Q ∂Q a = f w 2 ( s g ) · ∂v a Q ∂Q a , providing group status information for Q g generation that facili- tates efficient intra-group cooperation . v Q is the representation after w g 1 marked in Figure 4. The

two-layer mixing structure of the group mixing network estimates the group action-value Q g by w 1 which decides the group division and w 2 which carries group status information.

GoMARLestimates the global action-value Q tot by mixing all the Q g in a similar fashion. Concretely, the two layers of the total mixing network are generated by two hypernetworks, respectively taking group states s g and the full state s as inputs. Likewise, s g and s are deeply involved in the gradients, promoting inter-group cooperation . The architecture of the total mixing network is akin to the group mixing network and is omitted in Figure 4 for brevity. The TD-loss of the estimated Q tot is:

where ¯ Q tot is a target network with periodic updates. The overall learning objective of GoMARL is:

where θ = ( θ h , θ e , θ d , θ w ) . θ w denotes the parameters of hypernetworks producing all the mixing weights and biases. λ g and λ SD are two scaling factors.

Although containing two mixing networks, the total mixing-net size of GoMARL is smaller than the commonly used single monotonic mixing network [27], as verified in Section 5.1. This is mainly attributed to the input dimension reduction. The monotonic mixing network takes the global state s as input of all the hypernetworks. In contrast, we take specific group information ( i.e. , individual group-related info e , group state s g , and state s is only used in one hypernetwork). This parameter reduction offsets the increase of an extra mixing network. Compared with flat value factorization methods learning only with the global state, our method allows fine-grained learning guided by more direct signals embedded in the policy gradients, facilitating intra- and inter-group cooperation.

5 Experiments

Baselines. We compare GoMARL with prominent baselines to verify its effectiveness and efficiency. Hu et al. [8] fairly compared existing MARL methods without code-level optimizations and reported that QMIX [27] and QPLEX [38] are the top two value factorization methods. The authors also finetuned QMIX (denoted as Ft-QMIX in our paper), which attains higher win rates than the vanilla QMIX. VAST [24] learns value factorization for sub-teams based on apriori setting on group number. Therefore, we compare GoMARL with Ft-QMIX, QPLEX, and VAST to show its performance as a value factorization method. The baselines also include role-based methods ROMA [39] and the representative credit assignment method RIIT [8]. The latter combines effective modules of noticeable methods, and comparing them can further illustrate the superiority of GoMARL.

Experimental setup. All the methods are trained with 8 parallel runners for 10 Msteps and are evaluated every 10 K steps with 32 episodes. We report the 1 st, median, and 3 rd quartile win rates across 5 random seeds. Please refer to Appendix B for experimental setup details.

5.1 Performance on SMAC

Methods are evaluated on six challenging SMAC maps. The chosen maps involve homogeneous and heterogeneous teams with asymmetric battles, allowing a holistic study of all methods.

Overall performance. The performance comparison of all the methods in the six Hard and Super Hard SMAC maps is shown in Figure 5. As the results show, each baseline method only achieves satisfactory performance on some of the tasks with specific properties they specialize in; e.g. , RIIT performs well on MMM2 but converges much slower in other tasks; QPLEX's leaning is not efficient in 8m_vs_9m and corridor . Ft-QMIX significantly outperforms the vanilla QMIX and has more efficient learning than other baselines. GoMARL has a similar performance as Ft-QMIX in 8m_vs_9m and corridor . However, the superiority of GoMARL can be clearly validated in all the other maps.

Parameter size for value mixing. Many methods use larger mixing networks with stronger expressiveness and fitting abilities to obtain superior performance. However, they often fail when compared to baselines that use a mixing net of the same size [8]. As shown in Table 1, our dual-hierarchical mixing architecture has fewer parameters when there are a large number of agents (when n > 5 ). VAST is not compared since it utilizes a linear summation of local utilities. The results of GoMARL are the

80

3s5z vs 3s6z

QPLEX

RODE

ROMA

RIIT

VAST

80

3s5z vs 3s6z

70

3s5z vs 3s6z

QPLEX

60

RODE

ROMA

50

80

3s5z vs 3s6z

70

QPLEX

RODE

ROMA

RIIT

80

3s5z vs 3s6z

QPLEX

QPLEX

RODE

80

80

70

3s5z vs 3s6z

3s5z vs 3s6z

70

2

4

6

8

10

Steps

(

×

10

6

)

0

10

20

30

40

50

60

70

80

Test Win Rate (%)

5m vs 6m

2

4

6

8

10

Steps

(

×

10

6

)

0

20

40

60

80

100

Test Win Rate (%)

3s vs 5z

2

4

6

8

10

Steps

(

×

10

6

)

0

20

40

60

80

100

Test Win Rate (%)

8m vs 9m

0

2

4

6

8

10

Steps

(

10

6

)

0

20

40

60

80

100

Test Win Rate (%)

corridor

2

4

6

8

10

Steps

(

10

6

)

0

20

40

60

80

100

Test Win Rate (%)

MMM2

0

2

0

10

20

30

GoMARL

0

2

4

6

Steps

(

×

10

6

0

10

20

30

40

VAST

GoMARL

0

2

4

6

8

Steps

(

×

10

6

)

0

10

20

30

40

50

RIIT

VAST

GoMARL

0

2

4

6

8

10

Steps

(

×

10

6

)

0

10

20

30

40

50

ROMA

RIIT

VAST

GoMARL

0

2

4

6

8

10

Steps

(

×

10

6

)

0

10

20

30

40

50

60

70

Test Win Rate (%)

QPLEX

RODE

ROMA

RIIT

VAST

GoMARL

0

2

4

6

8

10

Steps

(

×

10

6

)

0

10

20

30

40

50

60

70

Test Win Rate (%)

QPLEX

RODE

ROMA

RIIT

VAST

GoMARL

0

2

4

6

8

10

Steps

(

10

6

)

0

10

20

30

40

50

60

70

80

Test Win Rate (%)

3s5z vs 3s6z

0

2

4

6

8

10

Steps

(

×

10

6

)

0

10

20

30

40

50

60

70

80

Test Win Rate (%)

QPLEX

RODE

ROMA

RIIT

VAST

GoMARL

0

2

4

6

8

10

Steps

(

×

10

6

)

0

10

20

30

40

50

60

Test Win Rate (%)

RODE

ROMA

RIIT

VAST

GoMARL

×

×

Test Win Rate (%)

×

average of the five runs since each run may learn a slightly different grouping with various group numbers. GoMARL outperforms other methods despite using fewer mixing parameters, highlighting its inherent superiority over methods relying on stronger mixing networks.

Component analysis and ablation study. We next conduct detailed component studies to analyze how GoMARL improves efficiency and enhances performance. GoMARL contains three key components: (1) an automatic grouping mechanism that progressively divides the team into proper groups; (2) specialized agent networks that generate diversified policies while sharing all the parameters; and (3) sufficiently informative signals integrated into the gradients to promote efficient cooperation. We respectively study each module on three Super Hard maps to show how they influence performance.

(1.1) Ablation study of the grouping mechanism. We validate our grouping mechanism by comparing GoMARL with other intuitive alternatives. All methods utilize the same architecture as GoMARL to reflect how grouping itself influences performance. As shown in the top row of Figure 6, setting all agents as a group in corridor converges faster but has significant variances. It may be due to the hard exploration of efficient cooperation without grouping guidance. Another two intuitive groupings, each agent a group and an equal division into two groups {{ a 1 , a 2 , a 3 } , { a 4 , a 5 , a 6 }} , have minor variance. However, their learning efficiency is affected since inappropriate groupings fail to promote cooperative behaviors. MMM2 contains heterogeneous agents; thus, a natural grouping is to keep the agents of the same type in one group. This natural grouping outperforms most alternatives but is inferior to our dynamic grouping adjustment. 3s5z_vs_3s6z is another scenario with heterogeneous agents, and the natural grouping of setting homogeneous agents as a group is also studied. It performs nearly the same as GoMARL because our mechanism learns the grouping precisely according to the agents' type in this scenario. We also analyze this map with a grouping containing three groups {{ a 1 , a 4 , a 5 } , { a 2 , a 6 , a 7 } , { a 3 , a 8 }} (1s2z, 1s2z, 1s1z); however, this balanced grouping fails to form effective collaboration. We can see from these results that appropriate grouping facilitates efficient cooperation and accelerates learning. Our grouping mechanism automatically learns adaptive grouping in different tasks and assists GoMARL with superior performance and efficiency.

(1.2) Learned grouping analysis. To demonstrate whether the learned grouping makes sense, we further visualize the trained strategy in a corridor battle, as illustrated in Figure 7. Six allied Zealots

60

70

60

60

50

40

Test Win Rate (%)

80

Test Win Rate (%)

Test Win Rate (%)

)

3s5z vs 3s6z

4

6

Steps

10

8

(

×

10

6

)

10

8

2

4

6

8

10

Steps

(

×

10

6

)

0

10

20

30

40

50

60

70

80

Test Win Rate (%)

Component study on 3s5z vs 3s6z

Each agent a group

All agents a group

Three groups ([145][267][38])

Same type of agents a group

Dynamic grouping

2

4

6

8

10

Steps

(

10

6

)

0

10

20

30

40

50

60

70

80

Test Win Rate (%)

Ablation on 3s5z vs 3s6z

GoMARL w/o group-related agent info

GoMARL w/o group state information

GoMARL

0

1

2

3

4

5

Steps

(

×

10

6

)

0

20

40

60

80

100

Test Win Rate (%)

Component study on corridor for 5M steps

Each agent a group

All agents a group

Two groups ([123][456])

Dynamic grouping

0

1

2

3

4

5

Steps

(

×

10

6

)

0

20

40

60

80

100

Test Win Rate (%)

Component study on MMM2 for 5M steps

Each agent a group

All agents a group

Same type of agents a group

Dynamic grouping

0

1

2

3

4

5

Steps

(

10

6

)

0

20

40

60

80

100

Test Win Rate (%)

Ablation on corridor for 5M steps

GoMARL w/o group-related agent info

GoMARL w/o group state information

GoMARL

0

1

2

3

4

5

Steps

(

10

6

)

0

20

40

60

80

100

Test Win Rate (%)

Ablation on MMM2 for 5M steps

GoMARL w/o group-related agent info

GoMARL w/o group state information

GoMARL

×

×

×

Agent2

Agent2

18Enemies

Agent1&2died.

Agent3-6

Agent3-5survived

Agent1

6Enemies

Agent1

Agent 3-6fight asa group

ThelastEnemy

Agent3-6

Agent3-6

tokill survivingenemies.

fight twenty-four Zerglings on this Super Hard map. The massive disparity in unit numbers between the two sides implies that the whole team must refrain from launching an attack together. The only winning strategy is to sacrifice a small number of agents who leave the team and attract the attention of most enemies. Taking this opportunity, our large force eliminates the rest of the enemies. The surviving agents then use the same tactic to keep attracting several enemies and kill them together. In this battle, Agent 1 and 2 sacrifice themselves to attract most enemies and bring enough time for the team to eliminate the remaining enemies. Other agents fight as a subgroup and successfully kill all the surviving enemies. GoMARL learns a double-group setting in this battle, where Agent 1 and 2 are in the same group while the others are set in another group. This grouping is explicable in light of the combat situation, and this reasonable grouping guidance contributes to our superior performance.

(2) Component study of the specialized agents. We transplant our specialized agent network (SAN) into other baselines to verify module effectiveness. ROMA is not included since its actors are produced by its learned roles, and the replacement will invalidate the method. As shown in Figure 8, all methods are improved when equipped with SAN. However, even with SAN, all baselines fail to surpass GoMARL.

0

20

40

60

80

Test Win Rate (%)

Component study of the Specialized Agent Network

120

2019 Olympics Top Scorers

GoMARL w/o SAN

GoMARL

100

120

2019 Olympics Top Scorers

VAST

VAST+SAN

80

100

120

Medals

2019 Olympics Top Scorers

RIIT

RIIT+SAN

60

80

100

120

Medals

2019 Olympics Top Scorers

QPLEX

QPLEX+SAN

40

60

80

100

120

Medals

Ft-QMIX+SAN

80

60

100

USA

India

China

Russia

Germany

USA India China Russia Germany Countries 20 40 USA India China Russia Germany Countries 20 Countries As shown in the learning curves in Appendix C, although dynamic grouping may reduce learning efficiency in the early stage of training, the learned grouping will significantly accelerate learning in the later stage. Therefore, the automatic grouping module and SAN are both crucial to GoMARL.

(3) Ablation study of the informative group-related signals. Our method models individual grouprelated information e and group state s g to integrate them into the gradients. The former adjusts local utilities for cooperative behaviors and realizes policy diversity, while the latter fosters efficient group cooperation. We ablate them respectively to validate their effectiveness. Concretely, we replace the group state s g with the full state s and remove the group-related information e to degenerate the specialized agent networks into vanilla agent networks. As shown in the bottom row of Figure 6, both signals greatly improve learning efficiency. On the one hand, the agent info e carrying group-related information is trained to encourage inter-group diversity, enabling extensive cooperative exploration and learning acceleration. On the other hand, the fused s g summarizes informative group-related status that facilitates efficient group-wise coordination. Although the global state s provides complete

2019 Olympics Top Scorers

20

40

Test Win Rate (%)

80

80

70

60

50

3s5z vs 3s6z

QPLEX

RODE

ROMA

RIIT

3s5z vs 3s6z

QPLEX

RODE

ROMA

RIIT

VAST

3s5z vs 3s6z

80

80

3s5z vs 3s6z

70

70

QPLEX

60

0

Test Win Rate (%)

8

80

70

60

50

40

×

10

RODE

QPLEX

80

3s5z vs 3s6z

70

60

3s5z vs 3s6z

QPLEX

RODE

ROMA

0

2

0

10

20

30

GoMARL

0

2

4

6

Steps

(

×

10

6

)

0

10

20

30

40

VAST

GoMARL

0

2

4

6

8

Steps

(

×

10

6

)

0

10

20

30

40

50

RIIT

VAST

GoMARL

10

20

30

40

50

60

70

Test Win Rate (%)

QPLEX

RODE

ROMA

RIIT

VAST

GoMARL

10

20

30

40

50

60

70

QPLEX

RODE

ROMA

RIIT

VAST

GoMARL

0

2

4

6

8

10

Steps

(

10

6

)

0

10

20

30

40

50

ROMA

RIIT

VAST

GoMARL

0

10

20

30

40

50

60

Test Win Rate (%)

RODE

ROMA

RIIT

VAST

GoMARL

2

4

6

8

10

Steps

(

10

6

)

0

10

20

30

40

50

60

70

80

Test Score Reward (%)

academy 3 vs 1 with keeper

2

4

6

8

10

Steps

(

10

6

)

0

10

20

30

40

50

60

Test Score Reward (%)

academy counterattack easy

2

4

6

8

10

Steps

(

10

6

)

0

20

40

60

80

Test Score Reward (%)

academy pass and shoot with keeper

0

80

0

×

2

Test Win Rate (%)

×

Test Win Rate (%)

4

6

0

2

4

6

8

×

10

Passes

The keeper prevent

Agent 1's shooting.

Passes to Agent 3

Agent 3 shoots

The keeper saves the ball.

Agent 4 gives

a second shoot

and scores.

DIRECTION ACTION

0TRQ

sprint

sprint

shot

SCORE:0-0

BALLONNEI):

Agent1

DIRECTIONACTION

sprint

shot

Opponent

player

SCORE:O-0

BALL OWNED:

Agent1

DIRECTION ACTION

RQ

Idle

BALL OWNED:

DIRECTIONACTION

Agent3

sprint

player

Opponent

SCORE:0-0

Keeper

Agent2

Opponent

Keeper

Agent3

player

Agent3

Keeper

Agent4

Agent3

Agent4

knowledge, it is burdensome for agents to extract effective guidance that promotes policy learning. Therefore, utilizing our specific group-wise signals e and s g is more efficient than the global state s .

5.2 Performance on Google Research Football

We also test GoMARL on three challenging Google Research Football (GRF) offensive scenarios. Agents in GRF need to coordinate timing and positions for organizing offense to seize fleeting opportunities, and only scoring leads to rewards. Therefore, the GRF tasks are more complicated than the SMAC battles, and the comparison in GRF is a secondary proof of our method's effectiveness.

Performance. Figure 9 shows the performance of all methods in GRF. GoMARL, Ft-QMIX, and QPLEX all perform well in pass_and_shoot_with_keeper . However, the advantages of our method become increasingly evident in the other two environments requiring more coordination. Only GoMARL achieves over 50% of the score reward in all scenarios. The superior performance with significant efficiency in the second testbed further demonstrates the transferability of our method.

Visualization. The trained strategy in counterattack_easy is visualized to validate if the learned group division makes sense. GoMARL learned a reasonable grouping G = {{ a 1 , a 2 } , { a 3 , a 4 }} for this complex goal, in which the first group brought the ball into the penalty area through smooth coordination, while the second group created two shoots and the final goal through skillful cooperation, as shown in Figure 10. Appendix D provides a detailed discussion of this visualization.

6 Conclusion and Future Work

Grouping is essential to the efficient cooperation of multi-agent systems. Instead of utilizing apriori knowledge, this paper proposes an automatic grouping mechanism that gradually learns reasonable grouping as training proceeds. Based on the dynamic group division, we further model informative group-related signals to achieve fine-grained value factorization and encourage policy specialization, promoting efficient intra- and inter-group coordination. With these novelties, our method GoMARL achieves impressive performance with high efficiency on the SMAC and GRF benchmarks.

The automatic grouping mechanism of GoMARL is currently grounded in the value decomposition process, rendering it inapplicable to policy-based methods. Our future research will delve into automatic grouping based on policy gradients to enhance the learning efficiency of the policy-based methods. Furthermore, the current version of GoMARL is only designed for one-stage and one-level grouping, and it possesses the potential to achieve finer granularity in group division. We will investigate multi-stage and multi-level grouping (i.e., group-within-a-group) in the future to pursue better performance and higher learning efficiency in more complex environments.

Test Win Rate (%)

10

3s5z vs 3s6z

4

6

8

Steps

(

×

10

6

10

)

8

References

Noelie Bonjean, Wafa Mefteh, Marie-Pierre Gleizes, Christine Maurel, and Frédéric Migeon. Adelfe 2.0. Handbook on Agent-Oriented Design Processes , pages 19-63, 2014.

Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, pages 2974-2982, 2018.

Sven Gronauer and Klaus Diepold. Multi-agent deep reinforcement learning: A survey. Artificial Intelligence Review , pages 1-49, 2021.

Jayesh K. Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multi-Agent Systems , pages 66-83, 2017.

Tarun Gupta, Anuj Mahajan, Bei Peng, Wendelin Böhmer, and Shimon Whiteson. Uneven: Universal value exploration for multi-agent reinforcement learning. In International Conference on Machine Learning , pages 3930-3941. PMLR, 2021.

David Ha, Andrew Dai, and Quoc V. Le. Hypernetworks. arXiv preprint arXiv:1609.09106 , 2016.

Pablo Hernandez-Leal, Bilal Kartal, and Matthew E. Taylor. A survey and critique of multiagent deep reinforcement learning. In International Conference on Autonomous Agents and MultiAgent Systems , volume 33, pages 750-797, 2019.

Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, and Shih-wei Liao. Riit: Rethinking the importance of implementation tricks in multi-agent reinforcement learning. arXiv preprint arXiv:2102.03479 , 2021.

Maximilian Hüttenrauch, Adrian Šoši´ c, and Gerhard Neumann. Guided deep reinforcement learning for swarm systems. arXiv preprint arXiv:1709.06011 , 2017.

Shariq Iqbal, Robby Costales, and Fei Sha. Alma: Hierarchical learning for composite multiagent tasks. In Advances in Neural Information Processing Systems , volume 35, pages 71557166, 2022.

Shariq Iqbal, Christian A. Schroeder de Witt, Bei Peng, Wendelin Böhmer, Shimon Whiteson, and Fei Sha. Randomized entity-wise factorization for multi-agent reinforcement learning. In International Conference on Machine Learning , pages 4596-4606. PMLR, 2021.

Shariq Iqbal and Fei Sha. Actor-attention-critic for multi-agent reinforcement learning. In International Conference on Machine Learning , pages 2961-2970. PMLR, 2019.

Raphaël Jeanson, Penelope F. Kukuk, and Jennifer H. Fewell. Emergence of division of labour in halictine bees: Contributions of social interactions and behavioural variance. Animal Behaviour , 70(5):1183-1193, 2005.

Jiechuan Jiang and Zongqing Lu. The emergence of individuality. In International Conference on Machine Learning , pages 4992-5001. PMLR, 2021.

Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing , 190:82-94, 2016.

Karol Kurach, Anton Raichuk, Piotr Sta´ nczyk, Michał Zaj ˛ ac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, and Sylvain Gelly. Google research football: A novel reinforcement learning environment. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 4501-4510, 2020.

Youngwoon Lee, Jingyun Yang, and Joseph J. Lim. Learning to coordinate manipulation skills via skill behavior diversification. In International Conference on Learning Representations , 2020.

Kemas M Lhaksmana, Yohei Murakami, and Toru Ishida. Role-based modeling for designing agent behavior in self-organizing multi-agent systems. International Journal of Software Engineering and Knowledge Engineering , 28(01):79-96, 2018.

Chenghao Li, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang. Celebrating diversity in shared multi-agent reinforcement learning. In Advances in Neural Information Processing Systems , volume 34, pages 3991-4002, 2021.

Kathryn Macarthur, Ruben Stranders, Sarvapali Ramchurn, and Nicholas Jennings. A distributed anytime algorithm for dynamic task allocation in multi-agent systems. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 25, pages 701-706, 2011.

Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. In Advances in Neural Information Processing Systems , volume 32, pages 7611-7622, 2019.

Frans A. Oliehoek and Christopher Amato. A Concise Introduction to Decentralized POMDPs , volume 1. Springer, 2016.

Frans A. Oliehoek, Matthijs T.J. Spaan, and Nikos Vlassis. Optimal and approximate q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research , 32:289-353, 2008.

Thomy Phan, Fabian Ritz, Lenz Belzner, Philipp Altmann, Thomas Gabor, and Claudia LinnhoffPopien. Vast: Value function factorization with variable agent sub-teams. In Advances in Neural Information Processing Systems , volume 34, pages 24018-24032, 2021.

Marc'Aurelio Ranzato, Y-Lan Boureau, and Yann LeCun. Sparse feature learning for deep belief networks. In Advances in Neural Information Processing Systems , volume 20, pages 1185-1192, 2007.

Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems , volume 33, pages 10199-10210, 2020.

Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning , pages 4295-4304. PMLR, 2018.

Stuart J. Russell and Andrew L. Zimdars. Q-decomposition for reinforcement learning agents. In International Conference on Machine Learning , pages 656-663. PMLR, 2003.

Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. In International Conference on Autonomous Agents and Multi-Agent Systems , pages 2186-2188, 2019.

Jeff Schneider, Weng-Keen Wong, Andrew Moore, and Martin Riedmiller. Distributed value functions. In International Conference on Machine Learning , pages 371-378, 1999.

Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip H.S. Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533 , 2020.

Tianmin Shu and Yuandong Tian. M3rl: Mind-aware multi-agent management reinforcement learning. In International Conference on Learning Representations , 2019.

Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning , pages 5887-5896. PMLR, 2019.

Jianyu Su, Stephen Adams, and Peter Beling. Value-decomposition multi-agent actor-critics. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 11352-11360, 2021.

Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value-decomposition networks for cooperative multi-agent learning based on team reward. In International Conference on Autonomous Agents and Multi-Agent Systems , pages 2085-2087, 2018.

Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In International Conference on Machine Learning , pages 330-337, 1993.

Lipeng Wan, Zeyang Liu, Xingyu Chen, Xuguang Lan, and Nanning Zheng. Greedy based value representation for optimal coordination in multi-agent reinforcement learning. In International Conference on Machine Learning , pages 22512-22535. PMLR, 2022.

Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. In International Conference on Learning Representations , 2021.

Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: Multi-agent reinforcement learning with emergent roles. In International Conference on Machine Learning , pages 9876-9886. PMLR, 2020.

Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles to decompose multi-agent tasks. In International Conference on Learning Representations , 2021.

Tonghan Wang, Jianhao Wang, Yi Wu, and Chongjie Zhang. Influence-based multi-agent exploration. In International Conference on Learning Representations , 2019.

Chao Wen, Xinghu Yao, Yuhui Wang, and Xiaoyang Tan. Smix ( λ ): Enhancing centralized value functions for cooperative multi-agent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 7301-7308, 2020.

G. Wittemyer and Wayne M. Getz. Hierarchical dominance structure and social organization in african elephants, loxodonta africana. Animal Behaviour , 73(4):671-681, 2007.

Zhao Xu, Yang Lyu, Quan Pan, Jinwen Hu, Chunhui Zhao, and Shuai Liu. Multi-vehicle flocking control with deep deterministic policy gradient method. In International Conference on Control and Automation , pages 306-311. IEEE, 2018.

Yaodong Yang, Jianye Hao, Guangyong Chen, Hongyao Tang, Yingfeng Chen, Yujing Hu, Changjie Fan, and Zhongyu Wei. Q-value path decomposition for deep multiagent reinforcement learning. In International Conference on Machine Learning , pages 10706-10715. PMLR, 2020.

Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.03939 , 2020.

Dayong Ye, Minjie Zhang, and Yun Yang. A multi-agent framework for packet routing in wireless sensor networks. Sensors , 15(5):10026-10047, 2015.

Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. In Advances in Neural Information Processing Systems , volume 35, pages 24611-24624, 2022.

Yifan Zang, Jinmin He, Kai Li, Haobo Fu, Qiang Fu, and Junliang Xing. Sequential cooperative multi-agent reinforcement learning. In International Conference on Autonomous Agents and Multi-Agent Systems , pages 485-493, 2023.

Kaiqing Zhang, Zhuoran Yang, and Tamer Ba¸ sar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control , pages 321-384, 2021.

Meng Zhou, Ziyu Liu, Pengwei Sui, Yixuan Li, and Yuk Ying Chung. Learning implicit credit assignment for cooperative multi-agent reinforcement learning. In Advances in Neural Information Processing Systems , volume 33, pages 11853-11864, 2020.

A The Insight of the Group Shifting Scheme

The proposed automatic grouping mechanism dynamically adjusts the group division as the training proceeds. In the beginning, all agents belong to one group. As introduced in Section 4.1, we examine the learned w i 1 of each agent every c timesteps to check if the grouping needs adjustment. If there are agents who contribute little to their group, it indicates that these agents do not belong to their current group. All these selected agents are assigned to the following group until they appropriately contribute to where they belong. A new group is built for selected agents in the last group.

The proposed scheme ensures that agents who belong to the same group will not be misassigned to different groups after training. For the example in the right side of Figure 1, agent a i is first selected after α 1 · c timesteps' training, and the initial grouping G = { a 1 , a 2 , · · · , a n } changes to G ′ = {{ a 1 , · · · , a i -1 , a i +1 , · · · , a n } , { a i }} . Subsequently, after another training period, at timestep α 2 · c , two agents a j and a k in the first group are selected simultaneously to be moved out of the first group. At this point, they are automatically placed in the second group, i.e. , the group to which agent a i belongs, and G ′′ = {{A -i,j,k } , { a i , a j , a k }} . Instead of placing a j and a k in a brand new group, our group shifting scheme ensures that they have the opportunity to train with a i together in one group, determining if a i , a j , and a k (or two of them) are supposed to be in a group.

Later, agent a l in the first group and agent a i , a k in the second group are chosen at timestep α 3 · c . The selection of a i and a k indicates that agent a j cannot cooperate well with them. Therefore, a i and a k are set in a new group (they should not return to the first group since they have been proved inappropriate for the first group), while a j stays alone in the second group. Agent a l from the first group is assigned automatically to the second group { a j } . If a l is supposed to be with a i and a k , it will be selected afterward. Because a j fails to form efficient cooperation with { a i , a k } , and if a j also cannot cooperate well with a l , then a l will be selected later on and set in the third group { a i , a k } .

The group shifting scheme of GoMARL assigns the selected agents to their following group and ensures that each group is trained for a fixed period of cooperative attempts. As analyzed above, each agent can be grouped with appropriate agents that can efficiently work together after training.

B Experiment Details

B.1 Detailed Experimental Setup

We compare all the methods in six Hard and Super Hard StarCraft II micromanagement tasks (SMAC) [29] and three challenging Google Research Football (GRF) [16] scenarios. When resources are available, parallel training can facilitate faster convergence of methods [8] and is very common in RL and MARL communities [39, 45, 11]. GoMARL and all the baseline methods in our paper are trained with 8 parallel runners for 10M steps in both testbeds. Therefore, the timestep is not comparable to some papers that conduct experiments with only one 'episode" runner. We use one NVIDIA Titan V GPU for training. We evaluate each method every 10K steps with 32 episodes and report the 1st, median, and 3rd quartile win rates across 5 random seeds. The detailed setting of GoMARL's hyperparameters is shown in our source code 1 .

Many algorithms introduce implementation tricks when they are built. These code-level optimizations were studied in depth in [31, 48, 8] and were shown to have a significant impact on algorithm performance. To ensure fair comparisons, our experiments are based on the PyMARL2 [8] framework proposed to compare algorithms fairly. Please refer to PyMARL2's open-source implementation for further training details and fair comparison settings. In addition, some methods implement specific parameter tuning in diverse scenarios, and is unfair for comparison. Therefore, we fixed parameters of all methods for all scenarios in our experiments.

Lastly, adding vanilla-QMIX in our experiments is not intended to unfairly compare GoMARL and QMIX. After all, GoMARL is already better than the finetuned version of QMIX (denoted as Ft-QMIX), and there is no need to compare it with vanilla-QMIX to show our superiority. We added vanilla-QMIX's learning curves in Figure 5 only to avoid potential misunderstanding caused by the performance difference between Ft-QMIX in our paper and that of vanilla QMIX in other papers.

B.2 Detailed Information about SMAC Tasks

In each SMAC micromanagement problem, a group of units controlled by decentralized agents cooperates to defeat the enemy agent system controlled by handcrafted heuristics. Each agent's partial observation comprises the attributes (such as health , location , unit_type ) of all units shown up in its view range. The global state information includes all agents' positions and health , and allied units' last actions and cooldown , which is only available to agents during centralized training. The agents' discrete action space consists of attack[enemy_id] , move[direction] , stop , and no-op for the dead agents only. A particular unit, Medivac, has no action attack[enemy_id] but has action heal[enemy_id] . Agents can only attack enemies within their shooting range. Proper micromanagement requires agents to maximize the damage to the enemies and take as little damage as possible in combat, so they need to cooperate with each other or even sacrifice themselves. We follow the default setup of SMAC in our experiments, and more settings, including rewards and observation information, can be acquired from the original paper [29] or open-source implementation.

Based on the performance of baseline algorithms, the tasks in SMAC are broadly grouped into three categories: Easy , Hard , and Super Hard . The key to winning some Hard or Super Hard battles is mastering specific micro techniques, such as focusing fire , kiting , avoiding overkill , et cetera. The battles can be symmetric or asymmetric, and the group of agents can be homogeneous or heterogeneous. Here we provide some characteristics of the scenarios to help gain insights into the good or poor performance of the methods:

3s_vs_5z is a Hard asymmetric battle between two homogeneous teams. The allied Stalkers have to master the kiting technique and disperse in the area to kill the Zealots that chase them one after another. This map faces the delayed reward problem; however, it is not very strict about micro-cooperation between agents because of agents' scattering.

The asymmetric scenarios 5m_vs_6m and 8m_vs_9m are two Hard maps offering a substantial challenge. The allied agents must learn to focus fire without overkill and to correctly position themselves with considerable precision to overcome the enemy team with more agents. The 5m_vs_6m battle is relatively more difficult than 8m_vs_9m due to fewer agents, so the enemy force with one more agent have a greater advantage.

corridor is a Super Hard map that needs extensive exploration. Six allied Zealots fight twenty-four Zerglings on this task. The massive disparity in unit numbers between the two sides implies that the whole team cannot launch an attack together. The only winning strategy is to sacrifice a small number of agents who leave the team and attract the attention of most enemies, and the large force eliminates the rest of the enemies. The surviving agents then keep repeating the same tactic till the whole team kills all the enemies.

MMM2 is a representative Super Hard asymmetric battle between two heterogeneous teams with three kinds of units. One Medivac, two Marauders, and seven Marines have to battle against a team with one more Marine. Marauder has greater attack damage and health than Marine but with a longer cooldown . Medivac has no damage but can heal other agents.

3s5z_vs_3s6z is a Super Hard map that requires breaking the bottleneck of exploration, where three Stalkers and five Zealots battle against three Stalkers and six Zealots. This map requires more cooperation between agents and is the most challenging task in SMAC.

B.3 Detailed Information about GRF Academy Scenarios

Google Research Football (GRF) Academy includes several scenarios which can be commonly found in football games. Agents need to coordinate timing and positions for organizing offense to seize fleeting opportunities, and only scoring leads to rewards. Each agent's partial observation contains the absolute positions and moving direction of the ego-agent, relative positions and moving directions of other agents, and the ball. The global state information includes the absolute positions and directions of all agents and the ball. Agents have a discrete action space of 19, including moving in eight directions, three kinds of ball pass, two kinds of dribble, two kinds of sprint, shot, sliding, stop-moving and do-nothing. Proper cooperation requires agents to pass and shoot effectively with a good direction at proper timing, thus making GRF scenarios much more complicated than SMAC's. Other details can be acquired from the original paper [16] or the open-source implementation.

2

4

6

8

10

Steps

(

10

6

)

0

10

20

30

40

50

60

70

80

Test Win Rate (%)

Ablation of the Specialized Agent Network

GoMARL w/o SAN

GoMARL

2

4

6

8

10

Steps

(

×

10

6

)

0

10

20

30

40

50

60

70

80

Test Win Rate (%)

Ablation of the Specialized Agent Network

2

4

6

8

Steps

(

×

10

6

)

0

10

20

30

40

50

60

70

80

Test Win Rate (%)

Ablation of the Specialized Agent Network

QPLEX

QPLEX with SAN

2

4

6

8

10

Steps

(

×

10

6

)

0

10

20

30

40

50

60

70

80

Test Win Rate (%)

Ablation of the Specialized Agent Network

RIIT

RIIT with SAN

2

4

6

8

10

Steps

(

10

6

)

0

10

20

30

40

50

60

70

80

Test Win Rate (%)

Ablation of the Specialized Agent Network

VAST

VAST with SAN

×

×

Here we provide an introduction of the three scenarios we utilized to help gain insights into the good or poor performance of all the methods in our paper:

academy_pass_and_shoot_with_keeper includes two of our players trying to score from the edge of the box. One of our agent is on the side with the ball and next to a defender. The other agent is at the center, unmarked, and facing the opponent keeper.

academy_3_vs_1_with_keeper contains three of our agents and two opponent players (a defender and a keeper). Our agents try to score from the edge of the penalty area. One stands in the middle, while the others are located on both sides of the area. Initially, the agent at the center keeps the ball and directly faces the defender.

academy_counterattack_easy contains four of our agents and two opponent players (a defender and a keeper). Agents are initialized far from the penalty area and stand evenly in an arc centered on the goal. The second agent from the top initially keeps the ball and has to pass it to a teammate at the appropriate time to avoid interception.

C Performance of Methods with and without the Specialized Agent Network

The proposed specialized agent network (SAN) is highly transferable. To further validate its effectiveness, we perform the specialized agent network on other baseline methods in Section 5.1 to see if they can perform better. ROMA is not included in this study since ROMA's agent networks are produced by its learned roles, and the replacement will invalidate the main idea of ROMA. As shown in Figure 8, our specialized agent networks significantly improve methods' performance and learning efficiency in the Super Hard SMAC task 3s5z_vs_3s6z . Compared to agent networks utilizing the vanilla parameter-sharing mechanism that limits policy diversity, our specialized agent network adjusts the local utility with the individual group-related information and enables various styles of behaviors related to the group status, encouraging extensive exploration and accelerating learning. The complete learning curves are shown in Figure A11.

As Figure A11 illustrates, the performance of all methods is further improved when equipped with our specialized agent network. Specifically, the learning efficiency of Ft-QMIX is boosted. The variance of RIIT is markedly reduced, and the win rate is increased by about 10%. The improvement of VAST is the most obvious; both learning efficiency and win rate are enhanced, and the variance is very clearly reduced. QPLEX's improvement is not very obvious; however, it obtains a slightly higher learning speed and achieves the highest win rate among all the baseline methods.

Most importantly, even equipped with our specialized agent network, all baseline algorithms fail to surpass GoMARL in terms of learning efficiency and the final win rate. GoMARL with vanilla agent networks looks much inferior to GoMARL with the specialized agent network, so it is worth questioning whether the dynamic grouping module is ineffective. The dynamic group learning may

Passes

The keeper prevent

Agent 1's shooting.

Passes to Agent 3

Agent 3 shoots

The keeper saves the ball.

Agent 4 gives

a second shoot

and scores.

STEP:9

00:18FRQ0RBAO

SCORE:O-0

SCORE:0-0

PLAYER SPRINT DRIBBLEDIRECTION ACTION

BALL ONNED:

PLAYER SPRINT DRIBBLEDIRECTION ACTION

1

1

0

shot

sprint

1

0

00

0

release_sprint

0

sprint

Agent 1

0

0

shot

Opponent

Agent1

player

Agent 2

Opponent

player

Agent3

Keeper

Agent3

(b)

SCORE:O-0

PLAYER SPRINT DRIBBLEDIRECTIONACTION

BALL OWNED:

PLAYER SPRINT DRIBBLEDIRECTIONACTION

BALL OWNED:

0

idle

0

0000

Opponent

00

Keeper

player

Agent3

Agent3

Keeper

Agent4

Agent4

(p)

reduce efficiency to a certain extent in the early stage of training compared to QPLEX, however, in the middle and late training stages, the learned grouping will have a significant effect when equipped with the proposed specialized agent network. Therefore, the two modules of GoMARL, the automatic grouping module and the specialized agent network, are both crucial to the performance of GoMARL.

D Visualizations of the Learned Policies in GRF

The trained strategy in Google Research Football is visualized to validate if the learned grouping makes sense. In Section 5.2, we visualize a match of academy_counterattack_easy with a more complex strategy to prove the rationality of the grouping GoMARL learned. Here, we provide a detailed discussion of the visualization in Figure A12 (a zoom-in version of Figure 10 for clarity).

As shown in Figure A12(a), Agent 2 holds the ball at the beginning and faces an opponent rushing toward him. Agent 2 passes the ball to Agent 1 to prevent the ball from being stolen. Subsequently, in (b), Agent 1 carries the ball and tries to break through. However, the opponent goalkeeper blocks his attacking route, and Agent 1 continues passing after a short carry. Agent 3 makes a good run and catches the ball smoothly but shoots quickly in Figure A12(c) since the opponent is close. The goalkeeper easily saves this hasty attack. Agent 4 in (d), who learns excellent coordination with Agent 3, stops the ball and immediately adds another shot to create the goal.

GoMARL's automatic grouping module learns a reasonable grouping G = {{ a 1 , a 2 } , { a 3 , a 4 }} for this complex goal, in which the first group successfully brings the ball into the penalty area through smooth coordination, while the second group creates the final goal through skillful cooperation.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper claims that grouping is beneficial for the efficiency of team coordination. To verify this claim, the authors propose a new learning framework called GoMARL, which achieves efficient cooperation by learning automatic grouping without domain knowledge. The joint action-values are factorized in a group-oriented way, which makes this work different from other related works. The group is generated dynamically and automatically. Meanwhile, a policy learning strategy is proposed to utilize the proposed grouping mechanism. The aim of this policy is to let agents in a same group specialize in similar policies, and agents in different groups have diverse strategies. The proposed solution is evaluated in StarCraft II micromanagement tasks and Google Research Football.\n\nSTRENGTHS:\nThe overall idea of introducing grouping into multi-agent value function factorization is quite novel. I’m also happy to see that it is executed well in many challenging scenarios. \n\nThe authors also provide detailed case studies of the learned groups and their performance in both SMAC and Google Research Football. I find these materials very helpful to show that grouping is helpful in multi-agent cooperation tasks.  \n\nThe experiments are also very detailed, showing that GoMARL performs well on multiple domains. I highly appreciate the authors’ efforts in designing ablation studies on each component of their proposed solution, which makes it very intuitive to verify the contribution of each component.\n\nWEAKNESSES:\nAlthough the overall idea of grouping is intuitive, it is quite challenging to grasp the detailed implementation procedures. For instance, in Line 139: “if agent $a_{j_{i}}$ in group $g_j$ establishes consistency with all the other agents $a_{-j_{i}}$ in group $g_j$, …, then $a_{j_i}$ should not be in group $g_j$ anymore”, this statement represents the standard of grouping, but it is quite abstract. Is it supported by natural systems on grouping or is it a compromise due to the restriction of GMM principle?\n\nSome statements can be rephrased in a more straightforward manner. For instance, in Line 270: “GoMARL maintains two mixing networks, and if it has more parameters for value mixing is a natural question”, this representation is quite ambiguous.\n\nQUESTIONS:\nGrouping is common in reality, but it also restricts the whole policy space into a sub-space where the requirements of grouping holds. Will this restriction influence the overall outcome in some cases?\n\nLIMITATIONS:\nThe authors do not appear to discuss the limitations of their work. I suggest that the authors discuss more scenarios where grouping is beneficial and where the contribution of grouping is limited.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-12"}

comments: SUMMARY:
This paper proposes an automatic grouping mechanism and a GGM principle to build a value function factorization. And the authors model informative group-related signals to encourage specialization in policies and inter-group coordination. The authors evaluate the performance of all methods on the SMAC and GRF benchmarks.

STRENGTHS:
1. The idea of grouping agents is sound.
2. The paper is well-written and easy to follow.

WEAKNESSES:
1. As the authors claim that the method is capable of the correct selection of non-optimal (sacrificial) actions, does this imply that the value decomposition enables the representation of various types of payoff matrices and facilitates decentralized execution? If this is the case, it would be worthwhile to discuss how this method can potentially address the issue of relative overgeneralization. Additionally, it might be beneficial for the authors to test their approach on a simple matrix game, such as

   |      | a_1 | a_2 |a_3|
   |---------|---------|---------|---------|
   | a_1 | 3 | -7 |   -5|
   | a_2 | -7 | 0 |   0|
   | a_3 | -5 | 0 |   2|

2. The absence of a theoretical analysis regarding the representation capacity of the proposed value decomposition weakens the contribution of the paper. It would greatly enhance the strength of the work if the authors could provide a thorough theoretical analysis to support and validate the effectiveness of their value decomposition approach.
3. What is the difference between e^i and role in other role-based methods? 
4. The performance of ROMA and RODE is very poor. Could the author give a further explanation why role-based methods are not suitable for these tasks?

QUESTIONS:
Please answer the questions in the weaknesses.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-06-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers fully cooperative MARL in the CTDE setting. Specifically, this work focuses on the notion of groups and proposes three components to improve learning performance: \n- specialized agent network (SAN): relies on parameter-sharing across agents, but the shared policy is group conditioned and regularized such that it is similar for agents in a same group and different across groups. \n- three stage value mixing: (1) w_1, from agent-level to group-level with agent information. (2) w_2, from group-level to group-level, but incorporating group information. (3) from group-level to total-level (mixing weights and the information used is unclear).\n- automatic and dynamic group formation during learning. \n\nThe resulting method GoMARL is compared against several baselines on two environments and several challenging tasks. It is shown to outperform the baselines. Authors also propose ablation to investigate the importance of SAN, incorporating different level of information in the value mixing, and automatic group formation. Finally, the learned groups and strategies are qualitatively analyzed and discussed by looking at how agents behave in selected scenarios.\n\nSTRENGTHS:\n- Improving performance of cooperative MARL in the CTDE setting is of significant importance for the research community. \n- Relying on group for cooperation and coordination is interesting and well motivated. \n- The paper is well written and does a good job at explaining the method despite its complexity. \n- Empirical evaluation is thorough and highlight significant and reliable performance improvements. \n- The method relies on less network parameters than the baselines. \n- SAN improvement is transferable to other methods.\n\nWEAKNESSES:\n- Clarity: I have trouble understanding several points related to the GGM principle. \n    - l.176: I cannot understand how \"sacrificing self-interest for greater team benefit\" could occur in the fully cooperative setting (shared reward) since individual agent's reward and team reward are exactly the same. If an agent killing itself increases the team reward, it would thus be incentivized to do so by its own individual reward. Therefore, I do not understand why consistency would be lost inside the group:\n        - Definition 2: I do not understand why individual-group consistency is lost.\n        - l.140: I do not understand why one would enforce group inconsistency by kicking out consistent agents out of the group.\n        \n- Limitation: I think that the assumption that agents in a group must have similar policies while being different to agents in another group could be overly restrictive. Imagine the setting of two groups each building half of a bridge: both groups need varied roles (resource gatherer, architect and builder for instance). My understanding is that grouping in that case is still relevant for credit assignment but impedes policy specialization inside each group. \n\n- Clarity: Section 4.3. The method is quite complex and has lots of moving part, so I believe it would be very valuable to summarize it by explicitly expressing Q_tot in terms of individual Q and highlighting the different mixing levels, weights and the information used (agent's, group's, total, etc.). For instance, on what are the total mixing weights conditioned?\n\n- Comparison: I think it is important to report GoMARL performance against MAPPO's given that [1] considers the same environments. Also, MAPPO's centralized value function could leverage GoMARL to see if it improves credit assignment and performance.\n\nQUESTIONS:\n- Figure 1.: shouldn't G''' be {{A}, {a_i, a_l}, {a_j, a_k}} since a_i wasn't selected but a_j was?\n- I found using $a$ as agent index a bit confusing, as usually $a$ is for action and agent index is $i$. \n- could it be that no stable grouping is achieved? Like varying team strategy or any other reason?\n- equation (4): does the h correspond to the same sample, i.e. the same situation but observed by different agents, or does it also correspond to different samples of the replay buffer?\n\n### Summary\nI think this is a very good paper, yet I would like to have a better understanding of it before increasing my score. Therefore, I encourage authors to address my concerns (see. Weaknesses). \n\n### references \n[1] Yu, Chao, et al. \"The surprising effectiveness of ppo in cooperative multi-agent games.\" Advances in Neural Information Processing Systems 35 (2022): 24611-24624.\n\n\n### EDIT\nI have updated my score from 6 to 7 post-rebuttal\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces group-oriented MARL (GoMARL). GoMARL leverages a grouping mechanism that automatically creates sub-teams in a cooperative multi-agent team in an end-to-end fashion by maximizing the expected global return. \n\nIn my opinion this approach is a nice, balanced solution for CTDE frameworks where the issues of fully centralized and decentralized models are greatly alleviated by instead looking at the coordination problem at a sub-team level, rather than individual agents, or one giant connected team. A major contribution and perhaps the most interesting part of the approach is the automatic grouping which does not require domain knowledge or a priori determined group number. The paper is very well-written and is easy to follow and understand. Extensive evaluations and ablation studies are carried out which confirms the applicability and efficiency of the approach.\n\nI only have a few minor comments and questions:\n\n- Since automatic grouping is being tackled, can this approach be directly applied to cooperative heterogeneous teams where each agent has a class-specific action-space (or observation-space) and objective. In such scenarios, despite having local separate objectives, classes of agents must collaborate towards a greater global objective/mission. Can this approach be reliable enough to include such class-dependent grouping integrated, for instance when specific classes/types of agents must be grouped for the whole thing to work?\n\n- The process of end-to-end learning for automatically clustering agents can be highly complex and computationally expensive. I can see that with increasing number of agents, the possible combinations of sub-groups could grow exponentially, making the task of finding the optimal group indexes a significant challenge. Can authors comment on this please?\n\n- Can inter-agent communication be leveraged here, for instance as a part of input/observation when needed? Would it require a similar locality and grouping?\n\n- Limitations of the proposed approach are never discussed in the paper. Please add a separate subsection at the end to cover potential known limitations of the method.\n\nSTRENGTHS:\nSee above\n\nWEAKNESSES:\nSee above\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nNo. Limitations are not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}