Self-Organized Group for Cooperative Multi-agent Reinforcement Learning
Authors: Jianzhun Shao
Zhiqiang Lou
Hongchang Zhang
Yuhang Jiang
Shuncheng He
Xiangyang Ji
Venue: NeurIPS 2022 Accept (2022)
Keywords: multi-agent reinforcement learning

Abstract: Centralized training with decentralized execution (CTDE) has achieved great success in cooperative multi-agent reinforcement learning (MARL) in practical applications. However, CTDE-based methods typically suffer from poor zero-shot generalization ability with dynamic team composition and varying partial observability. To tackle these issues, we propose a spontaneously grouping mechanism, termed Self-Organized Group (SOG), which is featured with conductor election (CE) and message summary (MS). In CE, a certain number of conductors are elected every $T$ time-steps to temporally construct groups, each with conductor-follower consensus where the followers are constrained to only communicate with their conductor. In MS, each conductor summarize and distribute the received messages to all affiliate group members to hold a unified scheduling. SOG provides zero-shot generalization ability to the dynamic number of agents and the varying partial observability. Sufficient experiments on mainstream multi-agent benchmarks exhibit superiority of SOG.

Self-Organized Group for Cooperative Multi-agent Reinforcement Learning
Jianzhun Shao, Zhiqiang Lou, Hongchang Zhang, Yuhang Jiang, Shuncheng He,
Xiangyang Ji Department of Automation
Tsinghua University, Beijing, China
{sjz18, lzq20, he hc-zhang19, jiangyh19, hesc16)@mails. . edu. cn xyj vyji@tsinghua.. edu.
Abstract
Centralized training with decentralized execution (CTDE) has achieved great success in cooperative multi-agent reinforcement learning (MARL) in practical applications. However, CTDE- methods typically suffer from poor zero-shot generalization ability with dynamic team composition and varying partial observ- ability. To tackle these issues, we propose a spontancously grouping mechanism, termed Self Self-Organized Group (SOG), which is featured with conductor election (CE) and message summary (MS) mechanism. In CE, a certain number of con- ductors are elected every T time-steps to temporally construct groups, each with conductor- follower consensus where the followers are constrained to only commu- nicate with their conductor. In MS, each conductor summarize and distribute the received messages to all affiliate group members to hold a unified scheduling. SOG provides zero-shot generalization ability to the dynamic number of agents and the varying partial observability. Sufficient experiments mainstream multi-agen benchmarks exhibit superiority of SOG.
Introduction
Cooperative multi-agent deep reinforcement learning algorithms (MARL) have been ubiquitously applied to real-world scenarios, such autonomous vehicle teams [45], sensor networks [48), and social science [3]. Recently MARL has gained extraordinary performance in various multiplayer games like Dota 3 StartCraft 30) and soccer [16). The framework of centralized training with decentralized execution (CTDE)
28 is one the popular frameworks for solving cooperative multi-agent tasks. Centralized training renders the CTDE paradigm better agent cooperation while independent execution endows the multi-agent system with more efficiency and scalability.
Classical CTDE algorithms like QMIX [28] and MADDPG [1] are confined to the fixed size of agents. However, the number of involved agents tends to vary in real-world multi-agent scenarios. Recently, some methods introduce the attention mechanism 44 to train the varying number of agents simultaneously [12 [T.. They only seek for a solution covering a range of varying team size, unable to provide the generalization ability to the cases not in the range. Agarwal et al. [[] and Liu et al. [18 further introduce the communication mechanism to provide the adaptability on the dynamic team composition (i.e., the team size varies).
In the paper, we propose method called "Self-Organized Group (SOG)"to possess zero-shot generalization for multi-agent reinforcement learning, which provides strong adaptability to the scenarios with varying number agents and even varying sight of agents. spirit of the individuals' cooperation in social science [10) and the leader-following formation in multi-agent system [9), part of the agents are elected as conductors at intervals, and for each conductor a temporary group is
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

formed with conductor-follower consensus. In each group, the followers can only communicate with the conductor, while the conductor summarizes all the received messages and distributes the refined message to all group members aiming for a unified target. Our insight is that, an organized group under the unified command of a conductor can better adapt to an unseen scenario than individuals. To be specific, each agent's 's behavior is prone to diverging from the training pattern when presented with an unseen scenario. They may behave in contradictory ways and pose a threat to the system's stability, On the contrary, conductor could send unified commands, making each agent behave in order, thus avoiding the overall collapse of the training pattern. In the meantime, our method largely reduces the bandwidth and the cost required for communication. For efficient and economical message passing, we also design a variational message summarizer (MS), which compresses the local observation information and helps predict the future trajectories.
Since the conductor navigates the information propagation in the whole group, Conductor Elec- tion (CE) plays critical role in SOG. Except for the simple random CE which satisfies the fully decentralized execution paradigm, we further propose heuristic variant: determinantal point pro- cess (DPP) [22] -based CE, and learning-basee variant: policy gradient (PG) based CE. DPP-based CE chooses conductors in the principle of maximum entropy and avoids the occurrence of homoge- neous conductors. PG-based CE formulates the CE as learning process and takes the long-term benefits of the selection into consideration.
We conduct experiments on three commonly used multi-agent benchmarks, including resource collection task, predator-prey task, and a set of customized StarCraft micromanagement tasks. To validate the zero-shot generalization ability of our method, we evaluate trained models in more complicated scenarios than training. We not only increase the number of agents for evaluation, but also adjust the agents' utility, e.g., reducing the sight range of each agent. The results show that SOG has better zero-shot generalization ability, not only for the dynamic team composition, but also for the unseen environment condition, than current state-of-the-art methods on all three benchmarks.
The contribution of our paper is summarized as follows: (1) We propose Self-Organized Group, mechanism for enhancing agents' zero-shot generalization ability. (2) We design two conductor election ways for reasonable group forming and a variational message summarizer for efficient and economical message passing. (3) We achieve better generalization ability on three commonly used multi-agent benchmarks than current mate-of-the-aa methods.
2 Related Work
Centralized Training with Decentralized Execution. We mainly focus on the typical paradigm of centralized training with decentralized execution (CTDE) for cooperative multi-agent tasks in this part. It combines the advantages of independent Q-learning 38 and joint action learning 4. A series of work concentrate on factorizing Q functions 35 28 and deriving theoretical guarantee for the policy
optimality (3 41 [29] Some other works use actor-critic methods with a centralized critic 7|
Although the CTDE paradigm shows great empirical performance in many multi-agent tasks, it may fail on some simple situations without communication on execution due to the partial observability of the local agent [18]. In addition, traditional CTDE methods usually concentrate on invariable environment conditions and fixed team size.
Communication for MARL. Many efforts have been made for communicating agents in multi- aggent systems (32). Recently some works assume the information can propagate among homogeneous 19, some of which use the graph neural network Some works focus on communication between heterogenous agents [25 but the role of each agent is usually fixed, therefore limiting the model's generalization ability. These works usually pre-define the number of agents, and can hardly handle the dynamic team composition. Liu et al. [18] uses a predefined coach with global information to instruct dynamic number of agents. However, global coach that can communicate with all agents is not often available in real-world domains. In contrast to these methods, our method enables all agents to become conductor, while reserves the efficiency and scalability of CTDE.
agents during decentralized execution 66
36
49
34
to handle the message passing (23
24
26
24
Transfer Learning and Curriculum Learning. An important topic for multi-agent systems the adaptation to the mutable environment. A bunch of works attempt to deal with the increasingly challenging environment by designing set of currieula 2 20 44). Another line of works utilize the attention mechanism or graph neural networks to handle the diversity of the local observation [4

---

[2 Wang et al. [43] allocates roles with limited action space to agents, and the study shows the model'ss is ability of task transfer by increasing the action space for role clusters. Iqbal et [[2] trains simultaneously
multiple tasks by randomly masking out some entities the agent observes. Unlike these methods, our method bypasses the overlarge training burden of curriculum learning. and is more robust in the dynamic team composition and varying environment conditions.
3 Background
In our work, consider fully collaborative multi-agent task with agents, which can be modeled as a decentralised partially observable Markov decision process (Dec-POMDP) [27] S is the true state of the environment. At each time step 1, each agent i E I (1, m.,....]]] m. chooses an action a; E A, which is an element of a joint action E A A". P(s++1|81: a+) S x - [0. is the state transition function of the environment. All agents share the same reward function r(s, a) S A + R. 7 E [0. 1) is the discount factor. Due to the partial observability, each agent has its local observations o' O drawn from the observation function (s, i) S x I + O. In our setting, we assume agents can communicate within its sight range. Each agent may receive a message p E U from its neighbors. Each agent chooses an action by its stochastic policy (a pp, p²) I X U - A(O0, 1)), where p r (O A)* denotes the action-observation history of agent i, and p is the action-observation histories of all agents. The agents' joint policy æ induces a joint action-value function: Q" (st, ar) E P [Re at), where R = & is the discounted accumulated reward. The goal of our method is find the optimal joint policy x² such that Q" (s, a) 2 Q" (s, a), for all x and (s, a) E A. In practice, to handle the input with dynamic team size, detailed in Appendix A.
G = (8, A. P. Z, O, n, 7), where
the entity- wise input rather than the vector input,
We use the popular centralized training with decentralized execution (CTDE) paradigm g 28 as the baseline, also allowing the communication between adjacent agents. During training, the method has the access to the full state and each agent's 's action- obborvvtttnovnoonn history. During testing (execution). each agent only has the access to its own action cestrnnobservattion history and communication messages from neighbors.
Method
In this section, we first highlight the importance of the communication during decentralized execution, and propose self-organized group for decentralized communication. Next we introduce the learning objective of the communication message. Finally we present the whole training procedure together with the model architecture.
4.1 Downsides of CTDE
Consider simple scenario in Fig. [a). The task is fulfilled only when the three buttons pressed synchronously. Therefore, agents A, B, and C need to press the three nonrecur- ring buttons respectively. The button I and 2 are ten meters apart, while button is at the midpoint of them. The sight range of each agent is a circle with radius six meters. As a result, the agent who in charge of the button 2 can't san't perceive if some agent is pressing the button 1, due to sight range limitation. If we use the common centralized training with de- centralized execution (CTDE) method, like QMIX 288, to deal with this task, these agents can not efficiently coordinate beyond the individual sight range, since each agent's execution is conditioned on its own action-observation history. The communication between agents is one of the easy and flexible solutions to handle the partial observability.
Agents
Button
Button
Conductor (with hat)
Button
Button
Sight range
Message
Button
Button
Button
(b)
Figure 1: The illustration of simple multi-agent scenario.

---

To extend the CTDE paradigm based on the communication mechanism, we now present the Self- Organized Group (SOG), mechanism to group all agents and select conductor for each group.
4.2 Self-Organized Group
Aiming for a cooperative multi-agent task with dynamic team composition, every ' time-steps, we elect certain number of conductors and the groups are constructed accordingly. In this section we first show the formulation of SOG. Then we present the message summarizer in Sec. 4.2.1 and discuss in Sec. 4.2.2 the strategies for conductor election. An example group is illustrated as in Fig. [b), where agent C is appointed as the conductor, with the collected information from A and B to broaden the sight range, Since our main task lies in MARL, simplify the communication procedure, assuming that all the messages can be delivered accurately without delay and each agent can handle all the received messages at the same time. Our goal is to build a communication mechanism that satisfies the following properties:
Lightweight. The communication message should be brief and informative, e.g., three- dimensional summary vector of the agent's current state.
Robust. The group should be able to handle the varying number of group members and the unfamiliar environmental conditions (e.g. the varying sight range).
The process of group organization is as fol- lows (Fig. 2): every T time-steps, some agents are elected to be conductors. We allow multiple conduc- tors in the same time. Then conductors send group invitations to all agents within their sight ranges. The non-conductor agents who receive one invitation then send their personal messages Ç back to the conductor. Here the personal message Ç is delicately designed by message summarizer, described in Sec. 4.2.1 For non-conductor agents who receive multiple group in- vitations, they randomly choose one as the conductor and reply. The conductor and its interactive agents temporarily form group. For those who receive no group invitation or leave the sight of the constructor during in- grog communication, they form I-agent group of themselves.
Conductor
Conductor
Broup
Condustor
Condustor
Group
Group
(b)
(c)
Figure 2: (a) Conductors are elected, and send group invitations to the neighbors.
The process of group organization.
(b) Followers choose the conductor and send the summa- rized messages. (c) Groups are formed and conductors send unified targets to the group members.
After the group is formed, the conductor processes the messages from all the followers (including itself). Then it sends the processed message Ç back to each member that belongs to group j- Each agent only needs to communicate with its group's conductor, therefore it greatly reduces the bandwidth required for communication. We use a non-parametric message mixer for each conductor:
M²
e
1
M S Group(j),
MI
where M3 is the size of group j. In this way, the conductor takes no extra computational cost for message mixing and distributing procedure, and each agent can play the role of the conductor without changing its network structure.
The communication takes place every T time-steps. Each agent i uses its local observation o, and the message μ² to predict its action. je¹ is defined as:
# = IT(t,T) E + (1- I(t,T)) C. i Group(j),
where I[tt, 7') the indicator whether is divisible by T.
Through such mechanism, agents can share the information of the whole group with the minimal communication cost and coordinate more efficiently. We present the quantitative analysis on the communication cost in Appendix H, which is 1/M of the cost of the fully-connected communication like Agarwal et al. [1. where M is the average number of neighbors for each agent.
In unfamiliar scenarios, the conductor integrates the followers' information and all group members execute unified command, driven by a delicate message summarizer. Such a in-group communication

---

mechanism empowers the system better stability compared to the an--conductor-followii formation. With time-varying conductor election, SOG shows strong zero-shot generalization ability to the dynamic team composition and unseen environmental conditions.
4.2.1 Message Summary (MS)
After the communication protocol is built, we now present the content the communication message, Inspired by recent works on role learning and state summary by variational inference [42] 18 17]. we propose a message summarizer for agents to summarize their local observations as brief latent variables. A summarizer can extract valuable information from the whole trajectory, and reduce the computational complexity.
We first aim to distill the information about the agent i' s transition of future T-steps. Let Tf = (of+ (o- -1. -0 (++T-1)- The personal message is random Gaussian variable of C' dimensions sampled from an encoder, i.e., Çi fo())). We maximize the mutual information between & and of conditioned on of. I [(C); of(o() has the following lower bound:
I(C:: T/(o)) 2 Eot.cton [log go x;)] + H({ of):
(3)
where go is the variational estimator, which is only used for centralized training. We define the opposite of the lower bound as future predictor loss CFp. The derivation is defered to Appendix D.
In addition, we expect the personal message to help discard the irrelevant information while retain the future trajectory's s information, which can endow the agent with a small state representation and accelerate training. We take advantage of the conditional entropy bottleneck (CEB) objective [5]. We maximize the mutual information between the personal message Ç and the future trajectory Ti, and minimize the mutual information between the current observation of and the personal message Çi conditioned on the future trajectory TŞ simultaneously. The objective has the following lower bound:
go (o T() K EE (o(G)oo,,
(e(C)))
((T;; I(o:: (lrr)
log
log
(4)
(o T/)
where (-) is the encoder for personal message, and K is the size of the mini-batch sampled for training. We define the opposite of the lower bound as LCEB. See Appendix D for detailed derivation.
4.2.2 Conductor Election (CE)
In this section, we introduce three conductor elec- tion mechanisms: Random CE, Determinantal Point Process (DPP) [22] based CE, and Policy Gradi- ent (3 based CE. Note that CE differs from the
CE
Cerraited bad Training
including agores
Q
h-/
concept of leader election (LE) (39) in distributed system. We elect conductors to obtain better agent cooperation for maximizing the team reward, and we assume perfect communication channel. LE is usu- ally deployed in real multi-agent systems with the probability of agent disconnection, with the target to maintain the cognitive consistency among agents.
Mixer
Never
Q
MS
h
V
V
Zer
do.t
GRU
hi
Random CE. Each agent is elected as conduc- tor with an independent and identical probability pl, which is fully decentralized. The expectation of the number of groups can be controlled by pt. Since all group members need to communicate with their neighbors only for message passing, SOG with ran- dom CE doesn't It need centralized commander during execution. Therefore, it can be parallelized easily during execution.
Uask
MHA
MHA
FC
Agreet
Figure: 3: diagram of the model structure. "MHA" means multi-head attention. We use this module to deal with the entity-wise input for dynamic team composition, detailed in Ap- pendix A. h is the hidden state for GRU cell.
based CE. In this part we expect to increase the generalization ability of SOG by maximizing the intra-group diversity. We achieve this by maximizing the diversity between conductors considering the conductor is the structural center of a group. Specifically, we formalize the conductor election as determinantal point process P.
5

---

Definition 1. A determinantal point process (DPP) P is probability distribution defined on the power set 2 of a discrete finite basic point set set = .,,,..,,. E R=*= measures the similarity
between any point pairs in Y. Let B be random subset drawn by P. then VC S.,, we have
det (T.c)
P(B=C) det (L + +1)
(5)
where Lc == [Lsglase is the submatrix of L, whose entries are indexed by the elements in C, and I is an n identity matrix. det(.) means the determinant of a matrix.
In practice, we use all agents to construct the point set y. Correlations are always non- non-positive in DPP [15]. The more similar two points are, the less possible they will appear in a subset sampled by DPP. Therefore, once we need to elect conductors, we just sample a subset B by DPP and use all elements in B as the conductors, which greatly reduces the probability to elect conductors with similar observations (e.g., two agents close to each other). For the similarity matrix L, we use the cosine similarity of agents' personal feature V
L = -= V VTI V2 V2,..,,
V"j "V,,
(6)
Another concern for DPP the high computational cost for calculating the determinant of all submatrices. Instead of the traditional sampler by Schmidt orthogonalization 47 we find method to calculate all determinants in parallel with GPU, which greatly accelerates the computation. Details can be found in Appendix B.
PG-based CE. The conductor election might affect the group's accumulative rewards in the following time-steps. Therefore, we regard the conductor election task as a reinforcement learning problem. We use policy π (alss) I x² (als) with the input of global state decide the probability of each agent to become conductor, and do policy gradient on it. The gradient V VLERR can be written as:
Es so00 {C= Er(enai)) Σ i
Vlog log T (a;ls))
where i is the time that CE happens, i.e., every T ttmmesttep. We omit the ' dependence on the parameter w of the neural network for brevity. Electing conductors by PG takes the long-term benefits of the selection into consideration, and in experiments it shows great generalization ability.
4.3 Algorithm Outline
As shown in Fig. 3 our model is constructed using centralized training with decentralized execution paradigm. We utilize the entity-based input like in Iqbal et al. [12]. so that the model can deal with the dynamic number agents. The partial observability unseen entities. All agents share the same parameters. It's worth noting that the variational estimator le designed include information of future -tspps. Since at time-step it takes the output of GRU cell hü as input, 4s should be used calculate the loss with message before T' steps, i.e. Gi The local Q function uses the message Ç sent by the conductor as part of input if it is in group when communication takes place, and uses personal message Ç in place at other time. Let μ4 denote the set of the messages used for local Q prediction at time-step f.. The mean square Bellman error objective for Q-learning is follows:
local agents is assured by masking out the
LRL (0) Er aet.eeettaaaaaaaaaa ((ta))))))))) (
r max Q (Te+1, a' al1p411) 1 Q60 (Te, a aelue)) )
(8)
a'
where 0 the parameters of the target network, and D the replay buffer. The overall loss can be written as:
fal = LL + X1 + >2LCEB.
where X3 and 2 are hyper-parameters. We summarize the training procedure in Algorithm |
5
Experiments
In this section we design experiments answer the following questions: (1) Whether the Self- Organized Group helps the agents coordinate better than fully-connected communication methods?

---

Algorithm Self-Organized Group
Initialize 0, Φ, v. Set learning rate +- 7, communication interval T. 01 d1, 22, tmax for each episode iteration do for 1, 1,2, .... tmaz do if t mod T * O then
For each agent i, 14 + Sample(ff (o()) else
Flect conductors and form groups {G₂)
For each agent i, < < Sample(fu(ot) For each conductor j. 5t <- Miz((G), For each agent i, 14 G(j) end if
Generate tuple (st; at, r2: 1 St. Se+13 by executing So(Te; Me) end for
D DU (s,, T1, Sr, lmm
Mmar + 0 V
v + + + (FFF(( + V WyLEn((D) + + nl (F VoLCEn(D))
end for
(2) Can Self- Self-Organized Group promote zero-shot generalization to the dynamic team composition and the varying partial observability, even on complicated tasks? (3) What's the effect the conductor election way? (4) What factor contributes to Self Seff-Organized Group most? We test idea three commonly used multi-agent benchmarks: Resource Collection, Predator-Prey. and StarCraft II micromanagement tasks. The previous two are built the multi-agent particle environment (21), and the last is modified from Iqbal et al. [2]. The number the agents is not fixed, sampling from a pre-defined set at the beginning of each episode in all scenarios. For all the three tasks the testing environments are more complicated than the training ones. Agents need to handle the situations that not encountered during training. Each experiment repeated 3 5 times with different seeds.
In experiments, the name "SOG" means our method with random CE, whose execution is decentral- ized. "SOG_ "_p_p"dpddppppppp and "SOG rI" means the SOG with DPP- CE and PG-based CE, respectively, Except for our method, we evaluate four state-of-the-en CTDE methods that are suitable for dynamic team composition: A-QMIX [46), REFIL 112. EMP [1 and COPA [8]. For fair comparison, we extend the original EMP's decentralized-traile training structure to CTDE. For A A-QMIX and REFIL, agents have communication during execution, while for EMP and COPA, agents communicate with their neighbors or a global coach. Since many MARL methods with communication are not suitable for dynamic team composition (e.g., CommsNet (34) and G2ANet (19), we extract one scenario of Resource Collection and show results in Appendix G.I. We also compare SOG with graph-based methods including NCC (23). MAGIC 26 and Gated-ACML 24 in Appendix
5.1 Resource Collection
This scenario is modified from the environment described in Liu et al. 118] Agents need to collect resources from 6 resource points and transport the goods home. The number of the agents for training is uniformly sampled from f2.3,4.5), while for testing is sampled from (6,7,8). Each agent has a sight range SR. Entities including other agents and resource points that exceed agent i's SR are invisible agent i. For more details of the environment, please refer to Appendix F.I.
We train on two kinds of SR: 0.5 and I.O. We show in Fig. 4 (a-b) the results of all methods. SOG significantly outperforms the other methods when SR = 0.5 and SR 1.0. Given that the testing scenario includes 6/7/8 agents while the training scenario only has 2/3/4/5, the communication mechanism of SOG greatly strengthens SOG's zero-shot generalization ability for dynamic team composition. The CE methods have no obvious difference when the team size varies.
Then we test the zero-shot generalization ability in different environmental conditions. We save the previously trained models with a fixed sight range 0.5 or 1.0 and evaluate them on different agent sight range settings. Each setting is repeated for 160 testing episodes and the results are averaged over 3 models with different seeds. We show the performance each method in Fig. cc-d). Except for the difficult setting SR 0.2 that all methods have poor performance, SOG and SOG rl can achieve
7

---

QMIX
REFIL
SDG
5h=1.0
Train 5R
180
300
653
Neturn
1200
Retu
168
203
(mii
T (mil)
Dost
5R
Figure 4: (a-b) The average episode returns in the scenario Resource Collection with kinds of sight ranges (SR). (c-d) Their performance on different test sight ranges from SR 0.2 to SR 1.0.
similar even better performance in different scenarios ranging from SR 0.5 to SR 1.0 than in training scenarios. SOG rl even performs better transferring from SR = 1.0 to SR = 0.8. In the meantime, other methods show notable performance drop when transferred from training settings to others. The results show SOG's strong adaptation to unseen scenarios.
5.2 StarCraft Micromanagement Tasks
A.MMX
Symoerticc
E347
RETIL
5OG, 4
$00_500
900
Stamano
(b)
Fienry Typea
Agunk Typak
Altuckhor
50G
50G
Figure 5: (a) Results of the test winning rates on StarCraft2. (b) Models' performance when transferred unseen scenarios. (c) Visualization of agents' behaviour time step 20.
We apply method to the StarCraft multi-agen challenge (SMAC) (30) We the map designed by Iqbal et al. [12] and Liu et al. [18]. The maps randomly initialize 3-8 agents with enemies of the same number at the start of each episode. Agents initialized together with the sight range [12] or divided into 2-4 groups with the sight range 18). We call the previous one "G(ather)" and the latter "D(isperse)". In the scenario "Disperse", the enemies divided into 1-2 groups, which means each group of enemies is stronger than that of the agents. We limit the number of agents to 3-5 in training procedure and test the model with agent number 6-8. Similar Iqbal et al. [12] Liu et al. 18), we also incorporate the imaginary objective.
We first explore the adaptation to the dynamic team composition. We show in Fig. 5 5) the results on maps. Our method has the highest test winning rate against the preset AI when testing the model with larger size of agents and enemies than training. And the PG-based CE improves the performance on 3-8sz. Combining the results in Resource Collection, we conclude that PG-based CE can improve agents' zero-shot generalization ability dynamic team composition in many cases, due to the more reasonable conductor election mechanism than random CE.
Then, we study the adaptation to the varying environment conditions of our method. We save the models trained on map 3-8s7_D with the sight range which a hard case that agents are split and have restricted horizons. Then we evaluate the models on other initial conditions and sight ranges. The results are shown on Fig. (].).. The initial conditions (Gather or Disperse) have no obvious effect on the performance for all four methods. However, when agents can more entities than training,

---

other three methods all show large performance drop while our method has comparable performance to that in training. This is probably due to the unified messaging passing mechanism within the group. An interesting phenomenon is that the random CE has comparable or better performance to rl-based CE on 2 scenarios. We speculate this is due to the similar pattern of random CE in different environment settings, and analyse this result in detail in Appendix G.2. The results show that our method can better adapt to the varying partial observability.
To further validate the effect of our method, we visualize the situation for four methods (SOG, SOG-rl, SOG without MS, and EMP3) on the same SMAC map in Fig. c).. We can see that agents trained by SOG learn to gather and concentrate fire on certain enemies surrounded by agents. SOG-rl sacrifices a group of agents exchange the the elimination of 3 enemies. SOG without message summary learns to gather, but the target of each agent's attack is not focused. EMP3 does not even learn to attack simultaneously. The results show that CE and MS both promote better agent cooperation.
5.3 Predator Prey
In this scenario, agents play the role of predators, aiming to catch some preys with random walk. The training number of predators is sampled from (3,4), along with only 1 prey. During execution we initialize 5 or 6 predators, as well as 1 or 2 preys. For environment details, please refer to Appendix F.2. a relatively casy-to-learn scenario, and we make ablations on it.
First, we explore the effect of the message summarizer. Except for the LFP and LCEB, we also try an entropy Wang et al. 42 It aims to maximize the entropy of the message to encourage exploration. As shown in Table I the combination of LFP and LCEB obtains the best aver- age testing return, exceeding the performance of CTDE method A-QMIX, and communication method EMP. Although the entropy regularizer helps stabilize the training (the variance is reduced when combined with the regularizer), it makes no contribution to the aver- age performance. Therefore we use the Lrp + LCEB as our default loss for message summarizer. Notice that EMP has similar performance to SOG on training scenarios. Nevertheless, SOG performs better when transferred to complicated evaluation scenarios.
Method
CFP
ROBB
Reg SR
CR
EMP A-QMIX
9w0.62
9,37+2.32
regularizer similar to the implementation
8.03+1 4
7,57+2,44
7.95a0.61
7.2+1.15
8.21
7.53x1.72
SOG
9.1020.56
9.98&1.15
7.7920.09
7.3120.08
Table I: The results of different losses on Predator-Prey. "Reg" means the entropy regularizer. "SR" is the average return of 160 testing episodes on simple training sce- narios, while "CR" is the counterpart on complicated evaluation scenarios.
Then we analyze the choice related to communication in T including the number of groups in expectation, the message dimension, and the communication interval. The number of group 2 has better performance than number 1 or 4. Since the rule that a prey needs be caught by 3 predators simultaneously, it is suitable to divide 5 or agents into 2 groups. When the message dimension is reduced from 10 to 3. the results have no obvious drop. However, decrease greatly when the message dimension is set to 1. The results show that 3-dimensional message sufficient for communication for Predator-Pree Prey task. As for communication interval, 7 is little better than T = 4, and T 10 is not enough for sufficient communication.
Group Num Test Return
1
2(D)
4
7.4+1.888
9.90+3.20
7.37+3.79
Msg dim Test Return
1
3(D)
10
6.30:20
9.90u3.20
10.77=0.91
T
2 1040.79
4(D)
10
Test Return
9.9013.200
7.511.05
Table 2: Ablation studies. "(D)" means the default setting.
Discussion on Limitations
As stated in the section 4.2.2, we only test SOG on the perfect communication channel. Its perfor- mance may fall down when faced with broken communication channel. Another limitation for random CE is that the expectation of the size of each group is decided by pre-defined hyperparameter, i.e., the probability of agent elected as a conductor. When transferred to unseen scenarios, if the size of each group differs a lot from the training condition, it may cause the performance drop. We try to git rid of the hyperparameter by introducing Dpp-based CE and RL-based CE, but they both require a centralized conductor elector, which is not full CTDE.

---

7 Negative Societal Impact
There is possibility that our method could be employed in real-world multi-agent systems such as UAV formation or intelligent warehouse management. Utilizing the policy our method derives directly is risky, since there is domain gap between the training virtual environment and real-worle scenarios. The practitioners are supposed to restrict their policy under human supervision to avoid harmful options.
8 Conclusion
In this paper we propose Self-Organized Group (SOG) for cooperative multi-agent reinforcement learning. In SOG, certain number of agents are randomly elected be conductors and the corresponding groups are constructed with conductor-follover consensus, allowing the groups to be re-organized every T time-steps. We find the organized group under the unified command of conductor embeds the multi-agent system with stronger zero-shot generalization ability compared the traditional CTDE methods with tully-connected communication mechanism. Furthermore. we derive variational message summarizer for efficient and economical message passing, and we propose DPP-based and PG-based conductor election strategies for better group organization. We take experiments on three commonly used multi-agent benchmarks. SOG shows better zero-shot generalization ability not only for the dynamic team composition, but also for the varying partial observability all three benchmarks.
9
Acknowledgement
This work was supported by the National Key R&D Program of China under Grant 2018AAA0102801,, National Natural Science Foundation of China under Grant 61620106005.
References
[1] Agarwal, A., Kumar, S., and Sycara, K. Learning transferable cooperative behavior in multi- agent teams. arXiv preprint arXiv: 01202, 2019.
[2] Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., and Mordatch, I. Emergent tool use from multi-agent autocurricula. arXiv preprint arXiv: arXiv:/909.0 1909.09522 2019.
[3] Berner, C., Brockman, G., Chan, B., Cheung,
V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv: 066600 2019.
[4] Claus, C. and Boutilier, C. The dynamics of reinforcement learning in cooperative multiagent systems. AAAIIAAI, 1998(746-752):2, 1998.
[5] Fischer, I. The conditional entropy bottleneck. Entropy, 22(9):999, 2020.
[6] Foerster, J., Assael, I. A., De Freitas, N., and Whiteson, S. Learning to communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems, 29: 2137-2145, 2016.
[7] Foerster, Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[8] Gupta, J. K., Egorov, M., and Kochenderfer, M. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems, pp. 66-83. Springer, 2017.
[9] Han, T., Guan, Z.HH., Chi, M., Hu, B., Li, T., and Zhang, X.-H. Multi-formation control of nonlinear followigo multi-agent systems. ISA transactions, 69: 69:140-147, 2017.
[10] Hooper, P. I.,, Kaplan, S., and Boone, J. theory leadership in human cooperative groups Journal of Theoretical Biology, 265(4):633-64) 2010.
10

---

[11] Hu, S., Zhu, F., Chang, X., and Liang, X. Updet: Universal multi- ageett rl via policy decoupling with transformers. In International Conference on Learning Representations, 2020.
[12] Iqbal, S., De Witt, C. A. S., Peng, B., Böhmer, W. Whiteson, S., and Sha, F. Randomized entity-wise factorization for multi-agent reinforcement learning. In International Conference on Machine Learning, pp. 4596-46960 PMLR, 2021.
[13] Jaques, N., Lazaridou, A., Hughes, E.,, Gulcehre, C., Ortega, P., Strouse, D., Leibo, J. Z., and De Freitas, N. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International Conference on Machine Learning, pp. 3040-3049. PMLR, 2019.
[14] Jiang, J., Dun, C., Huang, I., and Lu, Z. Graph convolutional reinforcement learning. arXiv preprint arXiv: 1.XV:08000.0902.. 2018.
[15] Kulesza, A. and Taskar, B. Determinantal point processes for machine learning. arXiv preprint arXiv:1207.6083, 2012.
[16] Kurach, K., Raichuk, A., Stañczyk, P., Zajac, M., Bachem, O., Espeholt, L.,, Riquelme, C.,
Vincent, D., Michalski, M., Bousquet, O., et al. Google research football: A novel reinforcement learning environment. arXiv preprint arXiv: /rrXiiv/997..1 /907.
180, 2019.
[17] Lee, K.-H., Fischer, I., Liu, A., Guo, Y., Lee, H., Canny, J., and Guadarrama, S. Predictive information accelerates learning in rl. arXiv preprint wrXiv:2007.12400 2020.
[18] Liu, B., Liu, Q., Stone, P., Garg, A., Zhu, Y., and Anandkumar, A. Coach-player multi-agent reinforcement learning for dynamic team composition. arXiv preprint arXiv:2105.08692, 2021.
[19] Liu, Y.. Wang, W., Hu, Y., Hao, J., Chen, X., and Gao, Y. Multi-agent game abstraction via graph attention neural network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7211-7218, 2020.
[20] Long, Q. Zhou, Z., Gupta, A., Fang, F., Wu, Y., and Wang, X. Evolutionary population curriculum for scaling multi-agent reinforcement learning. arXiv preprint arXiv:2003. 10423, 2020.
[21] Lowe, R.. Wu, Y. I., Tamar, A., Harb, J., Abbeel, O. P., and Mordatch, I. Multi-agent actor-critic for mixed cooperative-compeitiis environments. In Advances in neural information processing systems, pp. (379-6339 2017.
[22] Macchi, O. The fermion process- model of stochastic point process with repulsive points. In Transactions of the Seventh Prague Conference on Information Theory, Statistical Decision Functions, Random Processes and of the 1974 European Meeting of Statisticians, pp. 391-398. Springer, 1977.
[23] Mao, H., Liu, W., Hao, J., Luo, J., Li, D., Zhang, Z., Wang. J., and Xiao, Z. Neighborhood cognition consistent multi-agent reinforcement learning. In Proceedings the AAAI conference on artificial intelligence, volume 34, pp. 7219-7226, 2020.
[24] Mao, H., Zhang, Z., Xiao, Z., Gong. Z., and Ni, Y. Learning agent communication under limited bandwidth by message pruning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5142-5149, 2020.
[25] Mirsky, R., Macke, W., Wang, A., Yedidsion, H., and Stone, P. A penny for your thoughts: The value of communication ad hoc teamwork. Good Systems- Published Research, 2020.
[26] NIu, Y., Paleja, R. R., and Gombolay, M. C. Multi-agent graph-attention communication and teaming. In AAMAS, pp. 964 1449973,22 2021.
[27] Olichock, F. A., Amato, C., al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016.
[28] Rashid, T., Samvelyan, M., De Witt, C. S., Farquhar, G., Foerster, J., and Whiteson, S. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint 18Xiv:1803.11440 2018
11

---

[29] Rashid, T., Farquhar, G., Peng. B., and Whiteson, S. Weighted qmix: Expanding monotonic value function factorisation. arXiv e-prints, pp. arXiv-2006, 2020.
[30] Samvelyan, M., Rashid, T., de Witt, C. S., Farquhar, G., Nardelli, N., Rudner, T. G., Hung, C.-M., Torr, P. H., Foerster, J., and Whiteson, S. The starcraft multi-agent challenge. arXiv preprini arXiv: 1902.04042,, 2019.
[31] Seraj. E., Wang, Z.,, Paleja, R., Martin, D., Sklar, M., Patel, A., and Gombolay, M. Learning efficient diverse communication for cooperative heterogeneous teaming. In Proceedings of the s1st International Conference on Autonomous Agents and Multiagent Systems, pp. 1173-1182, 2022.
[32] Singh, M. P. Agent communication languages: Rethinking the principles. Computer, 31(12): 40-47, 1998.
[33] Son, K., Kim, D., Kang, W. J., Hostallero, D. E., and Yi, Y. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, pp. 5887-5896. PMLR, 2019.
[34] Sukhbaatar, S., Fergus, R., al. Learning multiagent communication with backpropagation. Advances in neural information processing systems, 29:2244-2252, 2016.
[35] Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W. M., Zambaldi, V. F., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls, K., et al. Value-decomposition networks for cooperative multi-agent learning based on team reward. In AAMAS, pp. 2085-2087, 2018.
[36] Suttle, W., Yang. Z., Zhang, K., Wang. Z., Basar, T., and Liu, J. multi--gent off- aff-policy actor- critic algorithm for distributed reinforcement learning. Pupersnnsn,,,,ne 53(2):1549-1555: 2020.
[37] Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018.
[38] Tan, M. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning, pp. 330-337, 1993.
[39] Van Steen, M. and Tanenbaum, A. S. Distributed systems. Maarten van Steen Leiden, The Netherlands, 2017.
[40] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008 2017.
[41] Wang, J., Ren, 7., Liu, T., Yu, Y., and Zhang, C. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008.0 1062, 2020.
[42] Wang, T., Dong, H., Lesser, V., and Zhang, C. Roma: Multi-agent reinforcement learning with emergent roles. Proceedings of the 37th International Conference on Machine Learning, 2020.
[43] Wang, T., Gupta, T., Mahajan, A., Peng, B., Whiteson, S., and Zhang, C. Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.0152: 2020.
[44] Wang, W., Yang, T., Liu, Y., Hao, J., Hao, X., Hu, Y., Chen, Y., Fan, C.,, and Gao, Y. From few to more: Large-scale dynamic multiagent curriculum learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7293-7300, 2020.
[45] Xu, Z., Lyu, Y., Pan, Q., Hu, J., Zhao, C., and Liu, S. Multi-vehicle flocking control with deep deterministic policy gradient method. In 2018 IEEE 14th International Conference Control and Automation (ICCA), pp. 306-311. IEEE, 2018.
[46] Yang, Y., Hao, J., Liao, B., Shao, K. Chen, G., Liu, W., and Tang, H. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.03933 2020.
12

---

[47] Yang, Y. Wen, Y. Wang, J., Chen, L., Shao, K., Mguni, D., and Zhang, W. Multi-agent determinantal q-learning, In International Conference Machine Learning, pp. 10757-10766. PMLR, 2020.
[48] Zhang, C. and Lesser, V. R. Coordinating multi-agent reinforcement learning with limited communication. In AAMAS, pp. 1101-1108, 2013.
[49] Zhang, K., Yang. Z., Liu, H., Zhang, T., and Basar, T. Fully decentralized multi-agent reinforce- ment learning with networked agents. In International Conference Machine Learning, pp- 5872-5881. PMLR, 2018.
[50] Zhou, M., Liu, Z., Sui, P., Li, Y., and Chung, Y. Y. Learning implicit credit assignment for multi-agent actor-critic. arXiv preprint arXiv:2007.02529, 2020.
13


Reviews:
[{"review_id": "uk-w6pBHPG8", "reviewer": "Reviewer_Ur4a", "summary": "This paper approaches the problem of allow members in a team to group together by allowing “conductor agents” to temporally construct groups and then allowing messages to be passed from agents to conductors (with a variational message summariser) in a unified scheduling approach. They show that in the case of having a dynamic number of agents, and also having partial observability, this method has good zero-shot generalisation performance on standard MARL benchmark tasks like predator-prey.\n", "questions": "1. Are there any insights on the differences between the message summariser using an entropy regulariser? It intuitively seems like it should work but from the results stated it is unclear why it does not\n2. Are there results on increasing the message dimension upwards of 3? Do you have thoughts on why the different message dimensions (and ability to store more bits of information) do not impact performance?\n", "limitations": "I think the conductor selection might benefit from some prior information/thought put into selecting the conductors rather than randomly selecting.\n\nNit: some typos throughout the paper, particularly occurrences of “aims->aim” e.g., line 196, 204\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. This paper is clearly written and the motivation of this work is very sound\n2. The authors spend some amount of time first outlining the downsides of centralised training and decentralised execution and why communication between agents is important and can help handle partial observability.\n3. The motivation for self-organised grouping in order to build communication mechanisms that satisfy properties of being lightweight and robust is also clearly laid out.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "mBPLnQ4Vqgo", "reviewer": "Reviewer_AmRN", "summary": "This paper develops a multi-agent reinforcement learning approach with attention to enable zero-shot generalization across scales of homogeneous agents. The approach, self-organized groups (SOG), develops mechanisms for a decentralized team to form ad hoc groups. Within each group, a conductor is elected. The conductor then provides messages to each of the other members of the group. A message summary procedure is added to maximize the mutual information between personal messages and future agent trajectories. Results across StarCraft and Predator-Prey show promising results. Ablation studies are provided to show the contribution of various conductor election schemes. Zero-shot results are provided for scaling the number of agents in Starcraft.", "questions": "- In Lines 137-139, the paper seems to suggest that agents that communicate are not part of the CTDE paradigm; however, communication (at least communication that is range-limitted) can be to be compliant with the CTDE paradigm. For example, see Ding et al., (NeurIPS’20). Does the paper mean to suggest otherwise?\n-How well does the approach perform as a function of the communication range (and associated ability of the conductor to communicate with just one or all of the agents? \n\nDing, Z., Huang, T. and Lu, Z., 2020. Learning individually inferred communication for multi-agent cooperation. Advances in Neural Information Processing Systems, 33, pp.22069-22079. ", "limitations": "The paper says that limitations were addressed and to see Section 4.2. However, word “limit” and “limitation” doe not appear anywhere in the paper except in regards to limitations of prior work or the ranges of agents tested (Line 319). As such, it is hard to see where limitations are sufficiently addressed as the checklist seems to intend. Note: The paper does state some assumptions in Line 148 (“assuming that all the messages can be delivered accurately without delay and each agent can handle all the received messages at the same time”) and Line 218 (“we assume a perfect communication channel”). However, the reproducibility checklists seems to differentiate between limitation and assumptions (though, they are semantically related). It would have been nice if the paper had proactively addressed the shortcomings of the work, when it would fail, when it might be disadvantageous to use the approach, when societal harm might occur (e.g., military applications), etc.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ Section 3 provides a helpful background to formally define the problem being addressed.\n+ Figure 3 provides a helpful illustration describing the model architecture.\n+ Figure 4 shows promising results. It might be helpful to choose a color scheme that more clearly reflects that SOG and its ablations are a group, where as COPA, EMP, etc. are baselines.\n+ The approach (and ablations) are evaluated in computational studies in StarCraft and Predator-Prey. \n+The evaluations show positive results in answering three key questions: (1) does the approach improve coordination vs. fully-connect communication models, (2) can the approach enable zero-shot generalization, and (3) how does performance vary with the CE selection mechanism?\n\nWeaknesses:\n- The color scheme from Figure 4 to Figure 5 is inconsistent. In one, COPA is red; in the other SOG is red.\n- The paper seems to have accidentally missed some prior work that have addressed related concepts of scalability and message efficiency.\n1) Wang et al. (ICML) develop a MARL approach, called IMAC, that uses variational information reasoning to learn communication protocols, similar to the message summary mechanism in SOG. As such, it would have been helpful to benchmark against IMAC – particularly to compare the ability to learn efficient, information-rich messages.\n2) Niu et al., (AAMAS’21) developed a MARL approach, called MAGIC, with a centralized scheduler and message processor that performs message aggregation. The Niu et al. approach is scalable (using attention), but it has the weakness that it brings some centralization as it relies on an agent to act as a message aggregator and scheduler. It would be helpful if this paper would benchmark against MAGIC, as the problem setup is analogous (or provide sufficient justification for why HetNet is not a suitable baseline).\n3) Seraj et al., (AAMAS’22) developed a CTDE MARL approach (with communication) for heterogeneous agent teams, called HetNet. The approach scales (again, using attention) with the number of agents of various types and compresses messages to binary. It would be helpful if this paper would benchmark against HetNet (or provide sufficient justification for why HetNet is not a suitable baseline).\n\nWang, R., He, X., Yu, R., Qiu, W., An, B. and Rabinovich, Z., 2020, November. Learning efficient multi-agent communication: An information bottleneck approach. In International Conference on Machine Learning (pp. 9908-9918). PMLR.\n\nNiu, Y., Paleja, R.R. and Gombolay, M.C., 2021, May. Multi-Agent Graph-Attention Communication and Teaming. In AAMAS (pp. 964-973).\n\nSeraj, E., Wang, Z., Paleja, R., Martin, D., Sklar, M., Patel, A. and Gombolay, M., 2022, May. Learning efficient diverse communication for cooperative heterogeneous teaming. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems (pp. 1173-1182).\n\nWriting:\n-“ which is featured with conductor election (CE) and message summary (MS)” is not grammatically quite correct. Perhaps “a message summary (MS) mechanism” would be better.\n-Line 16 should be “have” instead of “has”\n-Line 21, no comma before “while”\n-Line 49 should have an endash before “based”\n-Line 135 “like QMIX [25]” should be offset with commas\n-Line 350 should have a comma after “First”\n-Line 354 should not have a period after “Table”\n-Line 372, it is helpful not to start a sentence with “but”\n-Line 328 should have a comma after “Then”\n-Line 344 – it is best not to have contractions\n\n====== Post-rebuttal ======\n\nI have raised my score to reflect the improvements the authors have made in their paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "He2V3in1Cnq", "reviewer": "Reviewer_646o", "summary": "This paper proposes a Self-Organized Group mechanism for Cooperative MARL, which is claimed to have zero-shot generalization ability with dynamic team composition and varying partial observability. The mechanism first selects a condutor for each group; then the condutor summarizes the messages (by conditional entropy bottleneck objective) from all agents within this group, and sends the summarized message to all agents; finally, the agents generate actions based on the observation and the summarized message. For the condutor selection, the authors proposed Random, DPP-based and Policy Gradient-based methods. The authors conducted experiments on three environments to show the effectiveness of the proposed methods.", "questions": "* What is the relation between the proposed Self-Organized Group mechanism (i.e., both Conductor Election and Message Summary) and the so claimed zero-shot generalization ability? \n* Could the authors give some results by comparing with [1, 5] or other more strong baselines? Because [1] has a message prunning mechanism, which is like the T-timestep communication in the current paper. Similarly, the group-based multi-agent Message Summary is similar to the Neighborhood Cognition Consistent proposed in [5], and [5] can also be applied to dynamic team composition.  ", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths  \n* Clear paper writting. In general, this paper is easy to read and follow.   \n* Interesting methods for Conductor Election.\n\n\n\n\n\nWeaknesses  \n* The methods and the claims are not related. I do not see the direct relation between the proposed Self-Organized Group mechanism (i.e., both Conductor Election and Message Summary) and the so claimed zero-shot generalization ability with dynamic team composition and varying partial observability. In my opinion,  the zero-shot generalization ability is mostly because of the mean message mixer (i.e., Eq 1) and the MHA-based model structure (i.e., Fig 3).  \n* Missing a lot of related works. For example, the Conductor is similar to the centralizer/coordinator used in many MARL communication methods [1,2]. The entropy bottleneck based Message Summary is also related with entropy bottleneck based communication message generation [3, 4]. The group-based multi-agent Message Summary is also similar to the Neighborhood Cognition Consistent proposed in [5], where summarized message is equal to the consistent cognition. But the authors missed these works.  \n* Due to the above reason, I found the baselines used in the experiments are not strong. For example, the authors mentioned \"Since many MARL methods with communication are not suitable for dynamic team composition\", but as far as I know, [1, 4, 5] and many other MARL methods can be applied for dynamic team composition.\n\n\n[1] Learning agent communication under limited bandwidth by message pruning. AAAI 2020.  \n[2] Learning multi-agent communication with double attentional deep reinforcement learning. AAMAS 2020.  \n[3] Deep multi agent reinforcement learning for autonomous driving. 2020.  \n[4] Multi-agent Communication with Graph Information Bottleneck under Limited Bandwidth. 2021.  \n[5] Neighborhood cognition consistent multi-agent reinforcement learning. 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Fo8ZepA2bCe", "reviewer": "Reviewer_xiGw", "summary": "This work proposes a novel group-based communication scheme for MARL which is trained under the CTDE paradigm. Agents are periodically grouped together with each group having a single conductor which receives messages from each agent and sends back a summary of all its received messages. Messages are received through a variational encoder-decoder architecture and aim to encode information about the local trajectory of agents. Three approaches to select conductors within groups based on random selection, a policy gradient RL agent and determinal point process are proposed. The resulting Self-Organized Group (SOG) algorithm is evaluated in varying environments and demonstrates generalisation ability over varying numbers of agents and varying partial observability across training and testing tasks without further fine-tuning.", "questions": "1. Is the evaluated SOG algorithm based on REFIL as the MARL foundation with the additional communication paradigm? This does not quite come across clearly in Section 4.3.\n2. Do all the evaluated baselines also apply parameter sharing across all agents? I would expect that parameter sharing helps significantly, in particular when communication is applied as agents follow the same communication and therefore already have a “common language” without further coordination. If not, all baselines should be (re-)evaluated with parameter sharing.\n3. In Figure 4a, b, it appears that training performance in the Resource Collection task with larger sight range (1.0) is worse than in the task with shorter sight range (0.5)? This appears strange given that inputs are of the same size and in the shorter sight range case more values would simply be masked to 0s. Why do you think this appears?\n4. What does shading in all training learning curves represent (standard deviation, variance, confidence intervals, …)? Same for tables 1 and 2.\n5. In line 321 regarding the SMAC experiments, it is stated that a “imaginary objective” is applied as in prior work - what does this objective contain and incentivize?\n6. In lines 167ff you state - “For those who receive no group invitation or leave the sight of the constructor during in-group communication, they form a group of themselves.” - it is unclear to me whether this means each of these agents is in a group alone or all these agents are together in a group? Latter would allow agents to communicate beyond boundaries of visibility.", "limitations": "The authors have sufficiently discussed limitations of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n1. Generalisation and communication for cooperative MARL are very relevant and difficult problems.\n2. The idea to group agents and efficiently communicate among agents based on agent proximity to improve coordination is intuitive and well motivated. The method is clearly outlined and well depicted as three intertwined components.\n3. Generalisation to tasks with a different number of agents and varying degrees of partial observability are considered in the conducted evaluation.\n4. Experiments are conducted in three different environments and comparison is made to several MARL algorithms with similar architecture and communication. Message learning components are ablated in Table 1 and further experiments investigating generalisation to varying degrees of partial observability at constant number of agents are presented in the appendix.\n\n# Weaknesses\n**Major:**\n1. I have concerns regarding the novelty and significance of this work. The core components of SOG appear to be (1) identifying groups of agents to communicate through conductor election, (2) encoding messages through a variational encoder optimized with a mutual information objective, (3) summarising/ aggregating messages, and (4) being invariant to number of agents due to architecture design. Out of these, it appears to me that only (1) and (3) are novel with (4) following from the proposed architecture of REFIL [12], and (2) being proposed similarly for NDQ [A]. Also, it is worth noting that prior work already considered the problem of identifying which agents should communicate with each other for most effective communication [A, B, C, D]. The aggregation of messages in (3) is simply a sum and the conductor election process (1) was shown to be comparable to just randomly selecting an agent as conductor in experiments of this work.\n\n2. In l. 325ff the authors state that the RL-based selection of conductors for communication improves generalisation performance compared to the random selection of conductors based on their experimental results, but the results are not conclusive and in some cases even indicate better performance for the random selection process.\n\n**Minor:**\n\n3. For results in the predator prey environment, it is stated that “SOG performs better when transferred to complicated evaluation scenarios” (l. 364) but generalisation returns are very similar to baselines with large variance indicating no significant difference across algorithms.\n4. The downsides to CTDE described is mostly a downside of a lack of information (no specific failure/ downside of CTDE). Also the described failure case should be investigated with a simple small experiment of the described task to verify the statements, QMIX and similar algorithms are not designed for these scenarios but agents might implicitly follow a pattern to still effectively coordinate.\n\n[A] Wang, Tonghan, Jianhao Wang, Chongyi Zheng, and Chongjie Zhang. \"Learning nearly decomposable value functions via communication minimization.\" International conference on learning representations, 2020.\n\n[B] Du, Yali, Bo Liu, Vincent Moens, Ziqi Liu, Zhicheng Ren, Jun Wang, Xu Chen, and Haifeng Zhang. \"Learning correlated communication topology in multi-agent reinforcement learning.\" In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pp. 456-464. 2021.\n\n[C] Jiang, Jiechuan, and Zongqing Lu. \"Learning attentional communication for multi-agent cooperation.\" Advances in neural information processing systems, 2018.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept