Stateful Active Facilitator: Coordination and Environmental Heterogeneity in Cooperative Multi-Agent Reinforcement Learning
Authors: Dianbo Liu
Vedant Shah
Oussama Boussif
Cristian Meo
Anirudh Goyal
Tianmin Shu
Michael Curtis Mozer
Nicolas Heess
Yoshua Bengio
Venue: ICLR 2023 poster (2023)

Abstract: In cooperative multi-agent reinforcement learning, a team of agents works together
to achieve a common goal. Different environments or tasks may require varying
degrees of coordination among agents in order to achieve the goal in an optimal
way. The nature of coordination will depend on properties of the environment—its
spatial layout, distribution of obstacles, dynamics, etc. We term this variation
of properties within an environment as heterogeneity. Existing literature has not
sufficiently addressed the fact that different environments may have different levels
of heterogeneity. We formalize the notions of coordination level and heterogeneity
level of an environment and present HECOGrid, a suite of multi-agent RL
environments that facilitates empirical evaluation of different MARL approaches
across different levels of coordination and environmental heterogeneity by providing
a quantitative control over coordination and heterogeneity levels of the
environment. Further, we propose a Centralized Training Decentralized Execution
learning approach called Stateful Active Facilitator (SAF) that enables agents to
work efficiently in high-coordination and high-heterogeneity environments through
a differentiable and shared knowledge source used during training and dynamic
selection from a shared pool of policies. We evaluate SAF and compare its performance
against baselines IPPO and MAPPO on HECOGrid. Our results show
that SAF consistently outperforms the baselines across different tasks and different
heterogeneity and coordination levels.

Published as a conference paper at ICLR 2023
STATEFUL ACTIVE FACILITATOR: COORDINATION AND ENVIRONMENTAL HETEROGENEITY IN COOPERATIVE MULTI-AGENT REINFORCEMENT LEARNING
Dianbo Liu
Vedant Shah
Oussama Boussif
Cristian Meo
Anirudh Goyal!
Tianmin Shu
Michael Mozer
Nicolas Heess
Yoshua Bengio
ABSTRACT
In cooperative multi-agent reinforcement learning, a team of agents works together achieve common goal. Different environments or tasks may require varying degrees of coordination among agents in order to achieve the goal in an optimal way. The nature of coordination will depend on the properties of the environment- its spatial layout, distribution of obstacles, dynamics, etc. We term this variation of properties within environment heterogeneity. Existing literature has not sufficiently addressed the fact that different environments may have different levels of heterogeneity. We formalize the notions of coordination level and heterogene- ity level of an environment and present HECOGrid, a suite of multi-agent RL environments that facilitates empirical evaluation of different MARL approaches across different levels of coordination and environmental heterogeneity by pro- viding a quantitative control over coordination and heterogeneity levels of the environment. Further, we propose a Centralized Training Decentralized Execution learning approach called Stateful Active Facilitator (SAF) that enables agents to work efficiently in high-coordination and high-heterogeneityy environments through a differentiable and shared knowledge source used during training and dynamic selection from shared pool of policies. We evaluate SAF and compare its per- formance against baselines IPPO and MAPPO on HECOGrid. Our results show that SAF consistently outperforms the baselines across different tasks and different heterogeneity and coordination levels. We release the code for HECOGrid as well all our experiments.
INTRODUCTION
Multi-Agent Reinforcement Learning (MARL) studies the problem of sequential decision-making in an enviromment with multiple actors. A straightforward approach to MARL is to extend single agent RI. algorithms such that each agent learns an independent policy (Tan) 1997p. de Witt et al. (2020) recently showed that PPO, when used for independent learning in multi-agen settings (called Independent PPO or IPPO) is in fact capable of beating several state-of-the-an approaches MARL on competitive benchmarks such as StarCraft (Samvelyan et al. 2019). However, unlike most single- agent RL settings, learning in multi-agent RL setting is faced with the unique problem of changing environment dynamics other agents update their policy parameters, which makes it difficult to learn optimal behavior policies. To address this problem of environment non non-stationarity. a class of approaches called Centralized Training Decentralized Execution (CTDE) such as MADDPG (Lowe) lt al. 2017), MAPPO (Yu et al., (021). HAPPO and HTRPO (Kuba et al. 2021) was developed. This usually consists of a centralized critic during training which has access to the observations of every agent and guides the policies of each agent. In many settings, MARL manifests itself in the form
"Fqual Contribution, 'Mila, Quchee AI institute, 2BITS, Pilani, STU Delft, *MIT, 'Google Re- search, Brain Team, 'Deepmind. "CIFAR Chair. Corresponding authors: liudi andiaabb@gmma com, vedantshahh2122eam i_ com, oussama bouss i f@mi la.queeee. https:://gi thub. com veds www.hhccorrroooo d and https: //gi //gittttttuut httpuub.cooo//gggoooooooo

---

Published as a conference paper at ICLR 2023
of cooperative tasks in which all the agents work together in order to achieve common goal. This requires efficient coordination among the individual actors in order to learn optimal team behavior. Efficient coordination among the agents further aggravates the problem of learning in multi-agent settings.
Benchmark
Cooperative
Partial
Image Obs.
Coordination Control
Heterogeneity Control
Obs.
SMAC
x
X
MeltingPot
MPE
V
x
X
X
SISL
V
X
x
X
DOTA 2
X
HECOGrid
Table 1: Comparison between our newly developed HECOGrid environments and widely used
multi-agent reinforcement learning environments including SMAC /Vinyals al., [217)) MeltingPot (Leibo et al. 2021), MPE (Lowe et al. 2017). SISL (Gupta et al. 2017) and DOTA2 Berner et al.
2019)
Another challenge in practical multi-agent learning problems is heterogeneity in the environment.
Environment heterogeneity in reinforcement learning has previously been studied in the context of federated learning in Jin et al. (2022) which considers the problem of jointly optimizing the behavior of n agents located in n identical environments (same state space, same action space, and same reward function) with differing state-transition functions. However, in some real-world multi-agent learning problems, environment properties such as structure, dynamics, etc, may also vary within an environment, compared to varying across different environments. Unmanned guided vehicles (UGVs) used for search and exploration may encounter different conditions, such as different distribution of obstacles different terrains leading to differing dynamics in different regions. Warehouse robots coordinating to pick up a bunch of items might have to work in conditions varying from one section of the warehouse to another such different organization of aisles. Similarly, as argued in Jin et al. (2022). an autonomous drone should be able to adapt to different weather conditions that encounters during its flight. We build upon the formulation of Ji al. (2022) to address the broader problem of heterogeneity within the environment.
We formally define two properties of an environment: heterogeneity, which is a quantitative measure of the variation in environment dynamics within the environment, and coordination which is a quantitative measure of the amount of coordination required amongst agents to solve the task at hand (we formally define heterogeneity and coordination in Section 3. The difficulty of an environment can vary based on the amount of heterogeneity and the level of coordination required solve it. In order to investigate the effects of coordination and environmental heterogeneity in MARL, we need to systematically analyze the performance of different approaches on varying levels of these two factors. Recently, several benchmarks have been proposed to investigate the coordination abilities of MARL approaches, however, there exists no suite which allows systematically varying the heterogeneity of the environment. A quantitative control over the required coordination and heterogeneity levels of the environment can also facilitate testing the generalization and transfer properties of MARL algorithms across different levels of coordination and heterogeneity. A detailed analysis of the existing benchmarks can be found in Appendix A.I
Previous MARL benchmarks have largely focused on evaluating coordination. As a result, while, there has been a lot of work which attempts addressing the problem of coordination effectively, environment heterogeneity has been largely ignored. Heterogeneity so far has been an unintentional implicit component in the existing benchmarks. Hence, the problem of heterogeneity hasn't 't been sufficiently addressed. This is also apparent from our results where the existing baselines do not perform very competitively when evaluated on heterogeneity, since they were mainly designed to address the problem of coordination. Moreover, the fact that heterogeneity has been an unintentional implicit component of existing benchmarks, further strengthens our claim that heterogeneity is an essential and exigent factor in MARL tasks. Coordination and heterogeneity are ubiquitous factors for MARL. We believe that explicitly and separately considering these two separate factor and isolating them from other factors contributing to environment difficulty, will help motivate more research in how these can be tackled.
2

---

Published as a conference paper at ICLR 2023
To address these limitations, we propose HECOGrid, procedurally generated Multi-Agent Bench- mark built upon MARLGrid (Ndousse et al., 202I). HECOGrid consists of three different envi- ronments which allow the testing of algorithms across different coordination and environmental heterogeneity levels. Each environment consists of N agents and M treasures, where the goal is to maximize the total number of treasures picked up by the agents in one episode. c cagents are required to pick up single treasure, where is the coordination level of the environment. The environment is spatially divided into h zones, and the environment dynamies vary from zone to zone. h is the level of heterogeneity of the environment. c, h. NN, and M are controllable parameters. Table presents a qualitative comparison between HECOGrid and other commonly used Aulti-Agent RL environment suites.
HECOGrid also allows complete control over size of the map, number of obstacles, number of treasures, number of agents in addition to the coordination and heterogeneity levels. This provides ease of use of environments from small to very large scale with respect to the aforementioned parameters. This allows HECOGrid to be used as a standard challenging benchmark for evaluating not only coordination and heterogeneity but lot of other factors. A lot of existing benchmarks (Vinyals et al. 2017 Samvelyan et al. 2019 [erner et al. (019) focus moving away from toy-like grid world scenarios, using more complex scenarios with high dimensional observation and action spaces, continuous control, challenging dynamics and partial observability. Although HECOGrid has partial and image observations, it has a relatively small and discrete action space. Hence, HECOGrid cannot be used to test how an algorithm fares in continuous control and high dimensional action space scenarios. However, in most of existing standard benchmarks, it is non-trivial to modify environment parameters and hence it is difficult to perform wide range of generalization and robustness studies. Melting Point (Leibo et al.) 202T allows evaluating for out of distribution generalization, where the OOD scenarios can be defined by changing the background population of the environment. Unlike HECOGrid however, the physical layout of the environment (substrate) however, cannot be changed. HECOGrid, providing complete control over these environment parameters, make easy to perform wide range of experiments.
Further, to enable efficient training of MARL agents in high coordination and heterogeneity, we introduce novel approach called Stateful Active Facilitator (SAF). SAF uses shared knowledge source during training which learns to sift through and interpret signals provided by all the agents before passing them to the centralized critic. In this sense, the knowledge source acts as information bottleneck and helps implement a more efficient form of centralization by refining the information being passed the critic. Further, recent work in modular deep learning (Goyal et al.) 2021a b Rahaman et al.) (02T) has shown that different neural modules trained on a common objective lead to the emergence of specialist neural modules which help in improving performance via decomposition of the task. We hypothesize that a similar form of modularity can also be helpful in tackling the problem of heterogeneity. Instead of each agent using an individual monolithic policy, we propose the use of a pool of policies that are shared across different agents (Goyal et al. (021a). At each time step, each agent picks one policy from the pool which is used to determine its next action where the selection being conditioned on the current state of the agent. During execution the parameters the shared pool of policies can be distributed to each agent which can then operate in a completely decentralized manner. Hence our method falls under the umbrella of Centralized Training Decentralized Execution (CTDE) methods.
Contributions. We introduce set of cooperative MARL environments with adjustable coordination and heterogeneity levels. We also propose SAF- which consists of a shared knowledge source which is empirically shown to improve performance in high-level coordination settings, and pool of policies that agents can dynamically choose from, which helps in tackling environmental heterogeneity. We show that the proposed approach consistently outperforms established baselines MAPPO (Yu et al.) 2021) and IPPO (de Witt et al.) (0200) on environments across different coordination and heterogeneity levels. The knowledge source is the key to improved performance across different coordination levels whereas further ablation studies show that the pool of policies is the key to good performance across different levels of environmental heterogeneity.
3

---

Published as a conference paper at ICLR 2023
2
RELATED WORK
Centralized Training Decentralized Execution (CTDE). These approaches are among the most commonly adopted variations for MARL in cooperative tasks and address the problem of environment non-stationarity in multi-agent RL. They usually involve a centralized critic which takes in global information, i.e. information from multiple agents, and decentralized policies whose learning are guided by the critic. Lowe et al. (2017) first proposed an extension of DDPG (Lillierap et al. 20T6) to multi-agent framework by using shared critic and agent specific policies during training, and
decentralized execution. Yu et al. (2021) proposes the extension PPO (Schulman et al. 2017) to a
multi-agent framework in a similar manner. Kuba et al. (202T) extends trust region learning to
cooperative MARL setting in a way that the agents do not share parameters. |Foerster
.Frerseers.Foerster et al. (2018)
uses the standard centralized critic decentralized actors framework with counterfactual baseline, Li
et al. (2021) uses an information theory-based objective to promote novel behaviors in CTDE-based
approaches. Value Decomposition (Sunehag et al.) 2018 Rashid al.) 2018: Wang et al. 2020
Mahajan et al. 2019), (Rashid et al., 2018) approaches learn a factorized state-action value function.
Sunchag et al. (2018) proposes Value Decomposition Networks (VDN) which simply add the state- action value function of each agent to get the final state-action value function. |Rahhdd et al.] (2018) uses mixing network to combine the action-valus functions of each agent in a non-linear fashion.
Coordination in Multi Agent Reinforcement Learning. There have been several definitions of coordination in MARL, most which come from the agents' perspective. Guestrin et al. (2002) and Busoniu et al. (2008) define coordination as the ability of agents to find optimal joint actions. Kapetanakis & Kudenko (004) defines coordination as consensus among agents. Choi & Ahn (2010) defines coordination as agents' ability to achieve a common goal. In contrast, analyze coordination levels from the angle of the RL environment. Developing MARL algorithms that train coordinated policies requires sufficient exploration due to the presence of multiple equilbria. A large section of recent approaches revolves around explicitly taking into account the states and actions of other agents by learning differentiable communication channels between agents in order train coordinated policies. Foerster et al. (2016) proposes DRQN-based communication protocols DIAL and RIAL. /Suhhbaatar et al./ (2016) proposes CommNet which uses a shared recurrent module which calculates the state of each agent conditioned on the previous state and the mean of messages received from all the other agents. Jiang & Lu (218)) and Das et al. (2019) use attention based communication protocols where attention is used to transmit and integrate messages sent by other agents. Similar to our work, Kim et al. (2020a) attempts to learn coordinated behavior in a CTDE setting without introducing explicit communication, by using the mutual information of the agents' actions.
Environmental Heterogeneity in MARL. Environmental heterogeneity is a relatively uncharted land in MARL and has only been explored to very limited extent in RL as a whole. JJin et al. (022) analyzed environmental heterogeneity in a federated learning setting. The authors define heterogeneity different state transition functions among siloed clients in the federated system, while for each client the environment is homogeneous. In another more recent study by Xie & Song (2022). heterogeneity of initial state distribution and heterogeneity of environment dynamics are both laken into consideration. Our problem is also closely related to Hidden Parameter Markov Decision Processes (HiP-MDPs) (Doshi Velez & Konidaris 2016) which consider a set of closely related MDPs which can be fully specified with a bounded number of latent parameters. Our approach can also be seen as being related to the multi-task reinforcement learning setting, where the goal is to learn an optimal policy that can be generalized to a set of closely related tasks. In all of the above works, heterogeneity is considered to be arising from variations across different environments or tasks. In contrast, we focus on heterogeneity within an environment.
Extended related works can be found in appendix A.I
PRELIMINARIES
Notation. In this work, we consider multi-agent version of decentralized Partially Observable
Markov Decision Processes PDcDP) (Oliehoek & Amato 2016). The environment is defined as (N, O, 0, A, T, II, R, 7). {1. [1.....N N) denotes set of N > agents and S is the set of global states. A = A₁ X .... x An denotes the joint action space and a,, E A, refers to the action of agent i
time step I. O O
x On denotes the of partial observations where o E O stands for
4

---

Published as a conference paper at ICLR 2023
partial observation of agent i time step t. The joint observation o C is given by the observation function O (a,, $t+1] - P(o [at, St+1) where at, $t+1 and 0141 are the joint actions, states and observations respectively. II is the set of policies available to the agents. To choose actions at timestep 1, agent i uses a stochastic policy To; (a,,a//u, conditioned on its action-observation history hy.t (or,0; di.0,---5 -,,,0---,,,,11111 (i, Actions from all agents together produce the transition the next state according to transition function T (3t, C1,t,... @N.,) + P($++1|5r, 01.t. .l....... (N,,). the global reward function conditioned on the joint state and actions. At timestep t, the agent team receives reward r. R(s A1.t. @N,t) based on the current joint state and the joint action a1, AN Y is the discount factor for future rewards.
R: S x A
Coordination level in cooperative MARL task. Here, we quantitatively define the coordination level required in a cooperative MARI task used in this study. Recall that global reward function conditioned on the joint state and actions. At time step I, the agent team receives a reward r. R(s,, ae) based on the current total state of all agents &, and joint action a, Just as in the previous section, the joint state is factorized as follows: 8, (Se,t: .1, SN.)) where Se,, is the state of the external environment and 5;, is the local state of agent at timestep and E N.
x H+ IR is the
Let 9 C. N denote a subset of [ k agents and different subsets can overlap. We define Rg(se, at) = Rigss,t, of, ag) as the joint reward that can only be obtained when subset of k agents cooperate, where sf. = { lieg and af {as.t) We can then write the joint reward as the sum of rewards contributed by all subset of agents:
R(st, a;) = Σ
Rg(
GCN
Hence, the level of coordination et can be defined as the positive reward that can be obtained at time t if no less than et agents involved in it:
3G C C s.t = k: Rg s6 ag) > =0}
G: = min
Where | is the number of elements in G. The global coordination level of the environment can then simply be defined as: c = max This means that if there's 's at least one task in the environment that must be solved using the largest number of agents, then that number of agents (c)) is defined as the coordination level of that environment.
It is worth mentioning the difference between the problem we explore and the formulation in previous studies such as (Zhang & Lesser 2013).. We define coordination in way that some rewards require least & agents to coordinate with each other obtain but different subsets of agents Gj do not have to be disjoint, i.c., one agent can be involved in obtaining more than one reward in single time step.
Heterogeneity level of cooperative MARL task. Another aspect like to explore is the hetero- geneity of the RL environment. It worth pointing out that the heterogeneity of the RL environment different from the heterogeneity agents or heterogeneity of policies as explored in previous studies (Mondal et al. (2022); Kapetanakis & Kudenko (2004).
For simplification, we define heterogeneity unwrapped into multi-agent setting. We assume that the environment has a collection of K different state-transition functions (Tx (st,aa) + 8t+1 I At each timestep f, whenever the agent takes an action, its next state is governed by one of the K state-transition functions, and that choice is decided according to some (possibly latent) variable Vt. K is then defined as the level of heterogeneity of the environment, if K 1 then the environment is said to be homogeneous. In this study, implement v4 as the position of the agent in the environment, which means the state- transition function depends on where the agent is in the environment.
single-agent RL environment, which can easily
5

---

Published as a conference paper at ICLR 2023
(a)
(b)
(c) onty colected Hhan supportig
A beesure can co collected Seere Wt saurty
One ugant on
Agent
Rll camning bey ind coled
A Bneeuoe sat aga uradirecenly
g
Figure 1: HECOGrid: Three cooperative multi-agent reinforcement learning environments developed in this study that allow quantitative control of coordination and environmental heterogeneity levels adjust difficulty of cooperative tasks: (a) reamTogether environment, (b) TeamSupport environment and (c) KeyForTreasuz environment.
HECOGRID: MARL ENVIRONMENTS FOR VARYING COORDINATION AND ENVIRONMENTAL HETEROGENEITY LEVELS
We introduce a set of three cooperative MARL environments, which we collectively call HECOGrid, that allows manipulation of coordination and environmental heterogeneity levels in addition to several other properties. HECOGrid consists of TeamSupport, TeamTogether and KeyForTreasure environments shown in Fig. T In this section, we describe each HECOGrid environment in detail.
Team Together Environment. The cooperative team task in this environment is to collect many treasures as possible in limited number of time steps. Each treasure is presented as a bright box in the environment and becomes grey once collected. In order for treasure to be collected, certain number of agents need to step onto simultaneously. The number of agents required for collection is the level of coordination of the task.
TeamSupport Environment.
This environment similar to TeamToget her except that in order for agent collect treasure, instead of being on the treasure together simultaneously with other agents, it needs to step onto the box and with a certain number of agents within fixed distance (set to 2 by default) support the collection. This number of agents required for collection support (including the agent that actually collects) is defined as the level of coordination of the task. Rewards are distributed equally across all agents in the whole team.
KeyFoTTreasure Treasure Environment. This environment is similar to TeamSuppor's except that an agent can only collect treasure if is carrying a key, and collect the treasure, certain number of agents need to be on the box simultaneously. This additional key-searching step increases the difficulty of the task. an agent picks up a key, its color changes.
In all the environments, all the rewards are distributed equally across all agents on the team.
Environmental heterogeneity in HECOGrid. We implement environmental heterogeneity by dividing the grid into K zones. For each zone, the transition function T is different. Concretely, each action leads to a different state depending on which zone the agent is in (e.g. action number may make the agent turn left, right, move forward perform some other action depending what zone of the grid the agent is present in).
SAF: THE STATEFUL ACTIVE FACILITATOR
In this section we describe the proposed method that consists of two main components: Knowledge Source (KS) that enhances coordination among the agents and a policy pool (PP) that allows agents to dynamically select policy, enabling agents to exhibit diverse behaviors and have distinct goals, as well as handling heterogeneous environments.
After receiving the local observations, the agents produce messages conditioned on the observations and send them to the KSS These messages are integrated into the KS via soft attention mechanism. This enables the KS to sift through the information shared by the agents and filter out the irrelevant information, before sending it back to the agents. The agents then utilize this message to define informative state which is then used by the critic. Hence, the KS acts information

---

Published as a conference paper at ICLR 2023
Knowledge Source
Pollcy Pool
Figure 2: STATEFUL ACTIVE FACILITATOR (SAF) algorithm: during training, agents read and write from Knowledge source (KS) and pick policy from the shared policy pool. First, each agent generates a message and competes for write-access into the KS. Next, all agents read messages from the KS and use it with their internal state as inputs into critic during training. Each agent selects their policy from pool of policy using trainable signature key and their states. After training, the execution conducted in a decentralized manner as the policy is conditioned on the observation only.
bottleneck, which by filtering out irrelevant information, aids the centralized critic in coordinating the agents better. Further, each agents dynamically selects a policy from the shared pool of policies conditioned on its current state. The current state of an agent is simply an encoding of the local observation received by the agent. By using pool of policies, we aim to train specialists which are suited to tackle different environmental conditions, hence aiding in tackling heterogeneity.
Technically, SAF can be obtained by augmenting any of the existing Centralized Training Decentral- ized Execution (CTDE) with the KS and PP. The setup used in our experiments closely resembles that of MAPPO Yu et al. 2021) with the centralized critic augmented with the KS and the policies of agents replaced by shared pool of policies. SAF is trained in the same manner MAPPO, where the centralized critic and the KS are used for guiding the policies. These are not used during execution, which allows is to retain the CTDE nature. We train SAF end end by using the same loss function and standard implementation practices as discussed in MAPPO in Yu et al. (2021).
In the following subsection, explain each step. For further details, see Algorithm 1 in Ap- pendix A.3
Step 1: Generating the messages. Each agent receives partial observation O each time step t. These observations are encoded into messages which are written into the KS by a common encoder go: m = go (ot,)) m/, Rdd.. We denote the set of messages generated by the agents at time step
t by M/:
M = sis N)
Step 2: Writing into the Knowledge Source. The messages M; generated in step one are distilled into a latent state which we term Knowledge Source or KS. We represent the KS state at time step / by F,, F, consists of slots (lo, [,, .. each dimension di so that F, E Lxdd
The messages in M. compete with each other to write into each KS's state slot via a cross-attention mechanism. The query, in this case, a linear projection of the F;, i.e. Q F.W", whereas the keys and values are linear projections of the messages Mt. KS state is updated as:
@LM:W9T Vd.
F. softmax
M.W
After this, self-attention is applied to the KS using transformer encoder tower constituting Perceiver-Fe architecture (Jaegle et al. 2022).
Step 3: Reading from the Knowledge Source. The KS makes the updated state available to the agents should they deem to use it. We again utilize cross attention to perform the reading operation. All the agents create queries Qi [ &
N] N.d, where of W Si, and
7

---

Published as a conference paper at ICLR 2023
St. (w()),) encoded partial observations Generated queries matched with the keys F,W" E Rxdd from the updated state of SAF. As result, the attention mechanism can be written as:
Qi
M = softmax
F.W
(I)
vd
where M,- = (m S S N). Consequently, the read messages are used to define a more informative state .AF 9 ([ss,,, m,,r)), (,s,s,,,,,,,l)), where Jo is parameterized as a neural network. Finally, the new state SAF is used by the critic to compute values. Interestingly, since sAAF is exclusively used by the critic, which is only used during training, SAF do not uses communication during execution.
Step 4: Policy Selection. In order perform policy selection for each policy we define an associated signature key which is initialized randomly at the start of the training: kn = (kx= S u S U}. These keys are matched against queries computed as deterministic function of the encoded partial observation Gift policy = (8i,e), where Speel is parametrized as neural network.
policy (kn))
index; GumbelSoftmax
dmm
As result of this attention procedure, agent selects a policy π rindex. This operation is performed independently for each agent, i.e. each agent selects a policy from the policy pool. Therefore, it does not involve communication among different agents.
EXPERIMENTS
In this section, we design empirical experiments to understand the performance of SAF and its potential limitations by exploring the following questions: (a) how much difficulty do high levels of coordination and environmental heterogeneity cause cooperative MARL tasks? (b) does SAF perform well when coordination or/and heterogeneity levels are high? (c) Is SAF robust to changes of coordination and heterogeneity levels? and (d) SAF has two components. How does each component contribute to the performance at high coordination and heterogeneity levels?
Baseline Methods. We compare SAF with two widely used algorithms with related architectural designs and similar number of parameters to SAF, namely Independent PPO (IPPO) (de Witt et al.) 2020) and multi-agent PPO (MAPPO) (Yu et al. 2021) (Table A.3). In IPPO, each agent has its own actor and critic and does not share information with other agents. In MAPPO, instead of being trained a decentralized manner, the critic takes information from all agents in each step as inputs during training, and agents operate in decentralized manner without sharing information during execution. Since, the agents in our environments are homogeneous, we use the parameter sharing for MAPPO, where the the actor and critic parameters are shared across all agents (Christianos al.) 2021; Terry et al. 2020). SAF has similar training and execution strategy as MAPPO but uses an added component- the KS before passing information the critic during training, and a shared pool of policies instead of a single shared policy for each agent.
High levels of coordination and environmental heterogeneity. To understand how much diffi- culty high levels of coordination in the environment cause, we conducted experiments in all three HECOGrid environments for coordination levels 1 to 3 with heterogeneity level set 1. We train all methods for 10M steps for all experiments. Our results, performance of all three methods decreases dramatically as coordination levels increase. Performance of all three methods show more than 50% decrease in performance at coordination level 2, as compared with coordination level 1, in all environments. At a coordination level 3, all methods fail to show meaningful behavior. These observations indicate that tasks requiring more agents to work together for reward collection are extremely challenging in a cooperative MARL setting.
shown in Figure 3 show that the
To understand how much difficulty high levels of environmental heterogeneity cause, conducted experiments in all three HECOGrid environments for heterogeneity levels 1 to with coordination level set to 1. All methods show decrease in performance the environments become more
8

---

Published as conference paper at ICLR 2023
Team Togetherrr
Teams upportÉ nm
KeyforlreasurrE
80
50
Eptocic Retums
Eplcodic Returns
60
Epbodic
20-
0
Coordination
Coordination
Coordination
#PC
NAPPD
SAF
FPC IAPPO
S4F
FF0 MAPPO
SAF
Figure 3: Test-time results for SAF, MAPPO and IPPO on TeamTogether TeamSupport and KeyForTreasute environments on varying levels of coordination. The heterogeneity level is fixed I. Performance all algorithms decreases as coordination levels increase with SAF showing better performance across all environments.
heterogeneous, though to smaller extent compared with coordination levels (see Figure 4 We provide further results for experiments performed in cases where coordination and heterogeneity levels are high simultaneously in Appendix A.2
Team Togetheeterr
TeamSupportE
KryfForlreasuunnv
Epsodic Reture
Epsodic Retures
66.
Epsodic Retures
60
Heterogeneity
Heterogenelty
Heterogenelty
Figure 4: Test-time results for SAF, MAPPO and IPPO on TeanTogether, TeamSupport and KeyForTreasure environments varying levels of heterogeneity. The coordination level is fixed at 1. All algorithms show decreased performance as heterogeneity increases. SAF shows better performance in cases.
Contribution of each component of SAF. In the method section, design SAF by hypothesizing that in order to tackle coordination and environment heterogeneity. two key elements are necessary: the of shared knowledge source (KS) and a shared pool of policies (PP) from which the agents can dynamically choose. We wish to understand how much each component contributes to the performance of in different scenarios. To investigate this question, we conduct experiments using different ablated versions of SAF in cooperative tasks with different levels of heterogeneity and coordination. As seen in Figure in the Appendix, our experimental results indicate that the knowledge source contributes to the performance of SAF in all cooperative tasks while the shared pool of policies significantly improves the performance of the agents in heterogeneous environments and has minimal contribution to tasks requiring high coordination.
CONCLUSION
this work, we explore coordination and heterogeneity levels cooperative MARL environments by developing set of environments, HECOGrid, which allows full quantitative control over coordination and heterogeneity levels. Moreover, we propose novel algorithm that enables agents to perform well in difficult environments with high levels of coordination and heterogeneity. Our experimental results suggest that high coordination and heterogeneity do make cooperative tasks challenging and SAF method allow agents to gain better performance in these environments.

---

Published as a conference paper at ICLR 2023
8 ETHIC STATEMENT AND REPRODUCIBILITY
To the best of the authors' knowledge, this study does not involve any ethical issues. authors aim to maximize the reproducibility of the study. The codes of this project including the new environment constructed will be released in the camera-ready version. In the methods section, notions align with existing literature. A detailed description of each step in the SAF algorithm is given in the method section and full algorithm is provided in the appendix.
ACKNOWLEDDGEMENT AND AUTHOR CONTRIBUTION
Y.B. supervised the project and contributed to conceptualization, presentation and manuscript writing. N.H. contributed to the conceptualization and experimental design. M.M. contributed to conceptu- alization, experimental design and manuscript writing. T.S. contributed to reinforcement learning task design. A.G. contributed to project initialization, conceptualization, coordination, experimental design, method development and manuscript writing. C.M. contributed to implementation, exper- imental design, method development and manuscript writing. D.L., V.S, and O.B. contributed to project initialization, conceptualization, implementation, experimental design, method development and manuscript writing.
In addition, we thank CIFAR, Mila Quebec Al institute, University of Montreal, BITS Pilani, TUDelft, Google, Deepmind, MIT, and Compute Canada for providing all the resources make the project possible.
REFERENCES
Bernard Baars. Cognitive Theory of Consciousness, Cambridge University Press, 1988.
Peter Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi. Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 2018.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv: 2019.
Kahina Boutoustous, Guillaume J Laurent, Eugen Dedu, Laëtitia Matignon, Julien Bourgeois, and Nadine Le Fort-Piat. Distributed control architccture for smart surfaces. In 2010 IEEE/RS/ International Conference on Intelligent Robots and Systems, pp. 2018-2024. IEEE, 2010.
Michael Bowling and Manuela Veloso. An analysis of stochastic game theory for multiagent reinforcement learning. Technical report, Carnegie-Melloo Univ Pittsburgh Pa School of Computer Science, 2000.
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18-42, 2017.
Lucian Busoniu, Robert Babuska, and Bart De Schutter. A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38(2):156-17,, 2008.
Young-Cheol Choi and Hyo-Sung Ahn. survey on multi-agent reinforcement learning: Coordination problems. In Proceedings of 2010 IEEE/ASME International Conference on Mechatronic and Embedded Systems and Applications, pp. 81-86. IEEE, 2010.
Filippos Christianos, Georgios Papoudakis, Arrasy Rahman, and Stefano V. Albrecht. Scaling multi-agent reinforcement learning with selective parameter sharing. In ICML, 2021.
Abhishek Das, Théophile Gervet, Joshua Romoff. Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In International Conference on Machine Learning, pp. 1538-155 1546. PMLR, 2019.
10

---

Published as a conference paper at ICLR 2023
Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip H. S. Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge?? 2020.
S. Dehaene, H. Lau, and S. Kouider. What is conseiousness, and could machines have it? Science, 35866362)4486-492, 2017.
Finale Doshi-Veler and George Konidaris. Hidden parameter markov decision processes: A semi- parametric regression approach for discovering latent task parametrizations. IJCAI: proceedings of the conference, volume 2016, pp. 1432. NIH Public Access, 2016.
Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, and Shimon Whiteson. Learning to commu- nicate with deep multi-agent reinforcement learning. In NIPS, 2016.
Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI, 2018.
A. Goyal, A. Lamb, P. Gampa, P. Beaudoin, S. Levine, C. Blundell, Y. Bengio, and M. C. Mozer. Object files and schemata: Factorizing declarative and procedural knowledge in dynamical systems. In International Conference on Learning Representations, 2021a.
Anirudh Goyal, Aniket Didolkar, Nan Rosemary Ke, Charles Blundell, Philippe Beaudoin, Nico- las Heess, Michael Mozer, and Yoshua Bengio. Neural production systems. arXiv preprint arXiv:2103.01937, 2021b.
Anirudh Goyal, Aniket Didolkar, Alex Lamb, Kartikeya Badola, Nan Rosemary Ke, Nasim Rahaman, Jonathan Binas, Charles Blundell, Michael Mozer, and Yoshua Bengio. Coordination among neural modules through a shared global workspace. arXiv preprint arXiv:2103.01 197, 2021c
Anirudh Goyal, Alex Lamb, Jordan Hoffmaann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. ArXiv, abs/1909.10899 2021d.
Carlos Guestrin, Michail Lagoudakis, and Ronald Parr. Coordinated reinforcement learning. In ICML,, volume 2, pp. 227-234. Citeseer, 2002.
Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems, pp. 66-83. Springer, 2017.
Andrew Jaegle, Sebastian Borgeaud, -aptistt Alayrac, Carl Doersch, Catalin Ioneseu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Henaff, Matthew Botvinick, Andrew Zisserman. Oriol Vinyals, and Joao Carreira. Perceiver IO: A general architec- ture for structured imputs & outputs. In International Conference on Learning Representations. 2022. URL
Jiechuan Jiang and Zongging Lu. Learning attentional communication for multi-agent cooperation. In NeurlPS, 2018
Hao Jin, Yang Peng, Wenhao Yang, Shusen Wang, and Zhihua Zhang. Federated reinforcement learning with environment heterogeneity. In International Conference on Artificial Intelligence and Statistics, pp. 18-37. PMLR, 2022.
Spiros Kapetanakis and Daniel Kudenko. Reinforcement learning of coordination in heterogeneous cooperative multi-agent systems. In Adaptive Agents and Multi-Agent Systems II, pp. 119-131. Springer, 2004.
Woojun Kim, Whiyoung Jung, Myungsik Cho, and Youngehul Sung. A maximum mutual information framework for multi-agent reinforcement learning. arXiv preprint arXiv:2006.0273 2020a.
Woojun Kim, Jongeui Park, and Youngchul Sung. Communication in multi-agent reinforcement learning: Intention sharing. In International Conference on Learning Representations, 2020b.
Thomas Kipf, Ethan Fetaya, Kuan-Chich Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems, arXiv preprint arXiv: 1802.v:800.044m6 04687, 2018

---

Published as a conference paper at ICLR 2023
Jakub Grudzien Kuba, Ruiqing Chen, Munning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning. arXiv preprint 11X1::2109.11211, 2021.
Joel Z. Leibo, Edgar A. Dučhez-Guzzáá, Alexander Sasha Vezhnevets, John P. Agapiou, Peter Sunehag, R. Koster, Jayd Matyas, Charlie Beattie, Igor Mordatch, and Thore Graepel. Scalable evaluation of multi-agent reinforcement learning with melting pot. ArXiv, abs/2107.06851 2021.
Chenghao Li, Chengjie Wu, Tonghan Wang. Jun Yang, Qianchuan Zhao, and Chongjie Zhang. Cele- brating diversity in shared multi-agent reinforcement learning. arXiv preprint arXiv:2106.02195, 2021.
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, abs/1509.02971, 2016.
Dianbo Liu, Alex Lamb, Kenji Kawaguchi, Anirudh Goyal, Chen Sun, Michael Curtis Mozer, and Yoshua Bengio. Discrete-valued neural communication. ArXiv, 0oss2100.00236,, 2021.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Picter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperatiee-ooppetiive environments. Neural Information Processing Systems (NIPS), 2017.
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. In NeurlPS, 2019.
Laetitia Matignon, Guillaume J Laurent, and Nadine Le Fort-Piat. Independent reinforcement learners in cooperative markov games: a survey regarding coordination problems. The Knowledge Engineering Review, 27(1):1-31, 2012.
Washim Uddin Mondal, Mridul Agarwal, Vaneet Aggarwal, and Satish V Ukkusuri. On the approxi- mation of cooperative heterogeneous multi-agent reinforcement learning (marl) using mean field control (mfc). Journal of Machine Learning Research, 23(129): 1-, 2022.
Kamal Ndousse, Douglas Eck, Sergey Levine, and Natasha Jaques. Emergent social learning via multi-agent reinforcement learning. In ICML, 2021.
Frans A. Olichock and Christopher Amato. A Concise Introduction Decentralized POMDPs. Springer International Publishing, 2016. doi: 10.1007/988-3331--88999 URL https::/doi
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning.
arXiv preprint arXiv:1711.0033,, 2017.
Nasim Rahaman, Muhammad Waleed Gondal, Shruti Joshi, Peter V. Gehler, Yoshua Bengio, Francesco Locatello, and Bernhard Scholkopf. Dynamic inference with neural interpreters. In NeurlPS, 2021.
Tabish Rashid, Mikayel Samvelyan, C. S. D. Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. ArXiv, abs/1803.1 1485, 2018.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung. Philip H. S. Torr, Jakob Foerster, and Shimon Whiteson. The stareraft multi-agent challenge, 2019.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61-80 2008
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
Murray Shanahan. A cognitive architecture that combines internal simulation with a global workspace. Consciousness and cognition, 15(2):433 449, 2006.
12

---

Published as a conference paper at ICLR 2023
Sainbayar Sukhbaatar, Arthur D. Szlam, and Rob Fergus. Learning multiagent communication with backpropagation. In NIPS, 2016.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech M. Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value-decomposition networks for cooperative multi-agent learning. ArXiv, abs/ 0tts//2060022966 2018.
Ming Tan. Multi-Agent Reinforcement Learning: Independent vs. Cooperative Agents, pp. 487-494. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1997. ISBN 1558604952.
Justin K Terry, Nathaniel Grammel, Ananth Hari, Luis Santos, and Benjamin Black. Revisiting parameter sharing in multi-agent deep reinforcement learning. arXiv preprint arXiv:2005. 13625,, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yea, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, al. Starcraft ii: new challenge for reinforcement learning. arXiv preprint 1rXiv:1708.04782, 2017.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling
multi-agent q-learning. arXiv preprint arXiv:208..01062 2020.
Yuanfei Wang, Fangwei Zhong, Jing Xu, and Yizhou Wang. Tom2c: Target-oriented multi-agent communication and cooperation with theory of mind, 2021.
Marissa A. Weis, Kashyap Chitta, Yash Sharma, Wieland Brendel, Matthias Bethge, Andreas Geiger, and Alexander S. Ecker. Unmasking the inductive biases of unsupervised object representations for video sequences, 2020.
Zhijie Xie and SH Song. Fedkl: Tackling data heterogeneity in federated reinforcement learning by penalizing kl divergence. arXiv preprint arXiv:2204.08/25, 2022.
Chao Yu. Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955, 2021.
Chongjie Zhang and Victor Lesser. Coordinating multi-agent reinforcement learning with limited communication. In Proceedings of the 2013 International Conference on Autonomous Agents and Multi-Agent Systems, AAMAS 13, pp. 1101-1108, Richland, SC, 2013. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978 199395..
Lianmin Zheng, Jiacheng Yang, Han Cai, Weinan Zhang, Jun Wang, and Yong Yu. Magent: A many- agent reinforcement learning platform for artificial collective intelligence. CoRR, abs/1712.00600. 2017. URL httL: htttt:://arivv o abs 17122.0600
Vladislav I Zhukovskiy and Konstantin N Kudryavtsev. Pareto-optinas nash equilibrium: Sufficient conditions and existence in mixed strategies. Automation and Remote Control, 1500)1500- 2016.
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 ADDITIONAL RELATED WORK
Information Bottleneck. With the emergence of modular deep learning architectures (Vaswani et al., 2017; Goyal et al. 2021b; (Scasellii et al.) 2008 [Bronstein et al., 2017; Kipt et al. 2018 Battaglia et al. 2018) which require communication among different model components, there has been development of methods that introduce a bottleneck in this communication to a fixed bandwidth which helps to communicate only the relevant information. (Liu et al. 202T) use VQ-
VAE (Oord et al., 2017) to discretize the information being communicated. Inspired by the theories in cognitive neuroscience (Baars) 1988; Shanahan 2006 Dehaene et al. 2017). 7). (Goyal et al.) (221c))
proposes the use of generic shared workspace which acts bottleneck for communication among different components of multi-component architectures and promotes the emergence of specialist components. We use SAF, which is similar to the shared workspace that different agents compete to write information to and read information from.
Communication in MARL.. Communication involves deciding which message to be shared and determining how the message-sending process is implemented. Foerster et al. (2016) and Sukhbaatar al. (016)) implemented learnable inter-agent communication protocols, Jiang & Lu| (2018) first proposed using attention for communication where attention is used for integrating the received information as well as determining when communication is needed. [aa et al. (2019) uses multiple rounds of direct pairwise inter-agent communication in addition to the centralized critic where the messages sent by each agent are formed by encoding its partial observation, and the messages received by each agent are integrated into its current state by using soft-attention mechanism. Kim et al. (2020b) uses intentions represented as encoded imagined trajectories as messages where the encoding done via a soft-attention mechanism with the messages received by the agent. WWang et al. (2021) trains a model for each agent to infer the intentions of other agents in a supervised manner, where the communicated message denotes the intentions of each agent. The above-mentione approaches require a computational complexity that is quadratic in the number of agents whereas our approach has computational complexity that is linear in the number of agents. Moreover, we show that our approach is able to outperform several standard baselines using messages which can be computed as simply encoding each agent's ' partial observation. Weis al. (200)) developed transformer-basee multi-agent reinforcement learning method that models MARL decision decision-making as a sequential model.
Coordination in MARI and the Pareto-optimal Nash equilibrium. In the field of MARL, coordination is usually defined as the ability for agents to make optimal decisions achieve a common goal by finding an optimal joint action in dynamic environmen & Ahn (2010); Kapetanakis & Kudenko (2004) One way to find the optimal joint actions by a group of agents is by studying the Pareto-optimal Nash equilibrium Zhukovskiy & Kudryavtsey (2016), which describes the optimal solution as one in which no agent's 's expected gain can be further increased without compromising other agents' gain. However, there exist several challenges in cooperative MARL systems to achieve Pareto-optimal solutions. In the following sections, we are going to explain three of these challenges, which are the ones we seek tackle in this study as well as their links to coordination and environmental heterogeneity levels.
Environmental heterogeneity and the non-stationarity problem MARL, the transition prob- abilities associated with the action of a single agent change over time as the action dynamics of the other agents change (Bowling& & Veloso, 2000). To solve this problem of non-stationarity, most recent MARL methods follow the Centralized Training Decentralized Execution paradigm. The most extreme case of centralized training is when all agents share the same set of parameters. However, parameter sharing also assumes that all agents have the same behaviors, which is not true when there heterogeneity either among the agents themselves or in the environment. Previous studies use indicator-based methods to personalize shared policy in a group of heterogeneous agents(Terry ct al. 2020), however, environmental heterogeneity has been less explored in the literature (Jin et al. 2022
Environmental heterogeneity and the alter-exploration problem Another problem environmental heterogeneity may cause in cooperative MARL is the alter-exploration problem. The balance between exploration and exploitation is crucial for all reinforcement learning tasks. In cooperative MARL this problem arises when exploration of one agent may penalize other agents and their corresponding
14

---

Published as a conference paper at ICLR 2023
policies during training the cooperative agents share reward et al. (2010); Matignon et al. (2012). Environmental heterogeneity could potentially lead to worse alter-exploration problems as there tend to be more unseen states for exploring agent which may result in higher and more frequent penalties. In this study, we seek to solve the above-mentioned problem using a combination of inter-agent communication via active facilitator and shared pool of policies.
Existing Environments in MARL. Some of the existing benchmarks based on online multiplayer games, attempt to move away from the toy-like grid world setting for MARL environments in favor of more realistic environments, by making use of high-dimensional observation and action spaces, continuous action spaces, challenging dynamics, and partial observability (Vinyals al., 2017 Berner et al. 2019: Leibo et al.] [021] Zheng et al., (017). These benchmarks focus on decentralized control in cooperative tasks and agents are heterogeneous (i.e., different types of agents having different abilities). In principle, it is possible to vary the levels of coordination and heterogeneity since the difficulties of different environments vary. However, there is no well-defined notion of the two concepts and these can't It be varied in a controlled fashion. MeltingPot, which was recently proposed in (Leibo et al., 202T) focuses on test-time generalization abilities of a group of agents includes a wide range of scenarios: competitive games, games of pure common interest, team-based competitive games, and mixed motion games which stress test the coordination abilities of the agents. However, similar other benchmarks, there is no systematic decomposition nor a quantitative notion of the concepts of coorlination and environmental heterogeneity.
A.2 ADDITIONAL RESULTS
In this section, we show the training curves for SAF, MAPPO and IPPO on KeyForTreasure, TeamSupport and TeamTogether environments. We additionally present more ablation results in the Out-of-Disttriution setting.
Performance when increasing coordination or/and heterogeneity levels. SAF shows significant performance improvement upon MAPPO and IPPO at coordination levels 1 and 2 (Figure 3p. In addition, SAF shows faster performance increase at an early stage of the training process (figure A.2) This suggests potential advantages for training agents using SAF in cooperative tasks requiring high coordination. At most heterogeneity levels, SAF shows performance improvement upon MAPPO and IPPO in all HECOGrid environments and a faster increase in performance at early stages of the training (Figure and A.2). This suggests potential advantages for training agents using SAF in a cooperative environment which are heterogeneous. In addition to manipulating coordination and heterogeneity levels separately, experiments are conducted to understand if SAF can perform well in environments in which both parameters are high. In the relatively easy TeamSupport environment with both coordination and heterogeneity set at 2, 3 and 4, SAF again shows improved performance over IPPO and MAPPO (Figure (lc)). Figure [6(a] shows the training curves for SAF, MAPPO, and IPPO on TeyForTreasure TeamSupport and Team TeamTogether environments for a coordination level of 2. It shows gap in performance between SAF and the baselines and this gap is further enlarged when it comes to heterogeneity level of 2 (see Figure 6(b) which shows that SAF is effectively able to handle changes in the environment's 's dynamics.
Robustness to changes in coordination and heterogeneity levels. In most real-world applications, such as robots in warehouses, the coordination levels as well as environmental heterogeneity levels can change over time and may even be unknown to the agents. Therefore, the agents' robustness to such changes is important. To understand if agents trained with SAF can still function well in these out-of-distribution (OOD) settings, we conduct experiments test the agents' performance on T TeamSupport and TeamT TamTogeteer environments with heterogeneity or coordination levels that are different as compared to the ones used during training. First, we train the agents in environments with coordination level 2 and a heterogeneity level of I and test their performance at coordination levels between and 3, and a heterogeneity level of As shown in Figure SAF shows better transfer in the more difficult TeamTogether environment than other methods but fails to perform as well as MAPPO in the TeamSupport environment. Next, we train the agents in environments with a coordination level of I and heterogeneity level of 2 and test their performance at coordination level of and heterogeneity levels between and 5. As shown in figure 7 SAF shows superior performance in TeamToget her environment and matches the performance of MAPPO in TeanSupport envi- ronments. This suggests that SAF has similar robustness to changes in coordination and heterogeneity levels as some of the widely used baselines in the MARL community.
15

---

Published as a conference paper at ICLR 2023
Episodic Returs
80
TeamTogetherE nvv
Teams upportEnn
Keyf Ev
60
60-
60
80
Episadic
20:
20
Heterogene By
Heterogeneity
Heterogeneity
(a)
TeamTogetherEnw
Teams En
KeyF Fn
Episadic Returns
86
60
66-
60-
80
Episodic
20
20
18
Coordination
Coordination
Coordination
(b)
Figure 5: Ablation study to understand the contribution of the Knowledge Source (KS) and the shared policy pool (PP) to the performance of in HECOGrid environments. In the legend, KS indicatess SAF without the Pool of Policies whereas KS-PP essentially means SAF (a) Performance of the KS and the pool of policies against baselines in increasing levels of heterogeneity. (b) Performance the KS and the pool of policies against baselines in increasing levels of coordination. KS contributes to performance all settings and PP especially improves performance heterogeneous environments.
Ablation Study for OOD generalization Figure 8(a) shows test-time generalization results on the TeamTogether and TeanSupport environments where the training coordination level was set to 2 and the heterogeneity was set to 1. The pool of policies in SAF is important in getting good performance especially when it's tested on levels of coordination not seen during training. Moreover, Figure ((b) further validates that the pool of policies is important in handling varying environment dynamies as SAF was trained on a heterogeneity level and a coordination level of and the results on unseen levels of heterogeneity are better than SAF trained without pool of policies. These ablations show that the introduced pool of policies SAF key to its performance.
Comparison with QPLEX baseline Performance of QPLEX and QPLEX with shared pool of policies are compared in different environments with agents. The reason only pool of policies but not share knowledge source used is because QPLEX already has similar mechanism. The results suggest that a shared pool of policies among agents improve learning efficiency when coordination level high (Figure
A.3 ALGORITHM
In this section, we summarize the components of SAF and its computations in the form of pseudo- code outlined in algorithm [
16

---

Published as a conference paper at ICLR 2023
tnffofieenndmm Co?
TeanSusperttrs with He1
Tea te wthle1
73445
13715
stop
598
PPD MAPPO SA
PPD MAPPO SAT
IPPO MAPPO SAT
(a)
Te 2
TeorsSppantf Ht Cnl
TeenTToghhe fry with 2
Ephrede Rumaan
Tears upportEnv with 2 2 H
TeamSupport ny with 3-3
TeamS upamSuppoot my with H H
30
Epbodic Returss
Epbrodic Returns
Epbodic Returns
0.01
-0.5
0.06100
2.50106
500100 step
7.56106
10040
0.06400
2.50105
5.00006 sleg
7.56106
100000
0.06.00
100
2.56405
5,00006 step
7.59106
10040
NAPPO
(c)
Figure 6: Examples of training curves for SAF, MAPPO and IPPO on the KeyForTreasure, TeamSupport and TeamTogether at different coordination and Heterogeneity levels. At the initial stage of the SAF show faster increase in performance. After convergence, SAF shows improved performance in most tasks compared to IPPO and MAPPO.
TeamTagatherEmt TV
TeamSupportEnn
TeamTogotherEn
TeamSSpportEnn
Eploodic Retumis
60.
Epicodic Rotums
Eploodic Roatums
Episodic Rotums
Coordinatonn
Coordinatio
Heterogenelly
Reterogenely
IPPO MAPPO SAF
IPPD MAPPO
SAF
IPPO MAPPO SAF
IPPD MARPO SAF
Figure 7: Out-of-Distribuut generalization study understand the robustness of SAF to changes in coordination and heterogeneity levels in HECOGrid environments. Agents are trained at certain heterogeneity and coordination levels, and tested on unseen levels. general, SAF matches MAPPO in robustness to shifts in coordination or heterogeneity level
17

---

Published as a conference paper at ICLR 2023
TeamTogetherEnn
TeamS Caportin
40
Epbooccc Retums
Retums
60
30
40
20
20
10
Coordination
Coordination
t KS KS_PP
#
KS vá KP_PP
(a)
TeamTogetherEnv
TeamS upportE
80
Episodic Retums
80
pisodiic Retunss
40
40
20
20
Heterogeneity
Heterogeneity
KS KS_PP
KS_PP
(b)
Figure 8: Ablation study for Out-of-Distributie generalization. In order to understand the robustness of different ablated models to shifts either coordination or heterogeneity levels, models trained with coordination level 2 and heterogeneity level 1 are tested across different coordination levels (Figure 8(a)). similar manner, The models trained on coordination level and heterogeneity level 2 are tested across different heterogeneity levels (Figure [(b). KS indicates SAF without pool of policies whereas, KS PP means SAF.
A.4 IMPLEMENTATION DETAILS
In this section, we present the necessary implementation details for reproducing our results. We first present the algorithm's 's hyperparameters, next, we present architectures used for each algorithm, finally, we present the environments hyperparameters.
A.4.1 ALGORITHMS HYPERPARAMETERS
In this section, present the relevant hyperparameters related to training the algorithms showcased in our paper. The hyperparameters shown in this section are kept fixed throughout all three environ- ments. MAPPO, IPPO and SAF are all trained using Proximal Policy Optimization and Table A.4. summarizes the training hyperparameters for each algorithm. IPPO and MAPPO are trained with Generalized Advantage Estimation (GAE) while SAF is not. All algorithms are trained with Adam optimizer with fixed learning rate of 0.0007 throughout training, weight decay of 0 and 10
A.4.2 ARCHITECTURAL HYPERPARAMETERR
In this section, we present the exact architectures along with the hyperparameters that were used for each algorithm.
Actor and Critic Architectures Dilliggtallsstttess the pytorch-style implementations of the actor and critic architectures. While MAPPO shares parameters for both the actor and the critic, IPPO trains separate networks for both the actor and the critic. SAF uses the same architecture (and the same byperparameters) as IPPO and MAPPO for the actor network, with the difference that SAF initializes pool of policies for each agent. Listing 2 shows pytorch-style implementation of CNN that acts as our feature extractor. For IPPO and MAPPO, each agent's 's observation fed to the CNN generate feature vector NxC where N 10 is the number of agents and C 64
18

---

Published as a conference paper at ICLR 2023
TeanTTogether Wi O
TeanTTaether with Os
toledd faaurn
Eplard
006455
234-34
13440
63645
23e-2
2:3-:0
13m0
gpace coho ppa
splec oploo_pp
(a)
Teardiusport with Co
TrarSusport th Co
Epbodk
Epocck
Stap
Stap
guex oply
gpits 8 qpte 106
(b)
Epradd Patura
Epract: Paaurra
2.5ed0
Step
Step
oploo_ss
(c)
Figure 9: Performance comparison between QPLEX baseline and QPLEX with pool of policies in different environments with 5 agents. The reason only pool of policies but not share knowledge source was used is because QPLEX already has similar mechanism. The results suggest that shared pool of policies among agents improve learning efficiency when coordination level is high
19

---

Published as a conference paper at ICLR 2023
for to T do
Step 1: Each agent having state information or, (encoded partial observation), generates message.
Vi (I...., N), M = - go(or,) M = (mi,, ma.1 m2,..... m(,,)
Step 2: KS integrates information from all agents
Q F.W"
F. + softmax (Q(;:W"))
M.
vag
KS state Ft is then updated trough self-attention
Step 3: Information from KS is made available each agent
q.. W read Si,t, Vi
FWW""
M. softmax
K Vd.
F.W"
Step 4: Policy Selection from the pool
policy
polten (K?
Vi E NI, index; GumbelSoftmax
Vdm
as ,f
end
Algorithm 1: Detailed algorithm for learning MARL policies with SAF
Method
No. of parameters
МАРPО
2,477,296
IPPO
1,740,016
SAF (Our Method)
2,698,342
Table 2: Comparing the number of parameters used in our implementations of the discussed ap- proaches
SAF
MАPPO
IPPO
Learning rate
0.0007
0.0007
0.0007
PPO update epochs
10
10
10
Number of minibatches
1
1
1
Discount rate 7
0.99
0.99
0.99
GAE
No
Yes
Yes
GAE's A
0.95
0.95
Entropy loss coefficient
0.01
0.01
0.01
Value loss coefficient
0.5
0.5
0.5
Advantage Normalization
Yes
Yes
Yes
Value loss clipping value
0.2
0.2
0.2
Gradient norm clipping value
10
10
Value loss coefficient
0.5
0.5
0.5
Optimizer
Adam
Adam
Adam
Optimizer's 's epsilon (E)
le-5
le-5
le-5
Weight decay
0
Table 3: Hyperparameters used for training the MARL algorithms across all the HECOGrid environ- ments.
20

---

Published as a conference paper at ICLR 2023
the hidden dimension. The feature vector 2 is fed the actor network to get the action probabilities. For IPPO, the feature vector 2 is fed as is to the critic to get the value function, while for MAPPO, vector 2 concatenate dim=-1) IR gN is formed by concatenating feature vectors from all agents and then fed to the critic, that's why we make the distinction between different critic architectures in Listing
from torch import nn
r_agents 10
def layer_init (layer, std= (2), bias_const-0.0)=
nn. wwmiit.constat......mmmm www.ccoantttaaeer.......o. (layer.bias, bias_cons return layer
Llayer.weight std)
10 actor's output is a vector 7 channels which corresponds to the number of actions.
11 actor = (equuntntial
(nn. (64, 128)),
Tn.Tanh(),
14
layer_init (nn.Linea) Linear (128, 128)),
15
nn.Tanh
16
Jayer_init (nn. Linear (128, 7))
18
19
critic_ippo nn. an.Sequential
20
layer_init Lnn.Li (64, 128)),
Tanhanh() (),
layer_inis Linerr (128, 128)),
23
Tn.Tanh (),
24
layer_init (nn. Linear (128, 1))
25
critic_mappe nn. Sn..eunentalll
28
layer_init (nn. (64 n_agents, 128)),
29
nn. mn.Tanh (),
layer_init (nn. Linear (128, 128)),
Tanhanh()
32
layer init (nn. Linear (128, 1))
33
Listing I: Pytorch-style implementation of the actor and critic architectures with the hyperparameters used in the paper for IPPO and MAPPO. We also provide implementation for the orthogonal initialization scheme.
from torch import import torch. nn, functional as
class CNN (nn. Module):
def
init_(
self, in in__hhanels=3,
channels=132, 641,
kernel_sizes-144 14, 3],
strides-12, 2], hidden_layer=512, out_size=64):
11
12
13
()___nit___))))
15
16
sclf.convl mn.Convd channels(O], kernel_sizes (0),
strides
17
self.conv/ (cCCnned [0], channels(1), sizeel_sizes (1],
strides
self. lineri near (2304, hidden_layer)
21

---

Published as a conference paper at ICLR 2023
self linear2 Linei (hidden. layer, size)
20
21
def (self, inputs):
22
X (s convi (inputs / 255.))
F.relu (self. (x))
x.reshape (x. sx. [0], -1)
25
X F.relu (self. (x))
26
x = self, wwneernnnnnee (x)
27
return
Listing 2: Pytorch-style implementation of the CNN that generates feature vector from the observations. The feature vector is then input to the actor and the critic.
Mapping function Architectures SAF makes use MLPs architectures to encode the partial observations, messages, policies and combine encoded state and encoded messages. Each agent's observation is encoded with a ge that is CNN (see Listing 2 which generates a feature vector z. The agent's states are derived from the observation encoding using 9u which is the State Projector. 9e projects the concatenation of the agent's s state and message. Finally, Jpsel is an MLP that encodes z and is used to select policy from pool of policies. See Table A.4.2 for an overview of the MLP architectures and their hyperparameters.
90
Iw
9psel
g
FC(64)
FC(64)
FC(64)
FC(64)
FC(128)
FC(T28)
FC(128)
FC(128)
Input State [64]
Input State [64]
Input State [64]
Input State [128]
Table 4:
Mapping functions used to encode partial observations, messages, policies and to combine encoded state and encoded messages.
Knowledge Source Hyperparameters We present the hyperparameters used in the Perceiver-Ii architecture (Jaegle et al. 2022) that makes up the knowledge source. We use 2 Perceiver layers and use L slots for the knowledge source. The number of policies in the pool is set to 4. Table A.4.2 summarizes the hyperparameters used to define both the Perceiver Encoder and the Cross Attention Layer.
Perceiver-I0 Hyperparameters
Values
PerceiverEncoder Hyperparameters
latent dimension
num latent channels
64
cross attention channels
64
self attention heads
self attention layers per block
self attention blocks dropout
No
CrossAttention Hyperparameters
query dimension
64
key, value dimension
64
num query. key channels num value channels dropout
64
64
No
Table 5: Hyperparameters used to define the Perceiver- architecture used within the Know]-
edge Source.
22

---

Published as a conference paper at ICLR 2023
A.4.3 ENVIRONMENT HYPERPARAMETERS
This section presents the hyperparameters for our three environments excluding coordination/hetero- geneity levels since those are experiment dependent and are clarified in the main text. The agents
TeamTogether TeamSupport KeyForTreasure
Gym Observation Space
Box (0, 255, (28, 28, 3) dtype=uint8)
Gym Action Space
Discrete (7)
Number of treasures
100
100
100
Grid size
30
30
30
Max. number of steps/episode
50
50
50
Partial View Size
7
7
7
View Tile Size
4
Clutter Density
0.1
0.1
0.1
Hyperparameters used the paper for all three environments. controls how much of the grid the agent can see.
The partial view size parameter
23


Reviews:
[{"review_id": "p4fgKV8NBx", "reviewer": "Reviewer_UJ6m", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose HECOGrid, a suite of MARL environments. The authors claim that two properties that affect the performance of MARL algorithms in the environment are coordination and heterogeneity. The authors formally define these two metrics. HECOGrid allows the user to choose the level of coordination and heterogeneity in the environment. The authors also propose Stateful Active Facilitator (SAF), a cooperative MARL algorithm. They evaluate SAF in HECOGrid, for varying levels of coordination and heterogeneity against IPPO and MAPPO.", "summary_of_the_review": "Overall, I believe that the paper presents some worthy contributions. However, I have several concerns about the reproducibility of this work. Additionally, the writing seems a bit rushed, since the paper is not well organized. Additionally, the lack of some results in the figures, as well as, the lack of the source code shows that this work is in its early stage. Therefore, I recommend rejection. I would happily increase my score if the authors address my concerns.\n", "strength_and_weaknesses": "Strengths \n\n- The paper proposes an interesting suite of environments that would be useful to the MARL community.\n- The evaluation of SAF is detailed and the evaluation baselines are appropriate.\n \n-------------------------------------------------------------------------------\nWeaknesses \n- In Table 1, MPE has scenarios with partial observability, e.g. simple_speaker_listener\n- Page 3, paragraph Notation: Why are the policies of the agents conditioned on the observation and not the history of observations?\n- SAF is not properly described in the text. What is the actor's loss? Is it trained using the MAPPO loss? Is it an on-policy or an off-policy algorithm?\n- The used hyperparameters are not mentioned anywhere in the paper.\n- How does SAF compare to SOTA Value Decomposition methods, for example, QPLEX?\n- Do you use parameter sharing for the algorithms? If not, why? It has been shown in several studies that parameter sharing significantly improves the achieved returns of cooperative MARL with homogeneous agents.\n- The paper claims that one of its main contributions is HECOGrid, a suite of MARL environments. However, the authors have not included the source code of HECOGrid.\n- While state/observation heterogeneity is a feature currently missing from many MARL environments, I believe that agent heterogeneity is a more interesting and fundamental challenge (for the MARL community).\n- It seems like some IPPO results are missing from Figure 4.\n- Most of the experiments are performed for coordination level 1, which means that only a single agent is required to solve the task. That primarily shows that SAF does not \"enables agents to work efficiently in high-coordination and high-heterogeneity environments\". It seems that the main benefits of SAF come from the centralized computation of the value function. Can the authors elaborate more on why they believe that SAF outperforms the related baselines?\n- In Figure 6, what do KS, and KS-PP mean? Does KS mean SAF without PP? What is KS-PP? Is it just SAF? Or is it SAF without KS? If this is the case, how much is the contribution of KS to SAF?\n- I would expect to see more clarifications on why SAF is better compared to baselines. This is not presented in the paper. I can only see the algorithm description without much justification behind the design choices.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jWsC1K3o0S", "reviewer": "Reviewer_PAoT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This explores training MARL under heterogeneous environments with different coordination difficulties. The authors first define heterogeneous environments and different levels of coordination. Then three Grid environments are designed to support different levels of coordination. Then the author proposed an attention-based method to solve the designed tasks. ", "summary_of_the_review": "The work investigates the training of MARL under different heterogeneous environments and coordination levels. The authors have a good attempt to define the coordination level of MARL, though it seems incomplete. The relationship between the environment heterogeneity and coordination and motivation should be discussed more in the paper.", "strength_and_weaknesses": "**Strength**\n- This work tries to study and define the level of coordination in MARL, which can be a good perspective to evaluate various algorithms. \n- The paper is overall easy to follow. \n\n**Weakness**\n- For the definition of the coordination level, it seems each time step is considered in isolation, what about coordinations that are temporal, such as the actions that need to be taken by one agent after another agent has taken a certain action?\n- A high-level diagram should be provided for the proposed algorithm which would make it simpler for the readers to understand the overall network architecture. \n- It would a good idea to further motivate how heterogeneous environments can happen in a multi-agent environment. Why would different agents have different transition functions (designed as the change of action order)?  How does that map to a real-world environment?\n- The topic of the work is related to coordination and hetergeneous environment, but it is not so well motivated why these two are explored simultaneously in a single paper. The designed environments seem to focus more on the coordination level while the designed algorithm (dynamically selects model) seems to focus on the hetergeneous property.\n- From the evaluation result, it seems the designed solution performs better for different hetergeneous environments, but has less improvement when the coordination level is higher. How does the proposed algorithm improve coordination?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UTVeOFrpT0s", "reviewer": "Reviewer_y8q8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of Multi-agent Reinforcement Learning (MARL). This paper specifically addresses two components of MARL. First, the paper designs and presents three benchmark domains to explore the limits of modern MARL algorithms in terms of their ability to handle environment heterogeneity and the need to actually cooperate (rather than independently acting to complete the team's task). Second, the paper develops a new algorithm for MARL specifically designed for heterogeneity and cooperation. The paper evaluates their approach against baseline methods in the new benchmark domains, presenting numerous, positive results.", "summary_of_the_review": "This paper contributes three new domains to explore the limits of MARL in terms of heterogeneity and cooperation. These domains will likely be high-impact for the community. The paper also contributes an algorithm. The algorithm is under-described, unfortunately. Either, the paper must make more room for details about the technical algorithm (and adding baselines to demonstrate the technical contribution), or the paper needs to be refocused on evaluating MARL algorithms on these domains to show which methods are capable of heterogeneity/cooperation and why/why not.", "strength_and_weaknesses": "Strengths:\n-This paper clearly states its contributions on Page 3 (and in the abstract).\n-Unlike many papers in MARL, this paper contributes not only an algorithm but also a set of MARL environments that push MARL algorithms' ability to handle environment heterogeneity and test the need of these algorithms to truly coordinate.\n-The domains presented (TeamTgoether, TeamSupport and KeyForTreasure) are intuitive and do, indeed, appear to require coordination (unlike some MARL domains). Though, it is important to note that these are not the only domains that *benefit* from coordination (e.g., Predator-Prey [1]). \n-The paper shows positive results for SAF vs. MAPPO and IPPO on the proposed benchmark environments while varying the degree of coordination (Figure 3) and heterogeneity (Figure 4).\n-The paper also provides generally positive (but somewhat mixed results) for out-of-distribution generalization (Figure 5).\n-The breadth and depth of the empirical results in this paper (not including the appendices) is impressive.\n\nWeaknesses:\n-The paper could be improved with a DEC-POMDP setup formalism to help explain the terminology/line-of-inquiry.\n-The description of the Stateful Active Facilitator (SAF) is relatively short. While more details are in the appendix, it would be helpful if pseudocode was provided (Algorithm 1) in the main paper. Figure 2 does not help much in describing the approach and could be removed to make room for a better illustration or more text-based details. Many approaches work by generating and integrating messages -- how does this approach really differ? The technical uniqueness is unclear here in the algorithmic formulation. The selection of predefined(?) policies appears unique, but details are missing about how these policies are enumerated and trained.\n-The paper is missing important baselines in the MARL literature, including [2], [3], [4], and [5]\n\n[1] Singh, A., Jain, T. and Sukhbaatar, S., 2018. Learning when to communicate at scale in multiagent cooperative and competitive tasks. arXiv preprint arXiv:1812.09755.\n\n[2] Rashid, T., Samvelyan, M., Schroeder, C., Farquhar, G., Foerster, J. and Whiteson, S., 2018, July. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International conference on machine learning (pp. 4295-4304). PMLR.\n\n[3] Seraj, E., Wang, Z., Paleja, R., Martin, D., Sklar, M., Patel, A. and Gombolay, M., 2022, May. Learning efficient diverse communication for cooperative heterogeneous teaming. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems (pp. 1173-1182).\n\n[4] Iqbal, S. and Sha, F., 2019, May. Actor-attention-critic for multi-agent reinforcement learning. In International conference on machine learning (pp. 2961-2970). PMLR.\n\n[5] Mao, W., Yang, L., Zhang, K. and Basar, T., 2022, June. On improving model-free algorithms for decentralized multi-agent reinforcement learning. In International Conference on Machine Learning (pp. 15007-15049). PMLR.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "U6kuoKzZQVU", "reviewer": "Reviewer_RJCt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors explore coordination and heterogeneity levels of cooperative MARL environments. In particular, they formalize the notions of coordination level and heterogeneity level of an environment and present HECOGrid, a suite of multi-agent RL environments that facilitates empirical evaluation of different MARL approaches across different levels of coordination and environmental heterogeneity by providing a quantitative control over coordination and heterogeneity levels of the environment. They also proposed a new method named SAF (Stateful Active Faciliator) to enable agents to perform well in difficult environments with high levels of coordination and heterogeneity. They evaluated SAF and compare its performance against baselines IPPO and MAPPO on HECOGrid, and demonstrate the advantages of  the proposed SAF method.", "summary_of_the_review": "The authors formalized two MARL concepts (\"coordination\" and \"environmental heterogeneity\"), developed new MARL environments HECOGrid incorporating these two concepts, and proposed a novel algorithm SAF whose advantages were demonstrated in empirical evaluations in HECOGrid. The proposed method is heuristic in nature, and the contribution is mainly on empirical side and minor from theoretical side. \n\nI found the paper is generally written clear, and it does contain some valuable novel results which might be appropriate for publication on ICLR, though I'd like to see some more clarifications/explanations from the authors to better justify the contributions & motivation, as described in above \"strength and weakness\" section.", "strength_and_weaknesses": "Strength:\n\n[1] The problem of studying coordination and heterogeneity levels of cooperative MARL environments is interesting, and the developed new environments HECOGrid are helpful for better quantitative understanding of different MARL approaches across different levels of coordination and environmental heterogeneity, which could be valuable for future RL research.\n\n[2] The proposed SAF method seems to be valid, and the provided empirical evaluations help demonstrate the advantages of the proposed SAF method.\n\n[3] The paper is written reasonably clearly and I find it easy to follow.\n\nWeakness/suggestions:\n\n[1] As mentioned in the paper, environmental heterogeneity has been less explored in the literature. To better justify the significance of the contribution & better understand the motivation, I'd love to see some more explanations/discussions how important/essential for such separate concepts of \"coordination\" and \"environmental heterogeneity\" for MARL (rather than a single concept of \"difficulty level\" in some previous MARL environments, which implicitly supports various levels of  \"coordination\" and \"environmental heterogeneity\"), and why this is super important from MARL research perspective. Such explanation/clarification would be helpful to justify that the contribution here is significant enough.\n\n[2] It would be valuable/helpful if the authors could elaborate a bit more about the supported scale & complexity for developed new environments HECOGrid, against previous MARL environments. Is  HECOGrid appropriate for all different scales (from small to super large) and complexities (from simple to super complicated) of MARL experiments, and how about the comparison with previous MARL environments? Table 1 shows that HECOGrid has some advantage over all the other previous MARL environments by enabling explicit control of \"environmental heterogeneity\", but I'd like to also see some more comprehensive/objective discussions on pros and cons of HECOGrid against other MARL environments. E.g., are there any scenarios where other MARL environments would be more appropriate/preferred for the experiments rather than HECOGrid? Is the main advantage for HECOGrid being that it allows explicit control of \"environmental heterogeneity\", and it'd be not that relevant for future research that is not focusing on \"environmental heterogeneity\"?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster