A Coach-Player Framework for Dynamic Team Composition
Authors: Bo Liu
qiang liu
Peter Stone
Animesh Garg
Yuke Zhu
Anima Anandkumar
Venue:  (2021)
Keywords: Multiagent reinforcement learning

Abstract: In real-world multi-agent teams, agents with different capabilities may join or leave "on the fly" without altering the team's overarching goals. Coordinating teams with such dynamic composition remains a challenging problem: the optimal team strategy may vary with its composition. Inspired by real-world team sports, we propose a coach-player framework to tackle this problem. We assume that the players only have a partial view of the environment, while the coach has a complete view. The coach coordinates the players by distributing individual strategies. Specifically, we 1) propose an attention mechanism for both the players and the coach; 2) incorporate a variational objective to regularize  learning; and 3) design an adaptive communication method to let the coach decide when to communicate with different players. Our attention mechanism on the players and the coach allows for a varying number of heterogeneous agents, and can thus tackle the dynamic team composition.  We validate our methods on resource collection tasks in multi-agent particle environment. We demonstrate zero-shot generalization to new team compositions with varying numbers of heterogeneous  agents. The performance of our method is comparable or even better than the  setting where all players have a full view of the environment, but no coach. Moreover, we see that the performance stays nearly the same even when the coach communicates as little as 13% of the time using our adaptive communication strategy.  These results demonstrate   the significance of a coach to coordinate players in dynamic teams.

Under review as a conference paper at ICLR 2021
A COACH-PLAYER FRAMEWORK FOR DYNAMIC TEAM COMPOSITION
Anonymous authors Paper under double-blind review
ABSTRACT
In real-world multi-agent teams, agents with different capabilities may join leave "on the fly" without altering the team's overarching goals. Coordinating teams with such dynamic composition remains challenging problem: the optimal team strategy may vary with its composition. Inspired by real-world team sports, we propose coach-player framework tackle this problem. We assume that the players only have a partial view of the environment, while the coach has a com- plete view. The coach coordinates the players by distributing individual strategies. Specifically, we 1) propose an attention mechanism for both the players and the coach; 2) incorporate a variational objective to regularize learning; and 3) design an adaptive communication method to let the coach decide when to communicate with different players. Our attention mechanism on the players and the coach allows for varying number of heterogeneous agents, and can thus tackle the dy- namic team composition. We validate our methods on resource collection tasks in multi-agent particle environment. We demonstrate zero-shot generalization new team compositions with varying numbers of heterogeneous agents. The per- formance of our method is comparable even better than the setting where all players have a full view of the environment, but no coach. Moreover, we see that the performance stays nearly the same even when the coach communicates lit- tle as 13% of the time using our adaptive communication strategy. These results demonstrate the significance of coach to coordinate players dynamic teams.
INTRODUCTION
Cooperative multi-agen reinforcement learning (MARL) is the problem of coordinating a team of agents to perform shared task. It has broad applications in autonomous vehicle teams (Cao et al. 2012), sensor networks (Choi et al.. 2009), finance (Lee et al., (007). and social science (Leibo) al. 2017). Recent works in multi-agent reinforcement learning (MARL) have shed light solv- ing challenging multi-agent problems such as playing StarCraft with deep learning models (Rashid al., 2018). Among these methods, centralized training with decentralized execution (CTDE) has gained much attention since learning in centralized way enables better cooperation while execut- ing independently makes the system efficient and scalable (Lowe et al.) 2017). However, most deep CTDE approaches for cooperative MARL are limited to a fixed number of homogeneous agents.
Real-world multi-agent tasks, on the other hand, often involve dynamic teams. For example, a soccer game, a team receiving a red card has one fewer player. In this case, the team may switch to a more defensive strategy. As another example, consider an autonomous vehicle team for delivery. The control over the team depends on how many vehicles we have, how much load each vehicle permits, as well as the delivery destinations. In both examples, the optimal team strategy varies according the team composition! i.e., the size of the team and each agent's capability. In these settings, it is intractable to re-train the agents for each new team composition, and it is thus desirable have zero-shot generalization new team compositions that not seen during training.
Recently, Iqbal et al. (2000) proposed a multi-head attention model for learning in environments with variable number of agents under the CTDE framework. However, in many challenging tasks,
'Team composition is part of environmental scenario (de Witt et al. 2019), which also includes other
environment entities. The formal definition is in Section 2.1

---

Under review as a conference paper at ICLR 2021
coach
universe of teams
team is changing
Strategy
CTDE
reward
players
action
Dynamic composition
Hierarchical
the environment
(a) Training
(b) Dynamic composition
(c) MARL
Figure 1: (a) training, we sample teams from set of compositions. The coach observes the entire world and coordinates different teams via broadcasting strategies periodically; (b) A team with dynamic composition can be viewed as a sequence of fixed composition team, thus the proposed training generalizes to dynamic composition; (c) Our method is at the star position within MARL
the CTDE constraint is too restrictive as each agent only has access to its own decisions and partial environmental observations test time See Section 3.1 for an example where this requirement causes failure to learn. The CTDE constraint can be relaxed either by 1) allowing all agents to com- municate with each other (Zhang al., 2018) or 2) having a special "coach" agent who distributes strategic information based on the full view of the environment (Stone & Veloso 1999). The former case is typically too expensive for many CTDE scenarios (e.g.. battery-powered drones or vehicles), while the latter case of having coach may be feasible (e.g., satellites or watchtowers to monitor the field in which agents operate). In this work, we focus on the latter approach of having a coach to coordinate the agents.
Specifically, we grant the coach with global observation while agents only have partial views of the environment. We assume that the coach distribute information various agents only in lim- ited amounts. We model this communication through a continuous vector, termed as the strategy vector, and it is specific each agent. We design each agent's decision module to incorporate the most recent strategy vector from the coach. We further propose a variational objective to regularize learning, inspired by (Rakelly al. 2019 Wang al.) 2020a). In order save costs incurred in receiving information from the coach, we additionally design an adaptive policy where the coach communicates with different players only as needed. To train the coach and agents, we sample dif- ferent teams from set of team compositions. Recall that the training is centralized under the CTDE framework. At execution time, the learned policy generalizes across different team compositions in zero-shot manner. Our framework also allows for dynamic teams whose composition varies over time (see Figure (a-)))
Summary of Results:
We (1) propose coach-player framework for dynamic team composition of heterogeneous agents; (2) introduce a variational objective to regularize the learning, which leads to improved performance; (3) design adaptive communication strategy to minimize communication from the coach to the agents. We apply our methods on resource-collcction tasks in multi-agent particle environments. We evaluate zero-shot generalization for new team compositions at test time. Results show comparable or even better performance against methods where players have full ob- servation but coach. Moreover, there almost no performance degradation even when the coach communicates as little as 13% of the time with the players. These results demonstrate the effective- ness of having coach dynamic teams.
2 BACKGROUND
2.1 PROBLEM FORMULATION
We model the cooperative multi-agent task under the Decentralized partially observable Markov Decision Process (Dec-POMDP) (Oliehoek et al.) 2016). Specifically, we build the setting of Dec-POMDP with entities dde Witt et al. 2019), which considers entity-based knowledge represen- tation. Here, entities include both controllabe agents and other environment landmarks. In addition, we extend the representation to allow agents to have individual characteristics, i.e., skill-level, phys- ical condition, etc. Therefore, Dec-POMDP with characteristics- as a tuple (S, U, P, A,
entities can be described A,C, C, m. 2 p, 7). E represents the space of entities. Ve E E, the entity has its state representation 8 E Rdd, The global state is therefore the set (5 le E E} E S.
RRiooouusyy speaking the players in our method occasionally receive global information from the coach. But players still exccute independently with local views while they benefit from the centralized learning.
2

---

Under review as a conference paper at ICLR 2021
A subset of the entities are controllable agents a E A S E. For both agents and non-agent entities, we differentiate them based on their characteristies c" E C For example, c" can be a continuous vector that consists of two parts such that only one part can be non-zero vector. That is, if is an agent, the first part can represent its skill-level or physical condition, and if is an non-agent entity, the second part can represent its entity type. A scenario is multiset of entities c fe E) E Ω and possible scenarios are drawn from the distribution p(c). In other words, scenarios are unique up to the composition of the team and that of world entities. Fixing any particular scenario c, it maps to normal Dec-POMDP with the fixed multiset of entities felc² E c).
Given a scenario c, at each environment step, each agent a can observe a subset of entities specified by an observability function m : A x 3 + {0, 1}. where m(a. e) indicates whether agent a can observe entity e $ Therefore, an agent's 's observation
a set 0 = {s*Im(a, e) = 1] E O. All agents can perform the joint action u = [u la E A} E U, and the environment will step according to the transition dynamics P(s' s, uz c). After that, the entire team will receive a single scalar reward R(ss u,; c). Starting from an initial state 8o, the MARL objective is to maximize the discounted cumulative team reward over time: G = Es...s..1. uu. 1 7're]. where y is the discount factor. Our goal is to learn team policy that generalize across different scenarios (different team compositions) and eventually dynamic scenarios (varying team compositions over time).
For optimizing G. Q-leaming is specific method that learns accurate action-valle function and makes decision based on that. The optimal action-value function Q satisfies the Bellman equality: Qt c) = r(s, u: c) + -B max Qt* , u': c) where Q!i denote the team's 'ss optimal Q-value. A common strategy is to adopt function approximation and parameterize the optimal Q!et with parameter 0. Moreover, due to partial observability, the history of observation- action pairs is often encoded to a compact vector representation, i.e., via recurrent neural net- work (eddsker & Jain 1999), in place of the state: Q (T,, Uy; c) = E [Q" ue: Uz:c]], c)l, where (of: of).. In practice, at each time step t. the recurrent neural network takes in (ul of) as the new input, where ==1 = 0 at / 0 ZZhu et al. 2017). Deep Q- learning (Mnih et al., 2015) uses deep neural networks to approximate the Q function, its objective in our case is:
=
{ A} and 7"
(Te ((0) = E Ee (++ max Q5" (Te-1, u'; (Te, un e) )
1
(1)
"'
Here, D is a replay buffer that stores previously generated off-policy data. Qui is the target network parameterized by a delayed copy of 9 for stability.
2.2 VALUE FUNCTION FACTORIZATION AND ATTENTION QMIX
Factorizing the action-value function Q into per agent value function has become popular ap- proach in centralized training and decentralized execution. Specifically, Rashid et al. (2018) pro- poses QMIX that factorizes Q" u) into (Q" ua E A} and combines them via mixing network such that Va, 8Qm
aQ" 2 0. The condition guarantees that individual optimal action u" is also the best action for the team. As a result, during execution, the mixing network can be removed and agents work independently according to their own Q" Attention QMIX (A-QMIX) (Igbal et al.) 2020) augments the QMIX algorithm with attention mechanism to deal with an indefinite num- ber of agents/entities. In partieular, for each agent, the algorithm applies the multi-head attention (MHA) layer (Vaswani et al., 2017) to summarize the information of the other entities. This infor- mation is used for both encoding the agent's state and adjusting the mixing network. Specifically, the input o is represented by two matrices: the entity state matrix X and the observability ma- trix M. Assume at the given scenario c, there exists n. entities, n. of which are the controllable agents, then X E Xde includes all entities encoding and the first nh rows belong to agents. M E {0, 13 X is binary observability mask and M. = m(a', e²) indicates whether agent i observes entity j. X is first passed through an encoder, i.e., a single-layer feed-forward net- work, and becomes X. Denote the k-th row of X as h2, then for the i-th agent, the MHA layer then takes h4 as the query and May = 1} as the keys to compute a latent representation of a' observation. For the mixing network, the same MHA layer will take X and the full observation
*c" is part of , but we will explicitly write out c² in the following for emphasis.
'An agent can always observe itself, i.e. m(a, a) = 1, 1. A.
3

---

Under review as a conference paper at ICLR 2021
matrix M where Mi if both e' and e² exist in the scenario c, and outputs the encoded global representation for each agent. These encoded representations are then used to generate the mixing network. We refer readers to Appendix B of Iqbal et al. (2020) for more details. While A-QMIX in principle applies to the dynamic team composition problem, it is restricted to fully decentralized execution with partial observation. We borrow the attention modules from A-QMIX but additionally investigate how to efficiently take advantage of the global information by introducing the coach.
lqbal et al. 2020)) proposes an extended version of A-QMIX, called Attentive-Imaginativ QMIX (AI-QMIX), which randomly breaks up the team into two disjoint parts for each agent's Q" to further decompose the Q value. While the authors demonstrate AI- -MII outperforms A A-QMIX on a gridworld resource allocation task and a modified StarCraft environment. As we will show in the experiment section, we find that AI-QMIX does not improve over A-QMIX by much while doubling the computation resource. For this reason, our method is mainly based on the A-QMIX framework, but extending to AI-QMIX straightforward.
METHOD
Here we present the coach-player architecture incorporate global information for adapting the team-level strategy across different scenarios We first introduce the coach agent that coordinates base agents with global information via broadcasting strategies periodically. Then we present the learning objective and and an additional variational objective to regularize the training. We finish by introducing a method to reduce the broadcast rate and provide analysis to support it.
3.1 ON THE IMPORTANCE OF GLOBAL INFORMATION
As the optimal team strategy varies according to the scenario c, which includes the team composi- tion, it important for the team to be aware of the scenario change promptly. In an extreme example, assume in a multi-agent problem where every agent has its skill-level represented by real number e E IR and there is task to complete. For each agent o, x E {0. 1 indicates whether a chooses to perform the task. The reward is defined as R(u;c) reward is proportional to the skill-level of the agent who performs it and the team got penalized if more than I agent choose to perform the task. If the underlying scenario is fixed, even all agents are unaware of others' capabilities. it is still possible for the team to gradually figure out the optimal strategy. By contrast, when c is subject to change, i.e. agents with different can join or leave, even if we allow agents to communicate via network, the information that particular agent joins or leaves generally takes d time steps to propagate where d is the longest shortest path from that agent to any other agents. Therefore, can that knowing the global information is not only beneficial but sometimes also necessary for coordination. This motivates the introduction of the coach agent.
max, ca -ta +1 - -E. u². In other words, the
3.2 COACH AND PLAYERS
We introduce coach agent and grant it with global observation. To preserve efficiency in the decentralized setting, we limit the coach agent to only distribute information via a continuous vector 2n E Rd (dg the dimension of strategy) to agent a. which we call the strategy, once every T time steps. T' is the communication interval. The team strategy is therefore represented as z = {22 la e A). Strategies are predicted via a function f parameterized by Φ. Specifically, we assume
z" ~ NN(",,
where (#4 = 4441 E A}, Σ = (aala E A}) fo(
(2)
Within the next T steps, agent a will act conditioned on the strategy 2a. Specifically, within an episode, at time t E [ulu = (mod T)}. the coach observes the global state st, and computes and distributes the strategies zt, for all agents. From time E [tx, tx T - 1). any agent will act according to its individual action-value Q" (T c.).
Denote f = max u/v = 0 (mod T)) and S t}. the most recent time step when the coach distribute strategies. The mean square Bellman error objective equation becomes
2
CRL( Φ) = Blonamnna (re (r 1r+ymma max Q4 (T411 u' u'(z),1ic) - Q8 us 2z;c) c)
4

---

Under review as a conference paper at ICLR 2021
hi
T-step with the strategies
h
TTT
TT
AAgetts
h2
GPL
GRI
GRU-NUL
MLP
Partial
M -
Multi heed Altertion
Mult-Hood Athontinn
Murti-kodd Afte ntoon
11
2
(oe
FC
St+T
FC
Qtot (Te
View M
Mess-
M ++7
Mokng Network
team
team
Seam
M Head
{2 (Te u ug)
Niu, Ex)
21+T
Toam
Figure 2: The coach-player network architecture. Here, GRU refers to gated recurrent unit (Chung et al. 2014]; MLP refers to multi-layer perceptron; FC refers to fully connected layer. Both coach and players use multi-head attention to encode information. The coach has full view while players have partial views. hf encodes agent a's history. hę combines the most recent strategy 2; -t-1%T to predict the individual utility Q". The mixing network combines all Q"s to predict Q
where z; (o) Ztii f5(8141jc) and Φ is the parameter of the target network for the coach's strategy predictor J. We build our network on top A A-QMIX but use a separate multi-head attention (MHA) layer encode the global states that the coach observes. For the mixing network, we also use the coach's 's output from the MHA layer for mixing the individual Q to form the team We provide more details in Appendix.
Que The entire architecture is described in Figure 3.2
3.3 REGULARIZING WITH VARIATIONAL OBJECTIVE
Inspired by recent work that applied variational inference to regularize the learning of latent space reinforcement learning (Rakelly et al. 2019; Wang et al. 2020a), we also introduce a variational objective to stabilize the training. Intuitively. assigned strategy. In other words, the received strategy should be identifiable from the agent's 's future trajectory. Therefore, the agent's s future observation-action pairs Ç (of+1. ui+1 o2 ui+2.-... 04 uf -1) We maximize the following variational lower bound:
agent's behavior should be consistent with its
propose to maximize the mutual information between the strategy and
log 9s (2(2 st)
I2f;; st) Esr.et .5
DKL p(22117, st), (<<2)
p(2418t)
(4)
18 (zf CP 8t)
Es, log (e 81) + (= st).
2 Es Es..e
log
p(=0(8:)
Here H-)) denotes the entropy and qe is the variational distribution parameterized by &. We fur-
ther adopt the Gaussian factorization for qe as in (Rakelly al. 2019). i.c. (2 (8 8:) oc
45 (=ff (s,, uz) I 45 (=flog ug), where each 17 Gaussian distribution. So predicts the ie and Eyg of multivariate normal distribution from which we calculate the log-probability of 24. In practice, =ē is sampled from So using the re-parameterization trick (Kingma & Welling 2013). The objective is Evar {) = -AilE sp [log qe (2f st)] Ag (2f (t)), where A and Ag are tunable coefficients.
(k)
3.4 REDUCING THE COMMUNICATION FREQUENCY
So far, we assume at every 7 steps the coach periodically broadcasts new strategies for all agents. In practice, broadcasting suffers communication cost or bandwidth limit. So it desirable to only distribute strategies when "necessary". To reduce the communication frequency, we propose an intuitive method that decides whether to distribute new strategies based on the f distance the old strategy to the one. In particular, at time step KT, l E Z, assuming the prior strategy for agent a is a the new strategy for agent a is
if | C)
- otherwise.
2 5
24
(S)
oold
5

---

Under review as a conference paper at ICLR 2021
For general time step the individual strategy for a is therefore 24. Here 3 is a manually spec- ified threshold. Note that we can train a single model and apply this criterion for all agents. By adjusting 3, one can easily achieve different communication frequencies. Intuitively. when the previous strategy is "close" to the current one, it should be more tolerant to keep using it. The intuition is concrete when the learned Q has relatively small Lipschitz constant. If we assume VT,, Uz 8t, 8f, C, 1|Q (T,, Ue. f(8;); c) - (st Ue; c)ll S K, where Qte is the optimal Q, and 1 24 IQu (Ta, Ue 2 cc QUe (T, u 12 2 "; ee)] S n| z we have the following:
Theorem 1. If the used team strategies ε, satisfies Va,t, t, || 2i ll B, denote the action- value and the value function of following the used strategies as Q and V, i.e. ((T(Zz;c)
max, Q(T u/z6; c)2 and define V similarly, we have
V - V(TllE))))l 2(nan)3 + x))
(6)
1-7
where ng is the number of agents and y is the discount factor.
We defer the proof to Appendix A The method described in equation 5 satisfies the condition in Theorem D and therefore when 3 is small, distributing strategies according to equation 5 result in much performance drop.
will not
EXPERIMENTS
We design the experiments to 1) verify the effectiveness of the coach agent; 2) investigate how performance varies with the interval T; 3) test if the variational objective is useful; and to 4) un- derstand how much the performance drops by adopting the method equation [5] We test our idea on resource collection task with different scenarios in customized multi-agent particle environ- ments (Lowe et al., 2017). In the following, we call our method COPA --Dach-and-PAyyee
4.1 RESOURCE COLLECTION
In Resource Collection, a team of agents coordinate to collect different resources spread out on a square map with width 1.8. There are 4 types of entites: the resources, the agents, the home and the invader. We assume there are 3 types of resources: (r)ed, (g)reen and (b)lue. In the world, always resources appear with 2 of each type. Each agent has 4 characteristics (c+, cg. vf,,), where cg represents how efficient collects the resource x and is the agent's s max moving speed. The agent's 's job is to collect the most amount of resources and bring them home, and catch the invader if it appears. If collects x, the team receives reward of 10 ch as reward. Holding any resource, agents cannot collect more and need to bring the resource home until going out again. Bringing resource home has 1 reward. Occasionally the invader appears and goes directly to home. Any agent catch the invader will have 4 reward. If the invader reaches home, the team is penalized by -4 reward. Each agent has actions: accelerate up/ / down left right and decelerate, and it observes anything within 0.2 distance. The maximum episode length is 145. In training, we allow scenarios to have 2 to agents, and for each agent, c,,, cg. of are chosen from (0.1, 0.5,0.9) 0.9) and the max speed ea from {0.3, 0.5, 0.7}. We design 3 testing tasks: 5-agent task, 6-agent task, and a varying-agent task. For each task, we generate 1000 different scenarios c. Each scenario includes n. agents, 6 resources and an invader. For agents, cf, Cg. of are chosen uniformly from the interval [0.1, 0.9) and v from [0.2, 0.8). For a particular scenario in the varying agent task, starting from agents, the environment randomly adds or drops an agent every steps as long as the number of agents remains in [2, 6). / a random variable from the uniform distribution ((( 12). See Figure for an example run of the learned policy.
Effectiveness of Coach We provide the training curve in Figure 4.1 (a) where the communication interval is set to T = 4. The black solid line is hard-coded greedy algorithm where agents always go for the resource they are mostly good at collecting, and whenever the invader appears, the closest agent goes for it. We see that without global information, A-QMIX and AI-QMIX are significantly below the hard-coded baseline. Without the coach, we let all agents have the global view every T steps in A- A-QMIX (periodic) but it barely improves over A-QMIX. A-QMIX (full) is fully centralized, i.e., all agents have global view. Without the variational objective, COPA is comparable against A-QMIX (full). With the variational objective, it becomes even better than

---

Under review as a conference paper at ICLR 2021
Home
Resources
Invader
Agent
1t
(ii) t
(iii) 10
( 19
(V) t= 21
(vi) t 30
Here, ca represented by rgb values, e = (r, g. b, v). For illustration, set agents rgb to be one-hot but it can vary in practice.
Figure 3: An example episode up to 30 with communication interval T
(i) an agent starts at home; (ii) the invader (black) appears while the agent (red) goes to the red resource; (ii) another agent is spawned while the old agent brings resource home; (iv) one agent goes for the invader while the other for resource; (v-vi) a new agent (blue) is spawned and goes for the blue resource while other agents (red) are bringing resources home.
A A-QMIX (full). Note that all baseline methods are scaled to have more parameters than COPA. The results demonstrate the importance of global coordination and the coach-player hierarchy.
Communication Interval To investigate how performance varies with T. train with dif- ferent T chosen from [2. 4, 8, 12, 16, 20, 24] in Figure 4.1.b). Interestingly, the performance peaks T 4, contradicting the intuition that smaller T is better. This shows the coach is more useful when it can make the agents behavior smooth/consistent over time.
120
120
100
greedy expert
A-DMIX
80
randem
80
(full)
Reward
A-QMIX
T-4
Al-OMIX
T-8
A-QMIX (pariodic)
A-QMIX (full)
T-16
20
COPA
20
T-20
Timestap (mil)
Timestop [mil)
(a) Comparison with baselines
(b) Ablations on
Figure 4: Training curves for Resource Collection. (a) comparison against A- A-QIIX, AI-QMIX and COPA without the variational objective. Here we choose T 4; (b) ablations on the communication interval T. All results are averaged over seeds.
Method
Bay. (n 5) Comm. Frequency
Eav.
Eav. (varying Frrequency
Frequency
Random Policy
6.9
N/A
10.4
N/A
2.3
N/A
Greedy Expert
N/A
AI-QMIX
90.5#
0.
109.3:1..6
0.
61.5=0.9
0.
A-QMIX
96.9++2.1
0.
115,1+22.1
66.2+116
QMIX (periodic)
93.1-20.4
0.25
104.2:122.6
0.25
68.91112.6
0.25
A-QMIX (full)
157.418.5
I.
179.6:19.8
114.3:26.2
COPA (2
175.62119
0.25
203
0.25
124.9=00.9
0.25
COPA (2
174.4=1.7
0.18
100.3=1.6
0.18
122. 122.8=15
0.18
(2
COPA (2 = 149.3=14
0.08
174.7=1.7
0.08
104.7=16
0.08
(7
109.4:3.66
0.04
130.6:t:40
0.04
80.6=200
Table 1: Generalization performance unseen environments with agents and dynamic team composition. Results are computed from models trained with 5 different seeds. Communication frequency is compared to communicating with all agents at every step.
1.0
Zero-shot Generalization We apply the learned model with T 4 to the 3 testing environments. Results are provided in Table 4.1 The communication frequency is calculated accord-
Relative reward
ing to the fully centralized setting. For instance, when T 4 and B = 0, it results in an average 25% centralization frequency.
static medium dynamic
As increase to suppress the distribution of strategies. see that the performance shows no significant drop till 13% cen-
tralization frequency. Moreover, apply the same model to
1.0 Relative comm. frequency
0.8
0.6 0.4 0.2 0.0
Figure 5: The varying sensitivity communication frequency.

---

Under review as a conference paper at ICLR 2021
environments that are dynamic to different extents. In the more static environment, resources always spawned at the same locations. In medium environment, resources are spawned randomly but there is invader. The more dynamic environment is the 3rd environment in Table 4.1 where the team is dynamic in composition and there exists the invader. Result is summarized Fig Figure 5 0, and the
Here, the x-axis is normalized according to the communication frequency when y-axis is normalized by the corresponding performance. As expected, as the environment becomes more dynamic, low communication frequency more severely downgrades the performance.
4.2 RESCUE GAME
Search-and-rescue is a natural application of multi-agent systems. In this section we further apply COPA to a rescue game. In particular, we consider a 10 x 10 grid-world, where each grid contains a building. At any time step, each building is subject to catch a fire. When building is on fire, it has an emergency level d² ~ [1001) 1). Within the world, at most 10 buildings will be on fire at the same time. Fortunately surveillance firefighters. Each robot a has a skill-level c* E [0.2, 1.0]. A robot has actions, moving up/down//eft/right and put out the fire. If a is at a building on fire and chooses to put out the fire, the emergency level will be reduced to c² + max(c² - c², 0). At each time step I, the overall- emergency given by cf Ex(r) since we want to penalize the existence of more emergent fire. The reward is defined as rt c - -cf- the amount of emergence level the team reduces. During training, we sample from 3 5 and these robots are spawned randomly across the world. Each agent's skill-level is sampled from [0.2, 1.0). Then a random number of 3 6 buildings will catch fire. During testing, we enlarge n to 2 8 agents and sample up to 10 buildings on fire. Interestingly, we find that A- A-QMIX with full observation
have (n is a random number from 2 to 8) robots who the
We summarize the result the Table 4.2
Random
Greedy
A-QMIX A-QMIX (full)
COPA (w/o L.x) COPA (L)
COPA (0.5) COPA (0.15)
Epi. Reward
1.4
7.0
5.400.5
1.620.5
9.0:1..6
10.71:0.6
11.1:0.8
8.910..5
Table 2: Average episodic reward over the same 500 Rescue games. Results are averaged over the same algorithm trained with different seeds. For COPA (æ), denotes the communication frequency. Greedy algorithm matches the k-th skillful agent for the k-th emergent building.
failed to learn. We conjecture this is because the team has too much information to process during training and therefore it is hard to search for a good policy. COPA consistently outperforms all baselines even with communication frequency low 0.15.
RELATED WORKS
In this section we briefly go over some related works in cooperative multi-agent reinforcement learn- ing and bierarchical reinforcement learning.
Centralized Training with Decentralized Execution Centralized training with decentralized ex- ecution (CTDE) assumes agents execute independently but uses the global information for training. A branch of methods investigates factorizable Q functions (Sunehag et al.) 2017] Rashid al.) 2018 Mahajan et al. 2019 Son et al., 2019) where the team Q is decomposed into individual utility func- tions. Some other methods adopt actor-critic method where only the critic is centralized (Foerster ct al. 2017; Lowe et al. 2017). However, most deep CTDE methods by structure require fixed-size teams and are often applied on homogeneous teams.
Methods for Dynamic Compositions Several recent works pay attention to transfer learning and curriculum learning in MARL problems where the learned policy is is warm start for new tasks (Carion et al. 2019; Shu & Tian 2018 Agarwal et al. 2019 Wang et al. 20206 Long et al. 2020). These works focus curriculum learning and mostly consider homogeneous agents. Hence the team strategy is relatively consistent. Iqbal et al. (2020) first adopt the multi-head attention mechanism for dealing with varying size heterogeneous team. But the heterogeneity comes from a small finite set of agent types (usually 2 to 3). Additionally, the method is fully decentralized and therefore less adaptive frequent change in team composition.
Ad Hoe Teamwork and Networked Agents Ad hoc teamwork studies the problem of quick adap- Barrett & Stone 2012). However, ad hoc teamwork
tation to unknown teams (Genter al. 2011
8

---

Under review as a conference paper at ICLR 2021
focuses the single ad hoc agent and often assumes control over the teammates and therefore is essentially a single-agent problem. Decentralized networked agents assume information can propa- gate among agents and their neighbors (Kar al. 2013 Macua et al. 2014 Suttle et al. 20199) Zhang et al. 2018). However, research in networked agents still mainly focus on homogeneous fixed-size teams. Although it is possible extend the idea for the dynamic team composition problem,
leave it as a future work.
Hierarchical Reinforcement Learning The main focus of hierarchical RL/MARL is decom- pose the task into hierarchies: a meta-controller selects either a temporal abstracted action (Bacon al. 2017), called an option, or a goal state (Vezhnevets et al. 2017) for the base agents. Then the base agents shift their purposes to finish the assigned option or reach the goal. Therefore usually the base agents have different learning objective from the meta-controller. Recent deep MARL methods also demonstrate role emergence (Wang et al., (020a) or skill emergence (Yang et al., 2019). But the inferred role/skill only conditioned on the individual trajectory. The coach in our method uses global information to determine the strategies for the base agents. To our best knowledge, we are the first to consider applying such hierarchy for teams with varying number of heterogeneous agents.
CONCLUSION
We investigated a new setting of multi-agent reinforcement learning problems, where both the team size and members' capabilities framework where the coach coordinates with global view but players execute independently with local views and the coach's s strategy. We developed variational objective regularize the learning and introduces an intuitive method to suppress unnecessary distribution of strategies. The experi- ment results across multiple unseen scenarios the Resource Collection task demonstrate the effec- tiveness of the coach agent. The zero-shot generalization ability of our method shows a promising direction to real-world ad hoc multi-agent coordination.
subject to change. To this end, we proposed a coach-player
REFERENCES
Akshat Agarwal, Sumit Kumar, and Katia Sycara. Learning transferable cooperative behavior in multi-agent teams. arXiv preprint arXiv: 0022, 2019.
Pierre- Bacon, Jean Harb, and Doina Precup. The option-crite critic architecture. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Samuel Barrett and Peter Stone. An analysis framework for ad hoc teamwork tasks. In Pro- ceedings of the 11th International Conference on Autonomous Agents and Multiagent Sys- tems (AAMAS 2012), June 2012. BRLhttt:///www cs. www.uueeaasee... wwt.eeeee//uuaaiieeeeeeee
Yongcan Cao, Wenwu Yu, Wei Ren, and Guanrong Chen. An overview of recent progress in the study of distributed multi-agent coordination. IEEE Transactions on Industrial informatics, 9(1): 427-438, 2012.
Nicolas Carion, Nicolas Usunier, Gabriel Synnaeve, and Alessandro Lazaric. A structured prediction approach for generalization in cooperative multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pp. 8130-810 (140-8810,220 2019.
Jongeun Choi, Songhwai Oh, and Roberto Horowitz. Distributed learning and cooperative control for multi-agent systems. Automatica, 45(12):2802-28144 2009.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 2014.
Christian Schroeder de Witt, Jakob Foerster, Gregory Farquhar, Philip Torr, Wendelin Boehmer, and Shimon Whiteson. Multi-agent common knowledge reinforcement learning. In Advances in Neural Information Processing Systems, pp. 9927-9939, 2019.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras. Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. arXiv preprint /705 08926, 2017.

---

Under review as a conference paper at ICLR 2021
Katie Genter, Noa Agmon, and Peter Stone. Role-based ad hoc teamwork. In Proceedings of the Plan, Activity, and Intent Recognition Workshop at the Twenty- Conference on Artificial Intel- ligence (PAIR-1I), August 2011. URL http:/ www. es utexas www/euuuuueseeeeeeee ai-aaalaaa l
PATRI SaIR11-katee ie
Shariq Iqbal, Christian A Schroeder de Witt, Bei Peng, Wendelin Böhmer, Shimon Whiteson, and Fei Sha. Ai-qmix: Attention and imagination for dynamic multi-agent reinforcement learning. arXiv preprint arXiv:2006.04220 04222, 2020.
Soummya Kar, José MF Moura, and H Vincent Poor. Qd-learning: A collaborative distributed strat- egy for multi- reinforcement learning through consensus+innovations IEEE Transactions on Signal Processing, 61(7):1848-188) 2013.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114 2013.
arXiv preprint
Jae Won Lee, Jonghun Park, O Jangmin, Jongwoo Lee, and Euyseok Hong. A multiagent approach to g-learning for daily stock trading. IEEE Transactions on Systems, Man, and Cybernetics-Par A: Systems and Humans, 37(6):864-877, 2007.
Joel Leibo, Vinicius Zambaldi, Mare Lanctot, Janusz Marecki, and Thore Graepel. Multi-agent reinforcement learning in sequential social dilemmas. arXiv preprint arXiv:1702.03037, 2017.
Qian Long. Zihan Zhou, Abhibav Gupta, Fei Fang, Yi Wu, and Xiaolong Wang. Evolu- tionary population curriculum for scaling multi-agent reinforcement learning. arXiv preprint arXiv:2003. 10423, 2020.
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Picter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-come competitv environments. In Advances neural infor- mation processing systems, pp. 6379-6390, 2017.
Sergio Valcarcel Macua, Jianshu Chen, Santiago Zazo, and Ali H Sayed. Distributed policy evalu- ation under multiple behavior strategies. IEEE Transactions on Automatic Control, 60(5): 1274, 2014.
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. In Advances in Neural Information Processing Systems, pp. 7613-7624, 2019.
Larry Medsker and Lakhmi C Jain. Recurrent neural networks: design and applications. CRC press, 1999.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 5177540) 2015.
Frans A Oliehoek, Christopher Amato, et al. A concise introduction to decentralized POMDPs, volume Springer, 2016.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinnorceement learning via prohabilistic context variables. In International conference machine learning, pp. 5331-5340, 2019.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foer- ster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint arXiv: 1445, 2018.
Tianmin Shu and Yuandong Tian. M 3 rl: Mind-awar multi-agent management reinforcement learning. arXiv preprint 10I.::1110.000 2018.
Kyunghwan Son. Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learn- ing to factorize with transformation for cooperative multi-agent reinforcement learning. arXiv preprint ar 1905 1905005408 2019.
10

---

Under review as a conference paper at ICLR 2021
Peter Stone and Manuela Veloso. Task decomposition, dynamic role assignment. and low-bandwidth communication for real-time strategic teamwork. Artificial Intelligence, 110(2)241-275, 1999.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. caloe-decomposition networks for cooperative multi-agent learning. arXiv preprint
007770002290 2017.
Wesley Suttle, Zhuoran Yang, Kaiqing Zhang, Zhaoran Wang, Tamer Basar, and Ji Liu. A multi- agent off-poliey actor-critic algorithm for distributed reinforcement learning. arXiv preprint MoXiv:/903306372. 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, pp. 5998-6008, 2017.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. arXiv preprint 170:11703.01161, 2017.
Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Multi-agent reinforcement learning with emergent roles. arXiv preprint arXiv:2003. 08039, 2020a.
Weixun Wang, Tianpei Yang, Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, Changjie Fan, and Yang Gao. From few to more: Large-scale dynamic multiagent curriculum learning. In AAAI, pp. 7293-7300, 2020b.
Jiachen Yang, Igor Borovikov, and Hongyuan Zha. Hierarchical cooperative multi-agent reinforce- ment learning with skill discovery. arXiv preprint arXiv:1912.03555 2019.
Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Başar. Fully decentralized multi- agent reinforcement learning with networked agents, arXiv preprint arXiv: 2018.
Pengfei Zhu, Xin Li, Pascal Poupart, and Guanghui Miao. On improving deep reinforcement learn- ing for pomdps. arXiv preprint MiXi:1100.009977 2017.
A APPENDIX
PROOF OF THEOREM
Here we expand the assumptions from Theorem @ and provide the proof for it. The two assumptions are:
Assumption 1. Denote the learned team action-value function as Qta, the learned coach strategy encoder as f and the true optimal action-value function as Q2et. We assume for any Tt, uy St, Sis C,
lQ (Te, Ur, ((:): Q" (St, uei c)ll2 S.K.
Ji=1 and the particular individual action-value at state with action {q Qa (5 ut D)= Then we assume unilaterally varying any des to d', i.e. all other q= remain the same, will not cause dramatic change of Q"" if d' stays closely to qhi:
Assumption 2. Denote the learned individual aation-value function as (Qa: na
1Q (9th)))) - Qq ai ".9')l S nm 9'l
(8)
for any agent a and Ve", Tf, uf, 28 with proper dimensions,
QQ ufl2f; e²) Q" u 22: e I nallet 28112-
In other words, assumption I assumes the learned Qui approximates the true optimal Q!d well combined with the learned coach strategy function f and assumption assumes the learned team action-value net has bounded Lipschitz constant. Next provide the proof for Theorem I
Note here only assume Q is accurate around the predicted strategy by for any strategy.

---

Under review as a conference paper at ICLR 2021
Proof. From assumption 2, it is easy to check that if || -2il2 I 3 for all then 1Qt uz( c) Q u 21, c) n2. For notation convenience, ignore the perscript of tot and the condition u = arg max, Q(T, w). Similarly we can define u as the action the optimal Q₂ takes and ú that
For state s, denote the action the learned policy take as ut,
Q takes. From assumption 1, we know that
(s u²) 2 Q(T, u x u²) 2 (s, u²) - 2x..
(10)
Therefore taking ut will result in at most 2k. performance drop at this single step. Similarly, denote 6o 777 then
@) @ ū) 6a uT) <o Q(T, u²) 260.
(11)
Hence Q. ū) (s, u²) - 2(co k). Note that this means taking the action ú in the place of u at state 8 will result in at most 2(eo + x) performance drop. This conclusion generalizes to any step Therefore,
each single step the performance is bounded within 2(cn + k), then overall
the performance is within 2(eo + +x)((1 1011-21 7).
NETWORK ARCHITECTURE
For all experiments, we use the same network architecture where all intermediate hidden layer have 128 dimensions. Note that this is possible since the only difference is the number entities, which does not influence our architecture when adopting an attention model. The architecture details follow exactly as in Appendix A of (Iqbal et 2020).
TRAINING DETAILS
To train the model, we set the max total number steps to million. Then we use the exponentially decayed e-greedy algorithm as our exploration policy, starting from c = 1.0 to e, = 0.05. We parallel the environment with 8 threads for training. Details on hyper-parameters are available in Section A
HYPER PARAMETERS
For all experiments, we use the same set of hyper-parameters. We provide them in the following table:
Name
Description
Value
[D]
replay buffer size number of heads in multi-head attention number parallel threads for running the environment
100000
Thead
4
Tobread
8
dh
the hidden dimension all modules
128
the discount factor learning rate optimizer a value in RMSprop value in RMSprop batch size
0.99
Ir
0.0003
R.Sprop
a
0.99
0.00001
Thhnthh
256
grad clip
clipping value of gradient how frequent do we update the target network
10
target update frequency
200 updates 0.001
\
A in variational objective
X2 in variational objective
0.0001
Table 3: parpeetrras in our experiments.
12


Reviews:
[{"review_id": "9LYsN51laG", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper studies the dynamic multi-agent team coordination problem, in which the optimal team strategy may change over time as the environment and the team members vary. The authors propose a coach-player framework in which only the coach has full information about the game while the players only have their own local information, separately. In this framework, the coach computes the optimal game plan and sends the plan to the players periodically, and the players play according to the received plan from the coach and their local information. Empirical studies demonstrate the effectiveness of the new framework.\n\nComments:\n\nThis paper is generally well-written and clear. The idea of having a coach to compute the optimal game plan to coordinate the players seems interesting and non-trivial. In the empirical study, COPA with variational objective outperforms the state-of-art benchmark. The authors also have an interesting observation that always communicating the optimal global game plan to the players is not always the optimal strategy.\n\n1. The reviewer is not familiar with A-QMIX. Is it correct that the main difference between A-QMIX (full) and COPA is that in A-QMIX, even though the players have access to the full information, they still compute for their own game plans instead of coming up with a joint optimal game plan for the team?\n\n2. As the authors claim that the coach is more useful when it can make the agents behavior smooth/consistent over time, the reviewer is wondering whether it would be helpful to communicate a smoothed strategy update to the players. More precisely, the current way is to set a threshold on the distance between z_t and z_old to limit the communication frequency. For example, if the coach always communicates, whether it would be helpful if the coach sends a convex combination between z_t and z_old (similar to ideas of online learning) to smooth the strategy update?", "extra_scores": {}, "extra_text": {}}, {"review_id": "7twWau2QY52", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the important problem of being able to deal with heterogeneous teams of agents (that might change over time) in cooperative multiagent reinforcement learning. In the strictly cooperative setting with identical problem representations among agents, this essentially boils down to a problem of decentralized learning and control, where, effectively, the improvements demonstrated by the paper over existing papers boil down to effectively reducing the ways in which non-local information needs to be communicated between a global observer (coach) and the individual learning (agents). The specific novelty of the approach is to introduce a variational objective to stablise training, and introduce a heuristic for reducing the communication frequency. \n\nAs is the case with many similar papers in deep reinforcement learning, much of the paper's approach is premised on pre-training high-dimensional function approximators with an enormous amount of data to learn an optimal strategy offline that can then be applied in sequential games, in this case a reasonably complex (yet toy) scenario. Only a small part of the paper focuses on the technical innovations while the bulk of it describes the application of DNN techniques to the specific coach-player mechanism in a specific game. In my view, this does not provide enough evidence to be able to assess the general applicability of the approach, in particular because it is only evaluated in a single domain and much of the reported performance could be attributed to fine-tuning of domain-specific parameters. \n\nThe paper is generally well-written and relatively easy to follow, though many of the design decisions are not very well explained, and there is no explicit problem formulation regarding what optimal communication strategies would look like, which would help evaluate the advances reported in the paper with respect to the overall fundamental problem. As is the case with many similar papers, while the existing work is surveyed, I find comparisons to other approaches somewhat unfair, as many of them propose actual algorithms that deal with the problem of coordination at runtime, rather than creating than making (as is proposed here) certain small modifications to immensely data-hungry function approximation methods that learn an effective coordination policy from data, and offer little insight on what it actually looks like that advances our understanding of the problem to build better future AI systems.\n \nThe language is not perfect in some places, but could be easily polished with additional proofreading.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7HV6wwG6dKc", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThe authors propose a coach-player framework for dynamic team composition of dynamic and heterogeneous agents based on deep Q learning with an attention mechanism and a variational objective to regularize the learning. The authors design an adaptive communication strategy to minimize communication from the coach to the agents. Using a resource-collection task in multi-agent particle environments, the authors evaluated zero-shot generalization for new team compositions at test time. Results show comparable or even better performance against methods where players have full observation but no coach. Interestingly, there is almost no performance degradation even when the coach communicates as little as 13% of the time with the players. \n\nReasons for score: \nAlthough the motivation, novelty compared with the related work, and a superiority of the proposed method in an experimental result were clear, there were unclear points in the background and methods (without sharing codes) and the authors performed only a simple experiment. I think the idea is interesting and contributed to this community, but for the above reasons it is difficult to provide a higher rating. \n\n\nPros:\n1. The motivation and novelty compared with the related work were clear. \n2. The contribution in this paper is to propose a coach-player framework for dynamic team composition of dynamic and heterogeneous agents based on deep Q learning with an attention mechanism and a variational objective to regularize the learning. \n3. In a resource-collection task in multi-agent particle environments, the proposed method clearly demonstrated the effectiveness of having a coach in dynamic teams.\n\nCons:\n1. There were many unclear points in Sections 2 and 3 (methodology, see below)\n2. The authors performed only one experiment in a resource-collection task.  It seems to be a relatively simple setting for me, but the investigation in another experiment will help us understand the effectiveness of the proposed method from more general perspectives.\n\nOther comments:\n\nRegarding deep recurrent Q-learning (Zhu et al., 2017), was u_0^a in \"tau^a = (o_0^a, u_0^a,... o_t^a) correct?  If correct, more formal expression and additional explanation may be required. \n\nDid u’ in eqs. (1) and (2) mean u_{t+1}? It may be confusing.\n\nWas 2.2 VALUE FUNCTION action-value function or Q function? \n\n“Speciﬁcally, the input o is represented by two matrices…” Was the input tau rather than o in 2.2? What is the input?\n\nI did not understand the first equation of eq. (4). It seems to be different from the definition of the mutual information that I know, and the idea of the referenced papers (Rakelly et al., 2019; Wang et al., 2020a). A careful introduction is required.\n\n======after rebuttal=========\n\nThank you for answering my questions.  I understood these points. The authors added a new simple experiment and a code, whereas the manuscript at the current stage can improve the clarity. All things considered, I increased the rating. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6eVTmfFr2zu", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper proposes to extend the centralized training, decentralized execution paradigm for cooperative multi-agent RL with a coach-player framework. The proposed framework would allow the coach to have access to the full observation, but only allow for limited communication of a continuous strategy vector to the other agents / players. It is argued, that the framework helps with dynamic composition of multi-agent teams, which encompasses variable numbers of agents, as well as heterogeneous agents.\nA variational objective is introduced, to force the player trajectories to be coherent with the strategy assigned by the coach. A concrete algorithm is proposed, that builds strongly upon [1]. Experimental evidence shows, that the coach-player framework with variational objective outperforms a baseline from this paper (referred to as A-QMIX) even if the latter is granted full observability for its agents.\n \nStrong:\nConsidering the addition of a coach agent with constraint communication makes intuitively sense and there very well might be scenarios, where this addition is useful to solve actual tasks. The paper provides experimental evidence, that structured communication about the coordination of decentralized agents is beneficial, even if the communication frequency is enforced to be sparse. The variational objective seems to be especially helpful. A proof is included, which shows that sparse communication results in only small loss compared to more frequent communication if the representation of the team strategy is relatively stable (wrt. changes in its L2 norm). The paper includes results which suggest good generalization performance of the proposed algorithm wrt. unseen team composition.\n \nWeak:\nMainly, I consider the papers experimental evaluation to be not very convincing. This might very well be due to the paper proposing a specific type of communication (with global knowledge), while disregarding to argue about different types of communication in MARL and only comparing to an approach that does not explicitly involve communication. This seems especially strange, since the multi-agent particle environment upon which the experiments build [2], was explicitly contrived to analyze communication (grounded in RL scenarios). The comparison of the proposed algorithm (referred to as COPA) to the baseline (A-QMIX) alone does not seem very insightful. Both show about the same performance if A-QMIX agents are provided with full observations and COPA is ablated wrt. its variational objective. That COPA’s performance for this task seems dependent on the variational objective (which is not further discussed), to me is an indication that important issues are not addressed in the paper / the experiments. Given its focus, I further argue that the paper lacks related literature with respect to learned communication in MARL [e.g. compare with listed literature in a current survey, 3].\n\nConclusion:\nIn its current form, I find the paper’s line of argumentation, as well as the experiments, to detached from relevant prior work wrt. learned communication within/for MARL. Further, I believe, the paper would benefit from disentangling the communication idea with the variational objective, which is more in line with hierarchical approaches (which is accounted for in the related work section).\n \n Other comments:\nThe AI-QMIX reference [1] changed significantly after the paper submission. An update of the reviewed paper with respect to this should be considered. Also, it is easy to get lost in the very involved notation. (Wrt. this: Are the subscripts to z consistent in Section 3.4; especially in the paragraph leading to Theorem 1?)\n \n[1] Iqbal, Shariq; Witt, Christian A. Schroeder de; Peng, Bei; Böhmer, Wendelin; Whiteson, Shimon; Sha, Fei (2020): Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning. In: arXiv preprint arXiv:2006.04222.\n[2] Mordatch, Igor; Abbeel, Pieter (2018): Emergence of Grounded Compositional Language in Multi-Agent Populations. In: AAAI Conference on Artificial Intelligence.\n[3] Hernandez-Leal, Pablo; Kartal, Bilal; Taylor, Matthew E. (2019): A survey and critique of multiagent deep reinforcement learning. In: Autonomous Agents and Multi-Agent Systems 33 (6), S. 750–797\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach for coordinating teams with dynamic composition consisting of an attention mechanism, regularization and communication. The clarity of the paper is currently low seemingly due to the conflated message of the multiple parts of the framework. Improvements to the text via the suggested edits of all reviewers should be a relatively quick fix, but the clearer placement of this piece within the wider literature may require additional experiments to compare against so would be a larger change. 

The reviewers did continue to discuss the paper after the end of the open discussion period with the authors and appreciated the additional experiments performed. In the absence of supporting theory, empirical results in a second domain significantly improve the evidence that the method may be more generally applicable. However, the new experiments raised new questions (included in the reviewers later replies) indicating more experiments in the second domain are needed which would require further peer review.

I hope the authors will take the constructive feedback provided here as intended; to improve the paper, submit the work again at a later stage when the second experimental domain is sufficiently explored to support the proposed framework and in doing so maximise its potential for impact. 
Decision: Reject