Promoting Coordination through Policy Regularization in Multi-Agent Deep Reinforcement Learning
Authors: Paul Barde
Julien Roy
Félix G. Harvey
Derek Nowrouzezahrai
Christopher Pal
Venue:  (2020)
Keywords: Reinforcement Learning
Multi-Agent
Continuous Control
Regularization
Coordination
Inductive biases

Abstract: A central challenge in multi-agent reinforcement learning is the induction of coordination between agents of a team. In this work, we investigate how to promote inter-agent coordination using policy regularization and discuss two possible avenues respectively based on inter-agent modelling and synchronized sub-policy selection. We test each approach in four challenging continuous control tasks with sparse rewards and compare them against three baselines including MADDPG, a state-of-the-art multi-agent reinforcement learning algorithm. To ensure a fair comparison, we rely on a thorough hyper-parameter selection and training methodology that allows a fixed hyper-parameter search budget for each algorithm and environment. We consequently assess both the hyper-parameter sensitivity, sample-efficiency and asymptotic performance of each learning method. Our experiments show that the proposed methods lead to significant improvements on cooperative problems. We further analyse the effects of the proposed regularizations on the behaviors learned by the agents.

Under review as a conference paper at ICLR 2020
PROMOTING COORDINATION THROUGH POLICY REGULARIZATION IN MULTI-AGENT DEEP REINFORCEMENT LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
A central challenge in multi-agent reinforcement learning the induction coordination between agents of team. In this work, we investigate how to promote inter-agent coordination using policy regularization and discuss two possible avenues respectively based on inter-agent modelling and synchronized sub-policy selection. We test each approach in four challenging continuous control tasks with sparse rewards and compare them against three baselines including MADDPG, a state-of-the-art multi-agent reinforcement learning algorithm. To ensure a fair comparison, we rely on thorough hyper-parameter selection and training methodology that allows a fixed hyper-parameter search budget for each algorithm and environment. We consequently assess both the hyper- parameter sensitivity, sample-effciency and asymptotic performance of each learning method. Our experiments show that the proposed methods lead significant improvements on cooperative problems. We further analyse the effects the proposed regularizations on the behaviors learned by the agents.
INTRODUCTION
Multi-Agent Reinforcement Learning (MARL) refers to the task of training an agent to maximize its expected return by interacting with an environment that contains other learning agents. represents challenging branch of Reinforcement Learning (RL) with interesting developments in recent years (Hernandez- et al. 2018). A popular framework for MARL is the use a Centralized Training and Decentralized Execution (CTDE) procedure (Lowe et al.) 2017]] Foerster al., 2018 Iqbal & Sha 2019: Foerster et al. 2019 Rashid et al. 2018). It is typically implemented by training critics that approximate the value of the joint observations and actions, which are used to train actors restricted to the observation of a single agent. Such critics, if exposed to coordinated joint actions leading high returns, can steer the agents' policies toward these highly rewarding behaviors. However, these approaches depend on the agents luckily stumbling on these actions in order to grasp their benefit. Thus, might fail in scenarios where coordination is unlikely to occur by chance. We hypothesize that in such scenarios, policy search could help discover coordinated behaviors more efficiently and supersede -peciiic reward shaping and curriculum learning.
conddiaationprommootn inductive biases on the
In this work, we explore two different priors for successful coordination and use these to regularize the learned policies. The first avenue. TeamReg, assumes that agent must be able predict the behavior of teammates in order to coordinate with them. The second, CoachReg, supposes that coordinating agents individually recognize different situations and synchronously use different sub- policies to react to them. In the following sections we show how to derive practical regularization terms from these premises and meticulously evaluate them[]
Our contributions are twofold. First, we propose two novel approaches that aim at promoting coordination in multi-agent systems. Our methods augment CTDE MARL algorithms with additional multi-agent objectives that act as regularizers and are optimized jointly with the main ittur-maximizztiion objective. Second, we design two new sparse-reward cooperative tasks in the
Source code for the algorithms and environments will be made public upon publication of this work.

---

Under review as a conference paper at ICLR 2020
multi-agent particle environment (Mordatch & Abbeel 2018). We use them along with two standard multi-agent tasks to present a detailed evaluation of our approaches against three different baselines. Finally, we validate our methods' key components by performing ablation study. Our experiments suggest that our TeamReg objective provides a dense learning signal that helps to guide the policy towards coordination in the absence of external reward, eventually leading it the discovery of high performing team strategies in a number of cooperative tasks, Similarly, by enforcing synchronous sub-policy selections, CoachReg enables to fine-tune a sub-behavior for each recognized situation yielding significant improvements on the overall performance.
2 BACKGROUND
2.1 MARKOV GAMES
In this work we consider the framework of Markov Games (Littman 1994), a multi-agent extension of Markov Decision Processes (MDPs) with N independent agents. A Markov Game is defined by the tuple (S, T, P. 1O', AA', R²),=)) S, T. and P respectively are the set of all possible states, the transition function and the initial state distribution. While these are global properties of the environment, O A' and Rs individually defined for each agent They are respectively the observation functions, the sets of all possible actions and the reward functions. At each time-step t, the global state of the environment given by 9t E S and every agent's 's individual action vector is denoted by a; E A'. To select their action, each agent i has only access to its own observation vector of which is extracted by its observation function Oi from the global state st. The initial global state is sampled from the initial state distribution P : S - [0, 1] and the next states of the environment 8t+1 are sampled from the probability distribution over the possible next states given by the transition function T S x S x A x x A - [0. 1]. Finally, at each time-step, each agent receives an individual scalar reward r from its reward function R S x S x A' x ... X A + R.
Agents aim at maximizing their expected discounted return E ΣT yrr over the time horizon T, where 7 E [0, 1] is a discount factor.
2.2 MULTI-AGENT DEEP DETERMINISTIC POLICY GRADIENT (MADDPG)
MADDPG eLwe al. 2017) is an adaptation of the Deep Deterministic Policy Gradient algorithm (DDPG) Lillicrap et al. 2015) to the multi-agent setting. It allows the training of cooperating and competing decentralized policies through the use of a centralized training procedure. In this framework, each agent possesses its own deterministic policy Je² for action selection and critic Q' for state-action value estimation, which are respectively parametrized by H and ø'. All parametric models are trained off-policy from previous transitions G. == (or; a, Fr. 0c+1) uniformly sampled from replay buffer D. Note that o, == [o]..... is the joint observation vector and a, = I , a;" is the joint action vector, obtained by concatenating the individual observation vectors of and action vectors of of all N agents. Each centralized critic is trained to estimate the
expected return for a particular agent using the Deep Q-Network (DQN) (Mnih et al., 2015) loss:
L C² = TR 2 (Q' an (ri + (0+1,, ai 1 (2 (
(I)
For a given set of weights w, we define its target counterpart w. updated from T + Tw + (1 - T ) where 7 is hyper-parameter Each policy is updated to maximize the expected discounted return of the corresponding agent i:
(0a = Eo.ND
a1) ai af a 00
(2)
Vo.Jp(("") = EOND
Vof (o, an) at )))
vidi
By taking into account all agents' observation- pairs when guiding an agent's policy, the value- functions are trained in a centralized, stationary environment, despite taking place in a multi-agent setting. In addition, this mechanism can allow to implicitly learn coordinated strategies that can then be deployed in decentralized way. However, this procedure does not encourage the discovery of coordinated strategies since high-reward behaviors have to be randomly experienced through unguided exploration. This work aims alleviating this limitation.
2

---

Under review as a conference paper at ICLR 2020
3 RELATED WORK
Many works in MARI, consider explicit e communication channels between the agents and distinguish between communicative actions (c.g. broadcasting given message) and physical actions (e.g. moving in a given direction) (Foerster et al. 2016 Mordatch Abbeel 2018 Lazaridou et al, 2016). Consequently, they often focus on the emergence of language, considering tasks where the agents must discover a common communication protocol in order to succeed. Deriving a successful communication protocol can already be seen as coordination in the communicative action space and can enable, to some extent, successful coordination in the physical action space (Ahilan & Dayan 2019). Yet, explicit communication is necessary condition for coordination as agents can rely on physical communication (Mordatch & Abbeel 2018 Gupta et al. 2017).
Approaches to shape RL agents' behaviors with respect other agents have also been explored. Strouse et al. (2018) use the mutual information between the agent's 's policy and goal-independent policy to shape the agent's behavior towards hiding or spelling out its current goal. However, this approach is only applicable for tasks with an explicit goal representation and is not specifically intended for coordination. Jaques et al. (2019) approximate the direct causal effect between agent's actions and use it as an intrinsic reward to encourage social empowerment. This approximation relies on each agent learning a model of other agents' policies to predict its effect on them. In general, this type of behavior prediction can be referred to agent modelling (or opponent modelling) and has been used in previous work to enrich representations (Hernandez- classify the opponent's play style (Schadd et al. (007). In our work, agent modelling is extended to derive a novel incentive toward team- aaaa-pedicttble behaviors.
et al. (001)), to stabilise the
learning dynamics (He et al. 2016)
Finally, Barton al. (2018) propose convergent cross mapping (CCM) to measure the degree of effective coordination between two agents. Although this may represent an interesting avenue for behavior analysis, it fails to provide a tool for effectively enforcing coordination must be computed over long time series which makes it an impractical learning signal for single- step temporal difference methods. In this work, we design two coordination-driven multi-agent approaches that do not rely on the existence of explicit communication channels and allow to carry the learned coordinated behaviors at test time, when all agents act in a decentralized fashion.
CCM
4 COORDINATION AND POLICY REGULARIZATION
Intuitively, coordination can be defined as an agent's behav- ior being informed by the one of another agent, i.e. structure in the agents' interactions. Namely, a team where agents act independently of one another would not be coordinated. To promote such structure, our proposed methods rely on ream-objectives as regularizers of the common policy gra- dient update. In this regard, our approach is closely related to General Value Functions and Auxiliary tasks (Sutton & Barto 2018) used in Deep RI. to learn efficient representa- tions (Jaderberg et al.) 2019). However, this work's 's novelty lies in the explicit bias of agents' policy towards either pre- dictability for their teammates or synchronous sub-policy selection. Pseudocodes of our implementations are provided in Appendix[[ (see Algorithms annd 2b
Illustration of TeamReg with two agents. Each agent's policy is equipped with additional heads that are trained to predict other agents' actions and every agent is regular- ized to produce actions that its team- mates correctly predict. Note that the method is depicted for agent I only to avoid cluttering.
Figure 1:
4.1 TEAM REGULARIZATION
The structure of coordinated interactions can be leveraged to attain a certain degree of predictability of one agent's behavior with respect to its teammate(s). We hypothesize that the reciprocal also holds i.e. that promoting agents' predictability could foster such team structure and lead to more coordinated behaviors. This assumption is cast into the decentralized framework by training agents to predict their teammates' actions given only their own observation. For continuous control, the loss is defined the Mean Squared Error (MSE) between the predicted
3

---

Under review as a conference paper at ICLR 2020
and true actions of the teammates, yielding Grammate-modelllee secondary objective. While the previous work of Hernandez-Leaa et al. (2019) focus on stationary, non-learning teammates and exclusively use this approach to learn richer internal representations,
propose to extend this objective to drive the teammates' behaviors closer to the prediction by leveraging a differentiable action selection mechanism. We call team-spirit this novel objective JT' between agents and j:
(0 (00)0')))= Bo,nD MSSE(a,) ,at) of=
(3)
LA'l a'p
-Borp
2
(a
-
(4)
k=
where Y is the policy head of agent i trying to predict the action of agent j. The total gradient for a given agent i becomes:
VorJistal
Vo.Jbo A Voest 04) A2 Voi 0*)
(5)
where d and X2 are tlyer-parametee that respectively weight how well an agent should predict its teammates' actions, and how predictable an agent should be for its teammates. We call TeamReg this dual regularization from team-spirit objectives. Figure summarizes these interactions.
4.2 COACHI REGULARIZATION
In order to foster structured agents interactions, this method aims teaching the agents recognize different situations and synchronously select corresponding sub-behaviors.
4.2.1 SUB-POLICY SELECTION
Firstly, to enable explicit sub-behavior selection, we pro- pose policy masks that modulate the agents' policy. A pol- icy mask u² a one-hot vector of size K with its jdh compo- nent set one. In practice, use policy masks to perform dropout (Srivastava et al., 2014) in a structured manner on Jn E RR the pre-activations of the first hidden layer In of the policy network и. To do so, we construct the vector ui which is the concatenation of C copies of u², in order to reach the dimensionality M = * K. The element-wise product u² h is then performed and only the units of h at indices m modulo K j j are kept for m = 0, ..... ....M... M - 1. In our contribution, each agent é generates eń, its own policy mask, from its observation of. Here, a simple linear layer I' is used to produce a categorical probability distribution p' of) from which the one-hot vector is sampled:
Illustration of CoachReg with two agents. central model, the coach, takes all agents' observations as input and outputs the current mode (policy mask). Agents are regular- ized to predict the same mask from their local observations only and op- timize the corresponding sub-policy.
Figure 2:
exp (o(o);)'),)
p'(ei = u' oo softmax(/ 0'));
(6)
K-1 =k=o exp (1) 6))))
To our knowledge, while this method draws similarity the options and hierarchical frameworks (Sutton & Barto 2018) Ahilan & Dayan 2019) and to policy dropout for exploration (Xie et al., 2018), it the first to introduce an agent induced modulation of the policy network by a structured dropout that is decentralized at evaluation and without an explicit communication channel. Although the policy masking mechanism enables the agent to swiftly switch between sub-policies it does not encourage the agents to synchronously modulate their behavior.
4.2.2 SYNCHRONOUS SUB-POLICH SELECTION
To promote synchronization we introduce the coach entity, parametrized by d, which learns to produce policy-masks ef from the joint observations, i.e. of (eg w). The coach is used training
4

---

Under review as a conference paper at ICLR 2020
time only and drives the agents toward synchronously selecting the same behavior mask. In other words, the coach is trained to output masks that (1) yield high returns when used by the agents and (2) are predictable by the agents. Similarly, each agent is regularized matches the coach's mask and (2) it derives efficient behavior when using the coach's mask. At evaluation time, the coach is removed and the agents only rely on their own policy masks. The policy gradient loss when agent is provided with the coach's 's mask is given by:
that (I) its private mask
(0', x)
Q' ae) )o) lof= =f (x
The difference between the mask of agent and the coach's 's one is measured from the Kullback-Leibler divergence:
JO"( v) Eo,DD [DLL (p" (-/o 1) (-lo), 6))]
(8)
The total gradient for agent i is:
ye Fionaaa0)) - V ('a + AI Vz v) V2
(9)
Vo JEPC(,0, 1) Eo, arnD an- Voru(d,,e; Q' at) ai==
with a and ' the regularization coefficients. Similarly, the coach is trained with the following dual objective, weighted by the A3 coefficient:
N
V Vo (w)
Σ (V JE. v)) + VE
(10)
N
In order to propagate gradients through the sampled policy mask we reparametrized the categorical distribution using the Gumbel-softme trick (Jang et al. (017)) with a temperature of 1. We call this coordinated sub-policy selection regularization CoachReg and illustrate it in Figure 2
5 TRAINING ENVIRONMENTS
All of our tasks are based on the OpenAl multi-agent particle environments (Mordatch & Abbeel, 2018). SPREAD and CHASE were introduced by (Lowe et al. (017). We use SPREAD as is but with sparse rewards only. CHASE is modified with a prey controlled by repulsion forces and only the predators are learnable, we wish to focus on coordination in cooperative tasks. Finally we introduce COMPROMISE and BOUNCE where agents are explicitly tied together. While non- zero return can be achieved in these tasks by selfish agents, they all benefit from coordinated strategies and optimal return can only be achieved by agents working closely together. Figure presents visualizations and brief description of all four tasks. A detailed description is provided in Appendix A In all tasks, agents receive as observation their own global position and velocity as well as the relative position of other entities. Note that work showcasing experiments on this environment often use discrete action spaces and (dense) reward shaping (e.g. the proximity with the objective) IIqall Shal 2019; Lowe et al. 2017 Jiang & Lu 2018). However, in our experiments, agents learn with continuous action spaces and from sparse rewards.
6 RESULTS AND DISCUSSION
The proposed methods offer a way to incorporate new inductive biases in CTDE multi-agent policy search algorithms. In this work, we evaluate them by extending MADDPG, state of the art algorithm widely used in the MARL litterature. We compare against vanilla MADDPG as well as two of its variations in the four cooperative multi-agent tasks described in Section [S] The first variation (DDPG) is the single-agent counterpart of MADDPG (decentralized training). The second (MADDPG + sharing) shares the policy and fulue-function models across agents.
To offer fair comparison between all methods, the hyper-parameter search routine is the same for each algorithm and environment (see Appendix D. For each cearch-epperiment (one per algorithm per environment), 50 randomly sampled typer-parameter configurations each using 3 training seeds (total of 150 runs) are used to train the models for 15, 000 episodes. For each algorithm-enve -evironmmenn pair, then select the best hyper-pas parrmmar configuration for the final comparison and retrain them on 10 seeds for twice as long. We give more details about the training setup and model selection in Appendix B and D.2 The results of the hyperparameter searches are given in Appendix D.5
5

---

Under review as a conference paper at ICLR 2020
Figure 3: Multi-agent tasks used this work. (a) SPREAD: Agents must spread out and cover set of landmarks. (b) BOUNCE: Two agents linked together by spring and must position themselves so that the falling black ball bounces towards a target. (c) COMPROMISE: Two linked agents must compete cooperate to reach their assigned landmark. (d) CHASE: Two agents chase a (non-learning) prey (turquoise) that moves w.r.t repulsion forces from predators and walls.
6.1 ASYMPTOTIC PERFORMANCE
From the average learning curves reported in Figure 4 we observe that CoachReg significantly improves performance on three environments (SPREAD, BOUNCE and COMPROMISE) and performs par with the baselines except on COMPROMISE, the only task with an adversarial component, where it significantly underperforms compared to the other algorithms. We discuss this specific case in Section 6.3 Finally, parameter sharing is the best performing choice on CHASE, yet this superiority restricted symmetrically and squeeze the prey into corner.
the last (CHASE). The same be said for TeamReg,
this task where the optimal play
SPREAD
BOUNCE
7.5
Return
5.0
0.0
5000
10000
15000
20000
25000
30000
5000
10000
15000
20000
25000
30000
COMPROMISE
CHASE
1200
30
Return
5000
10000
20000
25000
10000
15000
20000
25000
30000
Episodes
Episodes
DDPG
MADDPG
MADDPG
sharing
MADDPG TeamReg
MADDPG CoachReg
Figure 4: Learning curves (mean return agents) for all algorithms all four environments. Solid lines are the mean and envelopes are the Standard Error (SE) across the 10 training seeds.
6.2 ABLATION STUDY
Additionally to our two proposed algorithms and the three baselines, we present results for two ablated versions of our methods. The first ablation (MADDPG + agent modelling) is similar to TeamReg but with X2 = 0, which results only enforcing agent modelling (i.e. agent predictability is not encouraged). The second ablation (MADDPG policy mask) is structurally equivalent to CoachReg, but with A1,2,3 = O, which means that agents still predict and apply a mask to their own policy, but synchronicity is not encouraged. Figure 12 and [3] (Appendix D.6) present the results of the corresponding hyper-parametts search and Figure S shows the learning curves for our full regularization approaches, their respective ablated versions and MADDPG.

---

Under review as a conference paper at ICLR 2020
SPREAD
BOUNCE
7.5
160
5.0
2.5
0.0
0
5000
10000
15000
20000
25000
20000
5000
10000
15000
20000
25000
20000
COMPROMISE
CHASE
900
Return
20
600
1.0
5,00
10000
15000
200000
25000
30000
5000
10000
15000
20000
25000
30000
Episodes
Episodes
MADDPG
MADDPG agent modelling
MADD policy mask
MADDPG TeamReg
MADDPG CoachReg
Figure 5: Learning curves (mean return over agents) for the ablated algorithms on all environments. Solid lines are the mean and envelopes are the Standard Error (SE) across the 10 training seeds.
The use of unsynchronized policy masks might result in swift and unpredictable behavioral changes and make it difficult for agents to perform together and coordinate. Experimentallyy "MADDPG + policy mask" performs similarly or worse than MADDPG on all but one environment, and never outperforms the full CoachReg approach. However, policy masks alone seem enough to succeed on SPREAD, which is about selecting a landmark from a set. Regarding "MADDPG + agent modelling", does not drastically improve on MADDPG apart from on the SPREAD environment, and the full TeamReg approach shows improvement over its ablated version except on the COMPROMISE task, which we discuss in Section 6.3
6.3 EFFECTS OF ENFORCING PREDICTABLE BEHAVIOR
First, investigate the reason for TeamReg's 's poor performance on COMPROMISE. Then, we analyse how TeamReg might be helpful in other environments.
COMPROMISE is the only task with a competitive component (and the only one in which agents do not share their rewards). The two agents being linked, a good policy has both agents reach their landmark successively (maybe by simply having both agents navigate towards the closest landmark). However, one agent never reaches for its landmark, the optimal strategy for the other one becomes to drag it around and always go for its own, leading to a strong imbalance in the return cumulated by both agents. While this scenario very rarely occurs for the other algorithms, we found TeamReg to often lead to such domination cases (see Figure 14| in Appendix E). Figure 6 depicts the agents' performance difference for every 150 runs of the hyperparameter search for TeamReg and the baselines, and shows that (1) TeamReg is the only algorithm that does lead to large imbalances in performance between the two agents and (2) that these cases where one agent becomes dominant are all associated with high values of Ag. which drives the agents to behave in predictable fashion to one another. However, the dominated agent eventually gets exposed more and more to sparse reward gathered by being dragged (by chance) onto its own landmark, picks up the goal of the task and starts pulling in its own direction, which causes the average return over agents to drop as we see in Figure 4 This experiment demonstrates that using predictability-base bassd team-regularization
25
20
15
10 10-2
10:
10c
Figure 6: Average performance difference (A perr) between the two agents in COM- PROMISE for each 150 runs of the hyper- parameter searches (left). All occurrences of abnormally high performance difference are associated with high values of λ (right).
7

---

Under review as a conference paper at ICLR 2020
in competitive task be harmful; quite understandably, you might not want to optimize objective that aims at making your behavior predictable to your opponent.
On SPREAD and BOUNCE, TeamReg significantly improves the performance over the baselines. We aim to analyze here the effects of À on cooperative tasks and investigate if it does make the agent modelling task more successful (by encouraging the agent to be predictable). To this end, we compare the best performing nyper-parameter configuration for TeamReg on the SPREAD environment with its ablated versions. The average return and team-spirit loss defined in Section 4.1 are presented in Figure 7 for these three experiments.
Team Sprr
Rubur
80
0
2000
4000
6000
2006
10000
1200
14000
Eptandio
12990
À. OKr
1 OPF ON
1 DH
ON
Ag ON
Figure Comparison between enabling and disabling the regularizing weights A and X2 MADDPG+TeamRep on the SPREAD environment. Values are averaged over the 3 agents and over the seeds used in the hyper-parameter exploration.
Initially, due to the weight initialization, the predicted and actual actions both have relatively small norms yielding small values of team-spirit loss. As training goes on (~~1000 episodes), the norms of the action-vector increase and the regularization loss becomes more important. As expected, S.OFFIA,C OFF leads to the highest team-spirit loss as it not trained to predict the actions of other agents correctly. When using only the agent-modelling objective (A₁ ON),,, the agents significantly decrease the team-spirit loss, but it never reaches values as low as when using the full TeamReg objective. Finally, when also pushing agents to be predictable (AzON), the agents best predict each others' actions and performance is also improved. We also notice that the team-spirit loss increases when performance starts to improve i.e. when agents start to master the task (~8000 episodes). Indeed, once the reward maximisation signals becomes stronger, the relative importance of the second task reduced. We hypothesize that being predictable with respect to one-another may push agents to explore in more structured and informed manner in the absence of reward signal, as similarly pursued by intrinsic motivation approaches (Chentanez et al. 2005).
6.4 ANALYSIS OF SYNCHRONOUS SUB-POLICY SELECTION
(a) The ball is the left side of the target, agents both select the purple policy mask
(b) The ball on the right side the target, agents both select the green policy mask
1-0, C
5.
1-
1= 16,
1-50
50,
50,
1 66,
Figure 8: Visualization of different BOUNCE evaluation episodes. Note that here, the agents' colors represent their chosen policy mask. Agents have learned to synchronously identify two distinct situations and accordingly. The coach's masks (not used evaluation time) are displayed with the timestep at the bottom of each frame.
8

---

Under review as a conference paper at ICLR 2020
In this section we aim at experimentally verifying that CoachReg yields the desired behavior: agents synchronously alternating be- tween varied sub sub-policies. A special attention is given when the sub-policies are interpretable. To this end we record and analyze the agents' policy masks 100 different episodes for each task.
(a)
1.0
0.8
M
0.6
From the collected masks, we reconstructed the empirical mask distribution of each agent (see Figure [5] in Appendix [I.1) whose entropy provides an indication of the mask diversity used by given agent. Figure 9 (a) shows the mean entropy for each environment compared to the entropy of Categorical Uniform Distributions of size (k-CUD). shows that, on all the environments, agents use at least two distinct masks by having non-zero entropy. In addition, agents tend to alternate between masks with more variety (close to uniformly switching between 3 masks) on SPREAD (where there are agents and 3 goals) than on the other environments (comprised of 2 agents). To test if agents selecting the same policy mask at test time (without a coach), compute the Hamming proximity between the agents' mask sequences with - Dn where D2 is the Hamming distance, i.e. the number of timesteps where the two sequences are different divided by the total number of timesteps. From Figure | (b) observe that agents are producing similar mask sequences. Notably, their mask sequences are significantly more similar that the ones of two agent randomly choosing between two masks at each timestep. Finally, we observe that some settings result in the agents coming up with interesting strategies, like the one depicted in Figure g where the agents alternate between two sub- policies depending on the position of the target. More cases where the agents change sub-policies during episode are presented in Appendix F.T These results indicate that, addition to improving the performance coordination tasks, CoachReg indeed yields the expected behaviors. An interesting following work would be to use entropy regularization to increase the mask usage variety and mutual information to further disentangle sub-policies.
SPESS
sestect
Oliog
(b)
medk
nudde mer,
06
04
synchronously
SPREAD
Reuuce
OHASE
Figure 9: (a) Entropy of the policy mask distributions for each task, averaged over agents and training seeds. Hmaz,, is the entropy of a k- CUD. (b) Hamming Proxim- ity between the policy mask sequence of each agent aver- aged across agent pairs and seeds. rand, stands for agents independently sampling their masks from k-CUD. Error bars are SE across seeds.
6.5 ROBUSTNESS TO HYPER-PARAMETERS PARAMETERS
Stability across hyper-parameter configurations is recurring challenge in Deep RL. The average performance for each sampled configuration allow to empirically evaluate the robustness of an algorithm w.r.t. its hyper-parameters in Figures 10 12, and 13 in Appendix D.5 and D.6 Figure [I shows that while most algorithms can perform reasonably well with the correct configuration, our proposed coordination regularizers can improve robustness to hyper- parammeer despite the fact that they have more syper-parameters to search over. Such robustness can be of great value with limited computational budgets.
We share the full results of the hyper-parameter searches
6.6 ROBUSTNESS TO THE NUMBER OF AGENTS
To assess how the proposed methods perform when using an greater number of agents, we present additional experiments for which the number of agents in the SPREAD task is gradually increased from three to six agents. The results presented in Figure 18 (Appendix G show that the performance benefits provided by our methods hold when the number of agents is increased. Strikingly, we also note how quickly the performance of all methods drop when the number of agents rises. Indeed, with each new agent, the coordination problem becomes more and more difficult, and that might explain why our methods that promote coordination maintain a higher degree of performance in the case of 4 agents. Nonetheless, estimating the value function also becomes increasingly challenging as the input space grows exponentially with the number of agents. In the sparse reward setting, the complexity of the task soon becomes too difficult and none of the algorithms is able to solve it with six agents.

---

Under review as a conference paper at ICLR 2020
7 CONCLUSION
In this work we introduced two policy regularization methods to promote multi-agent coordination within the CTDE framework: TeamReg, which is based on inter-agent action predictability and CoachReg that relies on synchronized behavior selection. A thorough empirical evaluation of these methods showed that they significantly improve asymptotic performances on cooperative multi- agent tasks. Interesting avenues for future work would be to study the proposed regularizations on other policy search methods as well as to combine both incentives and investigate how the two coordinating objectives interact. Finally, limitation of the current formulation is that it relies single-step metrics, which simplifies off oollcc learning but also limits the longer-term coordination opportunities. A promising direction is thus to explore model-based planning approaches to promote long-term multi-agent interactions.
REFERENCES
Sanjeevan Ahilan and Peter Dayan. Feudal multi-agent hierarchies for cooperative reinforcement learning. arXiv preprint (9XIvv:900.0849) 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 2016.
Sean L Barton, Nicholas R Waytowich, Erin Zaroukian, and Derrik E Asher. Measuring collabo- rative emergent behavior in multi-agent reinforcement learning. In International Conference on Human Systems Engineering and Design: Future Trends and Applications, pp. 422-427. Springer, 2018.
Nuttapong Chentanez, Andrew G Barto, and Satinder P Singh. Intrinsically motivated reinforcement learning. Advances in neural information processing systems, pp. 1281-1288, 2005.
Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2137-2145, 2016.
Jakob Foerster, Francis Song, Edward Hughes, Neil Burch, lain Dunning, Shimon Whiteson, Matthew Botvinick, and Michael Bowling. Bayesian action decoder for deep multi-agent reinforcement learning. International Conference on Machine Leurning, 2019.
Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras. Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Jayesh K. Gupta, Maxim Egorov, and Mykel J. Kochenderfer. Cooperative multi-agent control using
deep reinforcement learning. In AAMAS Workshops, 2017.
He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé III. Opponent modeling in deep
reinforcement learning. In International Conference on Machine Learning, pp. 1804-1813, 2016.
Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor. Is multiagent deep reinforcement learning the answer or the question? a brief survey. arXiv preprint arXiv: 1810.0558. 2018.
Pablo Hernandez-Leal, Bilal Kartal, and Matthew E. Taylor. Agent Modeling Auxiliary Task for Deep Reinforcement Learning. In AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, 2019.
Shariq Iqbal and Fei Sha. Actor-attention-crite for multi-agent reinforcement learning. In International Conference Machine Learning, pp. 2961-29700 2019.
Max Jaderberg, Wojeiech M Czamecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, al. Human-level performance in 3d multiplayer games with population based reinforcement learning. Science, 364((444)) 865, 2019.
10

---

Under review as a conference paper at ICLR 2020
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparametrizztion with gumble-softmax. In International Conference on Learning Representations (ICLR 2017). OpenReview. net, 2017.
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, Dj Strouse, Joel Z Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International Conference on Machine Learning, pp. 3040-3049, 2019.
Jiechuan Jiang and Zongging Lu. Learning attentional communication for multi-agent cooperation. In Advances Neural Information Processing Systems, pp.
2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:14/2.6990, 2014.
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. arXiv preprint arXiv:1612.07182. 2016.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa,
David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.0m 1509.02971, 2015.
Michael L Littman. Markov games as framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pp- 157-163. Elsevier, 1994.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettiiee environments. Advances in Neural Information Processing Systems, pp. 6379-6390, 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland. Georg Ostrovski, et al. Human- level control through deep reinforcement learning. Nature, 518(75400:529, 2015.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-I0), pp. 807-814, 2010.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pp. 4292-4301, 2018.
Frederik Schadd, Sander Bakkes, and Pieter Spronck. Opponent modeling in real-time strategy games. In GAMEON, pp. 61-70, 2007.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way prevent neural networks from overfitting. The journal of machine learning research, 1911: 1958, 2014.
Daniel Strouse, Max Cleiman-Weines Josh Tenenbaum, Matt Botvinick, and David Schwab. Learning to share and hide intentions using information regularization. In Advances in Neural Information Processing Systems, pp. 10270-10281, 2018.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
George E Uhlenbeck and Leonard Ornstein. On the theory of the brownian motion. Physical review, 36(5):823, 1930.
Sirui Xie, Junning Huang, Lanxin Lei, Chunxiao Liu, Zheng Ma, Wei Zhang, and Liang Lin. Nadpex: An on-policy temporally consistent exploration method for deep reinforcement learning. arXiv preprint arXiv: wtX.x::1112.09022.. 2018.
11

---

Under review as a conference paper at ICLR 2020
A
TASKS DESCRIPTIONS
SPREAD (Figure 3a): In this environment, there are 3 agents (small orange circles) and landmarks (bigger gray circles). At every timestep, agents receive team-reward rr n c where is the number of landmarks occupied by at least one agent and e the number of collisions occurring at that timestep. To maximize their return, agents must therefore spread out and cover all landmarks. Initial agents' and landmarks' positions are random. Termination is triggered when the maximum number of timesteps is reached.
BOUNCE (Figure Bp): In this environment, two agents (small orange circles) are linked together with a spring that pulls them toward each other when stretched above its relaxation length. At episode's mid-time ball (smaller black circle) falls from the top of the environment. Agents must position correctly so as to have the ball bounce on the spring towards the target (bigger beige circle), which turns yellow if the ball's bouncing trajectory passes through it. They receive team-reward of r, = 0.1 if the ball reflects towards the side walls, r = 0.2 if the ball reflects towards the top of the environment, and ball's vertical position is fixed, their horizontal positions are random. Agents' initial positions are also random. Termination triggered when the ball is bounced by the agents or when the maximum number of timesteps is reached.
10 if the ball reflects towards the target. At initialisation, the target's 's and
COMPROMISE (Figure 3:: In this environment, two agents (small orange circles) are linked together with a spring that pulls them toward each other when stretched above its relaxation length. They both have a distinct assigned landmark (light gray circle for light orange agent, dark gray circle for dark orange agent), and receive a reward of r, 10 when they reach it. Once a landmark reached by its corresponding agent, the landmark is randomly relocated in the environment. Initial positions of agents and landmark are random. Termination is triggered when the maximum number of timesteps is reached.
CHASE (Figure Bt)): In this environment, two predators (orange circles) are chasing prey (turquoise circle). The prey moves with respect to a scripted policy consisting of repulsion forces from the walls and predators. At each timestep, the learning agents (predators) receive a team- reward of re = n where n is the number of predators touching the prey. The prey has a greater max speed and acceleration than the predators. Therefore, to maximize their return, the two agents must coordinate in order to squeeze the prey into a corner or a wall and effectively trap it there. Termination is triggered when the maximum number of time steps is reached.
B TRAINING DETAILS
In all of our experiments, updates. All models (actors, critics and coach) are parametrized by feedforward networks containing two hidden layers of 128 units. We use the Rectified Linear Unit (ReLU) (Nair & Hinton] 2010) as
the Adam optimizer (Kingma & Ba 2014) to perform parameter
activation function and layer normalization (Ba et al.) 2016) on the pre-activations unit to stabilize the learning. We use a buffer-size of 10th entries and batch-size of 1024. We collect 100 transitions by interacting with the environment for each learning update. For all tasks in our hyper- hyper-paranye parameter searches, we train the agents for 15, 000 episodes of 100 steps and then re-train the best configuration for each for the final evaluation. The scale of the exploration noise is kept constant for the first half of the training time and then decreases linearly to 0 until the end of training. We use discount factor y of 0.95 and a gradient clipping threshold of 0.5 in all experiments. Finally for CoachReg, we fixed K to 4 meaning that agents could choose between 4 sub-policies. Since policies' hidden layers are of size 128 the corresponding value for C is 32.
pair for twice as long (30, 000 episodes) to ensure full convergence
12

---

Under review as a conference paper at ICLR 2020
C ALGORITHMS
Algorithm Team
Randomly initialize critic networks Q' and actor networks Ju Initialize the target weights Initialize one replay buffer D for episode from 0 to number of episodes do Initialize random processes / for action exploration
Receive initial joint observation Og
for timestep from 0 to episode length do
Select action a; y²((); + NV for each agent
Execute joint action a, and observe joint reward r, and new observation 0,+1
Store transition (0,, a,, r, 0+41) D end for
Sample a random minibatch of M transitions from D
for each agent i do
Evaluate L' and JPc from Equations 1 and (
for each other agent (j i) do
Evaluate J from Equations 33
Update actor j with 0³ - 00 + as V A2 end for
Update critic with Ø² + 6² - as VoL
Update actor i with 6 + Hr + Vo. (JPG + x Eyn1 end for Update all target weights end for
Algorithm 2 Coach
Randomly initialize critic networks Q', actor networks y² and one coach network p Initialize N target networks Q" and n²
Initialize one replay buffer D
for episode from 0 to number of episodes do
Initialize random processes, ' for action exploration
Receive initial joint observation Og
for timestep from 0 to episode length do
Select action a; 1² (ot) Ni for each agent
Execute joint action a, and observe joint reward r; and new observation 0,+1
Store transition (o,, a,, Ft. 0c41) D
end for
Sample a random minibatch of M transitions from D for each agent i do
Evaluate L and JéG from Equations I1 and (2)
6² as V Update actor with 0² 6² as V JPG end for
Update critic with
for each agent do
Evaluate Jj. and JEpG from Equations (8 and
Update actor with 0² +- 0i + ax Vo. (Ai E A2 end for
Update coach with db + + as V EN (JEPG + \aJ%))
Update all target weights end for
13

---

Under review as a conference paper at ICLR 2020
D HYPER-PARAMETER SEARCH
D.1 HYPER- PARAMETER SEARCH RANGES
We perform searches over the following hyper-parameters: the learning rate of the ag, the learning rate of the critic we relative to the actor (as = We * az)), the target-network soft-update parameter т and the initial scale of the exploration noise Thoise for the Ornstein Uhlenbeck noise generating process (Uhlenbeck & Ornstein] 1930) as used by Lillicrap et al. (2015). When using TeamReg and CoachReg, we additionally search over the regularization weights Å A2 and λ3- The learning rate of the coach is always equal the actor's learning rate (i.c. ag aw), motivated by their similar architectures and learning signals and in order to reduce the search space. Table shows the ranges from which values for the ppremeerrmeterss drawn uniformly during the searches.
Table I: Ranges for hyper-parameter search, the log base is 10
RANGE
log((gg
8,
log(ws)
2, 2
log(T)
-3, 1
log(A1)
-3. 0
log(A2)
3,
(sg(
-1
Tnoise
0.3, 1.8
D.2 MODEL SELECTION
During training, policy is evaluated on set of 10 different episodes every 100 learning steps. At the end of the training, the model at the best evaluation iteration is saved as the best version of the policy for this training, and is re-evaluated on 100 different episodes to have better assessment of its final performance. The performance of a waperr-paameed parameter configuration is defined as the average performance (across seeds) of the policies learned using this of hyper-paramettr values.
14

---

Under review as a conference paper at ICLR 2020
D.3
SELECTED HYPER-PARAMETER PARAMETERS
Tables 3 and shows the best happee-ass found by the random searches for each of the environments and each of the algorithms.
Table 2: Best found hyper- ppper-prrameters for the SPREAD environment
HYPER- PARAMETER
DDPG
MADDPG
MADDPG+SHARING
MADDPG+TEAMREE
MADDPG+COACHRES
Og 5.3 + 10
2.1=10
9.0 10
2.5 10
1.2
Ga 0.05
53
0.71
82
0.083
0.076
0.098
0.0077
A
0.054
0.13
Ag
0.29
0.24
Ag
8.4
Qnooer
0.7
1.2
Table 3: Best found hyper-parameters for the BOUNCE environment
HYPER-PARAMETER
DDPG 8.1 + 10-4 2.4
MADDPG
MADDPG+SHARING
MADDPG--TEAMRGG
MADDPGHCOACIREG
3.8 + 10-5 87
+ 1
1.3 10-5
6.8 10
Glg
D.47
85
9.4
0.059
0.016
0.06
0.055
0.02
A
0.06
0.0086
Aa
0.0026
0.23
Ag
0.34
Amoiss
1.2
0.9
1.2
1.0
1.1
Table Best found hyper-parameters for the CHASE environment
HYPER HYPER- PARAMETER
DDPG 10
MADDPG
MADDPG+SHARING
MADDPG-TEAMREG
MADDPGHCDACHREG
Dg
9.7 10-1
1.8
4 0.031
32
64
0.79
85
90
0.021
0.032
0.055
0.011
A
0.06
0.0069
Ag
0.0026
0.86
Ag
0.76
Thoise
0.6
1.0
1.5
1.0
1.1
Table 5: Best found hyper-parameters for the COMPROMISE environment
HYPER PARAMETER
DDPG
MADDPG
MADDPG+SHARING
MADDPG-TEAWREE
MADDPGHCUACHREs
6.1 x 10
3.1 10
6.2 10
1.5
3.4 10
0.063
0.045
0.007
0.02
0.0037
A
0.0013
0.65
Ag
0.56
Ag
1.3
Quadee
1.3
1.6
15

---

Wip Under review as a conference paper at ICLR 2020
D.4 SELECTED
(ABLATIONS)
Tables and 9 shows the best hyper- syper--arameters found by the random searches for each of the environments and each of the ablated algorithms.
Table Best found eyper-parameters for the SPREAD environment
HYPER-PAR PATEER-PARAMERE PARAMETER
MADDPG+AGEN MODELLING
MADDPG+POLICY MASK
1,3 10
6.8 + 10
9.4
0.055
0.02
X
0.06
0
Ag
0
0
Ag Thonae
Table 7: Best found hyper-parameters for the BOUNCE environment
HYPER-PARAMPTER
MADDPG+AGBNT MODELLING
MADDPG+POLICY
0g
1,3
2.5 10
85
0.52
0.055
0.0077
A
0.06
0
Az Ag Thoise
1.0
1.3
Table 8: Best found hyper-parameters for the CHASE environment
HYPER-PARAMETER MADDPG+AGEN MODELLING
MADDPG+ POLICY MASK
2.5 10
6.8 + 10
W6
42
9.4
0.098
0.02
0.054
As Thosse
0
1.2
1.1
Table 9: Best found hyper-parameters for the COMPROMISE environment
SEYPER-PARAMERS
MADDPG+AGENT MODELLING
MADDPG+POLICY MASK
Cgg
1.2
2.5 10
0.52
0.0051
0.0077
Ag
0.0075
0
Av
0
0
A
0
Thunae
1.3
16

---

Under review as a conference paper at ICLR 2020
D.5 HYPER-PARANEE PARAMETER SEARCH RESULTS
The performance of each parameter configuration is reported in Figure yielding the performance distribution across hyper-parameters configurations for each algorithm on each task. The same distributions are depicted in Figure [1 using box-and-whisker plot. It can be seen that TeamReg and CoachReg both boost the performance of the third quartile, suggesting an increase in the robustness across prer-paaaaeter.
SPREAD
BOUNCE
average retum
150
1.00
COMPROMISE
CHASE
25
averag 280022
20
12
200
DDPG MADDPG
MADDPG sharing
MADDIG Teamneg
MADDPG Coacchligg
Figure 10: Hyper-parameter tuning results for all algorithms. There is one distribution per (algorithm. environment) pair, each one formed of 50 points (hyper-parameter configuration samples). Each point represents the best model performance averaged over 100 evaluation episodes and averaged over the 3 training seeds for one sampled hyper-parameters configuration (total of 300 performance values per sampled configuration).
SPREAD
BOUNCE
200
average return
150
200
0DPG
MADDPG
MADORS
MADORS
MADOPS
DOPS
MADGPS
MADOPG CHASE
MADDS
MADOPS
COMPROMISE
20
average
15
1.0
200
*sharing
+TeamReg
CoachReg
+sharlgg
+TeamReg
CCachRey
Figure 11: Summarized performance distributions of the sampled parer-parameters configurations for each (algorithm, environment) pair. The box-plots divide in quartiles the 49 lower-perforning configurations for each distribution while the score of the best-performing configuration is high- lighted above the box-plots by a single dot.
17

---

Under review as a conference paper at ICLR 2020
D.6
SEARCH RESULTS (ABLATIONS)
From Figure 13 seems that the "policy mask" or the "agent modelling" additions respectively
provide nearly the same robustness boosts CoachReg and TeamReg.
SPREAD
BOUNCE
200
150
average
50
COMPROMISE
CHASE
1000
average retur
800
220
15
600
10
MADDPG
MADDPG agent modelling
MADDPG policy mask
MADDPG + +TeamReg
MADDPG CoachReg
Figure 12: Hyper-paranen tuning results for ablated algorithms compared to their full approach counterparts and MADDPG. There is one distribution per (algorithm, environment) pair, each one formed of 50 points (hyper-parameter configuration sample). Each point represents the best model performance averaged over 100 evaluation episodes and averaged over the training seeds for one sampled syper-parrmmetess configuration (total of 300 performance values per sampled configuration).
SPREAD
BOUNCE
average return
selly mat
-OossTeg
kigen sigenmeeeding
speity mah
Coych
COMPROMISE
CHASE
830
420
NADEFG
NAIEPG
apelity
Figure 13: Summarized performance distributions of the sampled ppperr-arameters configurations for each (ablated algorithm, environment) pair. The box-plot, divide in quartiles the 49 lower- performing configurations for each distribution while the score of the best-performing configuration highlighted above the box-plot single dot.
18

---

Under review as a conference paper at ICLR 2020
E
THE EFFECTS OF ENFORCING PREDICTABILITY (ADDITIONAL RESULTS)
DDPG
MADDPG
15
agent
10
24
33
Seco
10003
19300
20003
25006
30333
S200
1c00
15085
20002
25300
T0333
MADDPG sharing
MADDPG TeamReg
agent aggent
agent
agent
30003
19300
20003 25000
1000 20003
25200
Figure 14: Learning curves for TeamReg and the three baselines on COMPROMISE. We see that while both agents remain equally performant they improve at the task for the baseline algorithms, TeamReg tends to make one agent much stronger than the other one. This domination is optimal as long as the other agent remains docile, the dominant agent can gather much more reward than it had to compromise. However, when the dominated agent finally picks up the task, the dominant agent that has learned policy that does not compromise see its return dramatically go down and the mean over agents overall then remains lower than for the baselines.
19

---

Under review as a conference paper at ICLR 2020
F ANALYSIS OF SUB-POLICY SELECTION (ADDITIONAL RESULTS)
F.1 MASK DENSITIES
We depict Figure 15 the mask distribution of each agent for each (seed, environment) experiment. Firstly, in most of the experiments, agents use at least 2 different masks. Secondly, for given experiments, agents' distributions are very similar, suggesting that they are using the same masks in the same situations and that they are therefore synchronized. Finally, agents collapse more to using only one mask on CHASE, where they also display more dissimilarity between one another. This may explain why CHASE is the only task where CoachReg does improve performance. Indeed, on CHASE, agents do not seem synchronized nor leveraging multiple sub-policies which are the priors to coordination behind CoachReg. In brief, observe that CoachReg is less effective in enforeing those priors to coordination of CHASE, an environment where it does not boost nor harm performance.
SPREAD
ROUNCF
COMPROMISF
CHASF
13
AGENT
AGEKT
AGENT
AGENT L
AGONT
AGINT
AGENT
AGONT
AGENT
6A
seed 10
610
seed
25.0
1.3
seed
AVIMI
A0/911 1
AUUN11 2
00281
08
AUAMI
ADRIT
AUEVT
40EMI
AUSV12
ACUR1
ACANI
AB2V11
5eed 5m
Seed 6a4
AGENT 1
ASDNT>
AGENT3
AGENT
AGENT
AGENT
ASONT1
MGDT
seed
4GENT
AGENT
AGENT
AGONT1
seed 9
mask1
mask2
mask3
mask4
Figure 15: Agent's 's policy mask distributions. For each (seed, environment) we collected the masks of each agents 100 episodes.
20

---

Under review as a conference paper at ICLR 2020
F.2 EPISODES ROLL-OUTS
We render here some episodes roll-outs, the agents synchronously switch between policy masks during episode. In addition, the whole group selects the mask as the that would have been suggested by the coach.
1-0 C
1
1-20
T-BUCH B
1-28 C-
(a) SPREAD
L.C.C*
1+2
1-17C*
1+
1+32,0
1+32
1+M,C+
1-32,0
1-0.0+
1-ALCH
1-91CC
1TALO11
1- 1-PLC
(b) COMPROMISE
Figure 16: Visualization sequences on different environments. An agent's color represent its current policy mask. For informative purposes the policy mask that the coach would have produced these situations would have happened during training displayed next the frame's 's timestep. Agents synchronously switch between the available policy masks.
21

---

Under review as a conference paper at ICLR 2020
F.3 MASK DIVERSITY AND SYNCHRONICITY (ABLATION)
As in Subsection 6.4 we report the mean entropy of the mask distribution and the mean Hamming proximity for the ablated "MADDPG policy mask" and compare the full CoachReg. With "MADDPG policy mask" agents are not incentivized to use the same masks. Therefore, in order assess if they synchronously change policy masks, we computed, for each agent pair, seed and environment, the Hamming proximity for every possible masks equivalence (mask of agent corresponds to mask 0 of agent 2, etc.) and selected the equivalence that maximised the Hamming proximity between the two sequences.
We can observe that while "MADDPG + policy mask" agents display a more diverse mask usage, their selection is less synchronized than with CoachReg. This is easily understandable as the coach will tend to reduce diversity in order to have all the agents agree on common mask, on the other hand this agreement enables the agents synchronize their mask selection. To this regard, it should be noted that "MADDPG policy mask" agents are more synchronized that agents independently sampling their masks from k-CUD, suggesting that, even in the absence of the coach, agents tend to synchronize their mask selection.
1.0
Hmox,
0.8
Hmax, k-
Hmao, k
H/Hmax
0.6
MADDPG Coachhag
0.4
MADDPG policy mask
0.2
0.0
SPREAD
BOUNCE
COMPROMISE
CHASE
(a)
rand
0.8
rand
rand
MADDPG
CoachReg
MADDPG
0.4
policy mask
SPREAD
BOUNCE
COMPROMISE
CHASE
(b)
Figure 17: Entropy of the policy mask distributions for each task and method, averaged over agents and training seeds. Hmaz.k is the entropy of k-CUD. (b) Hamming Proximity between the policy mask sequence of each agent averaged across agent pairs and seeds. rand, stands for agents independently sampling their masks from k-CUD. Error bars SE across seeds.
22

---

Under review as a conference paper at ICLR 2020
G ROBUSTNESS TO THE NUMBER OF AGENTS
We varied the number of agents present in the SPREAD task from three to six. For each algorithm we used the best performing hyper-parameter configuration from the hyper-parameter search performed on SPREAD with three agents and trained on ten different random seeds. Results are shown in Figure As expected the task becomes more complicated when the number of agents increases and no algorithm succeeds at the task with six agents. This difficulty is likely to be exacerbated by the sparse reward setting. However, the proposed methods still outperform the baselines showing that they do not disproportionately suffer from the increased regularization pressure of additional agents.
SPREAD AGENTS
SPREAD AGENTS
150
150
Return
20000
25000
30000
5000
25000
30000
SPREAD AGENTS
SPREAD AGENTS
150
150
Return
Episades
DDPG
MADDPG
MADDPG
sharing
MADDPG
TeamReeg
MADDPG CoachReq
Figure 18: Learning curves (mean return over agents) for all algorithms on the SPREAD environment for varying number of agents. Solid lines are the mean and envelopes are the Standard Error (SE) across the 10 training seeds.
23


Reviews:
[{"review_id": "rygkUJhntS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes two methods of biasing agents towards learning coordinated behaviours and evaluates both rigorously across two existing and two novel multi-agent domains of suitable complexity. The rigour of the empirical evaluation and its documentation is exemplary. \n\nThe related work section could be improved by including a section on the closely related work in the area of opponent modelling. This literature is touched on shortly afterwards alongside the related proposed method in Section 4.1 but only by a single citation. A more thorough review in Section 3 would improve the positioning of the paper.\n\nFrom figures 4 and 5 it is interesting to observe that only parameter sharing performs significantly better than the existing baseline method MADDPG on the CHASE environment. In particular I think it would improve the discussion of the results to elaborate further on why the proposed methods do not improve, but also do not detrimentally affect the learning performance of agents in this environment. \n\nIn Section 6.2 the authors note \"MADDPG + policy mask performs similarly or worse than MADDPG  on all but one environment and never outperforms the full CoachReg approach.\" I found it also interesting to note that in that one environment (SPREAD) adding a policy mask both outperformed MADDPG and was more sample efficient than CoachReg. Again what is it about this specific environment that causes the policy mask ablation to be sufficient, enabling it to match the asymptotic performance and learn quicker than the full CoachReg method proposed?\n\nAll references to papers both published and available on arxiv should cite the published version (e.g. Jacques et al. 2018 was published at ICML). Please revise all references if accepted.\n\nMinor Comments:\n1) Page 2 \"two standards multi-agent tasks\" -> standard\n2) Page 7 \"improve on MADDPG apart on the SPREAD environment\" -> apart from on \n3) Page 9 \"each sampled configuration allows to empirically evaluate\" -> allows us\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1eVV2HJqS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed two approaches to encourage cooperation among multi-agents under the *centralized training decentralized execution* framework. The main contribution of this paper is that they propose to allow agents to predict the behavior of others and introduce this prediction loss into the RL learning objective. One approach is teammate-regularization where each agent predicts the behavior of all others (and therefore makes the total model complexity increasing quadratically with the number of agents), while the other is a centralized coach-regularization. The performance of these two approaches are compared with MADDPG and its two variants on 4 games. Experiments show that their approaches surpass a few baselines in certain game settings. However, the novelty of this algorithm is not strong, given that the similar idea of 'predicting the behavior of other agents' can be found in related work with discrete action spaces. Experimental results also show unstable advantages of their approaches.\n\nMethodology:\n\nTeam-reg: One main difference between this approach and Jaques work: * Intrinsic social motivation via causal influence in multi-agent*  (or other MARL algorithms for discrete action spaces) is that in this work, each agent predict the action of other peers instead of the policy, since this algorithm is only based on DDPG with continuous action space. However, It seems that this idea is transferrable to discrete action spaces as well, so long as we change the MSE loss between actions to KL loss between policy over states. The novelty of this approach is not strong. Also, one hypothesis on which their work is based is that promoting agents’ predictability could foster such team structure and lead to more coordinated behaviors. Does predictability really improve cooperation? Could it be the other way around? Experiment results in the right-below penal (CHASE Game ) and left-above penal (SPREAD game) of Figure 5 actually strength my concern. \n\nCoach-reg: Descriptions of the coach regularization approach is quite vague. I doubt the motivation and effectiveness of this approach. More questions are as follows:\n\n* What is the role of the policy mask in deriving a better action? Is it just a dropout before the activation layer with a fixed $p$ proportion (which is related to the choice of $K$)? If so, this is like you first assume the policy network is somewhat overfitting, then alleviate this issue by letting the coach adjust which part to keep and which part to drop.  How would different $K$ values affect the performance?\n\n* Does this framework really improve cooperation? Intuitively, all agents will finally reach a point where they agree on the same policy mask distribution (which is the one generated by the coach). How does the same policy mask lead to an improvement in cooperation? Empirical or theoretical explanations are strongly needed in the main results, not in the appendix section.\n\nExperiments:\n\n- All variants of MADDPG in the experiments are weak baselines, assuming that $MADDPG + sharing$ means all agents share the same policy and Q-function. However, since this algorithm only works for continuous action space, available relate work to compare with is quite limited. All environments only include two agents. Experiments with more than two agents should be implemented. \n- Even for only 3 games (excluding the adversarial game setting), both branches failed to beat the *sharing* baseline for the CHASE game. For the ablation study, *agent-moduling* even works better than *TeamReg* for the CHASE game, so as the case when *policy mask* beats *CoachReg* in the SPREAD game (Figure 5). They seem to show that under these two proposed frameworks, the predictability of agents does not always encourage cooperation, at least for 2 out of 4 game settings mentioned in this section.\n- This work makes a lot of effort on the hyper-parameter tuning part, which however does not provide a systematic solution to the tradeoff between improving predictability and maximizing reward. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJgFjR-CKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Contribution:\n\nThis paper proposes two methods building upon MADDPG to encourage collaboration amongst decentralized MARL agents:\n - TeamReg: the agents are trained with an additional objectives of predicting other agents' actions, and make their own action predictable to them\n - CoachReg: the agents use a form of structured dropout, and at training time a central agent (the \"coach\") predicts the mask dropout mask that should be applied to all of the agents. An additional loss is provided so that the agent learn to mimic the coach's output from their own input so they can still apply the dropout at test time, when the coach is not available anymore.\n\nThese two methods are evaluated in 4 different MARL environments, and compared against their ablations and vanilla MADDPG.\n\n\nReview:\n\nThe paper is well written and easy to follow. It generally motivates well the design choices, both intuitively and experimentally, using numerous ablations. It includes the analysis and explanations of the failure modes, which is valuable in my opinion.\n\nThe two main limitations of the work are the following:\n - Limited scale of the experiments. The majority of the experiments contain only 2 agents, and the last one merely contains 3. It is unclear whether the additional losses proposed in this work would still perform correctly with more agents, since the regularization pressure will increase.\n - No comparison to SOTA MARL methods. The only baseline method presented here is MADDPG, upon which this work is built. Recent work tend to show that it is easily outperformed, hence comparison to stronger baselines (QMIX, COMA, M3DDPG, ...) would be advisable to assess the quality of the policy found.\n\n\nAbout the policy masks:\n- what are the value chosen for K and C?\n- Is there a reason why this particular form of dropout was chosen? Since it occurs after a fully-connected, it should be equivalent and more straight-forward to mask out C contiguous values.\n\n\nFinally, it seems to me that the two methods presented here (TeamReg and CoachReg) are not mutually incompatible. Is there a reason why you didn't to apply both of them simultaneously?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: After reading the reviews and discussing this paper with the reviewers, I believe that this paper is not quite ready for publication at this time. While there was some enthusiasm from the reviewers about the paper, there were also major concerns raised about the comparisons and experimental evaluation, as well as some concerns about novelty. The major concerns about experimental evaluation center around the experiments being restricted to continuous action settings where there is a limited set of baselines (see R3). While I see the authors' point that the method is not restricted to this setting, showing more experiments with more baselines would be important: the demonstrated experiments do strike me as somewhat simplistic, and the standardized comparisons are limited.

This might not by itself be that large of an issue, if it wasn't for the other problem: the contribution strikes me as somewhat ad-hoc. While I can see the intuition behind why these two auxiliary objectives might work well, since there is only intuition, then the burden in terms of showing that this is a good idea falls entirely on the experiments. And this is where in my opinion the work comes up short: if we are going to judge the efficacy of the method entirely on the experimental evaluation without any theoretical motivation, then the experimental evaluation does not seem to me to be sufficient.

This issue could be addressed either with more extensive and complete experiments and comparisons, or a more convincing conceptual or theoretical argument explaining why we should expect these two particular auxiliary objectives to make a big difference.
Decision: Reject