Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning
Authors: Qian Long*
Zihan Zhou*
Abhinav Gupta
Fei Fang
Yi Wu†
Xiaolong Wang†
Venue:  (2020)
Keywords: multi-agent reinforcement learning
evolutionary learning
curriculum learning

Abstract: In multi-agent games, the complexity of the environment can grow exponentially as the number of agents increases, so it is particularly challenging to learn good policies when the agent population is large. In this paper, we introduce Evolutionary Population Curriculum (EPC), a curriculum learning paradigm that scales up Multi-Agent Reinforcement Learning (MARL) by progressively increasing the population of training agents in a stage-wise manner. Furthermore, EPC uses an evolutionary approach to fix an objective misalignment issue throughout the curriculum: agents successfully trained in an early stage with a small population are not necessarily the best candidates for adapting to later stages with scaled populations. Concretely, EPC maintains multiple sets of agents in each stage, performs mix-and-match and fine-tuning over these sets and promotes the sets of agents with the best adaptability to the next stage. We implement EPC on a popular MARL algorithm, MADDPG, and empirically show that our approach consistently outperforms baselines by a large margin as the number of agents grows exponentially. The source code and videos can be found at https://sites.google.com/view/epciclr2020.

Published as a conference paper at ICLR 2020
EVOLUTIONARY POPULATION CURRICULUM FOR SCALING MULTI-AGENT REINFORCEMENT LEARNING
Qian Long"
Zihan Zhou'
Abhibav Gupta CMU, Facebook Al Research abhi Bohhinav@cs.coo cmu edu
CMU
SJTU
qi gianlongecs. cmu. edu
footoredoaa jtu. edu. cn
Fei Fang CMU
Yi Wu
Xiaolong Wang
OpenAl
UCSD xiw0 11w0122 Mww012@ucco 2@uucd.
fei feiffcs.. cmu.edu
jxwuyi@openai. com
ABSTRACT
In multi-agent games, the complexity of the environment can grow exponentially as the number of agents increases, so particularly challenging to learn good policies when the agent population is large. In this paper, we introduce Evolutionary Population Curriculum (EPC), a curriculum learning paradigm that scales up Multi- Agent Reinforcement Learning (MARL) by progressively increasing the population of training agents stage-wiss wise manner. Furthermore, EPC uses evolutionary approach to fix an objective misalignment issue throughout the curriculum: agents successfully trained in an early stage with small population are not necessarily the best candidates for adapting to later stages with scaled populations. Concretely, EPC maintains multiple sets of agents in each stage, performs mix-and-match and fine-tuning over these sets and promotes the sets of agents with the best adaptability to the next stage. We implement EPC a popular MARL algorithm, MADDPG, and empirically show that our approach consistently outperforms baselines by large margin as the number of agents grows exponentially. The source code and videos can be found at httttps://ssii /tttps://sitess geogle. com/ wwwwwwwwwwcccccctc.o............
INTRODUCTION
Most real-world problems involve interactions between multiple agents and the problem becomes significantly harder when there exist complex cooperation and competition among agents. Inspired by the tremendous success of deep reinforcement learning (RL) in single-agent applications, such as
Atari games (Mnih et al. (2013), robotics manipulation (Levine et al. (016), and navigation (Zhu et al. 2017: Wu et al., 2018 Yang al. 2019), has become a popular trend to apply deep RL techniques into multi-agent applications, including communication (Foerster et al.) 2016, Sukhbaatar et al. 2016)
Mordatch Abbeel 2018), traffic light control (Wu al. 2017 physical combats (Bansal et al.
2018). and video games (Liu et al. 2019 OpenAl 2018).
fundamental challenge for multi-agent reinforcement learning (MARL) is that, as the number of agents increases, the problem becomes significantly more complex and the variance of policy gradients can grow exponentially (Lowe et al. 2017). Despite the advances on tackling this challenge via actor-critic methods (Lowe et al. 2017 Foerster et al. 2018), which utilize decentralized actors and centralized critics to stabilize training, recent works still scale poorly and are mostly restricted to less than dozen agents. However, many real-world applications involve moderately large population of agents, such as algorithmic trading (Wellman et al., 2005), sport team competition (Hausknecht & Stone,, (015). and humanitarian assistance and disaster response (Meier 2015). where one agent should collaborate and/or compete with all other agents. When directly applying the existing MARL algorithms to complex games with large number of agents, as we will show in Sec. S.3. the agents may fail to learn good strategies and end up with little interaction with other agents even when collaboration is significantly beneficial. Yang et al. (2018) proposed a provably-converye field formulation to scale up the actor-critic framework by feeding the state information and the average value of nearby agents' actions to the critic. However, this formulation strongly relies on the assumption that the value function for each agent can be well approximated by the mean of local
mean-
"Equal contribution
I Equal advising

---

Published as a conference paper at ICLR 2020
pairwise interactions. This assumption often does not hold when the interactions between agents become complex, leading to a significant drop in the performance.
In this paper, we propose a general learning paradigm called Evolutionary Population Curriculum (EPC), which allows us to scale up the number of agents exponentially. The core idea of EPC is to progressively increase the population of agents throughout the training process. Particularly, we divide the learning procedure into multiple stages with increasing number of agents in the environment. The agents first learn to play in simpler scenarios with less agents and then leverage these experiences to gradually adapt later stages with more agents and ultimately our desired population.
There are two key components in our curriculum learning paradigm. To process the varying number of agents during the curriculum procedure, the policy/critie needs to be population pepulation-invariant. So, we choose a self-attention (Vaswani et al., 2017)) based architecture which can generalize to an arbitrary number of agents with fixed number of parameters. More importantly, we introduce evolutionary selection process, which helps address the misalignment of learning goals across stages and improves the agents' performance in the target environment. Intuitively, our within-stage MARL training objective only incentivizes agents to overfit a particular population in the current stage. When moving towards a new stage with larger population, the successfully trained agents may not adapt well to the scaled environment. To mitigate this issue, we maintain multiple sets of agents in each stage, evolve them through cross-set mix-and-mated and parallel MARL fine-tuning in the scaled environment, and select those with better adaptability to the next stage.
EPC is RL-algorithm agnostic and can be potentially integrated with most existing MARL algorithms. In this paper, we illustrate the empirical benefits of EPC by implementing it on popular MARL algorithm, MADDPG (Lowe et al. (017). and experimenting on three challenging environments, including a predator-prey-stvy
individual survival game, mixed apperatieeannd-coppele tle game, and a fully cooperative food collection game. We show that EPC outperforms baseline approaches by large margin on all these environments the number of agents grows even exponen- tially. We also demonstrate that our method can improve the stability of the training procedure.
bat-
RELATED WORK
Multi-Agent Reinforcement Learning: It has been long history in applying RL to multi-agent
games (Littman) 1994] Shoham et al. |003) Panait & Luke 2205; Wright et al. 2019). Recently deep RL techniques have been applied into the multi-agent scenarios to solve complex Markov games and great algorithmic advances have been achieved. Foerster et al. (2016) and He et al. (2016) explored multi-agent agent variant of deep Q-leaming: (Peng et al. (217)) studied fully centralized actor-critt variant; Foerster et al. (2018) developed a decentralized multi-agent policy gradient algorithm with a centralized baseline; Lowe et al. (2017) proposes the MADDPG algorithm which extended DDPG to the multi-agen setting with decentralized policies and centralized Q functions. Our population curriculum approach is general framework for scaling MARL which can be potentially combined with any of these algorithms. Particularly, we implement our method on top of the MADDPG algorithm in this paper and take different MADDPG variants as baselines in experiments. There are also other works studying large-scal scale MARL recently (Lin et al.. 2018 Jiang Lu| 2018 Yang et al. 2018, Suarez et al.] 2019), which typically simplify the problem by weight sharing and taking only local observations. We consider much more general setting with global observations and unshared- weight agents. Additionally, our approach is a general learning paradigm which is complementary to the specific techniques proposed in these works.
Attention-Bae Based Policy Archttecture: Attention mechanism is widely used in RL policy repre- sentation to capture object level information (Duan et al., [2177; Wang et al. 2018). represent relations (Zambaldi et al. 22018 Malysheva et al. [2088] Yang et al. 2019) and extract communication channels Jiang & Lu 2018). Iqbal& & Sha (2019) use attention basedd critic. In our work, we utilize an attention module in both policy and critic, inspired by the transformer architecture (Vaswani et al. 2017), for the purpose of generalization to an arbitrary number of input entities.
Curriculum Learning: Curriculum learning can be tracked back to Elman (1993), and its core idea is to "start small": learn the easier aspects of the task first and then gradually increase the task difficulty. It has been extended deep neural networks on both vision and language tasks (Bengio et al. 2009) and much beyond: Karras et al. (2017) propose to progressively increase the network capacity for synthesizing high quality images; Murali et al. (2018) apply a curriculum over the control space for robotic manipulation tasks; several works (Wu & Tian 2016 Florensa et al. 2017]
(Sukhbattar al, 2017 Wang al. 2019) have proposed to first train RI. agents easier goals
2

---

Published as a conference paper at ICLR 2020
and switch to harder ones later. Baker et al. (2019) show that multi-agent agget self-play also lead to autocurricula in open-ended environments. In our paper, we propose to progressively increase the number of the agents as curriculum for better scaling multi-agent reinforcement learning.
Evolutionary Learning: Evolutionary algorithms, originally inspired by Darwin's 's natural selection, has long history (Bäck & Schwefel 1993). which trains a population of agents in parallel, and let them evolve via crossover, mutation and selection processes. Recently, evolutionary algorithms have been applied to learn deep RL policies with various aims, such as enhance training scalabil- ity (Salimans et al. 2017). to tune hyper-parameters (Jaderberg et al. 2017). to evolve intrinsic dense rewards (Jaderberg et al. 2018), learn neural loss for better generalization (Houthooft et al. 2018), to obtain diverse samples for faster off-policy learning (Khadka & Tumer (018)) and to encourage exploration (Conti et al. 2018). Leveraging this insight, we apply evolutionary learning to better scale MARL: we train several groups of agents in each curriculum stage and keep evolving them to larger populations for the purpose of better adaptation towards the desired population scale and improved training stability. Czarmecki et al. (2018) proposed similar evolutionary paradigm to progressively increase agent capacity, i.e., larger action spaces and more parameters. Their work considers a fixed environment with an increasingly more complex agent and utilizes the traditional parameter crossover and mutation during evolution. By contrast, we focus scaling MARL, namely an increasingly more complex environment with a growing number of agents. More importantly, we utilize MARL fine-tuning as an implicit mutation operator rather than the classical way of mutating parameters, which is more efficient, guided and applicable to even a very small number of evolution individuals. A similar idea of using learning for mutation is also considered by Gangwani & Peng 2018 in the single-agent setting.
mix--andmatee training
3 BACKGROUND
Markov Games: We consider multi-agent Markov decision processes (MDPs) (Littman 1994).
Such an N-agent Markov game is defined by state space S of the game, action spaces AI, Ay and observation spaces 01, Ox for each agent. Each agent i receives a private observation correlated
with the state O; S - O. and produces an action by a stochastic policy To, : O, x A: r [0. 1 parameterized by 0,. Then the next states are produced according to the transition function T S. The initial state is determined by a distribution p : S + [0, 1]. Each agent
S A1 ... x An i obtains rewards as a function of the state and its action ri S x A, + I, and aims to maximize its own expected return R2 = -E 7 r'r!(s) (s*,, af), where 7 is a discount factor and T is the time horizon. To minimize notation, we omit subscript of policy when there is no ambiguity.
Multi-Agent Deep Deterministic Policy Gradient (MADDPG): MADDPG (Lowe al. 2017) is a multi-agent variant of the deterministic policy gradient algorithm (Silver et al. 2014). learns centralized Q function for each agent which conditions global state information to resolve the non-stationary issue. Consider N agents with deterministic policies μ "M......y.. where
H, O + A, is parameterized by 0:. The policy gradient for agent is:
Vo, J(.)) Ex.o~D (V. a, (,,, ai, aN) (x))
(I)
Here DD denotes the replay buffer while O.ft (x, ai, aN) is a centralized action-value function for agent i that takes the actions of all agents, di...., aN and the state information x (i.c., x (oi,.... ON) or simply x = s if s is available). Let x' denote the next state from the environment tran- sition. The replay buffer D contains experiences in the form of tuples (x,x', ai...., aN, Pi, Pl. .....,,.............. PNN). Suppose the centralized critic Qf is parameterized by 0; Then is updated via:
((b.)) = Rx [10) (x, a1, ,NN) - - == + O,'² (x',''.....,d'
d'v -((())) (2)
where μ¹ {aa , ... Mag } is the set of target policies with delayed parameters 0. Note that the centralized critic is only used during training. At execution time, each policy He, remains decentralized and only takes local observation O,,
4 EVOLUTIONARY POPULATION CURRICULUM
In this section, we will first describe the base network architecture with the self-attention mecha- nism (Vaswani et al. 2017] which allows incorporate flexible number of agents during training. Then we will introduce the population curriculum paradigm and the evolutionary selection process.
3

---

Published as a conference paper at ICLR 2020
Q
Value
fi aj)
FC
FC
FC
FC
Attention
Attention
Attention
f
f
f
..
f
-..
O4, Rg
Pg. 4q
O2. R2
Og. Ax
0 9
(a]
Figure Our population-invariane Q function: (a) utilizes the attention mechanism to combine embeddings from different observation action encoder fú; (b) is a detailed description for fi, which also utilizes an attention module to combine M different entities one observation.
4.1 OMPULATION-INVARAAN ARCHITECTURE
We describe our choice of architecture based on the MADDPG algorithm (Lowe et al. (017), which is population-invarian in the sense that both the Q function and the policy can take in an arbitrary number of input entities. We first introduce the Q function (Fig- D and then the policy.
We adopt the decentralized execution framework, so each agent has its own Q function and policy network. Particularly for agent i. its centralized function is represented as follows:
Offt (x, (x,aa....,..))
ho(los(f((o;; (4)),,;)) where U; attentionn(sllj Vj) (3)
Here Si(0),0)) uz) is an observation-actio encoder (the green box in Fig. [a)) which takes in the observation 0j and the action aj from agent j. and outputs the agent embedding of agent j: o denotes the global attention embedding (the orange box in Fig. [a)) over all the agent embeddings. We will explain 2 and later. 9i a 1-layer fully connected network processing the embedding of the ith agent's own observation and action. h1 is a 2-layer fully connected network that takes the concatenation of the output of gi and the global attention embedding i and outputs the final value.
Attention Embedding o;; We define the attention embedding & by weighted sum of each agent's embedding fi(oj, az) for j + i:
Vi- Σ Eassiiooj,a)
jF
The coefficient a; is computed by
exp (Bi,)) aij - ΣjS1 exp (Br.)) ff a;) W.
(5)
where We and We parameters to learn. BL, computes the correlation between the embeddings of agent and every other agent j via an inner product. Oiij is then obtained by normalizing 3r, by a softmax function. Since represent the observations and actions of other agents with weighted mean i from Eq. 4. we can model the interactions between agent and an arbitrary number of other agents, which allows us to easily increase the number of agents in our curriculum training paradigm.
Observation-Action Encoder f& We now define the structure of f((ay, ag) (Fig. [b). Note that the observation of agent j. oj. also includes many entities, i.e., states of all visible agents and objects in the game. Suppose 0, contains M entities, i.e., 0j [o;,1, oj. ]. M may also vary as the agent population scales over training procedure or simply during episode when some agents die.
Thus, we apply another attention module combine these entity observations together in a similar way to how U; is computed (Eq. 5).
In more details, we first apply an entity encoder for each entity type to obtain entity embeddings of all the entities within that type. For example, in oj, we can have embeddings for agent entities (green boxes in Fig. (b) and landmark/object entities (purple boxes in Fig. [b). Then we apply an attention module over each entity type by attending the entity embedding of agent j to all the entities of this type obtain attended type embedding (the orange box in Fig. Tb)) Next, concatenate
4

---

Published as a conference paper at ICLR 2020
all the type embeddings together with the entity embedding of agent well as its action embedding. Finally, this concatenated vector is forwarded to a fully connected layer to generate the output of (()))aa)) Note that in the overall critic network of agent i, the same encoder Si is applied to every observation-action pair so that the network can maintain a fixed size of parameters even when the number of agents increases significantly.
The policy network (o:0:) has a similar structure as the observation-action encoder fo,o, a,), which uses an attention module over the entities of each type in the observation o adapt to the changing population during training. The only difference in this network is that the action a; is not included in the input. Notably, we do not share parameters between the Q function and the policy,
Policy Network:
4.2 POPULATION CURRICULUM
We propose to progressively scale the number of agents in MARL with a curriculum. Before combining with the evolutionary selection process, we first introduce simpler version, the vanilla population curriculum (PC), where we perform the following stage-wise procedure: (i) the initial stage starts with MARL training over a small number of agents using MADDPG and our population- invariant architecture; (ii) we start new stage and double the number of agents by cloning each of the existing agents; (iii) apply MADDPG training on this scaled population until convergence; (iv) the desired number of agents is not reached, go back to step (ii).
Mathematically, given N trained agents with parameters 0 = ..., ON from the previous stage, we want to increase the number of the agents to 2NV with new parameters 0 = ( ON.... 02N) for the next stage In this vanilla version of population curriculum, we simply initialize 0 by setting 0. + 02 and 0N++ + 0z. and then continue MADDPG training on 0 get the final policies for the new stage. Although 6, and + are both initialized from 6,, as training proceeds. they will converge to different policies since these policies are trained in decentralized manner in MADDPG.
4.3 EVOLUTIONARY SELECTION
Introducing new agents by directly cloning existing ones from the previous stage has clear limitation: the policy parameters suitable for the previous environment are not necessarily the best initializatton for the current stage as the population is scaled up. In the purpose of better performance in the final game with our desired population, we need to promote agents with better adaptation abilities during early stages of training.
Therefore, we propose an evolutionary selection process to facilitate the agents' scaling adaption ability during the curriculum procedure. Instead of training a single set of agents, we maintain K parallel sets of agents in each stage, and perform crossover, mutation and selection among them for the next stage. This is the last piece in our proposed Evolutionary Population Curriculum (EPC) paradigm, which essentially population curriculum enhanced by the evolutionary selection process.
Specifically, we assume the agents in the multi-agent game have Ω different roles. Agents in the same role have the same action set and reward structure. For example, we have f roles in a predator- ceddtto--rey game, namely predators and prey, and Ω = 1 role of agents for fully cooperative game with homogeneous agents. For notation conciseness, we assume there are N1 agents of role 1, namely A = (#I...., MN, 1; N₂ agents of role 2, namely A₂ = { +I..,.... NN and so on. In each stage, we keep K parallel sets for each role of agents, denoted by A(" "..,, ((k) for role i, and take a 3-step procedure, i.c., mix-and-match (crossover), MARL fine-fntuuuning (mutation) and selection, as follows evolve these K parallel sets of agents for the next stage.
A
Mix-and-Mater (Crossover): In the beginning of a curriculum stage, we scale the population of agents from to 2N. Note that we have K parallel agent sets of size N, for role i, namely A (.....,(())
(K) We first perform mix-and-match over these parallel sets within every role i: for
each set A!5). we pair it with all the K sets of the same role, which leads to K(K + 1)/2 new scaled agent sets of size 2N, Given these scaled sets of agents, we then perform another mix-and-matte across all the l roles: we pick one scaled set for each role and combine these l2 selected sets to produce a scaled game with 2N agents. For example, in the case of s = 2, we can pick one agent
set (ft1) from the first role and another agent set A (k2) from the second role to form a scaled game.
'Generally, can scale up the population with any constant factor by introducing any amount of cloned agents. We use the factor of 2 as conerete example here for easier understanding.
5

---

Published as a conference paper at ICLR 2020
Thus, there are Cuux (K(K+1))/
+ 1)/2) different combinations in total through this mix-and-matee process. We sample C games from these combinations for mutation in the next step. Since we are mixing parallel sets of agents, this process can be considered as the crossover operator in standard evolutionary algorithms.
MARL Fine-TTuing (Mutation): In standard evolutionary algorithms, mutations are directly per- formed on the parameters, which inefficient in high-dimensional spaces and typically requires a large amount of mutants to achieve sufficient diversity for evolution. Instead, here we adopt MARL fine-tuning in each curriculum stage (step (iii) in vanilla PC) as our guided mutation operator, which naturally and efficiently explores effective directions in the parameter space. Meanwhile, due to the training variance, MARL also introduces randomness which benefits the overall diversity of the evolutionary process. Concretely, we apply parallel MADDPG training on each of the C' scaled games generated from the mix-and-mated step and obtain C mutated sets of agents for each role.
Selection: Among these C' mutated sets of agents for each role, only the best K mutants can survive. In the case of ! 1. the fitness score of set of agents is computed their average reward after MARL training. In other cases when Q 2 2, given particular mutated set of agents of specific role, we randomly generate games for this set of agents and other mutated sets from different agent roles. We take its average reward from these randomly generated games as the fitness score for this mutated set. We pick the top-K scored sets of agents in each role to advance to the next curriculum stage.
Algorithm 1: Evolutionary Population Curriculum
Data: environment E[N, (Aiisi)))) with N agents of Ω roles, desired population Na. initial population No. evolution size K, mix-and-matel size C Result: a set of Na best policies N + No:
(.....,K) for each role 1 <is ! initial parallel MARL training on K games, E(N, ) for s j S K: while N < Na do
(K) initialize K parallel agent sets A,1) "!.,,....A!"*
N - 2 x N;
for 1 و j do
for each role 1sis S:: f1. unif(1, K), A75) A(50 + +A(ia)
(mix-and-match);
MARL training in parallel on E(N, (1(") )isisn) for 1 S S C (guided mutation); for role S < do
for 1 SisC do
5(6) + EAppEN(1
avg. rewards on E(N, (A(1)) ...., 100) ..... A(ha)) })
(finnes);
1(x) top-K w.r.t. S, from A( ,...,,,, A(() (selection); return the best set of agents in each role, i.e., {A (4)) |k E
Finally, when the desired population is achieved, we take the best set of agents in each role based on their last fitness scores as the output. We conclude the detailed steps of EPC in Alg. Note that in the first curriculum stage, we just train K parallel games without mix-and-match or mutation. So, EPC simply selects the best from the K initial sets in the first stage while the evolutionary selection process only takes effect starting from the second stage. We emphasize that although we evolve multiple sets of agents in each stage, the three operators, mix--nnd-mate MARL fine-tuning and selection, are all perfectly parallel. Thus, the evolutionary selection process only introduces little influence on the overall training time. Lastly, EPC is an walgorithh--gnnssie learning paradigm that can be potentially integrated with any MARL algorithm other than MADDPG.
Overall Algorithm:
EXPERIMENT
We experiment on three challenging environments. including a predatory predatory-peyy-tyee prey-style Grassland game, a mixed- sixxedd-ooperative-aii wnd--e--aadd---tttieeeee AddeetiiiveAvvvsaaaial Battle game and a fully cooperative Food Collection game. We compare EPC with multiple baseline methods on these environments with different scales of agent populations and show consistently large gains over the baselines. In the following, we will first introduce the environments and the baselines, and then both qualitative and quantitative performances of different methods on all three environments.
Adversarial

---

Published as a conference paper at ICLR 2020
5.1 ENVIRONMENTS
All these environments are built on top of the particle-world environment (Mrddtcch & Abbeel 2018 where agents take actions in discrete timesteps in continous 2D world.
Grassland: In this game, we have Ω = roles of agents, Ns sheep and Nw wolves, where sheep moves twice fast as wolves. We also have a fixed amount of L grass pellets (food for sheep) as green landmarks (Fig. 2a). A wolf will rewarded when it collides with (eats) sheep,
Woll
Team
Sheep Food
Resource
(a) Grass Land
(b) Adverdeerrrial buttle
Food Collection
Figure 2: Environment Visualizations
and the (eaten) sheep will obtain a negative reward and becomes inactive (dead), A sheep will be rewarded when it comes across a grass pellet and the grass will be collected and respawned in another random position. Note that in this survival game, each individual agent has its own reward and does not share rewards with others.
Adversarial Battle: This scenario consists of L units of resources as green landmarks and two teams of agents (i.e., Ω = 2 for each team) competing for the resources (Fig. 2b). Both teams have the same number of agents (N1 N2). When an agent collects unit of resource, the resource will be respawned and all the agents in its leam will receive a positive reward. Furthermore, if there are more than two agents from team 1 collide with one agent from team 2, the whole team will be rewarded while the trapped agent from team 2 will be deactivated (dead) and the whole team will be penalized, and vice versa.
Food Collection:
This game has food locations and N fully cooperative agents (S2 = 1). The agents need to collaboratively occupy as many food locations as possible within the game horizon (Fig. 2c). Whenever food occupied by any agent, the whole team will get reward of 6/N that timestep for that food. The more food occupied, the more rewards the team will collect.
In addition, we introduce collision penalties as well as auxiliary shaped rewards for each agent in each game for easier training. All the environments are fully observable that each agent needs to handle lot of entities and react w.r.t. the global state, More environment details are in Appx.
5.2 METHODS AND METRIC
We evaluate the following approaches in our experiments: (1) the MADDPG algorithm (Lowe ct al.) 2017) with its original architecture (MADDPG); (2) the provably-converg mean-field algorithm (Yang et al. 2018) (mean-field); (3) the MADDPG algorithm with our population-invariant architecture (Att-MADDP) (4) the vanilla population curriculum without evolutionary selection (vanilla-PC); and (5) our proposed EPC approach (EPC). For EPC parameters, we choose K = for Grassland and Adversarial Battle and simply set it Cmax and enumerate all possible mix-and-mater combinations instead of random sampling. All the bascline methods are trained until the same amount of accumulative episodes as EPC took. More training details can be found in Appx. B
3 for Food Collection; for the mix-and aat-h size C,
For Grassland and Adversarial Battle with Ω = 2, we evaluate the performance of different methods by competing their trained agents against EPC trained agents. Specifically, in Grassland, let sheep trained by each approach compete with the wolves from EPC and collect the average sheep reward as the evaluation metric for sheep. Similarly, each method. In Adversarial Battle, since two teams are symmetric, we just evaluate the shared reward of one team trained by each baseline against another team by EPC as the metric. For Food Collection with Ω 1, since it is fully cooperative, we take the team reward for each method as the evaluation metric. In addition, for better visualization, we plot the normalized scores by normalizing the rewards of different methods between and in each scale for each game. More evaluation details are in C
take the same measurement for wolves from
5.3 QUALITATIVE RESULTS
In Grassland, as the number of wolves goes up, it becomes increasingly more challenging for sheep to survive; meanwhile, as the sheep become more intelligent, the wolves will be incentivized be more aggressive accordingly. In Fig. 3] we illustrate two representative matches for competition, including one using the MADDPG sheep against the EPC wolves (Fig. 3a), and the other between the EPC sheep and the MADDPG wolves (Fig. (b)..33). From Fig. [3a] we can observe that the MADDPG sheep can be easily eaten up by the EPC wolves (note that dark circle means the sheep is eaten). On
7

---

Published as a conference paper at ICLR 2020
(a) MADDPG sheep vs EPC wolves
(b) MADDPG wolves EPC sheep
Figure 3: Example matches between EPC and MADDPG trained agents in Grassland
(b) MADDPG Figure 4: Adversarial Battl: dark particles are dead agents.
(a) EPC
(a) EPC Figure 5: Food Collection
(b) MADDPG
the other hand, in Fig. 3b we can see that the EPC sheep learns to eat the grass and avoid the wolves at the same time.
In Adversarial Battle, we visualize two matches in Fig. with one over agents by EPC (Fig, 4aj and the other over agents by MADDPG (Fig. 44).. We can clearly see the collaborations between the EPC agents: although the agents are initially spread over the environment, they learn to quickly gather as group to protect themselves from being killed. While for the MADDPG agents, their behavior shows little incentives to cooperate or compete - these agents stay in their local regions throughout the episode and only collect resources or kill enemies very infrequently.
In Food Collection (Fig. 5). the EPC agents in Fig. Sallearn to spread out and occupy as many food as possible to maximize the team rewards. While only one agent among the MADDPG agents in Fig. 5b successfully occupies food in the episode.
5.4 QUANTITATIVE RESULTS
Quantitative Results in Grassland
In the Grassland game, we perform curriculum training by starting with 3 sheep and 2 wolves, and gradually increase the population of agents. We denote game with Ns sheep and Nw wolves by "scale Ns-Nw"". We start with scale 3-2 and gradually increase the game size to scales 6-4, 12-8 and finally 24-16.For the curriculum learning approach, vanilla-PC and EPC, we train over 10a episodes in the first curriculum stage (scale 3-2) and fine-tune the agents with 5 x 10 episodes after mix-and-match in each of the following stage. For other methods that train the agents from scratch, we take the same accumulative training iterations as the curriculum methods for a fair comparison.
We report the performance of different methods for each game scale in Fig. 6a Overall, there little differences between the mean-field approach and the original MADDPG algo- rithm while the using the population-invariant architecture (i.e., Att-MADDPG) generally boosts the performance of MADDPG. For the method with population curriculum, vanilla-PC performs almost the same as training from scratch MADDPG) when the number of agents in the environment is small (i.e., 6-4) but the performance gap becomes much more significant when the population further grows (i.e., 12-18 and 24-16). For our proposed EPC method, it consistently outperforms all the baselines across all the scales. Particularly, in the largest scale 24-16, EPC sheep receive 10x more rewards than the best baseline sheep without curriculum training.
Main Results:
Detailed Statistics: Besides rewards, we also compute the statistics of sheep to understand how the trained sheep behave in the game. We perform competitions between sheep trained by different methods against the EPC wolves and measure the average number of total grass pellets caten per episode, i.e, #grass eaten, and the average percentage of sheep that survive until the end of an episode, i... survival rate, Fig. [6b] We can observe that as the population increases, becomes increasingly harder for sheep to survive while EPC trained sheep remain high survival rate even the largest scale. Moreover, as more sheep in the game, EPC trained sheep consistently learn to eat more grass even under the strong pressure from wolves. In contrast, the amount of eaten grass of MADDPG approach (i.e., MtDMP) drastically decreases when the number of wolves becomes large.
Quantitative Results in Adversarial Battle
8

---

Published as a conference paper at ICLR 2020
scale
scale
scale 12-8
scale 24-16
0.75
MADOHO fiedd
graie naten
surviaal rate
60-
0.50
Scprc
as PC EPC
ALL AEL-MADDP
40-
0.00
Addi
6-1
32-624-26
3/2
12-024-36
(a) Normalized scores of wolves and sheep
(b) Sheep statistics
Figure 6: Results in Grassland. In part (a), we show the normalized scores of wolves and sheep trained by different methods when competing with EPC sheep and EPC wolves respectively. In part (b), we measure the sheep statistics over different scales (x-axis), including the average number of total grass pellets eaten per episode (left) and the average percentage of sheep that survive until the end of episode (right). EPC trained agents (yellow) are consistently better than any baseline method.
1.-
In this game, we evaluate on environments with different sizes of agent population N, denoted by scale N1-N2 where N1 N₂ N/2. We start the curriculum from scale 4-4 and the increase the population size to scale 8-8 (N 16) and finally 16-16 (N = 32). Both vanilla-PC and EPC take 5 x 10a training episodes in the first stage and then 2 x 10 episodes in the following two curriculum stages, We report the normalized scores of different methods Fig. 7 where agents trained by EPC outperforms all the baseline methods increasingly more significant as the agent population grows.
0.8
0.6
0.4
0.2
0.0
4-1
MADDPG
varilla PC
mean held
DPC
ALLADDOP
Fi Figure Adversarial Battle
Quantitative Results in Food Collection
In this game, we begin curriculum training with N = 3, namely 3 agents and food locations, and progressively increase the pop- ulation size N to 6, 12 and finally 24. Both vanilla-PC and EPC perform training on 5 x 10a episodes on the first stage of N = 3 and then 2 10a episodes in each of the following curriculum stage. We report the normalized scores for all the methods in Fig. X where EPC is always the best among all the approaches with clear margin. Note that the performance of the original MADDPG and the mean- field approach drops drastically as the population size N increases. Particularly. the mean-field approach performs even worse than the original MADDPG method. We believe this is because in this game, the agents must act according to the global team state collaboratively, which means the local approximation assumption in the mean-field approach does not hold clearly.
0.6
0.-
0.-
0.0
WAGDPG
vanilla-PC
mean field
EPC
ASs-MADIIING
Figure 8: Food Collection
Ablative Analysis
Stability Analysis: The evolutionary selection process in EPC not only leads to better final perfor- mances but also stabilizes the training procedure. We validate the stability of EPC by computing the variance over 3 training seeds for the same experiment and comparing with the variance of vanilla-PC. which is also obtained from 3 training sceds. Specifically, we pick the second stage of curriculum learning and visualize the variance of agent scores throughout the stage of training. These scores are computed by competing against the final policy trained by EPC. We perform analysis on all the 3 environments: Grassland with scale 6-4 (Fig. 9a), Adversarial Battle with scale 8-8 (Fig. 99b) and Food Collection with scale 6 (Fig. 9c). We can observe that the variance of EPC is much smaller than vanilla-PC in different games.
Convergence Analysis: To illustrate that the self-attention based policies trained from a smaller scale is able to well adapt to a larger scale via fine-tuning we pick particular mutant by EPC in the second curriculum stage and visualize its learning curve throughout ane-tuning for all the environments, Grassland (Fig. (9)), Adversarial Battle (Fig. 9e) and Food Collection (Fig. 99). The scores are computed in the same way as the stability analysis. By comparing to MADDPG and Att-MADDPG, which train policies from scratch, we can see that EPC starts learning with a much higher score, continues improve during fine-tuning and quickly converges a better solution. Note that all baselines are in fact trained much longer. The full convergence curves are in App. D. Generalization: We investigate whether the learned policies can generalize to a different test environment with even larger scale than the training ones. To do so, we take the best polices trained by different methods on the largest population and directly apply these policies to a new environment with doubled population by self-cloning. We evaluate in all the environments with EPC, vanilla-PC
9

---

Published as a conference paper at ICLR 2020
1.0
1.0
10-
0.8
0.5
Wolf score
0.6
0.6
sheep
0.4
0.4-
0.4
0.2
0.2
0.2
02-
0.5
1.5
episode (X101)
opicode (x10%)
opisoder (x200)
Top Kin ERC
vanila PC
Stability comparison: Grassland,
(b) Adversarial Bottle,
(c) Food Collection.
ncore
shoop score
0.8
0.6
0.2
0.2
0.0
opisode (x101)
apriode (x10")
episode
(x10f)
MACOPG A1S-WA2DDG
EPC
MADOPS AP-AMDERC
EPC
AIL-ADDON
(d) Normalized scores: Grassland,
(c) Adversarial Battle,
(f) Food Collection.
Figure 9: Ablation analysis the second curriculum stage in all the games over 3 different training seeds. Stability comparison (top) in (a), (b) and (c): We observe EPC has much less variance comparing to vanilla-PC. Normalized scores during fine-tuning (bottom) in (d), (e) and (f): This illustrates that EPC can successfully transfer the agents trained with a smaller population larger population by fine-tuning.
sheep scare
6.2
0.2
02-
0.-
0.0
0,0
0.-
0.0
HADDPG VC
EPC
MA-MADORG
varilla-P PC
BPC
AILMADORS Virlfa-PC
EPC
Al-MADOPO
WC
EPC
(a) Environment Generalization: Grassland.
(b) Adversarial Battle.
Food Collection.
Figure 10: Environment Generalization: We take the agents trained on the largest scale and test on an environment with twice the population. We perform experiments all the games and show that EPC also advances the agents' generalizability.
and Att MADDPG and measure the normalized scores of different methods, which is computed in the same way the fitness score. In all cases, observe large advantage of EPC over the other two methods, indicating the better generalization ability for policies trained by EPC.
CONCLUSION
In this paper, we propose scale multi-agent reinforcement learning by using curriculum learn- ing over the agent population with evolutionary selection. Our approach has shown significant improvements over baselines not only in the performance but also the training stability. Given these encouraging results on different environments, we believe our method is general and can potentially benefit scaling other MARL algorithms. We also hope that learning with a large population of agents can also lead the emergence of swarm intelligence in environments with simple rules in the future.
ACKNOWLEDGMER
This research is supported in part by ONR MURI N000141612007 ONR Young Investigator AG. FF also supported in part by NSF grant 185-185 research grant from Lockheed Martin, and U.S. Army Combat Capabilities Development Command Army Research Laboratory Cooperative Agreement Number 13-2-0045 (ARL Cyber Security CRA). The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed implied, of the funding agencies. We also sincerely thank Bowen Baker and Ingmar Kanitscheider from OpenAl for valuable suggestions and comments.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Thomas Bäck and Hans-Paul Schwefel. An overview of evolutionary algorithms for parameter optimization. Evolutionary computation, 1(1):1-23, 1993.
Bowen Baker, Ingmar Kanitscheider, Todor Markow, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. arXiv preprint urXiv:1909.077 079999977,, 2019.
Trapit Bansal, Jakub Pachocki. Szymon Sidor, Ilya Sutskever, and Igor Mordatch. Emergent complexity via multi-agent competition. In ICLR, 2018.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML, pp. 41-48. ACM, 2009.
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty- seeking agents. In NeurIPS, pp. 5027-5038, 2018.
Wojeiech Marian Czarnecki, Siddhant M Jayakumar, Max Jaderherg, Leonard Hasenclever, Yee whye Teh, Nicolas Heess, Simon Osindero. and Razvan Pascanu. Mix & match agent curricula for reinforcement learning. In ICML, pp. 1095-1103, 2018.
Yan Duan, Marein Andrychowicz, Bradly Stadie, Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning. In NIPS, pp. 1087-1098, 2017.
Jeffrey L Elman. Learning and development in neural networks: The importance of starting small. Cognition, 48(1):71-99, 1993.
Carlos Florensa, David Held, Markus Wulfmeeer, Michael Zhang, and Pieter Abbecl. Reverse curriculum generation for reinforcement learning. In CoRL, pp. 482-495, 2017.
Jakob Foerster. Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In NIPS, pp. 2137-2145, 2016.
Jakob N Foenster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfac- tual multi-agent agent policy gradients. In AAAI, 2018
Tanmay Gangwani and Jian Peng. Genetic policy optimization. In ICLR, 2018.
Matthew Hausknecht and Peter Stone. Deep reinforcement learning in parameterized action space. arXiv
preprint arXiv:1511. 04143, 2015.
He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé III. Opponent modeling in deep reinforcement learning. In International Conference Machine Learning, pp. 1804-1813, 2016.
Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly Stadie, Filip Wolski, OpenAl Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. In Neur/PS, pp. 1400-5400 2018
Shariq Iqbal and Fei Sha. Actor-ateention-criticc for multi-agent reinforcement learning. In ICML, pp. 2961-2970, 2019.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojeiech Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals. Tim Green, lain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint arXiv: /tti:://11..9000 1,0986,, 2017.
Max Jaderberg, Wojeiech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Human-level performance in first-person multiplayer games with population- hased deep reinforcement learning. arXiv preprint MrXivv180..0128/ 2018.
Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi aulti-agent cooperation. In NeurlPS, 2018.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint 1rXiv:1710.1000 10196, 2017.
Shauharda Khadka and Kagan Tumer. Evolution-guiie guideed policy gradient in reinforcement learning. In NeurlPS, pp. 1188-12000 1200, 2018
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1)1333-1373, 2016.
11

---

Published as a conference paper at ICLR 2020
Kaixiang Lin, Renyu Zhao, Zhe Xu, and Jiayu Zhou. Efficient large-scale fleet management via multi-agent deep reinforcement learning. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1774-1783, ACM, 2018.
Michael Littman. Markov games framework for multi-agent reinforcement leamiing In ICMI, volume 157. pp. 157-163, 1994.
Siqi Liu, Guy Lever, Nicholas Heess, Josh Merel, Saran Tunyasuvunakool, and Thore Graepel. Emergent coordination through competition. ICLR, 2019.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Picter Abbecl, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative environments. NIPS, 2017.
Aleksandra Malysheva, Tegg Tackyong Sung, Chac-Bong Sohn, Daniel Kudenko, and Aleksei Shpilman. Deep multi-agent reinforcement learning with relevance graphs. arXiv preprint arXiv: 125577 2018.
Patrick Meier. Digital humanitarians: how big data is changing the face of humanitarian response. Routledge, 2015.
Volodymyr Mnih. Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou. Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, arXiv preprint arXiv:1312.5600,, 2013,
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. In AAAI, 2018.
Adithyavairvaan Murali, Lerrel Pinto, Dhiraj Gandhi, and Abhinav Gupta. Cassl: Curriculum accelerated self-supervised learning. In ICRA, pp. 6453-6460. IEEE, 2018.
OpenAL. OpenAI Five. htttps://blog
con/ openai -ali--iiin five/ 2018.
Liviu Panait and Sean Luke. Cooperative multi-agent learning: The state the art. AAMAS, 11(3)387-434, 2005.
Peng Peng. Quan Yuan, Ying Wen, Yaodong Yang, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent budirectionally.cordiine nets for learning to play starcraft combat games, arXiv preprint 1703v 2, 2017.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and llya Sutskever. Evolution strategics as a scalable alternative to reinforcement learning. arXiv preprint 17X.v:1703.03864, 2017.
Yoav Shoham, Rob Powers, and Trond Grenager. Multi-agent reinforcement learning: a critical survey. Web manuscript, 2003.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In ICML, pp.
2014.
Joseph Suarez, Yilun Du Phillip Isola, and Igor Mordatch. Neural mmo: massively multiagent game environment for training and evaluating intelligent agents. arXiv preprint arxiv:1903.00784 2019.
Sainbayar Sukhhaatar, Roh Fergus, et al. Learning multiagent communication with backpropagation. In NIPS, pp. 2244-2252, 2016.
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self ellf-pla. arXiv preprint arXiv:1703.05400 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention you need. In NIPS, pp. 1998-6008, 2017.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv:1901.0155.. 2019.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018.
Michael P Wellman, Daniel M Reeves, Kevin M Lochner, Shih-Fen Cheng, and Rahul Suri. Approximate strategic reasoning through hierarchical reduction of large symmetric games. In AAAI, 2005.
Mason Wright, Yongzhao Wang, and Michael P Wellman. Iterated deep reinforcement learning in games: history-aware training for improved stability. In Proceedings of the 2019 ACM Conference Economics and Computation, pp. 617-636. ACM, 2019.
12

---

Published as a conference paper at ICLR 2020
Cathy Wu, Aboudy Kreidich, Eugene Vinitsky, and Alexandre M Bayen. Emergent behaviors in mixed-autonomy traffic. CoRL, pp. 598-407, 2017.
Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. Building generalizable agents with realistic and rich
3d enviromment. arXiv preprint arXiv:1801.0222 2018.
Yuxin Wu and Yuandong Tian. Training agent for first-person shooter game with actor-critic curriculum learning. In ICLR, 2016.
Wei Yang, Xiaolong Wang, Ali Farhadi, Abhinav Gupta, and Roozbeh Mottaghi. Visual semantic navigation using scene priors. In ICLR, 2019.
Yaodong Yang. Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi-agent reinforcement learning. In ICML, pp. 5567-5576, 2018.
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Relational deep reinforcement learning. arXiv preprint arXiv:1806.0 01830, 2018.
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target- driven visual navigation in indoor scenes using deep reinforcement learning. In /CRA, pp. 3357-3364. IEEE, 2017.
13

---

Published as a conference paper at ICLR 2020
A ENVIRONMENT DETAILS
In the Grassland game, sheep gets +2 reward when he cats the grass, reward when eaten by wolf. The wolf get +5 reward when eats sheep. We also shape the reward by distance, sheep will get less negative reward when it is closer to grass and wolf will get less negative reward when it is closer to sheep. This game is adapted from the original Predator-prey game in the MADDPG paper (Lowe et al. 2017) by introducing grass and allowing agent to die.
In the Adversarial Battle game, agent will get +1 reward when he cats the food, -6 reward when killed by other agents. If N agents kill an enemy, they will be rewarded +6/N. We shape the reward by distance. Agent will receive less negative rewards when is closer to other agents and grass. We want to encourage collision within agents and also will be easier for them to learn to eat. This game is adapted from the mrean-field MARL paper (Yang et al. 2018) by converting it from a grid world to particle-world, introducing food and only allowing 2-agent cooperative killing.
In the Food Collection game, there N agents and N food locations. Each agent will get shared +6/LV reward per timestep when one food is occupied by any agent. If one agent gets collision with another, all of the agents will get a punish of -6/N. We shape the reward by distance. Agents will receive less negative rewards when it gets closer to the food. Since the number of agents and food are equal, we want to avoid the collision within agents and let the agents to learn to occupy as many food as possible. This is exactly the same game as the Cooperative Navigation game in the MADDPG paper. We slightly change the reward function to ensure it is bounded w.r.t. arbitrary number of agents.
We use the normalized reward the score during evaluation. For a particular game with particular scale, we first collect the reward for each type of agents, namely the average reward of each individual of that type withour the shaped rewards. Then we re-scale the collected rewards by considering the lowest reward among methods as score 0 and highest reward as score
B TRAINING DETAILS
We follow all the parameertssssrsss in the original MADDPG paper (Lowe et al. 2017) for both EPC and all the baseline methods considered. Particularly, we use the Adam optimizer with leaningg rate 0.01, 31 0.9, 82 0.999 and 5 10 across all experiments. r 0.01 is set for target network update and y 0.95 used as discount factor. We also use a replay buffer of size 10" and we update the network parameters after every 100 samples. The batch size is 1024. All the baseline methods are trained for a number of episodes that cquals the accumulative number of episodes that EPC has taken.
We set K 2 in all the games during training except that K 3 in the food collection game due to computational constraints. During EPC training, in the grassland game, train the scale of 3 sheep wolf for 100000 episodes. We train another 50000 episodes every time the agents number doubles. In the adversarial battle game and food collection game, we train the first scale for 50000 episodes. We train another 20000 episodes every time the agents number doubles.
In the grassland game, the entity types are the agent itself, other shoep, other wolf and food. We thus have four types of entity encoders for each of those entity types. In the adversarial battle game, Similar to grassland game, the entity types agent itself, other teammates, enemies and food. We also have four types of entity encoders for each of those entity types. Since there only one group in the food collection game, the entity types agent itself, other teammates and food. We thus have three entity encoders our network.
C EVALUATION DETAILS
To evaluate the agents trained in the environment with Q = 2, we make two roles of agents trained with different approaches compete against each other. Each competition is simulated for 10000 cpisodes. The average normalized reward over the 10000 episodes will be used as the competition score for each side. Note that in experiments, we let all the methods compete against our EPC approach for evaluation. For adversarial battle game, we take the average score of two teams as the model's final evaluation score, since the two teams this game are completely symmetric.
In the food collection game, since there is only one role, we simply simulate the model for 10000 episodes. The average normalized reward over the 10000 episodes will be used as the score of the model.
D ADDITIONAL DETAILS ON EXPERIMENT RESULTS
D. FULL TRAINING CURVES FOR BASELINES
All the baseline methods are trained for a numher of episodes that equals the accumulative numher of episodes that EPC has taken.
The purpose of Figure 9e 9f is simply showing the transfer performance, i.e., the initialization produced by EPC from the previous stage effective and can indeed leverage past experiences to warm-start. The x-axis of
14

---

Published as a conference paper at ICLR 2020
the plot shrunk for visualization purpose. Here illustrate the complete convergence of baselines, i.c., ATT-MADDPG and MADDPG, in Figure [ [Tc the games respective. Although Att-MADDPP takes much longer time learn, its performance still worse than EPC.
1.0
0.8
sheep score
0.8
wolf score
0.6
0.6
0.4
0.4
0.2
0.2
10
15
5
10
15
episode (x10*)
episode (x10*)
MADDPG
Att-MADDPG
EPC
(a) Normalized scores: Grassland,
1.0
0.8
score
0.6
score
0.6
0.4
0.4
0.2
0.2
0.0
4
6
2
4
episode (x10a)
episode (x10*)
MADDPG
EPC
MADDPG
EPC
Att-MADDPG
Att-MADDPG
(b) Adversarial Battle,
(c) Food Collection.
Figure 11: Full learning curves on the second curriculum stage in all the games. EPC fine-tunes the policies obtained from the previous stage while MADDPG and Att-MADDPG are trained from scratch for much longer time.
D.2 Raw REWARD NUMBERS OF EVALUATION RESULTS
In this section, provide the actual rewards without normalization when comparing all the haselines with EPC. These corresponding the histograms reported in Figure 6a
Grassland game, wolf rewards, corresponding to Figure 6a
scale
MADDPG
mean field
Au-MADDPG
vanilla-PC
EPC
3-2
0.596
0.877
0,8145
0.8145
1.407
6-4
3.7735
0.9515
2.5905
2.001
3.7735
12-8
3.1915
3.385
10.2125
9.974
14.377
24-16
14.482
18.272
32.8945
47.6365
61.4245
Grassland game, sheep rewards, corresponding to sheep Figure 6a
scale
MADDPG
mean field
Att- Mt-MADDPG
vanilla-PC
EPC
3-2
-4,0026
-3.9947
2.66
2.66
8.3846
6-4
-20.2494
-20.5107
0.9892
1.1804
10.1455
12-8
-52.863
-53,6338
-4248801
-11.3736
3.3774
24-16
-119.1327
-118,3668
-111.0656
-70.1981
-44.1311
Adversarial Battle game, rewards of team corresponding Figure
15

---

Published as a conference paper at ICLR 2020
scale
MADDPG
mean field
At-MAADPG
vanilla-PC
EPC
4-4
7.51555
5.81165
22.6357
22.6357
26,86355
8-8
0.6692
-0,58115
43.7801
46.89595
65.75585
16-16
-46.6398
-35,5978
28,8336
109.4406
189.69775
Food Collection game, team rewards, corresponding to Figure
scale
MADDPG
mean field
Att- AttMMADDPG
vanilla-PC
EPC
55.06
42.74
61.6488
61.6488
64.822
6
17.01
3.37
49.3626
58,0014
63.7004
12
6.32
6.735
49,45755
52.3625
59,34
24
10.346075
7,830975
33,435
49.998025
59.47035
D.3 PAIRWISE COMPETITION RESULTS BETWEEN ALL METHODS IN COMPETITIVE GAMES
For visualization purpose, only illustrate the scores of the competitions between baselines and EPC in the main paper. Here provide the complete competition rewards hetween every pair of methods in both Grassland Adversarial Battle with the largest population of agents follows.
Here show the wolf rewards in Grassland with scale 24- 16. For wolves trained by each approach, we compare them against the sheep by all the methods. EPC wolves always have the highest rewards in the bottom Correspondingly, when different wolves compete against EPC sheep, they always obtain the lowest rewards the rightmost column.
sheep
MADDPG
mean--ieel
Att-MADDPG
Vanilla-PC
EPC
wolf
MADDPG
66.914
67.0945
66.34
23.048
14.482
mean-fiell field
75.7655
74.23
74.7375
28.3705
18.272
Att-MADDPG
103.22
103.326
98.07
49.557
32.8945
Vanilla-PC: 110.333
111.3735
101.8975
64.53
47.6365
EPC
120.9025
121.4325
115.956
82,381
61.4245
Here show the sheep rewards in Grassland with scale 24-16. For sheep trained by each approach, we compete them against the all different wolves. EPC sheep always have the highest rewards as in the last row. Correspond- ingly, when competing different sheep against EPC wolves, the rewards are always the lowest in the rightmost column.
wolf
MADDPG
mean-fieh field
Att-MADDPG
Vanilla-PC
EPC
sheep
MADDPG
-63,5096
-72.5443
-100.4636
-107.225
-119.1327
mean- mean-field field
663.7089
-71.0714
-100.6304
-10.89117
-11.55668
Att-MADDPG
-6294166
-71.3339
-95.0522
.99.1207
-111.0656
Vanilla-PC -5.7086
-7.8011
-31.9936
-49.5186
-70.1981
EPC
9.2135
10.5892
-6,9846
-27.3475
-44.1031
Here we show the rewards of team I Adversarial Battle with scale 16-16. For agents trained by each approach, we compare them as team against all different methods as team 2. When EPC agents as team I, no matter which opponent is, they always get the highest rewards in the last row. When other methods compete against EPC, the obtained rewards are always the lowest in the rightmost column.
compared
MADDPG
mean-field
At-MAADPG
Vanilla-PC
EPC
reported
MADDPG
61.4555
17.1591
2.8033
-29,9242
-46.6398
mean-lield
104.41315
59.01315
27.9004
11.1891
-35.5978
All-MADDPG
146.9829
117,3804
81.702
18.57425
28.8336
Vanilla-PC:
202.9163
155.2805
174.91965
123.7212
109,4406
EPC
339.63255
318.3223
256.8464
198.3621
189.69775
D.4 VARIANCE OF PERFORMANCE EVALUATIONS
We present the performance of all approaches in all three games with the largest scale. We train all the approaches with 3 different seeds and show the normalized scores with variance as following. We can that EPC not only gives better results but also much smaller variance.
E ADDITIONAL EXPERIMENTS ON THE ORIGINAL PREDATOR-PREY GAME
Grassland is adapted from the Predator-prey game introduced by the original MADDPG paper (Lowe et al. 2017). To further validate empirical results, additionally study the performances of different algorithms the unmodified Predator-prey game follows.
16

---

Published as a conference paper at ICLR 2020
1.0
1.0
0.8
0.8
0.8
score
score
0.6
0.6
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
MADDPG
mean-fleld
Ait MAD-M
vanilla- PC
EPC
HADDPG
mean-field fiel Att Att-MADDPG
vanilla-Pe
EPC
MADDPG
mean-field
Att MAD-G
vanill-PPe
EPC
(b) Normalized scores with vari- Adversarial battle
(c) Normalized scores with vari- ances in Food Collection with 24 agents
(a) Normalized scores with vari- ances in Grassland in scale 24-16
scale 16-16
We first report the normalized score in Fig. 13 by comparing all the methods against EPC. EPC is consistently better than all methods in all the scales.
scale 3-2
scale 6-4
scale 12-8
1.00
MADDPG
0.75
mean field
score
score
score
Att- Att-MADDPG vanilla-PC
0.50
0.25
EPC
0.00
predator
prey
predator
prey
predator
prey
Figure 13: Normalized scores in the original Predator-prey game
Besides, also report the reward numbers when competing against EPC. Since the Predator-prey game zero-sum game, simply report the predator rewards (the prey reward is exactly the negative value).
scate
MADDPG
mean field
Au-MADDPG
vanilla-PC
EPC
10.403
7.39
8.675
8.675
12.662
6-4
31,458
25.529
31,747
45.155
54,546
12-8
74.939
64.377
133.261
200.638
214.328
Furthermore, we also demonstrate results full pairwise competition between every two methods scale 12-8 below. Consistently, we can see that EPC predators always have the highest scores as in the last row, When competing against EPC prey, the lowest rewards are observed.
prey
MADDPG
mean-fiehh field
Att-MADDPG
Vanilla-PC
EPC
predator
MADDPG
229.887
249.4
247.423
122.878
74,939
mean- mean-field
207.022
210.838
228.469
107.811
64.377
Att-MADDPG
569.862
532.611
373.979
204.743
133.261
Vanilla-PC
758,293
737.067
521.486
303.86
200.638
EPC
827.298
764.417
519.319
299.505
214.328
17


Reviews:
[{"review_id": "ryxYvoLRYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new method of scaling multi-agent reinforcement learning to a larger number of agents using evolution. Specifically, the procedures (EPC) involves starting with a small number of agents, training multiple sets in parallel, and doing crossover to find the set of agents that generalize best to a larger number of agents. This is motivated by the intuition that agents that perform best in small groups may not be the ones that perform best in larger groups. These claims are empirically verified in three games based on the particle world set of environments. \n\nI’m a fan of ‘automatic curriculum learning’-style methods designed to gradually add complexity to improve final agent performance, and this paper is no exception. The proposed method is simple, but it makes sense. I like the fact that it is both RL algorithm agnostic, and that it can be largely executed in parallel, which means that it introduces only small training time overhead. I also like the proposed method of making the Q function agnostic to the number of agents and entities using attention (although whether these policies incorporate information from previous time steps, or if they can be made to do so). The experimental results are thorough, comparing to MADDPG, a simpler version of their curriculum without evolution, and a recently proposed method for scaling up MADDPG, showing that EPC consistently outperforms all of them, and is more stable. I think the complexity of the environments is also suitable for this style of paper, although it would be nice to see results in a more open and complex domain such as the recent NeuralMMO game.\n\nOverall, I think this is a good paper and I recommend acceptance.  \n\n\nSmall fixes:\n-\t‘asThank yoTha’ -> not sure what this means\n-\tN1 agents for of role 1 -> N1 agents of role 1\n-\t“We adopt the decentralized learning framework” --- even if each agent has their own Q function, if that Q function is centralized (uses the observation of all agents) then training is still centralized \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1lUoOjaYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review Update: Thank you for the detailed response, it raised my opinion of the paper as it reduced my concerns on the rigor of the experiments performed. I believe the changes increase the significance of the contribution and may help it reach a broader audience.\n\n--\nThis paper proposes evolving curriculums of increasing populations of agents to improve multi-agent reinforcement learning with large number of agents. The topic is of relevance to the ICLR community and the results are tending towards publishable contributions, but I have some concerns that I would like the authors to discuss in their rebuttal.\n\nThe method is evaluated on a good range of suitably challenging environments. However, why did the authors propose new challenges within the particle environments and not those used in the original publication? This change makes it harder to compare results across publications, whilst not seeming to add a significant change in the learning problem beyond what was present in the original benchmark tasks. The food collection task sounds like it may be equivalent to simple spread. Will the new environments be released as open source for others to build upon this work? \n\nThe method is compared against a good range of baseline methods and ablations of the proposed method. However, without grounding the results in a known environment it is hard to place whether the implementations of MADDPG and Mean-Field are fair reproductions. Presuming the authors are using the open source implementation of MADDPG (please confirm) this is most significant for Mean-Field particularly given its poor performance in Section 5.4. Please provide evidence that this is a fair comparison.\n\nTo evaluate the resultant agents in the competitive environments, all methods were placed into games against the authors proposed EPC method. Was this the same EPC opponents the evaluated EPC team were trained against? If so, this is an unfair advantage to the EPC team as it has time to optimize against this opponent whilst the other methods have not. Even if it is an EPC team from a different training run, there may be outstanding biases in the joint policies EPC tends towards that benefit the EPC team evaluated. This could be overcome by evaluating all methods in competition with all other methods.\n\nFor all experimental results, please quantify variance in performance as well as average value (currently only done in Figure 9 a and b). How many repeats of evaluation and training were performed for each? Without these details the claim on page 9 that \"EPC is always the best among all the approaches with a clear margin\" is too strong. Are these differences statistically significant? Additionally, please also include the maximum scores (where normalized score = 1.0) for all experiments, as presenting results with only normalized scores unnecessarily reduces the reproducibility of the work. \n\nFinally, in Appendix B, the authors provide a list of hyperparameter settings without discussion of how these were chosen. Were they optimised for one specific method or set to defaults from the literature? In particular, as the performance of Att-MADDPG is still improving at the end of the plot in Figure 9e, I am concerned that the #episodes was chosen to optimise the performance of EPC.\n\nOverall, this is an interesting approach with promising initial results. I believe the contribution would be significantly improved by addressing the issues above and look forward to the authors responses which could increase my rating to acceptance.\n\nThings that could improve the paper but did not impact the score:\n1) On page 5 it is noted that the authors do not share parameters between the Q-function and policy. It would improve the paper to justify why this choice was made.\n2) Page 5: \"N_1 agents for of the role\" -> N_1 agents of the role\n3) Page 5: \"as follows to evolved these K parallel sets\" -> evolve\n4) Page 7: \"resources asThank yoTha green landmarks\"\n5) Page 8: \"understand how the trained sheep behavior in the game\" -> how the trained sheep behave in the game\n6) Page 14: There are repeated grammatical issues in Appendix A e.g. \"is more closer to grass / sheep / other agents\" -> is closer to grass / sheep / other agents and \"will less negative reward\" -> will receive less negative rewards", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1lZmZmkqH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a kind of curriculum for large-scale multi-agent learning. The related work section mentions some obvious points of comparison (note: see also https://science.sciencemag.org/content/364/6443/859.abstract). However, the authors do not compare with ANY of this work (either in terms of algorithm design or performance). It is therefore difficult to evaluate the contribution. \n\nIn more detail, the paper combines RL, multi-agent learning and evolution. This is an extremely challenging domain, with many moving parts. How does this approach relate to the work of Salimans et al, Jaderberg et al, Houthooft et al, etc? Without detailed discussion and experiments it is impossible to tell if this is an advance. Improving on the baselines is a useful sanity check. Showing the work is an actual contribution requires comparing against other algorithms in the same space. \n\n---- ----\nAfter reading the rebuttal and other reviews and comments, I've modified my score to weak accept. The paper makes an interesting contribution that is distinct from other approaches.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a curriculum approach to increasing the number of agents (and hence complexity) in MARL.

The reviewers mostly agreed that this is a simple and useful idea to the MARL community. There was some initial disagreement about relationships with other RL + evolution approaches, but it got resolved in the rebuttal. Another concern was the slight differences in the environments considered by the paper compared to the literature, but the authors added an experiment with the unmodified version.

Given the positive assessment and the successful rebuttal, I recommend acceptance.
Decision: Accept (Poster)