Emergent Tool Use From Multi-Agent Autocurricula
Authors: Bowen Baker
Ingmar Kanitscheider
Todor Markov
Yi Wu
Glenn Powell
Bob McGrew
Igor Mordatch
Venue:  (2020)

Abstract: Through multi-agent competition, the simple objective of hide-and-seek, and standard reinforcement learning algorithms at scale, we find that agents create a self-supervised autocurriculum inducing multiple distinct rounds of emergent strategy, many of which require sophisticated tool use and coordination. We find clear evidence of six emergent phases in agent strategy in our environment, each of which creates a new pressure for the opposing team to adapt; for instance, agents learn to build multi-object shelters using moveable boxes which in turn leads to agents discovering that they can overcome obstacles using ramps. We further provide evidence that multi-agent competition may scale better with increasing environment complexity and leads to behavior that centers around far more human-relevant skills than other self-supervised reinforcement learning methods such as intrinsic motivation. Finally, we propose transfer and fine-tuning as a way to quantitatively evaluate targeted capabilities, and we compare hide-and-seek agents to both intrinsic motivation and random initialization baselines in a suite of domain-specific intelligence tests.

Published as a conference paper at ICLR 2020
EMERGENT TOOL USE FROM MULTI-AGENT AUTOCURRICULA
Bowen Baker
Ingmar Kanitscheider
Todor Markov OpenAl
Yi Wu
OpenAl
OpenAl
OpenAl
nowendopenad.com
Seggarropenai.ccmmm
Eodor8openai...
jww.y1
Glenn Powell-
Bob McGrew
Igor Mordatch Google Brain
OpenAl glenn@openai.com
OpenAl
bmnogrex@openaa.com
imordattch&rooogee.com
ABSTRACT
Through multi-agent competition, the simple objective of hide-and-seek, and stan- dard reinforcement learning algorithms scale, we find that agents create self- supervised autocurriculum inducing multiple distinct rounds of emergent strategy, many of which require sophisticated tool use and coordination. We find clear evi- dence of six emergent phases in agent strategy in our environment, each of which creates new pressure for the opposing team to adapt; for instance, agents learn to build multi-object shelters using moveable boxes which in turn leads to agents discovering that they can overcome obstacles using ramps. We further provide ev- idence that multi-agent competition may scale better with increasing environment complexity and leads to behavior that centers around far more human-reld numann-elevant skills than other self-supervised reinforcement learning methods such as intrinsic motivation. Finally, we propose transfer and fine-tuning as way to quantitatively evaluate targeted capabilities, and compare hide-and-seek agents to both in- trinsic motivation and random initialization baselines in a suite of domain-specific intelligence tests.
INTRODUCTION
Creating intelligent artificial agents that can solve a wide variety of complex human-relevant tasks has been long-standing challenge in the artificial intelligence community. Of particular relevance to humans will be agents that can sense and interact with objects in a physical world. One approach creating these agents is to explicitly specify desired tasks and train reinforcement learning (RL) agent to solve them. On this front, there has been much recent progress in solving physically grounded tasks, e.g. dexterous in-hand manipulation (Rajeswaran et al., 2017; Andrychowicz al., 2018) locomotion of complex bodies (Schulman et al., 2015; Heess et al., 2017). However, specifying reward functions or collecting demonstrations order to supervise these tasks can be
"This was large project and many people made significant contributions. Bowen, Bob, and Igor conceived the project and provided guidance through all stages of the work. Bowen created the initial environment, infrastructure and models, and obtained the first results of sequential skill progression. Ingmar obtained the first results of tool use, contributed to environment variants, created conain-specifte statistics, and with Bowen created the final environment. Todor created the manipulation tasks in the transfer suite, helped Yi with the RND baseline, and prepared code for open-sourcing. Yi created the navigation tasks in the transfer suite, intrinsic motivation comparisons, and contributed to environment variants. Glenn contributed to designing the final environment and created final renderings and project video. Igor provided research supervision and team leadership.
Work performed while OpenAT

---

Published as a conference paper at ICLR 2020
(a) Running and Chasing
(b) Fort Building
(c) Ramp Use
Hiders
Seekera
Reward
-50
150
250
300
400
500
Episodes (x10')
(d) Ramp Defense
(e) Box Surfing
(f) Surf Defense
Figure 1: Emergent Skill Progression From Multi-Agent Autocurricula. Through the reward signal of hide-and-seek (shown on the y-axis), agents go through distinct stages of emergence. (a) Seekers (red) learn to chase hiders, and hiders learn to crudely run away. (b) Hiders (blue) learn basic tool use, using boxes and sometimes existing walls to construct forts. (c) Seekers learn to use ramps to jump into the hiders' shelter. (d) Hiders quickly leam to move ramps to the edge of the play area, far from where they will build their fort, and lock them in place. (e) Seekers learn that they can jump from locked ramps to unlocked boxes and then surf the box to the hiders' shelter, which is possible because the environment allows agents to move together with the box regardless of whether they are on the ground or not. (f) Hiders learn to lock all the unused boxes before constructing their fort. We plot the mean over 3 independent training runs with each individual seed shown with dotted line. Please see openai. i com bll og/ ame aeeemmmrrreeeee wegenttttoo use for example videos.
time consuming and costly. Furthermore, the learned skills in these single-agent RL settings are inherently bounded by the task description; once the agent has learned to solve the task, there is little room to improve.
Due to the high likelihood that direct supervision will not scale to unboundedly complex tasks, many have worked on unsupervised exploration and skill acquisition methods such as intrinsic motivation. However, current undirected exploration methods scale poorly with environment complexity and are drastically different from the way organisms evolve on Earth. The vast amount of complexity and diversity Earth evolved due to co-evolution and competition between organisms, directed by natural selection (Dawkins & Krebs, 1979). When a new successful strategy or mutation emerges, it changes the implicit task distribution neighboring agents need to solve and creates new pressure
2

---

Published as a conference paper at ICLR 2020
for adaptation. These evolutionary arms races create implicit autocurricula (Leibo et al., 2019a) whereby competing agents continually create new tasks for each other. There has been much success in leveraging multi-agent autocurricula to solve multi-player games, both in classic discrete games such as Backgammon (Tesauro, 1995) and Go (Silver et al., 2017), as well as in continuous real-time domains such Dota (OpenAI, 2018) and Starcraft (Vinyals et al., 2019). Despite the impressive emergent complexity in these environments, the learned behavior is quite abstract and disembodied from the physical world. Our work sees itself in the tradition of previous studies that showcase emergent complexity in simple physically grounded environments (Sims, 1994a; Bansal et al., 2018; Jaderberg et al., 2019; Liu al., 2019); the success in these settings inspires confidence that inducing autocurricula in physically grounded and open-ended environments could eventually enable agents to acquire an unbounded number of human-relevant skills.
We introduce new mixed competitive and cooperative physics-based environment in which agents compete in simple game of hide-and-seek, Through only visibility- -basee reward function and competition, agents learn many emergent skills and strategies including collaborative tool use, where agents intentionally change their environment to suit their needs. For example, hiders learn to create shelter from the seekers by barricading doors constructing multi-object forts, and as a counter strategy seekers learn to use ramps to jump into hiders' shelter. Moreover, we observe signs of dy- namic and growing complexity resulting from multi-agent competition and standard reinforcement learning algorithms; we find that agents go through as many as six distinct adaptations of strategy and -trateyy,, which are depicted in Figure 1. We further present evidence that multi-agent co-adaptation may scale better with environment complexity and qualitatively centers around more human-interpretable behavior than intrinsically motivated agents.
However, as environments increase in scale and multi-agent autocurricula become more open-ended, evaluating progress by qualitative observation will become intractable. We therefore propose a suite of targeted intelligence tests to measure capabilities in our environment that we believe our agents may eventually learn, e.g. object permanence (Baillargeon & Carey, 2012), navigation, and construction. We find that for number of the tests, agents pretrained in hide-and-seek learn faster or achieve higher final performance than agents trained from scratch or pretrained with intrinsic motivation; however, we find that the performance differences are not drastic, indicating that much of the skill and feature representations learned in hide-and-seek are entangled and hard to fune-tune.
The main contributions of this work are: 1) clear evidence that multi-agent self-play can lead to emergent autocurricula with many distinct and compounding phase shifts in agent strategy, 2) evi- dence that when induced in physically grounded environment, multi-agent autocurricula can lead to human-relevant skills such as tool use, 3) a proposal to use transfer as a framework for evaluating agents in open-ended environments well as a suite of targeted intelligence tests for our domain, and 4) open-sourced environments and code" for environment construction to encourage further research in physically grounded multi-agent autocurricula.
RELATED WORK
There is a long history of using self-play in multi-agent settings. Early work explored self-play using genetic algorithms (Paredis, 1995; Pollack et al., 1997; Rosin & Belew, 1995; Stanley & Mi- ikkulainen, 2004). Sims (1994a) and Sims (1994b) studied the emergent complexity morphology and behavior of creatures that coevolved in simulated 3D world. Open-ended evolution was fur- ther explored in the environments Polyworld (Yaeger, 1994) and Geb (Channon et al., 1998), where agents compete and mate in 2D world, and in TieIaa (Ray, 1992) and Avida (Ofria & Wilke, 2004), where computer programs compete for computational resources. More recent work attempted to formulate necessary preconditions for open-ended evolution (Taylor, 2015; Soros & Stanley, 2014). Co-adaptation between agents and environments can also give rise to emergent complexity (Florensa al., 2017: Sukhbaatar al., 2018 8; Wang et al., 2019). In the context of multi-agent RL, Tesauro (1995). Silver et al. (2016), OpenAl (2018), Jaderberg et al. (2019) and Vinyals et al. (2019) used self-play with deep RL techniques to achieve super-human performance in Backgammon, Go, Dota, cypture-the-Flae and Starcraft, respectively. Bansal et al. (2018) trained agents in a simulated 3D physics environment to compete in various games such as sumo wrestling and soccer goal shooting- In Liu et al. (2019), agents learn to manipulate a soccer ball in 3D soccer environment and discover
Code he found github. comiobnncc///ppaaalaaa /mu1 i -agent t- eee--mmergge semergence-eel envi www.-eenvviniinnnne ronment..
3

---

Published as a conference paper at ICLR 2020
emergent behaviors such as ball passing and interception. In addition, communication has also been shown to emerge from multi-agent RL (Sukhbaatar et al., 2016; Foerster et al., 2016; Lowe et al., 2017: Mordatch & Abbecl, 2018).
Intrinsic motivation methods have been widely studied in the literature (Chentanez et al., 2005; Singh et al., 2010). One example is count-based exploration, where agents are incentivized to reach infrequently visited states by maintaining state visitation counts (Strehl & Littman, 2008; Bellemare al., 2016; Tang et al., 2017) or density estimators (Ostrovski al., 2017: Burda et al., 2019b). Another paradigm are transition-based methods, in which agents are rewarded for high prediction error in learned forward or inverse dynamics model (Schmidhuber, 1991; Stadie et al.. 2015; Mohamed & Rezende, 2015; Houthooft et al., 2016; Achiam & Sastry, 2017: Pathak et al.. 2017; Burda et al., 2019a; Haber et al., 2018). Jaques et al. (2019) consider multi-agent scenarios and adopt causal influence as a motivation for coordination. In our work, we utilize intrinsic motivation methods as an alternative exploration baseline to multi-agent autocurricula. Similar comparisons have also been made in Haber et al. (2018) and Leibo et al. (2019b).
Tool use is a hallmark of human and animal intelligence (Hunt, 1996; Shumaker et al., 2011), how- ever, learning tool use in RL settings can be hard exploration problem when rewards are unaligned. For example, in Forestier et al. (2017); Xie et al. (2019) a real-world robot learns to solve various tasks requiring tools. In Bapst et al. (2019), agent solves construction tasks in a 2-D environment using both model-based and model-free methods. Allen et al. (2019) uses a combination of human- designed priors and model-based policy optimization to solve collection of physics-based puzzles requiring tool use. However, in each of these works, agents were explicitly incentivized to inter- act with and use tools, whereas in our environment agents implicitly create this incentive through multi-agent competition.
HIDE AND SEEK
Agents are tasked with competing in two-team hide-and-seck game in a physics-based environ- ment. The hiders are tasked with avoiding line of sight from the seekers, and the seekers are tasked with keeping vision of the hiders. There are objects scattered throughout the environment that the agents can grab and also lock in place. There are also randomly generated immovable rooms and walls that the agents must learn to navigate. Before the game of hide-and-seel begins, the hiders are given a preparation phase where the seekers are immobilized, giving the hiders chance to run away or change their environment.
There are no explicit incentives for agents to interact with objects in the environment; the only supervision given is through the hide-and-seck objective. Agents are given a team based reward; hiders are given reward of if all hiders are hidden and -1 if any hider is seen by a seeker. Seekers are given the opposite reward, 1 if all hiders are hidden and +1 otherwise. To confine agent behavior to reasonable space, agents are penalized with a reward of -10 if they go too far outside of the play area (outside an 18 meter square). An episode lasts 240 timesteps, the first 40% of which are the preparation phase where all agents are given zero reward.
We simulate the environment in the MUJOCO physics engine (Todorov et al., 2012). The world is populated with 1 to 3 hiders, to 3 seekers, 3 to 9 movable boxes of which at least are elongated, movable ramps, and randomly generated static walls and rooms. We also experiment with a simpler, less randomized environment described in Appendix A.3, Agents observe the position, velocity, and size (in the case of the randomly shaped boxes) of objects and other agents. If entities are not in ofne-of-sight of the agent or not in a 135 degree cone in front of the agent, then they are masked out in the policy. Agents also have 30 range sensors arrayed evenly around them, similar to a lidar. In addition, each agent observes its own team and other agents' teams as well as how much time is left. in the preparation phase.
Agents are simulated as spherical objects and have 3 action types that can be chosen simultaneously at each time step. They may move by setting discretized force along their x and y axis and torque around their z-axis. They have a single binary action to grab objects, which binds the agent to the closest object while the action is enabled. Agents may also lock objects in place with single binary action. Objects may be unlocked only by agents on the team of the agent who originally locked the object. Agents may only grab or lock objects that are in front of them and within small radius.
4

---

Published as a conference paper at ICLR 2020
4 POLICY OPTIMIZATION
Agents are trained using self-play, which acts as a natural curriculum as agents always play oppo- nents of an appropriate level.
Agent policies are composed of two separate networks with different parameters a policy network which produces action distribution and critic network which predicts the discounted future turns. Policies are optimized using Proximal Policy Optimization (PPO) (Schulman et al., 2017) and Generalized Advantage Estimation (GAE) (Schulman et al.. 2015), and training is performed using rapid (OpenAI, 2018), large-scale distributed RL framework. We utilize decentralized execution and centralized training. At execution time, each agent acts given only its own observations and memory state. At optimization time, we use centralized omniscient value function for each agent, which has access the full environment state without any information masked due to visibility, similar to Pinto et al. (2017); Lowe et al. (2017); Foerster et al. (2018).
In all reported experiments, agents share the same policy parameters but act and observe indepen- dently; however, we found using separate policy parameters per agent also achieved all six stages of emergence but at reduced sample efficiency.
Poliey Architecture
Generate maski from finntal vitton cone sight
Eniity Embecdings
Chiulia 'D Convalution
Fully Connwited
ITBAR
Connected
nty-
other
xim
Masked
Arerage Pocting
LSTN
Connected
Morument
Oralebing
Locking
nampp
Accion
Figure 2: Agent Policy Architecture. All entities are embedded with fully connected layers with shared weights across entity types, e.g. all box entities are encoded with the same function. The policy is ego-centric so there is only one embedding of "self" and (#agents - 1) embeddings of other agents. Embeddings are then concatenated and processed with masked residual self-attention and pooled into fixed sized vector (all of which admits variable number of entities). æ and stand for state (position and orientation) and velocity.
We utilize entity-centric observations (Džeroski al., 2001: Diuk et al., 2008) and use attention mechanisms to capture object-level information (Duan et al., 2017; Zambaldi et al., 2018). As shown in Figure 2 we self-attention (Vaswani et al., 2017) based policy architecture entities, which is permutation invariant and generalizes to varying number of entities. More details be found in Appendix B.

---

Published as a conference paper at ICLR 2020
Object Movement
Object Locking
2.5
Number Looked
Distance (m)
2.0
155
2
0.5
100
200
300
400
600
100
200
500
400
500
Episodes (x109)
Episodes (x10%)
Max Box Movernnt
Mex Remp Moverentt
Box locked Number Box Locked - Prep
Number Ramp locked
Max Box Movement- Prep
Mex Ramp Movement- Prea
Number Ramp Locked Prep
Figure 3: Environment specific statistics used to track stages of emergence in hide-and-seck. We plot the mean across seeds with each individual seed shown in a dotted line, and we overlay the emergent phases of strategy: (1) Running and Chasing, (2) Fort Building, (3) Ramp Use, (4) Ramp Defense, (5) Box Surfing, (6) Surf Defense. We track the maximum movement of any box or ramp during the game well during the preparation phase (denoted with "Prep"). We similarly track how many objects of each type were locked at the end of the episode and preparation phase. As agents train, their interaction with the tools in their environment changes. For instance, the agents learn to build forts they move boxes and lock boxes much more during the preparation phase.
5 AUTO-CURRICULA AND EMERGENT BEHAVIOR
As agents train against each other in hide-and-seek, many as six distinct strategies emerge, each of which creates a previously non-existing pressure for agents to progress to the next stage. Note that there are no direct incentives for agents interact with objects strategies are solely a result of the autocurriculum induced by multi-agent competition. Figure shows the progression of emergent strategies agents learn in our environment (see Appendix A. for trajectory traces of each strategy²).
explore, but rather the emergent
Initially, hiders and seekers learn to crudely run away and chase. After approximately 25 million episodes of hide-and-seck, the hiders learn to use the tools at their disposal and intentionally modify their environment. They begin to construct secure shelters which to hide by moving many boxes together or against walls and locking them in place. After another 75 million episodes, the seekers also learn rudimentary tool use; they learn to move and use ramps to jump over obstacles, allowing them to enter the hiders' shelter. 10 million episodes later, the hiders learn to defend against this strategy; the hiders learn to bring the ramps to the edge of the play area and lock them in place, seemingly removing the only tool the seekers have at their disposal.
We originally believed defending against ramp use would be the last stage emergence in this environment; however, were surprised to find that yet two more qualitatively new strategies emerged. After 380 million total episodes of training, the seekers learn to bring a box to the edge of the play area where the hiders have locked the ramps. The seekers then the ramp move top of the box and surf it to the hiders' shelter. This is possible because the agents' movement action allows them apply force on themselves regardless of whether they are on the ground or not; they do this while grabbing the box under them, the box will move with while they are on top of it. In response, the hiders learn lock all of the boxes in place before building their shelter.
In all stages of strategy agents must learn to coordinate within their team. Similar to Liu et al. (2019), we use team-based rewards such that agents are required collaborate in order to succeed; however, in our work we require neither copulation-based training (Jaderberg et al., 2017) or evolved dense rewards (Jaderberg et al., 2019). Notably, hiders learn efficient division of labor; for instance, when constructing shelter they often separately bring their own box to the construction area. Furthermore,
"See openai com '1 og/ emergent-too
for sample vidcos.
'Note that the discovery of a new skill does not necessarily correlate to the reward of team changing. For example, the hider reward still decreases even after the discovery of surf defense, which is likely hecause teams with hiders often do have enough time lock all of the boxes in play.

---

Published as a conference paper at ICLR 2020
as hiders attempt to defend against box surfing by locking boxes in place during the preparation phase, we find that 2 and hider teams are able to lock 25% and 36% more boxes, respectively, than single hider team, indicating that larger teams are able to divide labor and accomplish objective a smaller team could not.
Many stages of emergent strategy be mapped behavioral shifts in the way agents interact with the tools in their environment, similar to Leibo et al. (2017); Perolat et al. (2017). We therefore track basic statistics about the agents' interaction with objects during training, shown in Figure 3. For instance, as the hiders learn to build forts, they move and lock boxes much more during the preparation phase. Similarly, as the seekers learn to move and use ramps, the ramp movement the main phase of the game increases, and as they learn to "box surf"' there is slight increase in the box movement during the main phase of the game. Finally, as the hiders learn to defend against this strategy by locking all boxes in place, the number of locked boxes in the preparation phase increases.
We found that scale plays a critical role in en- abling progression through the emergent au- The default
Batch Size
tocurricula in hide-and-seek. model, which uses a batch size of 64,000 and
45744850
140
# Episodes (x106)
120
Time (Hours))
1.6 million parameters, requires 132.3 million episodes (3 1.7 billion frames) over 34 hours of training to reach stage of the skill progres- sion, i.e. ramp defense. In Figure 4 we show the effect of varying the batch size in our agents ability to reach stage We find that larger batch sizes lead to much quicker training time by virtue of reducing the number of required optimization steps, while only marginally af- fecting sample efficiency down to batch size of 32,000; however, we found that experiments with batch sizes of 16,000 and 8,000 never con- verged.
180
100
360 dk
50
32k
64k
128k
Batch Size
Figure 4: Effect of Scale on Emergent Autocur- ricula. Number of episodes (blue) and wall clock time (orange) required to achieve stage 4 (ramp defense) of the emergent skill progression pre- sented in Figure chunks, each of which consists of 10 contiguous transitions (the truncation length for backpropaga- tion through time).
Batch size denotes number of
We find the emergent autocurriculum to be fairly robust as long as we randomize the en- vironment during training. reduced, find that fewer stages of the skill progression emerges, and at times less sophisticated strategies emerge instead (e.g. hiders can learn to run away and use boxes as moveable shields.); Appendix A.2 for more details. In addition, we find that design choices such the minimum number of elongated boxes or giving each agent their own locking mechanism instead of a team based locking mechanism can drastically increase the sample complexity. We also experimented with adding additional objects and objectives to our hide-and-seed environment as well as with several game variants instead of hide-and-sedd (see Appendix A. We find that these alternative environments also lead to emergent tool use, providing further evidence that multi-agent interaction a promising path towards self-supervised skill acquisition.
randomization is
EVALUATION
In the previous section we presented evidence that hide-and-seek induces a multi-agent autocur- riculum such that agents continuously learn new skills and strategies. As is the case with many unsupervised reinforcement learning methods, the objective being optimized does not directly in- centivize the learned behavior, making evaluation of those behaviors nontrivial. Tracking reward is an insufficient evaluation metric in multi-agent settings, as it can be ambiguous indicating whether agents are improving evenly or have stagnated. Metrics like ELO (Elo, 1978) or Trueskill (Herbrich et al., 2007) can more reliably measure whether performance is improving relative to previous pol- icy versions other policies in population; however, these metrics still do not give insight into whether improved performance stems from new adaptations or improving previously learned skills. Finally, using environment specific statistics such as object movement (see Figure 3) can also be ambiguous, e.g. the choice to track absolute movement does not illuminate which direction agents moved, and designing sufficient metries will become difficult and costly as environments scale.
7

---

Published as a conference paper at ICLR 2020
Agent Movement
Box Movement
Distance
5-
8
10
10
Samples (x109)
Samples (X10%)
Count-Dased, AAgent - Dux Position
Count-Daced, AAgent Full Count-Dased. 1-3 Agentss Full
RND, 1-3 Agents Full
Figure 5: Behavioral Statistics from Count-Basee Exploration Variants and Random Network Dis- tillation (RND) Across 3 Seeds. We compare net box movement and maximum agent movement between state representations for count-based exploration: Single agent, 2-D box location (blue); Single agent, box location, rotation and velocity (green); 1-3 agents, full observation space (red). Also shown is RND for 1-3 agents with full observation space (purple). We train all agents to convergence as measured by their behavioral statistics.
In Section 6.1, we first qualitatively compare the behaviors learned hide-and-seek to those learned from intrinsic motivation, common paradigm for unsupervised exploration and skill acquisition. In Section 6.2, we then propose suite of domain-specific intelligence tests to quantitatively measure and compare agent capabilities.
6.1 COMPARISON TO INTRINSIC MOTIVATION
Intrinsic motivation has become popular paradigm for incentivizing unsupervised exploration and skill discovery, and there has been recent success in using intrinsic motivation to make progress in sparsely rewarded settings (Bellemare et al., 2016; Burda et al., 2019b). Because intrinsically mo- tivated agents are incentivized to explore uniformly, it is conceivable that they may not have mean- ingful interactions with the environment (as with the "noisy-TV" problem (Burda et al., 2019a))- As proxy for comparing meaningful interaction in the environment, measure agent and object movement over the course an episode.
We first compare behaviors learned in hide-and-seel to count-based exploration baseline (Strehl & Littman, 2008) with an object invariant state representation, which computed in a similar way as in the policy architecture in Figure 2. Count-based objectives the simplest form of state density based incentives, where one explicitly keeps track of state visitation counts and rewards agents for reaching infrequently visited states (details can be found in Appendix D). In contrast the original hide-and-seek environment where the initial locations of agents and objects are randomized, we restrict the initial locations quarter of the game area ensure that the intrinsically motivated agents receive additional rewards for exploring.
We find that count-based exploration leads the largest agent and box movement if the state repre- sentation only contains the 2-D location of boxes: the agent consistently interacts with objects and learns navigate. Yet, when using progressively higher-dimensional state representations, such as box location, rotation and velocity or 1-3 agents with full observation space, agent movement and, in particular, box movement decrease substantially. This is a severe limitation because it indicates that, when faced with highly complex environments, count-based exploration techniques require identifying by hand the "interesting" dimensions state space that are relevant for the behaviors one would like the agents to discover. Conversely, multi-agent self-play does not need this degree of supervision. We also train agents with random network distillation (RND) (Burda et al., 2019b), an intrinsic motivation method designed for high dimensional observation spaces, and find it to perform slightly better than count-based exploration in the full state setting.
6.2 TRANSFER AND FINE-TUNING AS EVALUATION
We propose to use transfer to a suite of Nomain-spectio tasks in order to asses agent capabilities. To this end, we have created 5 benchmark intelligence tests that include both supervised and reinforce-
8

---

Published as a conference paper at ICLR 2020
Object Counting
Lock and Return
Requential
Blueprint Construction
Shelter Construction
10
Nomalized reverd
Aopureey
CA-
04
0.6
84
0A
84
0.0
0.0
0.0
Samples (XD7))
Samples 6:10P)
Samples (XT0")
Samples
Samples (x10P)
Pretrained in Hide-and and--eek
Trained From Soratch
Pretrained with Count-Based Intrinsic Motivation
Figure 6: Fine-tuning Results. We plot the mean normalized performance and 90% confidence inter- val across 3 seeds smoothed with an exponential moving average, except for Blueprint Construction where we plot over seeds due to higher training variance.
ment learning tasks. The tests use the same action space, observation space, and types of objects as in the hide-and-seek environment. We examine whether pretraining agents in our multi-agent envi- ronment and then fine-tuning them on the evaluation suite leads to faster convergence or improved overall performance compared to training from scratch or pretraining with count- based intrinsic mo- tivation. We find that 3 out of 5 tasks, agents pretrained in the hide-and-seek environment learn faster and achieve a higher final reward than both baselines.
We categorize the intelligence tests into domains: cognition and memory tasks, and manipulation tasks. We briefly describe the tasks here; for the full task descriptions, see Appendix C. For all tasks, reinitialize the parameters of the final dense layer and layernorm for both the policy and value networks.
Cognition and memory tasks:
In the Object Counting supervised task, we aim to measure whether the agents have a sense of object permanence; the agent is pinned to a location and watches right or left where they eventually become obscured by a wall. It is then asked to predict how many boxes have gone each side for many timesteps after all boxes have disappeared. The agent's policy parameters are frozen and we initialize a classification head off of the LSTT hidden state. In the baseline, the policy network has frozen random parameters and only the classification head off of the LSTM hidden state is trained.
boxes each randomly
to the
In Lock and Return we aim to measure whether the agent can remember its original position while performing new task. The agent must navigate an environment with 6 random rooms and 1 box, lock the box, and return to its starting position.
In Sequential Lock there are 4 boxes randomly placed in 3 random rooms without doors but with a ramp in each room. The agent needs to lock all the boxes in a particular order - a box only lockable when it is locked in the correct order which is unobserved by the agent. The agent must discover the order, remember the position and status of visited boxes, and use ramps to navigate between rooms in order to finish the task efficiently.
Manipulation tasks: With these tasks we aim to measure whether the agents have any latent skill representation useful for manipulating objects.
In the Construction From Blueprint task, there are cubic boxes in an open room and between and 4 target sites. The agent is tasked with placing a box each target site.
In the Shelter Construction task there are 3 elongated boxes, 5 cubic boxes, and one static cylinder. The agent is tasked with building shelter around the cylinder.

---

Published as a conference paper at ICLR 2020
Results: In Figure 6 show the performance on the suite of tasks for the hide-and-seek, count- based, and trained from scratch policies across 3 seeds. The hide-and-seek pretrained policy per- forms slightly better than both the count-based and the randomly initialized baselines in Lock and Retunn Sequential Lock and Construction from Blueprint; however, it performs slightly worse than the count-based baseline on Object Counting, and it achieves the same final reward but learns slightly slower than the randomly initialized baseline on Shelter Construction.
We believe the cause for the mixed transfer results is rooted in agents learning skill representa- tions that are entangled and difficult to fine-tune. We conjecture that tasks where hide-and-seek pretraining outperforms the baseline are due reuse of learned feature representations, whereas satter-than-baseliie transfer on the remaining tasks would require reuse of learned skills, which is much more difficult. This evaluation metric highlights the need for developing techniques to reuse skills effectively from policy trained in one environment to another. In addition, as future en- vironments become more diverse and agents must use skills in more contexts, we may see more generalizable skill representations and more significant signal in this evaluation approach.
In Appendix A.5 we further evaluate policies sampled during each phase of emergent strategy on the suite of targeted intelligence tasks, by which we can gain intuition as to whether the capabilities we measure improve with training. are transient and accentuated during specific phases, or gener- ally uncorrelated to progressing through the autocurriculum. Noteably, find the agent's memory improves through training as indicated by performance in the navigation tasks; however, perfor- mance in the manipulation tasks is uncorrelated, and performance in object counting changes seems transient with respect to source hide-and-seed performance.
7 DISCUSSION AND FUTURE WORK
We have demonstrated that simple game rules, multi-agent competition, and standard reinforcement learning algorithms at scale can induce agents to learn complex strategies and skills. We observed emergence of as many as six distinct rounds of strategy and counter-strategy, suggesting that multi- agent self-play with simple game rules in sufficiently complex environments could lead to open- ended growth in complexity. We then proposed use transfer a method to evaluate learning progress in open-ended environments and introduced suite of targeted intelligence tests with which to compare agents in our domain.
Our results with hide-and-seek should be viewed as a proof of concept showing that multi-agent autocurricula can lead to physically grounded and human-relevant behavior. We acknowledge that the strategy space in this environment is inherently bounded and likely will not surpass the six modes presented as is; however, because it built in high-fidelity physics simulator it is physically grounded and very extensible. In order to support further research in multi-agent autocurricula, are open- sppen-souriing our environment code.
Hide-and-seck agents require enormous amount of experience progress through the six stages of emergence, likely because the reward functions are not directly aligned with the resulting behav- ior. While we have found that standard reinforcement learning algorithms sample complexity in these systems will be an important line of future research. Better policy learn- ing algorithms or policy architectures are orthogonal to our work and could be used improve sample efficiency and performance on transfer evaluation metrics.
sufficient, reducing
We also found that agents were very skilled at exploiting small inaccuracies in the design of the environment, such as seekers surfing on boxes without touching the ground, hiders running away from the environment while shielding themselves with boxes, or agents exploiting inaccuracies of the physics simulations to their advantage. Investigating methods to generate environments with- out these unwanted behaviors is another import direction of future research (Amodei et al., 2016; Lehman et al., 2018).
ACKNOWLEDGMENTS
We thank Pieter Abbeel, Rewon Child, Jeff Clune, Harri Edwards, Jessica Hamrick, Joel Liebo, John Schulman and Peter Welinder for their insightful comments on this manuscript. We also thank Alex Ray for writing parts of our open sourced code.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Joshua Achiam and Shankar Sastry. Surprise-based intrinsic motivation for deep reinforcement learning. arXiv preprint urXiv:/703.0/755 2017.
Kelsey R Allen, Kevin A Smith, and Joshua B Tenenbaum. The tools challenge: Rapid trial-and- error learning in physical problem solving. arXiv preprint 0rXiv:1907.09620 2019.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in Al safety. arXiv preprint arXiv:/6066 660.0066556 2016.
Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pa- chocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. arXiv preprint arXiv: /808,00177, 2018.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.064400 2016.
Rene Baillargeon and Susan Carey. Core cognition and beyond: The acquisition of physical and numerical knowledge. In In Pauen (Fd.), Early Childhood Development and Later Outcome, pp. 33-65. University Press, 2012.
Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. Emergent com- plexity via multi-agent competition. In International Conference Learning Representations, 2018.
Victor Bapst, Alvaro Maachez-Gonzalez. Carl Doersch, Kimberly Stachenfeld, Pushmeet Kohli. Pe- ter Battaglia, and Jessica Hamrick. Structured agents for physical construction. In International Conference on Machine Learning, pp.
2019.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pp. 1471-1479, 2016.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A. Efros. Large-scale study of curiosity-driven learning. In International Conference on Learning Repre- sentations, 2019a.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019b.
AD Channon, RI Damper, al. Perpetuating evolutionary emergence. FROM ANIMALS TO ANI- MATS 5, pp. 534-539, 1998.
Nuttapong Chentanez, Andrew G Barto, and Satinder P Singh. Intrinsically motivated reinforcement learning. Advances in neural information processing systems, pp. 1281-1288, 2005.
Richard Dawkins and John Richard Krebs. Arms races between and within species. Proceedings of the Royal Society of London. Series Biological Sciences, 205(1 161):489-551, 1979.
Carlos Diuk, Andre Cohen, and Michael L Littman. An abject-oriented representation for efficient reinforcement learning. In Proceedings of the 25th international conference on Machine learning, pp. 240-247. ACM, 2008
Yan Duan, Marcin Andrychowicz. Bradly Stadie, Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning. In Advances in neural infor- mation processing systems. pp. 1087-1098, 2017.
Saão Džemski, Luc De Raedt, and Kurt Driessens. Relational reinforcement learning. Machine learning, 43(1-2).7-5,, 2001.
A.E. Elo. The rating of chessplayers, past and present. Arco Pub., 1978. ISBN 9780668047210.
URL https:ttp//.o //bokks googl
11

---

Published as a conference paper at ICLR 2020
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse cur- riculum generation for reinforcement learning. In Conference on Robot Learning, pp. 482-495 2017.
Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2137-2145, 2016.
Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Sébastien Forestier, Yoan Mollard, and Pierre-Yves Oudeyer. Intrinsically motivated goal explo- ration processes with automatic curriculum learning, arXiv preprint arXiv:1708. (000:100000190, 2017.
Nick Haber, Damian Mrowea, Stephanie Wang. Li Fei-Fei, and Daniel Yamins. Learning to play with aitrisicalllymmttive Systems, pp. 8388-8399, 2018.
self-aware agents. In Advances in Neural Information Processing
Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, Martin Riedmiller, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286: 0ww.000........... 2017.
Ralf Herbrich, Tom Minka, and Thore Graepel. Trueskill: bayesian skill rating system. In Ad- vances neural information processing systems, pp. 569-576, 2007.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural compulation, 9(8): 1735-1780, 1997.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pp. 1109-1117, 2016.
Gavin R Hunt. Manufacture and use of hook-tools by new caledonian crows. Nature, 379(6562): 249, 1996.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojeiech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, et al. Population based train- ing of neural networks. arXiv preprint 1rXiv: 2017.
Max Jaderberg, Wojciech M. Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castañeda, Charles Beattie, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nico- las Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo. David Silver, Demis Hassabis, Ko- ray Kavukcuoglu, and Thore Graepel. Human-level performance in 3d multiplayer games with population- based reinforcement learning. Science, 364(6443):859-8888 865, 2019. ISSN 0036-8075. doi: 10.1 hww.s/ceennc.aau6244.. URL. https: /ttes://sciieee sciencemag. org/ content/ 264//443// 889.
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcchre, Pedro Ortega, Dj Strouse, Joel Z Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International Conference on Machine Learning, pp. 3040-3049, 2019.
Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg. Julie Beaulieu, Peter J Bentley, Samuel Bernard, Guillaume Beslon, David M Bryson, et al. The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities. arXiv preprint arXiv: 180.003533.3 2018.
Joel Leibo, Vinicius Zambaldi, Mare Lanctot, Janusz Marecki, and Thore Graepel. Multi-agent reinforcement learning in sequential social dilemmas. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, pp. 464-473. International Foundation for Au- tonomous Agents and Multiagent Systems, 2017.
12

---

Published as a conference paper at ICLR 2020
Joel Z Leibo, Edward Hughes, Mare Lanctot, and Thore Graepel. Autocurricula and the emergence of innovation from social interaction: A manifesto for aulti-agent intelligence research. arXiv preprint arXiv:1903.0000 000440 2019a.
Joel Z Leibo, Julien Perolat, Edward Hughes, Steven Wheelwright, Adam Marblestone, Edgar Duéieez-uzzmán, Peter Sunehag, lain Dunning, and Thore Graepel. Malthusian reinforcement learning. Proceedings of the 18th International Conference on Autonomous Agents and Multi- Agent Systems, pp. 1099-1107. International Foundation for Autonomous Agents and Multiagent Systems, 2019b.
Siqi Liu, Guy Lever, Nicholas Heess, Josh Merel, Saran Tunyasuvunakool, and Thore Graepel. Emergent coordination through competition. In International Conference Leurning Represen- tations, 2019.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperafive-competiive environments. In Advances in Neural Information Pro- cessing Systems, pp. 6379-6390, 2017.
Shakir Mohamed and Danilo Jimenez Rezende, Variational information maximisation for intrinsi- cally motivated reinforcement learning. In Advances in neural information processing systems, pp. 2125-112211111 2233, 2015.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. In Thirty-Secone AAAI Conference on Artificial Intelligence, 2018.
Charles Ofria and Claus Wilke. Avida: A software platform for research in computational evolu- tionary biology. Artificial life, 10(2):19--299 2004.
OpenAl. OpenAl Five. https: /blog.lloo/. openai com/ hwtm//ppnaiiffinee 2018.
Georg Ostrovski, Marc G Bellemare, Aäron den Oord, and Rémi Munos. Count-based ex- ploration with neural density models. In Proceedings of the 34th International Conference on Machine Learning-Volume Volume 70, pp. 2721-2730. JMLR. org, 2017.
Jan Paredis. Coevolutionary computation. Artificial life, 2(4):355-375, 1995.
Deepak Pathak, Pulkit Agrawal, Alexci A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning, pp. 2778-27877 2017.
Julien Perolat, Joel Z Leibo, Vinicius Zambaldi, Charles Beattie, Karl Tuyls, and Thore Graepel. A multi-agent reinforcement learning model of common-pool resource appropriation. In Advances in Neural Information Processing Systems, 3643-3652, 2017.
Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojeiech Zaremba, and Pieter Abbeel. Asym- metric actor critic for image-based robot learning. arXiv preprint arXiv:1710.0654. 2017.
Jordan B Pollack, Alan D Blair, and Mark Land. Coevolution a backgammon player. In Artificial Life V: Proc. of the Fifth Int. Workshop on the Synthesis and Simulation of Living Systems, pp- 92-98. Cambridge, MA: The MIT Press, 1997.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: 1709. 10087, 2017.
Thomas S Ray. Evolution, ecology and optimization of digital organisms. Technical report, Citeseer, 1992.
Christopher Rosin and Richard Belew. Methods for competitive co-evolution: Finding oppo- nents worth beating. In ICGA, pp. 373-381, 1995.
Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building ral controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals to animats, pp. 222-227, 1991.
13

---

Published as a conference paper at ICLR 2020
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.024338 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Robert Shumaker, Kristina R Walkup, and Benjamin B Beck. Animal tool behavior: the use and manufacture of tools by animals. JHU Press, 2011.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint 172.08177, 2017.
Karl Sims. Evolving virtual creatures. In Proceedings of the 21 annual conference on Computer graphics and interactive techniques, pp. 15-22. ACM, 1994a.
Karl Sims. Evolving 3d morphology and behavior by competition. Artificial life, 144::333-37, 1994b.
Satinder Singh, Richard Lewis, Andrew G Barto, and Jonathan Sorg. Intrinsically motivated reinforcement learning: An evolutionary perspective. IEEE Transactions on Autonomous Mental Development, 2(2):70-82, 2010.
Soros and Kenneth Stanley. Identifying necessary conditions for open-ended evolution through the artificial life world of chromaria. In Artificial Life Conference Proceedings 14, pp. 793-800. MIT Press, 2014.
Bradly Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv:1507.00884 2015.
Kenneth O Stanley and Risto Miikkulainen. Competitive coevolution through evolutionary com- plexification. Journal of artificial intelligence research, 21:63-100, 2004.
Alexander L Strehl and Michael Littman. An analysis of mosel-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 74(8): 1309-1311, 2008.
Sainbayar Sukhbaatar, Rob Fergus, al. Learning multiagent communication with backpropaga- tion. In Advances in Neural Information Processing Systems, pp. 2244 2252, 2016.
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fer- gus. Intrinsic motivation and automatic curricula via asymmetric self-play. In International Conference on Learning Representations, 2018
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. In Advances in neural information processing systems, pp. 2753-2762, 2017.
Tim Taylor. Requirements for open-ended evolution in natural and artificial systems. arXiv preprint arXiv:1507.074033 2015.
Gerald Tesauro. Temporal difference learning and d--aamon. Communications of the ACM, 38(3): 58 58-68,11 1995.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 8026-5033 IEEE, 2012.
14

---

Published as a conference paper at ICLR 2020
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008. 2017.
Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, al. AlphaS- tar: Mastering the real-time strategy game StarCraft II. https: /teepmind//eedddd.//m/ com / blog/
-at.eeessrattt-a-aaattata-re-eee sta.ueettstrrrratit.aaaaaa 2019.
Rui Wang. Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv: 1901.00/90....3.. 2019.
Annie Xie, Frederik Ebert, Sergey Levine, and Chelsea Finn. Improvisation through physical un- derstanding: Using novel objects as tools with visual foresight. arXiv preprint arXiv: 2019.
Larry Yaeger. Computational genetics, physiology. metabolism, neural systems, learning, vision, and behavior or poly world: Life a new context. In SANTA FE INSTITUTE STUDIES IN THE SCIENCES OF ADDISON-WESLEY PUBLISHING CO, 1994.
VOLUME,, volume 17, pp. 263-263.
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Relational deep reinforcement learning. arXiv preprint arXiy: /80600/ 0/000/83,, 2018.
15

---

Published as a conference paper at ICLR 2020
Appendix
Table of Contents
A Further Emergence Results
17
A.1
Trajectory Traces From Each Stage of Emergent Strategy
17
A.2 Dependence of Skill Emergence on Randomness in the Training Distribution of Environments
17
A.3
Quadrant Environment
18
A.4
Further Ablations
19
A.5 Evaluating Agents at Different Phases of Emergence
19
Altemative Games Hide-and-Seek with Secondary Objectives
20
A.7
Zero-shot generalization
22
B Optimization Details
23
B.1
Notation
23
B.2 Proximal Policy Optimization (PPO)
23
B.3 Generalized advantage estimation
23
B.4 Normalization of ohservations, advantage targets and value function targets
24
B.5 Optimization setup
24
B.6 Optimization byperparameters
24
B.7 Policy architecture details
24
C Intelligence Test Suite Details
25
C.I Cognition and memory task
25
C.2 Manipulation task
25
Intrinsic Motivation Methods
26
D.1 Counted-Based Exploration
26
D.2 Random Network Distillation
27
D.3 Fine-tuning From Intrinsic Motivation Variants
27
16

---

Published as a conference paper at ICLR 2020
A FURTHER EMERGENCE RESULTS
A.1 TRAJECTORY TRACES FROM EACH STAGE OF EMERGENT STRATEGY
Running and Chasing
Fort building
Ramp use
Ramp defense
Box surfing
Surf defense
Figure A.1: Trajectory Traces From Each Stage of Emergent Strategy. Rows correspond to different strategies, columns to characteristic snapshots in chronological order within an episode that show- case the strategy. Running and chasing: Hiders (blue) try to hide behind walls and moveable objects, and seekers (red) chase and search for hiders. Fort building: During the preparation phase (first snapshots), hiders build fort out of 3 boxes and hide inside. Seekers are not able to overcome this barrier during the remainder of the trial. Ramp use: During the preparation phase, hiders build fort out of 2 boxes and wall (Ist and 2nd snapshot). After the preparation phase, seeker grabs ramp (3rd snapshot), drags it towards the fort (4th snapshot) and uses it to enter the fort (5th snap- shot). Ramp defense: Hiders prevent ramp use by dragging ramps away and locking them in place during the preparation phase (Ist, 2nd and 3rd snapshot). They then build fort in safe distance from the ramps (4th snapshot). After the preparation phase, seekers are not able to enter the fort (5th snapshot). Box surfing: A single hider locks both ramps in place and hides inside fort built from boxes and an interior wall during the preparation phase (Ist and 2nd snapshot). The seeker grabs a square box (3rd snapshot) and drags it next to one of the locked ramps to jump top (4th snapshot). It then "surfs" on the box to the edge of the fort and jumps inside (5th snapshot). Surf defense: During the preparation phase, hiders lock all ramps and boxes in place (Ist, 2nd and 3rd snapshot) and construct a fort out of 3 boxes (4th snapshot). After the preparation phase, seekers are not able to enter the fort (5th snapshot).
A.2 DEPENDENCE SKILL EMERGENCE ON RANDOMNESS THE TRAINING DISTRIBUTION OF ENVIRONMENTS
Discovering tool use is difficult exploration problem, as only a very specific set of actions will lead to an increase in reward. During training, we find that randomization over many components of our environment, such as the number of agents in each team, the number of boxes, the shape of boxes (square elongated), the initial location of agents and objects and the presence and location of interior and exterior walls helps emergence, despite leading to a broader training distribution. As successively reduce the amount of randomization, find that fewer stages of skill progression emerges, and with at times less sophisticated strategies (c.g. hiders learn to run away and use boxes moveable shields), Table A.
17

---

Published as a conference paper at ICLR 2020
initial
team size
# boxes
box shape
location
walls
emergence
1-3
3-9
random random random random fixed random random
random
random
stages
1
3-9
stages
1
7
6 stages
1-3
3-9
random random no walls
stages
1
3-9
random random no walls
2 stages"
1
7
fixed random no walls
2 stages"
1
7
fixed fixed no walls
stages
Hiders run away and use boxes as movable shields.
Table A.I: Number of stages of emergence for different levels of environmental randomization (batch size is held constant).
Running and chasing
Fort building
Ramp use
Ramp defense
Figure A.2: Sample Trajectory Traces From Each Stage Emergent Strategy in the Quadrant En- vironment. Rows correspond to different strategies, columns characteristic snapshots in chrono- logical order within an episode that showcase the strategy. Rumning and chasing: Hiders (blue) try hide behind walls and moveable objects, seekers (red) chase and search for hiders. Fort building: During the preparation phase (first snapshots), hiders push the two boxes into the two doors of the lower right room and lock them in place. Seekers are able to overcome this barrier during the remainder of the trial. Ramp use: During the preparation phase, hiders secure the room using the two boxes (Ist and 2nd snapshot). After the preparation phase, a seeker grabs the ramp (3rd snapshot), drags it towards the walls of the room (4th snapshot) and both seekers use the ramp to enter the room (5th snapshot). Ramp defense: Hiders prevent ramp use by grabbing the ramp during the preparation phase (2nd snapshot), pulling it inside the room (3rd snapshot), and then secure the doors using the blocks (4th snapshot). After preparation phase, the seekers are not able to enter the room (5th snapshot). Also, note that while one hider brings the ramp into the room, the other hider drags box out of corner and then passes it to its team mate (3rd and 4th snapshot), evidence of sophisticated coordination between hiders.
A.3 QUADRANT ENVIRONMENT
As well as the default random rooms environment described above, we also train and evaluate agents the much simpler quadrant environment (Figure A.2) for illustration purposes. This environment consists of a room in the lower right corner that is connected by 1-2 doors to the remainder of the environment. Unlike the main environment, the interior walls are fixed and only the location of the doors vary from episode to episode. Hiders seekers are only spawned outside the room. Two cubic boxes are spawned inside the room and one ramp is spawned inside outside the room. Both hiders and seekers receive negative reward if they leave the environment or if they bring an object outside the environment. Also, in contrast to the main environment, only boxes, but not ramps be locked in place.
spawned randomly in the environment, but
18

---

Published as a conference paper at ICLR 2020
The quadrant environment gives rise to similar skill progression in the main environment (Figure A.2): basic running and chasing, then fort building by hiders, ramp use by seekers, and ramp defense by hiders. Hiders create shelter by pushing boxes into the doors of the room in the lower right corner, preventing seekers from entering. Seekers drag ramps to the exterior walls of the room and then use them to jump inside. As ramps are not lockable in this environment, hiders instead defend against ramp use by pulling the ramp inside the room before securing the wall opening with boxes.
A.4
FURTHER ABLATIONS
Omniscient Value Function (Emergence Phase 3)
Entity Aggregation
89:2M Ep 22.1H
Episodes (x10%)
160
34.0
145.4MF
Episodes (x109)
80-
76.6M
19.7 Hr
-20
(HourS)
120-
Fp
30.8
30
Time (Hours)
60-
15
80
20
40
20
-5
40
10
Omniscient
Masked
Seif Attention
Pooling
Figure A.3: Effect of Omniscient Value Function and Pooling Architectures on Emergent Autocur- ricula. In blue we show the number of episodes required and in orange the wall clock time required achieve stage (ramp defense) of the emergent skill progression presented in Figure for varying batch and model sizes. Note that in our ablation comparing omniscient and masked value functions, the experiment with a masked value function never reached stage the allotted time, so here we compare timing to stage 3.
In Figure A.3 we compare the performance between a masked and omniscient value function as well a purely pooling architecture versus self self-attention. We find that using an omniscient value function, meaning that the value function has access to the full state of the unobscured environment, is critical to progressing through the emergent autocurricula at the given scale. We found that with the same compute budget, training with a masked value function never progressed past stage (ramp usage). We further found that our self-attention architecture increases sample efficiency as compared to an architecture that only embeds and then pools entities together with a similar number of parameters. However, because self-attention requires more compute despite having the same number parameters, the wall-clock time to convergence is slightly slower.
A.5 EVALUATING AGENTS AT DIFFERENT PHASES OF EMERGENCE
Object Counting
Look and Return
Sequential Lock
Blueprint Construction
Shelter Construction
10
Normalzad roward
Normalted roward
0.8
0.8
08
6.
Accurecy
06.
0.6
06
Normolized
0.4
84
04
0.4
0.2
8
0.2
0.
0.0
0.0
0.2
0.
06
0.5
10
Somples
Samples (x10*)
Somples (x10%)
Samples
Samples (x10P)
Phose 1
Phose 2
Phase 3
Phese
Phase 5
Phose 6
Bascline
Figure A.4: Fine-tuning From Different Phases in the Emergent Autocurriculum. We plot the mean performance on the transfer suite and 90% confidence interval across seeds and smooth perfor- mance and confidence intervals with an exponential moving average. We show the fine-tuning per- formance for a policy sampled at each of the six phases of emergent strategy (see Figure 1).
In Figure A.4 we evaluate policies sampled during each phase of emergent strategy on the suite of targeted intelligence tasks, by which we can gain intuition as to whether the capabilities we measure improve with training, are transient and accentuated during specific phases, or generally uncorre- lated to progressing through the autocurriculum. We find that the hide-and-seck agent improves on the navigation and memory tasks it progresses; notably on Lock and Return, the performance monotonically increases with emergence phase, and the policy from the phase performs 20% bet- ter than the policy from phase 1. However, performance on Object Counting transient; during
19

---

Published as a conference paper at ICLR 2020
Figure A.5: Example trajectory in hide-and-seek environment with additional food reward. During the preparation phase, hiders lock the ramps at the boundary of the play area (2nd snapshot) and construct a fort around the food reward (3rd and 4th snapshot). After the preparation phase, hiders can eat food and receive the additional food reward, because they are hidden from seekers inside the fort.
phase the hide-and-seek agent performs extremely well, much better than all baselines and other phases but loses this ability in later stages. Finally, we find that performance on the manipulation tasks is relatively uncorrelated the phases of emergence, and surprisingly the policy transferred from phase 1, the phase before any tool use emerges, performs comparably well to other phases.
A.6 ALTERNATIVE GAMES TO HIDE-AND-SS AND-SEEK WITH SECONDARY OBJECTIVES
A.6.1 HIDE-AND-SEEL WITH FOOD REWARD
In the main hide-and-seek environment, hiders build forts at locations that would give them the best chance of staying hidden from seekers. To test whether hiders could be incentivized by secondary objective to adapt the location of forts, we add additional food rewards and test whether hiders would bias the location of their forts towards the location of the food. In this environment, hiders can eat food and receive food rewards only under the following conditions: after the preparation phase, when all hiders are hidden from seekers and when the food is sufficiently close and visible to them. Therefore, just chasing after food would not be an effective strategy for hiders, because hiders receive neither hide-and-seek nor food rewards if or more hiders seen by seeker. Instead, hiders are incentivized to build forts around the location of food and then eat the food while being unobserved by seekers (figure A.5).
Food is distributed in the form of 5 food pellets that are spawned close to each other in rectangular area in the center of the environment whose side length is 1/4 of the room size. Each food pellet can provide a positive food reward of +1 for each time step. Food rewards are shared between hiders, irrespective of which hider eats a food item. Hiders (as well as seekers) can observe the locations of food pellets as separate entities; if food pellets are obstructed outside the field of view they are masked out like other type of objects.
As shown in figure A.6, this environment gives rise to four levels of skill progression, similar to the one of the main environment: basic running and chasing, then fort building by hiders, ramp use by seekers, and ramp defense by hiders. Moreover, hiders consume food, and food consumption is highly correlated with their ability to construct stable forts; food consumption decreases during the initial phase seekers get better at chasing and therefore prevent hiders from eating food. Food consumption then increases again as hiders learn to construct forts and shield themselves from the view of seekers, but plateaus once seekers learn use ramps. Finally, food consumption rises again as hiders get better at defending against ramp use.
A.6.2 HIDE-AND-SEE WITH DYNAMIC FOOD
In the food variant introduce in Sec. A.6.1, the location of food pellets is fixed throughout the episode. Here we consider a dynamic food variant such that a food pellet will be eaten up, i.e., disappear, when hider is close to it, and then new food pellet will show up in different location but still within the center region. More precisely, the game area is simply an empty room without outside walls containing 2 seekers, 3 hiders, 1 dynamic food and 8 elongated boxes. The food will be always located within a square in the center of the environment with side length 1/5 the game area size. We inherit the same reward and policy structures from the previous game.
In this game, merely building a center fort is not sufficient for obtaining the highest reward since the food might disappear and respawn outside the fort. The agent must ensure that the fort is large
20

---

Published as a conference paper at ICLR 2020
Reward
Food consumption
700
300
600
Reward
Food units
100
300
200
100
-100
100
200
330
400
0
Episodes (x10%)
200
300
400
If Episodes (x109)
Hiders
Seakers
Object Movement
Object Locking
2.5
Distance (m)
2.0
Number locked
1.5
1.0
10
0.5
0.6
0.
400
Episodes (x109)
Episodes (x109)
Max Box Movement
Max Ramp Moverent
Number Box Looked
Number Ramp Looked
Max Box Movemmntt- Prep
Max Remp Movernntt Prep
Number Box Locked - Prep
Number Remp Locked Propi
Figure A.6: Reward and environment specific statistics during emergence in hide-and-seek with secondary food rewards for hiders.
enough such that all the possible spawning positions of the food will be inside the fort. Such behavior does emerge after training for around 4.5 10t0 samples.
We also experimented on variants where the food spawning region has different side length. When the side length reduced to 1/6 of the game area, the same behavior emerges faster taking 1.5 x 10t0 samples. However, when the side length of the dynamic food region is increased to 1/4 of the game area, hiders converge to a policy that ignores the food and only builds a small fort to protect themselves.
A.6.3 FOOD PROTECTION GAME
In the previous two variants, introduce extra food reward to the hider in addition to the original hide-and-seek reward for promoting more goal-oriented behavior. Now we consider different game rule such that the competition between hiders and seekers only depends on the food collecting reward and show that this rule can also lead to tool use and complex behavior.
We consider an empty game area surrounded by walls that contains 50 food pellets randomly dis- tributed in center square of size 2/3 of the game environment. There are 3 hiders, seekers and 7 elongated boxes in the game, The only goal for seekers is to collect food. Once food pellet is collected by any of the seeker, a +3 reward will be given to all the seekers and then the food will disappear permanently from the game. The goal for the hider is protect the food from seekers and their reward is simply the negative value of seekers. Each episode consists of 200 time steps. The preparation period, in which only hiders can move, extends to the first 100 time steps. The last 60 steps of the game corresponds to food collecting period, during which the hiders cannot move. Additionally, after the preparation phase we also add a "boundary penalty" for hiders when they too close to the wall to ensure they stay within the food region.
Our initial motivation for this task was to promote hiders to learn to construct complex and large fort structures with a more direct competition pressure. For example, expected that the agents would leam to build a large fort that would cover as much food as possible. Unfortunately, such behavior did not emerge. However, after around x 1010 samples of training, the hiders surprisingly discover
21

---

Published as a conference paper at ICLR 2020
Figure A.7: Example trajectory in the food protection game. During the preparation phase, the hiders first explore the room to find seekers, then physically push the seekers the corner and eventually collaboratively grab elongated boxes to build a small fort surrounding the seekers to trap them there. To secure the trap, they even learn build the fort using two elongated boxes to prevent seekers from escaping.
Hiderreard
Hider reward
8 g 8 8 8
Hider reward
sb 66 50
10.
3
10
Number ramps
Number
Figure A.8: Zero-shot generalization to a larger number of hiders (left), ramps (center) and boxes (right). The dotted line denotes the boundary of the training distribution (1-3 hiders, 2 ramps, 1-9
boxes). Error bars denote standard error of the mean.
extremely efficient and effective strategy: they learn to physically 'push' the immobile seekers to the corner of the environment during the preparation phase and then collaboratively use elongated boxes trap the seekers in the corner. Figure A.7 illustrates a trajectory corresponding to this strategy. Interestingly, the hiders demonstrate strong collaborations. Since there are seekers and hiders, when the 2 hiders are pushing the seekers to the corner, the 3rd hider will follow the 'pushing' hiders distantly, and once the pushing hiders finish the job, this 3rd hider will quickly grab box to build the trap without blocking the escaping route of those pushing hiders. We emphasize that this behavior was completely surprising to us and further indicates that complex behavior can emerge from multi-agent self-play with wide range competition rules.
Lastly, we also experimented with different versions of the close-to-wall wall "boundary penalties". In all cases, we observe similar fort-building behaviors, namely blocking the seekers to corner, with tiny variations. When remove the penalty, the behavior emerges twice faster (3.5 x 10t6 samples) and hider will stay in the fort with the seekers instead of getting outside. When we add the penalty throughout the episode, the hiders grabbing an elongated box as a tool to push the seekers towards the corner. However, the behavior emerges twice slower in this setting, i..., 1.4 10 samples required.
learned to stay in the center region while
A.7 ZERO-SHOT GENERALIZATION
We find that the trained policies can zero-shot generalize to environments with slightly larger number of hiders, ramps or boxes in Figure A.8. The hider reward provides reasonably good measure for generalization performance, since the modifications in question make constructing safe shelter more difficult. First, we looked at increasing the number of hiders from 1-3 hiders, as in the training distribution, to 4-6 hiders; increasing the number of hiders decreases the hider reward because all hiders need to stay hidden from seekers. However, the decline hider reward is very gradual, indicating that the policy generalizes well to more than 4 hiders. A similar effect occurs when increasing the number of ramps because hiders need to secure more ramps from seckers. we increase the number of ramps from 2 to 3 or 4 the hider reward drops only gradually. Finally, we find hider performance is remarkably stable, though still slowly declines, when increasing the number of boxes.
22

---

Published as a conference paper at ICLR 2020
B OPTIMIZATION DETAILS
B.1 NOTATION
We consider the standard multi-agent reinforcement learning formalism of N agents interacting with each other in an environment. This interaction is defined by set of states S describing the state of the world and configurations of all agents, set of observations 01 ..... ON of all agents, set of AN
actions ,,, AA of all agents, a transition function T S AN + S determining the distribution over next states, and reward for each agent i which a function of the state and the agent's s action. Agents choose their actions according to a stochastic policy no, O' x A + [0, 1}, where 0, are the parameters of the policy. In our formulation, policies are shared between agents, To; = Th and the set of observations O contains information for which role (e.g. hider or seeker) the agent will be rewarded. Each agent aims to maximize its total expected discounted return R = >> -- >'ri, where H is the horizon length and y is a time discounting factor that biases agents towards preferring short term rewards to long term rewards. The action-value function defined as QT at) st a:]. while the state-value function is defined as V**(s) E R[Re]e (x)]. The advantage function A (st, at) := (st. - V (st)
describes whether taking action az is better or worse for agent when in state s, than the average action of policy TT.
B.2 PROXIMAL POLICY OPTIMIZATION (PPO)
Policy gradient methods aim to estimate the gradient of the policy parameters with respect to the discounted sum of rewards, which is often non-differentiable. A typical estimator of the policy gradient is g == E[Á, V log np], where Å is estimate of the advantage function. PPO (Schulman et al., 2017), policy gradient variant, penalizes large changes to the policy to prevent training instabilities. PPO optimizes the objective L min(,,
(l) (l)) - c, + e) where (0) Roid (ax [ax] denotes the likelihood ratio between new and old policies and clip((, 6, 1 + E) clips 4 (0) in the interval [1 e, +e].
To( [x)
B.3 GENERALIZED ADVANTAGE ESTIMATION
We Generalized Advantage Estimation (Schulman et al., 2015) with horizon length H to estimate the advantage function. This estimator is given by:
H Σ bG 1=0
AM
8:1 Tri (V(++1411 - V(8+t))
where 8.+z is the TD residual, 7, A E [0, are discount factors that control the bias-variance tradeoff of the estimator, ((t) is the value function predicted by the value function network and we set (s:) 0 s; is the last step of an episode. This estimator obeys the reverse recurrence relation AP 6, + 1
We calculate advantage targets by concatenating episodes from policy rollouts and truncat- ing them to windows of T 160 time steps (episodes contain 240 time steps). window (so...., 8T-1) was generated in single episode use the advantage targets (Å/N- 1 AZ=!). If new episode starts at time step we use the advantage targets A
(Å !1 AA =1j-11 AFT A -I=T- (...))
Similarly we use as targets for the value function (HH=T H-T-1 G#= for a window gen-
erated by single episode and (G ,,6 j-1 ...., G#= G AI-Tij ....., GH==) if new episode
starts at time step j. where the return estimator is given by G : AN +V,,). This value function estimator corresponds to the TD(A) estimator (Sutton & Barto, 2018).
23

---

Published as a conference paper at ICLR 2020
B.4 NORMALIZATION OF OBSERVATIONS, ADVANTAGE TARGETS AND VALUE FUNCTION TARGETS
We normalize observations, advantage targets and value function targets. Advantage targets are z- scored over each buffer before each optimization step. Observations and value function targets are z-scored using a mean and variance estimator that is obtained from a running estimator with decay parameter 1 - 10-6 per optimization substep.
B.5 OPTIMIZATION SETUP
Training is performed using the distributed rapid framework (OpenAI, 2018). Using current policy and value function parameters, CPU machines roll out the policy in the environment, collect rewards, and compute advantage and value function targets. Rollouts are cut into windows of 160 timesteps and reformatted into 16 chunks of 10 timesteps (the BPTT truncation length). The rollouts are then collected in training buffer of 320,000 chunks. Each optimization step consists of 60 SGD substeps using Adam with mini-batch size 64,000. One rollout chunk is used for at most 4 optimization steps. This ensures that the training buffer stays sufficiently on-policy.
B.6 OPTIMIZATION HYPERPARAMETERS
Our optimization hyperparameter settings are as follows:
Buffer size
320,000
Mini-batch size
64,000 chunks of timesteps
Learning rate
3 10
PPO clipping parameter
0.2
Gradient clipping
Entropy coefficient
0.01
0.998
0.95
Max GAE horizon length T
160
BPTT truncation length
10
B.7 POLICY ARCHITECTURE DETAILS
Lidar observations are first passed through a circular 1D-convolution and concatenated onto the agents representation of self, Tsets. Each object is concatenated with Jselr and then embedded with a dense layer where parameters are shared between objects of the same type, e.g. boxes share the same embedding weights. All the embedded entities then passed through residual self- attention block, similar to Vaswani et al. (2017) but without position embeddings, in the form of y dense(schl attention(z)) + z. We then average-pool entity embeddings and concatenate this pooled representation to Jself- Note that in the policy network the entities not observed by each agent masked away through self-attention and pooling. Finally, this pooled representation is passed through another dense layer and an LSTM (Hochreiter & Schmidhuber, 1997) before pulling off separate action heads for each of the 3 action types described in Section 3. We also add layer normalization (Ba et al., 2016) to every hidden layer of the policy network except the ID-convolution layer. We empirically observe that layer normalization leads to faster training and better transfer performance.
Size of embedding layer
128
Size of MLP layer
256
Size of LSTM layer
256
Residual attention layer
attention heads of size 32
Weight decay coefficient
10 6
24

---

Published as a conference paper at ICLR 2020
C INTELLIGENCE TEST SUITE DETAILS
All evaluation tasks in the intelligence test suite single-agent tasks. In order to align the obser- vations between the hide-and-seck environment and the evaluation tasks, we add fake hider obser- vations, preparation phase observations and set them all to O.
C.1
COGNITION AND MEMORY TASK
All the variations of cognition and memory tasks have horizon of 120 timesteps, and all boxes cuboids.
Object Counting: An agent is pinned in place and watches as boxes move either to left or right at random. The boxes disappear behind walls such that agent how many boxes went left or right far after all the boxes have disappeared. In this test we evaluate the quality of the existing representation by holding the agent's policy fixed and only train new classification head from the agent's s LSTM state. The classification head first passes the LSTM state through layernorm and single dense layer with 64 units. We then do 7-class classification predicting whether 0 through boxes have gone to the left.
longer them, and the agent is asked to predict
Lock and Return:
In this game, the agent needs to navigate towards a hidden box, lock it, and then return to its starting position.
The game area has 6 randomly generated connected rooms with static walls and box. When the box is locked, the agent will be given reward of +5. If the agent unlocks the box during the episode, a -5 penalty will be given. Additionally, if the box remains unlocked at the end of the episode, the agent will be given another -5 penalty. returns to its starting location within 0.1 radius and with the box locked. For promoting fast task accomplishment, we give the agent +1 reward for each timestep of success. We also introduce shaped reward with coefficient of 0.5 for easier learning: at each time step, the shaped reward is the decrement in the distance between the agent location towards the target (either the unlocked box or the starting location).
success determined when the agent
Sequential Lock:
There are boxes and the agent needs to lock all the boxes in an unobserved order sequentially. A box can be locked only if it is locked in the right order.
The game area is randomly partition into three rooms with 2 walls. The 4 boxes are randomly placed the game area. Each room has a ramp. The agent has to utilize the ramps to navigate between rooms. When box is successfully locked (according to the order), +5 bonus is given. If a box unlocked, -5 penalty will be added. When all the boxes get locked, the agent will receives a +1 per-timestep success bonus. We also use the same shaped distance reward as the lock and return task here.
C.2 MANIPULATION TASK
All variations of the manipulation task have boxes, but no ramps.
Construction from Blueprint:
The horizon is at most 240 timesteps, but an episode can end early if the agent successfully finishes the construction. The game area is empty room. The locations of the construction sites sampled uniformly at random (we use rejection sampling to ensure that construction sites do not overlap).
For each construction site, agents observe its position and that of its 4 corners. Since there are no construction sites in the ande-and-seek game and the count-based baseline environments, need to change our policy architecture to integrate the new observations. Each construction site observation concatenated with s and then emhedded through new dense layer shared across all sites. This
25

---

Published as a conference paper at ICLR 2020
dense layer is randomly initialized and added the multi-agent and count-basee policies before the start of training. The embedded construction site representations are then concatenated with all other embedded object representations before the residual self- attentio block, and the rest of the architecture the same as the one used in the hide-and-seek game.
The reward at each timestep is equal to reward scale constant times the mean of the smooth min- imum of the distances between each construction site corner and every box corner. Let there be construction sites and n boxes, and let dij be the distance between construction site corner i and box corner j. and let de be the smooth minimum of the distances from construction site corner to all box corners. The reward at each timestep follows the following formula:
An
An
di =
Σ
digends
c
odg
Vi
1,2,, 1.2..... 4k
f=1
4k
TEw
8d
Σ dj
/4k
Here, Sa is the reward scale parameter and a is the smoothness hyperparameter (a must be non- -oo gives us the regular min function). In addition,
positive; = 0 gives us the mean, and when all construction sites have a box placed within certain distance dmnn them, and all con- struction site corners have box corner located within dmai of them, the episode ends and all agents receive reward equal to 8, * k; where & is separate reward scale parameter. For our experiment, 8 and & is randomly sampled between 1 and 4 (inclusive) every episode. The hyperparameter values we use for the reward are the following:
1.5
P 0.05
min 0.1
8c
Shelter construction: The goal of the task is to build shelter around a cylinder that is randomly placed in the play area. The horizon is 150 timesteps, and the game area is an empty room. The location of the cylinder is uniformly sampled at random minimum distance away from the edges of the room (this is because if the cylinder is too close to the external walls of the room, the agents physically unable to complete the whole shelter). The diameter of the cylinder is uniformly randomly sampled between dmin and dmaa-- There are movable elongated boxes and movable square boxes. There are 100 rays that originate from evenly spaced locations the bounding walls of the room and target the cylinder placed within the room. The reward at each timestep is (-n 8), where n is the number of raycasts that collide with the cylinder that timestep and the reward scale hyperparameter.
We use the following hyperparameters:
0.001
dmin 1.5
dmaz 2
D INTRINSIC MOTIVATION METHODS
We inherit the same policy architecture as well as optimization hyperparameters as used in the hide- and-seek game.
Note that only the Sequential Lock task in the transfer suite contains ramps, so for the other 4 tasks remove ramps in the environment for training intrinsic motivation agents.
D.1
GOUNTED-BASED EXPLORATION
For each real value from the continuous state of interest, we discretize it into 30 bins. Then we randomly project each of these discretized integers into discrete embedding of dimension 16 with
26

---

Published as a conference paper at ICLR 2020
integer value ranging from to 9. Here discrete embeddings for the purpose of accurate hashing. For each imput entity, we concatenate all its obtained discrete embeddings as this entity's s feature embedding. An max-pooling is performed over the feature embeddings of all the entities belonging to each object type (i.e., agent, lidar, box and ramp) to obtain a entity-invariant object representation. Finally, concatenating all the derived object representations results in the final state representation to count.
We run decentralized version of the count-based exploration where each parallel rollout worker shares the same random projection for computing embeddings but maintains its own counts. Let (S)) denote the counts for state S in a particular rollout worker. Then the intrinsic reward is calculated by ((S))
D.2 RANDOM NETWORK DISTILLATION
Random Network Distillation (RND) (Burda al., 2019b) fixed random network, i.e., target network, to produce a random projection for each state while learns anther network, i.c., predictor network, fit the output from the target network on visited states. The prediction error between two networks is used as the intrinsic motivation.
For the random target network, we use the same architecture as the value network except that we remove the LSTM layer and project the final layer to 64 dimensional vector instead of a single value. The architecture is the same for predictor network. use the squared difference between predictor and target network output with an coefficient of 1.0 as the intrinsic reward.
D.3 FINE-TUNING FROM INTRINSIC MOTIVATION VARIANTS
We compare the performances different policies pretrained by intrinsic motivation variants on our intelligence suites in Figure D.I. The pretraining methods of consideration include RND and different count-based exploration variants with different state representations. For policies trained by count-based variants, as discussed the Section 6. we know that more concise state represen- tation for counts leads to better emergent object manipulation skills, i.c., only using object position better than using position and velocity information while using all the input as state representa- tion performs the worst. In our transfer task suites, we observe that polices with better pretrained skills perform better in 3 of the tasks except the Object Counting task and the Construction from Blueprint task. In Construction from Blueprint, policies with better skills adapt better in the early phase but may have a higher chance failure later this challenging task. Object Counting, the polices with better skills perform worse. Interestingly, this observation is consistent with the transfer result in the main paper (Figure 6). where the policy pretrained by count- based exploration outperforms the multi-agent pretrained policy. We conjecture that the Object Counting task exam- ines some factors of the agents that may not strongly relate to the quality of emergent skills, such as navigation and tool use.
27

---

Published as a conference paper at ICLR 2020
Object Counting
Lock and Return
Sequential Look
Blueprint Construction
Shelter Construction
Nomallddddwwo ioward
rewendd
Nomaaizadd roward
0.8
Nomolized Rg
0.8
0.6
0.6
0.4
0.4
02
122
82
0.0
0.0
0.2
0.3
0.6
0.5
5.00510 16 1226655
62635
0
Samples (X10P)
Samplea (c10P)
Samples (x10P)
Samples 6x10")
Samples (x10*)
Count: AAgant- Rox Position
Count Rasect
RND. Agents-
Figure D.I: Fine-tuning From Intrinsic Motivation Variants. We plot the mean performance the suite transfer tasks and 90% confidence interval across 3 seeds and smooth performance and confidence intervals with exponential moving average. We vary the state representation used for collecting counts: in blue we show the performance for a single agent where the state is defined on box 2-D positions, in green we show the performance of single agent where the state box position but also box rotation and velocity, in red we show the performance of 1-3 agents with the full observation space given hide-and-seek policy, and finally in purple show the performance also with 1-3 agents and full observation space but with RND which should scale better than count-based exploration.
28


Reviews:
[{"review_id": "rkxcmygpFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Review ICLR20, Emergent Tool Use...\n\nThis review is for the originally uploaded version of this article. Comments from other reviewers and revisions have deliberately not been taken into account. After publishing this review, this reviewer will participate in the forum discussion and help the authors improve the paper.\n\nI apologize in advance for being reviewer 2.\n\n## Overall\n\n**Summary**\n\nThe article introduces a new multi-agent physics environment called \"hide-and-seek\". The authors trained agents in this environment and studied the emergence of and changes in strategies. The authors also study the performance of these same agents in new \"targeted intelligence tests\" compared to training from scratch and compared to agents trained with curiosity.\n\n**Overall Opinion**\n\nI think the environment is very appealing and the paper is overall well-structured and demonstrates novel work. Therefore I'd recommend this paper to be accepted. That being said, there are glaring issues with some of the writing that need to be addressed before I think this work conforms to the standards of ICLR. However, if these issues are addressed, I have no issue increasing my review score.\n\nMain problems:\n\n- The majority of the paper presents essentially a case study of what happened during a single seed of policy training. For RL literature that's very uncommon and I think it's consensus that DRL is very sensitive to random seeds. I know that you do have additional seeds in the appendix, but why didn't you mention those in the main body of the paper? You seem to have found some robustness against multiple seeds, so why not show it? And also the fact that Figure 1 & 3 only apply to 1 seed is not mentioned. I think this is easy enough to fix - I suggest since you're already at 10 pages, to just bring in the additional seeds from the appendix and average over their performance in Fig.1&3.\n- The contributions section is overselling the work: (1) states that autocurricula lead to changes in agent strategy - Maybe I'm mistaken here but that sounds like a tautology. In other words, \"a self-generated sequence of challenges\" (\"Autocurricula\", according to [Leibo et al., 2019][1]) lead to changes in strategy. And (3) advertises \"a proposed framework for evaluating agents in open-ended environments\" and also \"a suite of targeted intelligence tests for our domain\". The former of those two is either not in the paper or you mean your section \"6.2 Transfer and Fine-Tuning as Evaluation\", which isn't novel (see e.g. [Alain & Bengio, 2016][2])\n- Your acknowledgments should be anonymized until publication. Otherwise, reviewers might draw conclusions which group published this work, thus violating the double-blind review procedure.\n\n[1]: https://arxiv.org/pdf/1903.00742.pdf\n[2]: https://arxiv.org/pdf/1610.01644.pdf\n\nLike I mentioned above, I think these are all easy to address, which should allow acceptance of this work. Here are some additional questions, comments, and nitpicks:\n\n## Specific comments and questions\n\n### Abstract\n\n- \"evidence that ... competition may scale better with increasing environment complexity\" - that's only shown in the appendix\n\n### Intro\n\n- You mention TD-Gammon as a game, but I think it's an algorithm for the game Backgammon, similarly to how \"Go\" is the game and \"AlphaGo\" is an algorithm for playing.\n\n### Rel. Work\n\n- all good\n\n### Hide And Seek\n\n- Arena boundaries: What's the penalty and what's \" too far outside the play area\"? And in all depictions, it looks like the geometry of the arena is elevated around the edges and the agents don't have a jump action, so how would they ever go out of borders? After watching the videos: Apparently, the jagged-looking arena boundary in the videos is purely cosmetic and agents can still access that space. This is unclear from just the paper and the renderings in Figure 1.\n\n### Policy Optimization\n\n- Policy network and fusion are underspecified: How do you deal with the varying number of agents, boxes, obstacles? Do you just set the x/v of the missing pieces to zero or is the observation actually of a different shape in case there are more/fewer objects or agents? How's the embedding done that is depicted in Figure 2? Also, I didn't see the embedding being mentioned in the text - any reason for that?\n- Figure 2 - This diagram is visually appealing but confusing and needs to be improved. Why does the agent's embedding say \"1\"? Why do the other agents' embeddings have a \"-1\" at the end of the orange box and the others don't? Is the agent's embedding concatenated with the other embeddings? If so, why and how (concat, sum, multiply, conditional batch norm, etc.)? In the center and on the right you use a blue block to indicate the agent's embedding and then at the bottom right you seem to use it as a network component or something (between the \"LSTM\")? If you're trying to signal that this is the agent's perception at different stages in the network, I'd use a different color to separate it from the agent's lidar and pos/vel. You don't mention that \"x,v\" stands for \"position, velocity\".\n \n### Auto-curriculum and Emergent Behavior\n\n- Figure 3: \"environment-specific\" (add dash). Draw skill development boundaries like in Fig.1.\n- How exactly does the \"surfing\" work? The seekers step (not jump, right, since there is no jumping?) onto the boxes and then what? The momentum propels the box forward? Do other seekers push the box? Their movement on top of the box somehow moves the box (this seems to be the case judging by the videos but this is the least physically plausible)? This is a super interesting adaptation but I'd suspect the physics simulation to have a bug/glitch that's being exploited here.\n- You mention in footnote 3 that the developmental stage and changes in reward aren't necessarily correlated. The same seems to be true for the metrics in Fig.3, which raises the question how did you come up with those boundaries for the different developmental stages in Fig.1? Did someone look at rollouts from the trained policy every couple of million steps? And do all agents learn new skills at the same time or is there a delay? From my understanding, they are all using the same policy and critic networks but maybe dependent on the proximity of an agent to an object/obstacle, it's easier or harder to execute.\n\n### Evaluation\n\n- clear and well-written, slightly too much content in the appendix and not enough in the main paper. Weird appendix numbering - A.6 appears in the main paper pages after A.7\n\n### Discussion and Future Work\n\n- all good\n\n### Appendix\n\n- I appreciate the TOC. I did not look into Appendix B-D because it's another 10 pages on top of the 10 pages of the article.\n\nAll in all an interesting work. Good luck with the rebuttal/discussion.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SkxqkHzRYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Summary\n\nThe authors report on an empirical study of emergent behavior of multiple RL agents learning to play hide-and-seek (a sparse reward task). The main point of this paper is that RL agents learning at scale (large number of samples, batch-size 64000). can learn to solve tasks with strategies that are human-interpretable (e.g., using ramps, boxes). Scale also requires various simplifications (e.g., keeping the learning setup as close as possible to a single-agent problem as possible).\n\nAgents are grouped in 2 teams (seekers, hiders). Each agent receives a team reward, e.g., it can be punished for events that it did not participate in, e.g., if a team-mate is seen by an opponent. If hiders are hidden, seekers also automatically see reward. The first 40% of the episode there is no reward to let hiders hide.\n\nThere is one actor model, all agents share weights. Hence this is self-play: hiders and seekers use the same agent model. Also, all agents use a central value function that can see the entire state (decentralized execution, centralized learning). This makes the setting basically a single-agent problem, with the only decentralized aspect being each actor model only receiving its own observation. Note that a large body of multi-agent RL work in fact uses agents that do not share weights, etc.\n\nOther features described:\n- Auto-curricula: e.g. agents find new strategies (using ramps, boxes) that other agents have to counteract.\n- Human-relevant skills: They report that the agent model learns multiple ways to interact with (objects in) the environment that are semantically interesting (resembles something humans might do).\n- Authors compare with policies learning via intrinsic motivation.\n- Evaluation through transfer learning shows some benefit of transfer of hide-seek agents to auxiliary tasks. However, it is not so clear how this evaluation informs future work on transfer learning (e.g., how would you pick evaluation tasks for a given train-task?) \n\n1. Decision (accept or reject) with one or two key reasons for this choice.\n\nReject.\n\nThe main point of the paper is empirical RL at scale. Although the learned behaviors are human-interpretable, this does not seem surprising given the fact that in many (large-scale) RL applications (Atari games, Go, DotA 2, Starcraft), it has been observed that RL agents can learn to manipulate and use their environment (which includes other agents!) in unexpected ways / find creative ways to exploit the reward function (see e.g. demos in https://www.alexirpan.com/2018/02/14/rl-hard.html). There has also been work on object-level RL [Agnew, Domingos 2018], which involves agents interacting with objects in the environment. Compared to this, the observation that RL agents learn human-interpretable uses of objects does not seem surprising.\n\nThe paper also does not give new insights in how to make large-scale RL ``'work'. For instance, there are no significant differences in algorithm / model structure from DotA / Starcraft agents that can inform future large-scale experiments.\n\nThe paper also does not introduce new concrete evaluation metrics that can apply to other tasks / RL problems, skill detection / segmentation methods to learn the structure of auto-curricula. Furthermore, the setup is very close to a single-agent problem (see above), and is far simpler in the multi-agent assumptions from other decentralized multi-agent work (Foerster 2018, Jacques 2019, etc).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJlxTZh6Yr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors in introduce a new competitive/cooperative physics-based environment in which different teams of agents compete in a visual concealment and search task with visibility-based team-based rewards (although There are no explicit incentives for agents to interact with objects in the environment). They show that, complex behaviour emerge as the episode progresses and agents are able to learn 6 emergent skills/(counter-)strategies (including tool use), where agents intentionally change their environment to suit their needs. Agents trained using self-play \n\nIn my opinion, this is an excellent paper which main contribution is to provide experimental evidence that relevant and complex skills and strategies can emerge from multi-agent RL competing scenarios.\n\nMinor comments:\n\n- Hide&seek rules and safety issues: is it not supposed that hiders and the seekers could not get together (i.e., hiders cannot push seekers or as we can see in some videos)? Furthermore, it is surprising (one would say worrying) that hiders identified the barriers as an impediment to the seeker (not only as a way to hide). I wouldn’t say that this is a “ human-relevant strategies and skills “ as the authors claim. Hider agents even double walled seekers!  \n\n- Have the authors thought about joining the Animal-AI Olympics (http://animalaiolympics.com/) competition? It would be a great opportunity to to test the skills of your agents in a further general testing scenario. They provide an arena (test-bed) which contains 300 different intelligent tests for testing the cognitive abilities of RL agents (https://www.mdcrosby.com/blog/animalaiprizes1.html) which have to interact with the environment. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes how multi-agent reinforcement learning at scale leads to the evolution of complex behaviors. Actually, "at scale" may be an understatement - a lot of computing power was used here. But the amount of compute used is not the point, rather the point is that complex and fascinating behavior can emerge from a long co-evolutionary process (though gradient-based RL is used here, the principle is the same) where the arms race forms an implicit curriculum. This is the existence proof that people in artificial life and adaptive behavior have been looking for for so long. 

Two reviewers were positive about the paper, with a third being negative because the paper does not give any new insights about how to do RL at scale. But that was not the stated aim of the paper, as the authors clarify in a response.

This paper will draw quite some attention and deserves an oral presentation.
Decision: Accept (Spotlight)