Emergent Tool Use From Multi-Agent Autocurricula

1 INTRODUCTION . Creating intelligent artificial agents that can solve a wide variety of complex human-relevant tasks has been a long-standing challenge in the artificial intelligence community . Of particular relevance to humans will be agents that can sense and interact with objects in a physical world . One approach to creating these agents is to explicitly specify desired tasks and train a reinforcement learning ( RL ) agent to solve them . On this front , there has been much recent progress in solving physically grounded tasks , e.g . dexterous in-hand manipulation ( Rajeswaran et al. , 2017 ; Andrychowicz et al. , 2018 ) or locomotion of complex bodies ( Schulman et al. , 2015 ; Heess et al. , 2017 ) . However , specifying reward functions or collecting demonstrations in order to supervise these tasks can be ∗This was a large project and many people made significant contributions . Bowen , Bob , and Igor conceived the project and provided guidance through all stages of the work . Bowen created the initial environment , infrastructure and models , and obtained the first results of sequential skill progression . Ingmar obtained the first results of tool use , contributed to environment variants , created domain-specific statistics , and with Bowen created the final environment . Todor created the manipulation tasks in the transfer suite , helped Yi with the RND baseline , and prepared code for open-sourcing . Yi created the navigation tasks in the transfer suite , intrinsic motivation comparisons , and contributed to environment variants . Glenn contributed to designing the final environment and created final renderings and project video . Igor provided research supervision and team leadership . †Work performed while at OpenAI time consuming and costly . Furthermore , the learned skills in these single-agent RL settings are inherently bounded by the task description ; once the agent has learned to solve the task , there is little room to improve . Due to the high likelihood that direct supervision will not scale to unboundedly complex tasks , many have worked on unsupervised exploration and skill acquisition methods such as intrinsic motivation . However , current undirected exploration methods scale poorly with environment complexity and are drastically different from the way organisms evolve on Earth . The vast amount of complexity and diversity on Earth evolved due to co-evolution and competition between organisms , directed by natural selection ( Dawkins & Krebs , 1979 ) . When a new successful strategy or mutation emerges , it changes the implicit task distribution neighboring agents need to solve and creates a new pressure for adaptation . These evolutionary arms races create implicit autocurricula ( Leibo et al. , 2019a ) whereby competing agents continually create new tasks for each other . There has been much success in leveraging multi-agent autocurricula to solve multi-player games , both in classic discrete games such as Backgammon ( Tesauro , 1995 ) and Go ( Silver et al. , 2017 ) , as well as in continuous real-time domains such as Dota ( OpenAI , 2018 ) and Starcraft ( Vinyals et al. , 2019 ) . Despite the impressive emergent complexity in these environments , the learned behavior is quite abstract and disembodied from the physical world . Our work sees itself in the tradition of previous studies that showcase emergent complexity in simple physically grounded environments ( Sims , 1994a ; Bansal et al. , 2018 ; Jaderberg et al. , 2019 ; Liu et al. , 2019 ) ; the success in these settings inspires confidence that inducing autocurricula in physically grounded and open-ended environments could eventually enable agents to acquire an unbounded number of human-relevant skills . We introduce a new mixed competitive and cooperative physics-based environment in which agents compete in a simple game of hide-and-seek . Through only a visibility-based reward function and competition , agents learn many emergent skills and strategies including collaborative tool use , where agents intentionally change their environment to suit their needs . For example , hiders learn to create shelter from the seekers by barricading doors or constructing multi-object forts , and as a counter strategy seekers learn to use ramps to jump into hiders ’ shelter . Moreover , we observe signs of dynamic and growing complexity resulting from multi-agent competition and standard reinforcement learning algorithms ; we find that agents go through as many as six distinct adaptations of strategy and counter-strategy , which are depicted in Figure 1 . We further present evidence that multi-agent co-adaptation may scale better with environment complexity and qualitatively centers around more human-interpretable behavior than intrinsically motivated agents . However , as environments increase in scale and multi-agent autocurricula become more open-ended , evaluating progress by qualitative observation will become intractable . We therefore propose a suite of targeted intelligence tests to measure capabilities in our environment that we believe our agents may eventually learn , e.g . object permanence ( Baillargeon & Carey , 2012 ) , navigation , and construction . We find that for a number of the tests , agents pretrained in hide-and-seek learn faster or achieve higher final performance than agents trained from scratch or pretrained with intrinsic motivation ; however , we find that the performance differences are not drastic , indicating that much of the skill and feature representations learned in hide-and-seek are entangled and hard to fine-tune . The main contributions of this work are : 1 ) clear evidence that multi-agent self-play can lead to emergent autocurricula with many distinct and compounding phase shifts in agent strategy , 2 ) evidence that when induced in a physically grounded environment , multi-agent autocurricula can lead to human-relevant skills such as tool use , 3 ) a proposal to use transfer as a framework for evaluating agents in open-ended environments as well as a suite of targeted intelligence tests for our domain , and 4 ) open-sourced environments and code1 for environment construction to encourage further research in physically grounded multi-agent autocurricula . 2 RELATED WORK . There is a long history of using self-play in multi-agent settings . Early work explored self-play using genetic algorithms ( Paredis , 1995 ; Pollack et al. , 1997 ; Rosin & Belew , 1995 ; Stanley & Miikkulainen , 2004 ) . Sims ( 1994a ) and Sims ( 1994b ) studied the emergent complexity in morphology and behavior of creatures that coevolved in a simulated 3D world . Open-ended evolution was further explored in the environments Polyworld ( Yaeger , 1994 ) and Geb ( Channon et al. , 1998 ) , where agents compete and mate in a 2D world , and in Tierra ( Ray , 1992 ) and Avida ( Ofria & Wilke , 2004 ) , where computer programs compete for computational resources . More recent work attempted to formulate necessary preconditions for open-ended evolution ( Taylor , 2015 ; Soros & Stanley , 2014 ) . Co-adaptation between agents and environments can also give rise to emergent complexity ( Florensa et al. , 2017 ; Sukhbaatar et al. , 2018 ; Wang et al. , 2019 ) . In the context of multi-agent RL , Tesauro ( 1995 ) , Silver et al . ( 2016 ) , OpenAI ( 2018 ) , Jaderberg et al . ( 2019 ) and Vinyals et al . ( 2019 ) used self-play with deep RL techniques to achieve super-human performance in Backgammon , Go , Dota , Capture-the-Flag and Starcraft , respectively . Bansal et al . ( 2018 ) trained agents in a simulated 3D physics environment to compete in various games such as sumo wrestling and soccer goal shooting . In Liu et al . ( 2019 ) , agents learn to manipulate a soccer ball in a 3D soccer environment and discover 1Code can be found at github.com/openai/multi-agent-emergence-environments . emergent behaviors such as ball passing and interception . In addition , communication has also been shown to emerge from multi-agent RL ( Sukhbaatar et al. , 2016 ; Foerster et al. , 2016 ; Lowe et al. , 2017 ; Mordatch & Abbeel , 2018 ) . Intrinsic motivation methods have been widely studied in the literature ( Chentanez et al. , 2005 ; Singh et al. , 2010 ) . One example is count-based exploration , where agents are incentivized to reach infrequently visited states by maintaining state visitation counts ( Strehl & Littman , 2008 ; Bellemare et al. , 2016 ; Tang et al. , 2017 ) or density estimators ( Ostrovski et al. , 2017 ; Burda et al. , 2019b ) . Another paradigm are transition-based methods , in which agents are rewarded for high prediction error in a learned forward or inverse dynamics model ( Schmidhuber , 1991 ; Stadie et al. , 2015 ; Mohamed & Rezende , 2015 ; Houthooft et al. , 2016 ; Achiam & Sastry , 2017 ; Pathak et al. , 2017 ; Burda et al. , 2019a ; Haber et al. , 2018 ) . Jaques et al . ( 2019 ) consider multi-agent scenarios and adopt causal influence as a motivation for coordination . In our work , we utilize intrinsic motivation methods as an alternative exploration baseline to multi-agent autocurricula . Similar comparisons have also been made in Haber et al . ( 2018 ) and Leibo et al . ( 2019b ) . Tool use is a hallmark of human and animal intelligence ( Hunt , 1996 ; Shumaker et al. , 2011 ) ; however , learning tool use in RL settings can be a hard exploration problem when rewards are unaligned . For example , in Forestier et al . ( 2017 ) ; Xie et al . ( 2019 ) a real-world robot learns to solve various tasks requiring tools . In Bapst et al . ( 2019 ) , an agent solves construction tasks in a 2-D environment using both model-based and model-free methods . Allen et al . ( 2019 ) uses a combination of humandesigned priors and model-based policy optimization to solve a collection of physics-based puzzles requiring tool use . However , in each of these works , agents were explicitly incentivized to interact with and use tools , whereas in our environment agents implicitly create this incentive through multi-agent competition . 3 HIDE AND SEEK . Agents are tasked with competing in a two-team hide-and-seek game in a physics-based environment . The hiders are tasked with avoiding line of sight from the seekers , and the seekers are tasked with keeping vision of the hiders . There are objects scattered throughout the environment that the agents can grab and also lock in place . There are also randomly generated immovable rooms and walls that the agents must learn to navigate . Before the game of hide-and-seek begins , the hiders are given a preparation phase where the seekers are immobilized , giving the hiders a chance to run away or change their environment . There are no explicit incentives for agents to interact with objects in the environment ; the only supervision given is through the hide-and-seek objective . Agents are given a team based reward ; hiders are given a reward of 1 if all hiders are hidden and -1 if any hider is seen by a seeker . Seekers are given the opposite reward , -1 if all hiders are hidden and +1 otherwise . To confine agent behavior to a reasonable space , agents are penalized with a reward of -10 if they go too far outside of the play area ( outside an 18 meter square ) . An episode lasts 240 timesteps , the first 40 % of which are the preparation phase where all agents are given zero reward . We simulate the environment in the MUJOCO physics engine ( Todorov et al. , 2012 ) . The world is populated with 1 to 3 hiders , 1 to 3 seekers , 3 to 9 movable boxes of which at least 3 are elongated , 2 movable ramps , and randomly generated static walls and rooms . We also experiment with a simpler , less randomized environment described in Appendix A.3 . Agents observe the position , velocity , and size ( in the case of the randomly shaped boxes ) of objects and other agents . If entities are not in line-of-sight of the agent or not in a 135 degree cone in front of the agent , then they are masked out in the policy . Agents also have 30 range sensors arrayed evenly around them , similar to a lidar . In addition , each agent observes its own team and other agents ’ teams as well as how much time is left in the preparation phase . Agents are simulated as spherical objects and have 3 action types that can be chosen simultaneously at each time step . They may move by setting a discretized force along their x and y axis and torque around their z-axis . They have a single binary action to grab objects , which binds the agent to the closest object while the action is enabled . Agents may also lock objects in place with a single binary action . Objects may be unlocked only by agents on the team of the agent who originally locked the object . Agents may only grab or lock objects that are in front of them and within a small radius .

Summary: The authors report on an empirical study of emergent behavior of multiple RL agents learning to play hide-and-seek (a sparse reward task). The main point of this paper is that RL agents learning at scale (large number of samples, batch-size 64000). can learn to solve tasks with strategies that are human-interpretable (e.g., using ramps, boxes). Scale also requires various simplifications (e.g., keeping the learning setup as close as possible to a single-agent problem as possible).