Emergent Tool Use From Multi-Agent Autocurricula

1 INTRODUCTION . Creating intelligent artificial agents that can solve a wide variety of complex human-relevant tasks has been a long-standing challenge in the artificial intelligence community . Of particular relevance to humans will be agents that can sense and interact with objects in a physical world . One approach to creating these agents is to explicitly specify desired tasks and train a reinforcement learning ( RL ) agent to solve them . On this front , there has been much recent progress in solving physically grounded tasks , e.g . dexterous in-hand manipulation ( Rajeswaran et al. , 2017 ; Andrychowicz et al. , 2018 ) or locomotion of complex bodies ( Schulman et al. , 2015 ; Heess et al. , 2017 ) . However , specifying reward functions or collecting demonstrations in order to supervise these tasks can be ∗This was a large project and many people made significant contributions . Bowen , Bob , and Igor conceived the project and provided guidance through all stages of the work . Bowen created the initial environment , infrastructure and models , and obtained the first results of sequential skill progression . Ingmar obtained the first results of tool use , contributed to environment variants , created domain-specific statistics , and with Bowen created the final environment . Todor created the manipulation tasks in the transfer suite , helped Yi with the RND baseline , and prepared code for open-sourcing . Yi created the navigation tasks in the transfer suite , intrinsic motivation comparisons , and contributed to environment variants . Glenn contributed to designing the final environment and created final renderings and project video . Igor provided research supervision and team leadership . †Work performed while at OpenAI time consuming and costly . Furthermore , the learned skills in these single-agent RL settings are inherently bounded by the task description ; once the agent has learned to solve the task , there is little room to improve . Due to the high likelihood that direct supervision will not scale to unboundedly complex tasks , many have worked on unsupervised exploration and skill acquisition methods such as intrinsic motivation . However , current undirected exploration methods scale poorly with environment complexity and are drastically different from the way organisms evolve on Earth . The vast amount of complexity and diversity on Earth evolved due to co-evolution and competition between organisms , directed by natural selection ( Dawkins & Krebs , 1979 ) . When a new successful strategy or mutation emerges , it changes the implicit task distribution neighboring agents need to solve and creates a new pressure for adaptation . These evolutionary arms races create implicit autocurricula ( Leibo et al. , 2019a ) whereby competing agents continually create new tasks for each other . There has been much success in leveraging multi-agent autocurricula to solve multi-player games , both in classic discrete games such as Backgammon ( Tesauro , 1995 ) and Go ( Silver et al. , 2017 ) , as well as in continuous real-time domains such as Dota ( OpenAI , 2018 ) and Starcraft ( Vinyals et al. , 2019 ) . Despite the impressive emergent complexity in these environments , the learned behavior is quite abstract and disembodied from the physical world . Our work sees itself in the tradition of previous studies that showcase emergent complexity in simple physically grounded environments ( Sims , 1994a ; Bansal et al. , 2018 ; Jaderberg et al. , 2019 ; Liu et al. , 2019 ) ; the success in these settings inspires confidence that inducing autocurricula in physically grounded and open-ended environments could eventually enable agents to acquire an unbounded number of human-relevant skills . We introduce a new mixed competitive and cooperative physics-based environment in which agents compete in a simple game of hide-and-seek . Through only a visibility-based reward function and competition , agents learn many emergent skills and strategies including collaborative tool use , where agents intentionally change their environment to suit their needs . For example , hiders learn to create shelter from the seekers by barricading doors or constructing multi-object forts , and as a counter strategy seekers learn to use ramps to jump into hiders ’ shelter . Moreover , we observe signs of dynamic and growing complexity resulting from multi-agent competition and standard reinforcement learning algorithms ; we find that agents go through as many as six distinct adaptations of strategy and counter-strategy , which are depicted in Figure 1 . We further present evidence that multi-agent co-adaptation may scale better with environment complexity and qualitatively centers around more human-interpretable behavior than intrinsically motivated agents . However , as environments increase in scale and multi-agent autocurricula become more open-ended , evaluating progress by qualitative observation will become intractable . We therefore propose a suite of targeted intelligence tests to measure capabilities in our environment that we believe our agents may eventually learn , e.g . object permanence ( Baillargeon & Carey , 2012 ) , navigation , and construction . We find that for a number of the tests , agents pretrained in hide-and-seek learn faster or achieve higher final performance than agents trained from scratch or pretrained with intrinsic motivation ; however , we find that the performance differences are not drastic , indicating that much of the skill and feature representations learned in hide-and-seek are entangled and hard to fine-tune . The main contributions of this work are : 1 ) clear evidence that multi-agent self-play can lead to emergent autocurricula with many distinct and compounding phase shifts in agent strategy , 2 ) evidence that when induced in a physically grounded environment , multi-agent autocurricula can lead to human-relevant skills such as tool use , 3 ) a proposal to use transfer as a framework for evaluating agents in open-ended environments as well as a suite of targeted intelligence tests for our domain , and 4 ) open-sourced environments and code1 for environment construction to encourage further research in physically grounded multi-agent autocurricula . 2 RELATED WORK . There is a long history of using self-play in multi-agent settings . Early work explored self-play using genetic algorithms ( Paredis , 1995 ; Pollack et al. , 1997 ; Rosin & Belew , 1995 ; Stanley & Miikkulainen , 2004 ) . Sims ( 1994a ) and Sims ( 1994b ) studied the emergent complexity in morphology and behavior of creatures that coevolved in a simulated 3D world . Open-ended evolution was further explored in the environments Polyworld ( Yaeger , 1994 ) and Geb ( Channon et al. , 1998 ) , where agents compete and mate in a 2D world , and in Tierra ( Ray , 1992 ) and Avida ( Ofria & Wilke , 2004 ) , where computer programs compete for computational resources . More recent work attempted to formulate necessary preconditions for open-ended evolution ( Taylor , 2015 ; Soros & Stanley , 2014 ) . Co-adaptation between agents and environments can also give rise to emergent complexity ( Florensa et al. , 2017 ; Sukhbaatar et al. , 2018 ; Wang et al. , 2019 ) . In the context of multi-agent RL , Tesauro ( 1995 ) , Silver et al . ( 2016 ) , OpenAI ( 2018 ) , Jaderberg et al . ( 2019 ) and Vinyals et al . ( 2019 ) used self-play with deep RL techniques to achieve super-human performance in Backgammon , Go , Dota , Capture-the-Flag and Starcraft , respectively . Bansal et al . ( 2018 ) trained agents in a simulated 3D physics environment to compete in various games such as sumo wrestling and soccer goal shooting . In Liu et al . ( 2019 ) , agents learn to manipulate a soccer ball in a 3D soccer environment and discover 1Code can be found at github.com/openai/multi-agent-emergence-environments . emergent behaviors such as ball passing and interception . In addition , communication has also been shown to emerge from multi-agent RL ( Sukhbaatar et al. , 2016 ; Foerster et al. , 2016 ; Lowe et al. , 2017 ; Mordatch & Abbeel , 2018 ) . Intrinsic motivation methods have been widely studied in the literature ( Chentanez et al. , 2005 ; Singh et al. , 2010 ) . One example is count-based exploration , where agents are incentivized to reach infrequently visited states by maintaining state visitation counts ( Strehl & Littman , 2008 ; Bellemare et al. , 2016 ; Tang et al. , 2017 ) or density estimators ( Ostrovski et al. , 2017 ; Burda et al. , 2019b ) . Another paradigm are transition-based methods , in which agents are rewarded for high prediction error in a learned forward or inverse dynamics model ( Schmidhuber , 1991 ; Stadie et al. , 2015 ; Mohamed & Rezende , 2015 ; Houthooft et al. , 2016 ; Achiam & Sastry , 2017 ; Pathak et al. , 2017 ; Burda et al. , 2019a ; Haber et al. , 2018 ) . Jaques et al . ( 2019 ) consider multi-agent scenarios and adopt causal influence as a motivation for coordination . In our work , we utilize intrinsic motivation methods as an alternative exploration baseline to multi-agent autocurricula . Similar comparisons have also been made in Haber et al . ( 2018 ) and Leibo et al . ( 2019b ) . Tool use is a hallmark of human and animal intelligence ( Hunt , 1996 ; Shumaker et al. , 2011 ) ; however , learning tool use in RL settings can be a hard exploration problem when rewards are unaligned . For example , in Forestier et al . ( 2017 ) ; Xie et al . ( 2019 ) a real-world robot learns to solve various tasks requiring tools . In Bapst et al . ( 2019 ) , an agent solves construction tasks in a 2-D environment using both model-based and model-free methods . Allen et al . ( 2019 ) uses a combination of humandesigned priors and model-based policy optimization to solve a collection of physics-based puzzles requiring tool use . However , in each of these works , agents were explicitly incentivized to interact with and use tools , whereas in our environment agents implicitly create this incentive through multi-agent competition . 3 HIDE AND SEEK . Agents are tasked with competing in a two-team hide-and-seek game in a physics-based environment . The hiders are tasked with avoiding line of sight from the seekers , and the seekers are tasked with keeping vision of the hiders . There are objects scattered throughout the environment that the agents can grab and also lock in place . There are also randomly generated immovable rooms and walls that the agents must learn to navigate . Before the game of hide-and-seek begins , the hiders are given a preparation phase where the seekers are immobilized , giving the hiders a chance to run away or change their environment . There are no explicit incentives for agents to interact with objects in the environment ; the only supervision given is through the hide-and-seek objective . Agents are given a team based reward ; hiders are given a reward of 1 if all hiders are hidden and -1 if any hider is seen by a seeker . Seekers are given the opposite reward , -1 if all hiders are hidden and +1 otherwise . To confine agent behavior to a reasonable space , agents are penalized with a reward of -10 if they go too far outside of the play area ( outside an 18 meter square ) . An episode lasts 240 timesteps , the first 40 % of which are the preparation phase where all agents are given zero reward . We simulate the environment in the MUJOCO physics engine ( Todorov et al. , 2012 ) . The world is populated with 1 to 3 hiders , 1 to 3 seekers , 3 to 9 movable boxes of which at least 3 are elongated , 2 movable ramps , and randomly generated static walls and rooms . We also experiment with a simpler , less randomized environment described in Appendix A.3 . Agents observe the position , velocity , and size ( in the case of the randomly shaped boxes ) of objects and other agents . If entities are not in line-of-sight of the agent or not in a 135 degree cone in front of the agent , then they are masked out in the policy . Agents also have 30 range sensors arrayed evenly around them , similar to a lidar . In addition , each agent observes its own team and other agents ’ teams as well as how much time is left in the preparation phase . Agents are simulated as spherical objects and have 3 action types that can be chosen simultaneously at each time step . They may move by setting a discretized force along their x and y axis and torque around their z-axis . They have a single binary action to grab objects , which binds the agent to the closest object while the action is enabled . Agents may also lock objects in place with a single binary action . Objects may be unlocked only by agents on the team of the agent who originally locked the object . Agents may only grab or lock objects that are in front of them and within a small radius .

Summary: Authors in introduce a new competitive/cooperative physics-based environment in which different teams of agents compete in a visual concealment and search task with visibility-based team-based rewards (although There are no explicit incentives for agents to interact with objects in the environment). They show that, complex behaviour emerge as the episode progresses and agents are able to learn 6 emergent skills/(counter-)strategies (including tool use), where agents intentionally change their environment to suit their needs. Agents trained using self-play