Neural MMO: A massively multiplayer game environment for intelligent agents

We present an artificial intelligence research platform inspired by the human game genre of MMORPGs ( Massively Multiplayer Online Role-Playing Games , a.k.a . MMOs ) . We demonstrate how this platform can be used to study behavior and learning in large populations of neural agents . Unlike currently popular game environments , our platform supports persistent environments , with variable number of agents , and open-ended task descriptions . The emergence of complex life on Earth is often attributed to the arms race that ensued from a huge number of organisms all competing for finite resources . Our platform aims to simulate this setting in microcosm : we conduct a series of experiments to test how large-scale multiagent competition can incentivize the development of skillful behavior . We find that population size magnifies the complexity of the behaviors that emerge and results in agents that out-compete agents trained in smaller populations . 1 INTRODUCTION . Life on Earth can be viewed as a massive multiagent competition . The cheetah evolves an aerodynamic profile in order to catch the gazelle , the gazelle develops springy legs to run even faster : species have evolved ever new capabilities in order to outcompete their adversaries . The success of biological evolution has inspired many attempts to emulate it in silico , ranging from genetic algorithms that bear only loose resemblance to natural processes , to full-blown simulations of “ artificial life ” . A recurring question has been : at what level of abstraction should we simulate the competitive game of life ? In recent years , the field of deep reinforcement learning ( RL ) has embraced a related approach : train algorithms by having them compete in simulated games ( Silver et al. , 2016 ; OpenAI ; Jaderberg et al. , 2018 ) . Such games are immediately interpretable and provide easy metrics derived from the game ’ s “ score ” and win conditions . However , popular game benchmarks are currently still limited : they typically define a narrow , episodic task , with a small fixed number of players . In contrast , life on Earth involves a persistent environment , an unbounded number of players , and a seeming “ open-endedness ” , where ever new and more complex species emerge over time , with no end in sight ( Stanley et al. , 2017 ) . Our aim is to develop a simulation platform ( see Figure 5 ) that captures important properties of life on Earth , while also borrowing from the interpretability and abstractions of human-designed games . To this end , we turn to the game genre of Massively Multiplayer Online Role-Playing Games ( MMORPGs , or MMOs for short ) . These games involve a large , variable number of players competing to survive and prosper in persistent and far-flung environments . Our platform simulates a “ Neural MMO ” – an MMO in which each agent is a neural net that learns to survive using RL . We demonstrate the capabilities of this platform through a series of experiments that investigate emergent complexity as a function of the number of agents and species that compete in the simulation . We find that large populations act as competitive pressure that encourages exploration of the environment and the development of skillful behavior . In addition , we find that when agents are organized into species ( share policy parameters ) , each species naturally diverges from the others to occupy its own behavioral niche . Upon publication , we will opensource the platform in full . 2 BACKGROUND AND RELATED WORK . Multiagent Reinforcement Learning Multiagent reinforcement learning has received increased attention in recent years ( Lowe et al. , 2017 ; Mordatch & Abbeel , 2017 ; Bansal et al. , 2017 ; Lanctot et al. , 2017 ; Yang et al. , 2018a ; Zheng et al. , 2017 ; OpenAI ; Jaderberg et al. , 2018 ) . Unlike single agent environments often well modeled by classic Markov Decision Processes ( MDPs ) , multiagent interaction often introduces nonstationarity in the transition dynamics of the environment due to the continual learning and co-adaptation of other agents . While previous work has attempted to analyze emergent complexity in interactions of groups of 2-10 agents ( Bansal et al. , 2017 ; OpenAI ; Jaderberg et al. , 2018 ) , we focus on large populations of agents in a complex world . Zheng et al . ( 2017 ) also analyze behavior of many agents , but the task setting is considerably simpler , and agents are directly rewarded for foraging or fighting . In contrast , our work focuses on complexity in a diverse environment where agents are only rewarded for survival ; doing so requires maintaining their health through food and water while navigating partially obstructed terrain and with various forms of combat . Yang et al . ( 2018b ) also simulate populations scaling to millions of learning agents , but they focus on predator-prey population dynamics in this setting . Artificial Life “ Artificial life ” projects aim to simulate the complexity of biological life ( Langton , 1997 ) , often framed as a multiagent competition . This setting can lead to the development of skilled behaviors ( Yaeger , 1994 ) and capable morphologies ( Sims , 1994 ) . We consider the setting of coevolution ( Ficici & Pollack , 1998 ) in which agents coadapt alongside others . Our platform can simulate a kind of artificial life , but at a relatively high level of abstraction . While some basic features of our environment ( movement , food , water ) are similar to those in ( HernándezOrallo et al. , 2011 ; Strannegrd et al. , 2018 ) , our environment is grounded in the established game genre of MMOs . Unlike most past work in this area , our platform is built around deep reinforcement learning , where each agent is a neural net trained with distributed policy gradients , and includes tools for visualizing properties specific to this setting . Game Platforms for Intelligent Agents The Arcade Learning Environment ( ALE ) ( Bellemare et al. , 2013 ) and Gym Retro ( Nichol et al. , 2018 ) provide 1000+ limited scope arcade games most often used to test individual research ideas or generality across many games . Strictly better performance at a large random subset of games is a reasonable metric of quality , and strikingly divergent performance can motivate further research into a particular mode of learning . For example , Montezumas Revenge in ALE is difficult even with reasonable exploration because of the associative memory required to collect and use keys . However , recent results have brought into question the overall complexity each individual environment ( Cuccu et al. , 2018 ) and strong performance in such tasks ( at least in those not requiring precise reflexes ) is not particularly difficult for humans . More recent work has demonstrated success on multiplayer games including Go ( Silver et al. , 2016 ) , the Multiplayer Online Battle Arena ( MOBA ) DOTA2 ( OpenAI ) , and Quake 3 Capture the Flag ( Jaderberg et al. , 2018 ) . Each of these projects has advanced our understanding of a class of algorithms . However , these games were limited to 2-12 players , are round based on the order of an hour , lack persistence , and lack the game mechanics supporting large persistent populations – there is still a large gap in environment complexity compared to the real world . MMORPGs The game genre we focus on in this paper is MMORPGs , which are role-playing games ( RPGs ) in which many human players take part . RPGs , such as Pokemon and Final Fantasy , involve reasoning over up to hundreds of hours of persistent gameplay – a much longer time horizon than in MOBAs . Like the real world , RPGs confront the player with problems that have many valid solutions , and choices that have long term consequences . MMOs are the ( massively ) multiplayer analogs to RPGs . They are typically run across several servers , each of which contains a copy of the environment and supports hundreds to millions of concurrent players . Good MMOs involve a curriculum of challenges that require increasingly clever usage of the game systems . Early game content is accessible to new players , but skills required for late game content are inaccessible ( and often incomprehensible ) to those not intimately familiar with the game . Players have to acquire resources and “ level up ” in order to reach more advanced stages of the game . Such a curriculum is present in many game genres , but only MMOs contextualize it within social and economic structures approaching the scale of the real world . 3 NEURAL MMO . We present a persistent and massively multiagent environment that defines foraging and combat systems over procedurally generated maps ( see Appendix ) . The purpose of the platform is to discover game mechanics that support complex behavior and agent populations that can learn to make use of them . We follow the iterative development cycle of human MMOs : developers create balanced mechanics while players maximize their skill in utilizing them . The initial configurations of our systems are the results of several iterations of balancing , but are by no means fixed : every numeric parameter presented is editable within a simple configuration file . Parameter Value Notes Training Algorithm Policy GradientsWilliams ( 1992 ) + Value function baseline Adam Parameters lr=1e-3 Pytorch Defaults Weight Decay 1e-5 Training stability is sensitive to this Entropy Bonus 1e-2 To stabilize training ; possibly redundant Discount Factor 0.99 No additional trajectory postprocessing A map consists of a set of discrete tiles , which are positions an agent can occupy . On each step of the simulations , agents may move one tile North/South/East/West . Agents competes for food tiles while periodically refilling their water supply from infinite water tiles . They may attack each other using any of three attack options , each with different damage values and trade offs . The environment assumes only that agents receive local game state and output a decision . The environment is agnostic to the source of that decision , be it a neural network or a hardcoded algorithm . We have tested up to 100M agent trajectories ( lifetimes ) on 100 cores in 1 week . The code base already contains additional module code for trade , gathering , crafting , item systems , equipment , communication , and trade to name a few . We are actively balancing and integrating these into the neural API . 4 ARCHITECTURE AND TRAINING . Input Agents observe local game state–all tiles within a fixed L1 distance of their current position , including tile terrain types and the visible properties of occupying agents . This is an efficient equivalent representation of what a human sees on the screen without requiring rendering . Output Agents output action choices for the next time step ( “ game tick ” ) . For the experiments below , actions consist of one movement and one attack . As described in Framework , movement options are : North , South , East , West , Pass ( no movement ) . Attack options are : Melee , Range , Mage . This is purely flavor ; for those not familiar with MMOs , each attack option simply applies a preset amount of damage at a preset effective distance . The environment will attempt to execute both actions . Invalid actions , such as moving into a stone wall , are ignored . Our policy architecture is detailed in the Appendix . We provide a simple preprocessor that embeds and flattens this stimulus into a single fixed length environment vector and a list of entity embeddings . We apply a linear layer to the preprocessed embeddings followed by three output heads for movement , attacks . There is also a standard value head which is trained to predict the discounted expected lifetime of the agent . Each head is also a linear layer . New types of action choices can be included by adding additional heads . We train with simple policy gradients plus a value baseline . Agents receive only a stream of reward 1 . Rewards are postprocessed by a discounting factor , producing returns equal to a discounted estimate of the agent ’ s time until death . We found it possible to obtain good performance without discounting , but training was less stable .

Summary: This paper proposes a multi agent life simulators as an environment for RL. The environment is procedurally generated, with possibly many different game dynamics including foraging, and combat. They train deep RL agents in this environment and show various emergent behaviors such as exploration, and niche development. Additionally, they propose a tournament competition scheme to evaluate different populations of agents against each other.