PettingZoo: Gym for Multi-Agent Reinforcement Learning

1 Introduction . Multi-Agent Reinforcement Learning ( MARL ) has been behind many of the most publicized achievements of modern machine learning — AlphaGo Zero [ Silver et al. , 2017 ] , OpenAI Five [ OpenAI , 2018 ] , AlphaStar [ Vinyals et al. , 2019 ] . These achievements motivated a boom in MARL research , with Google Scholar indexing 9,480 new papers discussing multi-agent reinforcement learning in 2020 alone . Despite this boom , conducting research in MARL remains a significant engineering ∗Swarm Labs †Department of Computer Science | University of Maryland , College Park ‡Department of Electrical and Computer Engineering | University of Maryland , College Park §Department of Mechanical Engineering | University of Maryland , College Park ¶Maryland Robotics Center | University of Maryland , College Park ‖Faculty of Electrical Engineering and Computer Science | Technical University of Berlin 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . challenge . A large part of this is because , unlike single agent reinforcement learning which has OpenAI ’ s Gym , no de facto standard API exists in MARL for how agents interface with environments . This makes the reuse of existing learning code for new purposes require substantial effort , consuming researchers ’ time and preventing more thorough comparisons in research . This lack of a standardized API has also prevented the proliferation of learning libraries in MARL . While a massive number of Gym-based single-agent reinforcement learning libraries or code bases exist ( as a rough measure 669 pip-installable packages depend on it at the time of writing GitHub [ 2021 ] ) , only 5 MARL libraries with large user bases exist [ Lanctot et al. , 2019 , Weng et al. , 2020 , Liang et al. , 2018 , Samvelyan et al. , 2019 , Nota , 2020 ] . The proliferation of these Gym based learning libraries has proved essential to the adoption of applied RL in fields like robotics or finance and without them the growth of applied MARL is a significantly greater challenge . Motivated by this , this paper introduces the PettingZoo library and API , which was created with the goal of making research in MARL more accessible and serving as a multi-agent version of Gym . Prior to PettingZoo , the numerous single-use MARL APIs almost exclusively inherited their design from the two most prominent mathematical models of games in the MARL literature—Partially Observable Stochastic Games ( “ POSGs ” ) and Extensive Form Games ( “ EFGs ” ) . During our development , we discovered that these common models of games are not conceptually clear for multi-agent games implemented in code and can not form the basis of APIs that cleanly handle all types of multi-agent environments . To solve this , we introduce a new formal model of games , Agent Environment Cycle ( “ AEC ” ) games that serves as the basis of the PettingZoo API . We argue that this model is a better conceptual fit for games implemented in code . and is uniquely suitable for general MARL APIs . We then prove that any AEC game can be represented by the standard POSG model , and that any POSG can be represented by an AEC game . To illustrate the importance of the AEC games model , this paper further covers two case studies of meaningful bugs in popular MARL implementations . In both cases , these bugs went unnoticed for a long time . Both stemmed from using confusing models of games , and would have been made impossible by using an AEC games based API . The PettingZoo library can be installed via pip install pettingzoo , the documentation is available at https : //www.pettingzoo.ml , and the repository is available at https : //github.com/ Farama-Foundation/PettingZoo . 2 Background and Related Works . Here we briefly survey the state of modeling and APIs in MARL , beginning by briefly looking at Gym ’ s API ( Figure 1 ) . This API is the de facto standard in single agent reinforcement learning , has largely served as the basis for subsequent multi-agent APIs , and will be compared to later . from ray.rllib.examples.env.multi_agent import MultiAgentCartPole env = MultiAgentCartPole ( ) observation = env.reset ( ) for _ in range ( 1000 ) : actions = policies ( agents , observation ) observation , rewards , dones , infos = env.step ( actions ) Figure 2 : An example of the basic usage of RLlib The Gym API is a fairly straightforward Python API that borrows from the POMDP conceptualization of RL . The API ’ s simplicity and conceptual clarity has made it highly influential , and it naturally accompanying the pervasive POMDP model that ’ s used as the pervasive mental and mathematical model of reinforcement learning [ Brockman et al. , 2016 ] . This makes it easier for anyone with an understanding of the RL framework to understand Gym ’ s API in full . 2.1 Partially Observable Stochastic Games and RLlib . Multi-agent reinforcement learning does not have a universal mental and mathematical model like the POMDP model in single-agent reinforcement learning . One of the most popular models is the partially observable stochastic game ( “ POSG ” ) . This model is very similar to , and strictly more general than , multi-agent MDPs [ Boutilier , 1996 ] , Dec-POMDPs [ Bernstein et al. , 2002 ] , and Stochastic ( “ Markov ” ) games [ Shapley , 1953 ] ) . In a POSG , all agents step together , observe together , and are rewarded together . The full formal definition is presented in Appendix C.1 This model of simultaneous stepping naturally translates into Gym-like APIs , where the actions , observations , rewards , and so on are lists or dictionaries of individual values for agents . This design choice has become the standard for MARL outside of strictly turn-based games like poker , where simultaneous stepping would be a poor conceptual fit [ Lowe et al. , 2017 , Zheng et al. , 2017 , Gupta et al. , 2017 , Liu et al. , 2019 , Liang et al. , 2018 , Weng et al. , 2020 ] . One example of this is shown in Figure 2 with the multi-agent API in RLlib [ Liang et al. , 2018 ] , where agent-keyed dictionaries of actions , observations and rewards are passed in a simple extension of the Gym API . This model has made it much easier to apply single agent RL methods to multi-agent settings . However , there are two immediate problems with this model : 1 . Supporting strictly turn-based games like chess requires constantly passing dummy actions for non-acting agents ( or using similar tricks ) . 2 . Changing the number of agents for agent death or creation is very awkward , as learning code has to cope with lists suddenly changing sizes . 2.2 OpenSpiel and Extensive Form Games . In the cases of strictly turn based games where POSG models are poorly suited ( e.g . Chess ) , MARL researchers generally mathematically model the games as Extensive Form Games ( “ EFG ” ) . The EFG represents games as a tree , explicitly representing every possible sequence of actions as a root to leaf path in the tree . Stochastic aspects of a game ( or MARL environment ) are captured by adding a “ Nature ” player ( sometimes also called “ Chance ” ) which takes actions according to some given probability distribution . For a full definition of EFGs , we refer the reader to Osborne and Rubinstein [ 1994 ] or Appendix C.2 . OpenSpiel [ Lanctot et al. , 2019 ] , a major library with a large collection of classical board and card games for MARL bases their API off of the EFG paradigm , the API of which is shown in Figure 3 . The EFG model has been successfully used for solving problems involving theory of mind with methods like game theoretic analysis and tree search . However , for application in general MARL problems , three immediate concerns arise with the EFG model : 1 . The model , and the corresponding API , is very complex compared to that of POSGs , and isn ’ t suitable for beginners the way Gym is—this environment API is much more complicated than Gym ’ s API or RLLib ’ s POSG API for example . Furthermore , due to the complexity of the EFG model , reinforcement learning researchers don ’ t ubiquitously use it as a mental model of games in the same way that they use the POSG or POMDP model . 2 . The formal definition only includes rewards at the end of games , while reinforcement learning often requires frequent rewards . While this is possible to work around in the API implementation , it is not ideal . 3 . The OpenSpiel API does not handle continuous actions ( a common and important case in RL ) , though this was a choice that is not inherent to the EFG model . It ’ s also worth briefly noting that some simple strictly turn based games are modeled with the single-agent Gym API , with the environment alternating which agent is controlled , [ Ha , 2020 ] . This approach is unable to reasonably scale beyond two agents due to the difficulties of handling changes in agent order ( e.g . Uno ) , agent death , and agent creation . 3 PettingZoo Design Goals . Our development of PettingZoo both as a general library and an API centered around the following goals . 3.1 Be like Gym . In PettingZoo , we wanted to leverage Gym ’ s ubiquity , simplicity and universality . This created two concrete goals for us : • Make the API look and feel like Gym , and relatedly make the API pythonic and simple • Include numerous reference implementations of games with the main package Reusing as many design metaphors from Gym as possible will help its massive existing user base to almost instantly understand PettingZoo ’ s API . Similarly , for an API to become standardized , it must support a large collection of useful environments to attract users and for adoption to begin , similar to what Gym did . 3.2 Be a Universal API . If there is to be a Gym-like API for MARL , it has to be able to support all use cases and types of environments . Accordingly , several technically difficult cases exist that have to be carefully considered : • Environments with large numbers of agents • Environments with agent death and creation • Environments where different agents can be chosen to participate in each episode • Learning methods that require access to specialty low level features Two related softer design goals for universal design are ensuring the API is simple enough for beginners to easily use , and making the API easily changeable if the direction of research in the field dramatically changes . 4 Case Studies of Problems With The POSG Model in MARL . To supplement the description of the problems with the POSG models described in Section 2.1 , we overview problems with basing APIs around these models that could theoretically occur in software games , and then examine real cases of those problems occurring in popular MARL environments . We specifically focus on POSGs here because EFG based APIs are extraordinarily rare ( OpenSpiel is the only major one ) , while POSG based ones are almost universal . 4.1 POSGs Don ’ t Allow Access To Information You Should Have . Another problem with modeling environments using simultaneous actions in the POSG model is that all of an agent ’ s rewards ( from all sources ) are summed and returned all at once . In a multi-agent game though , this combined reward is often the composite reward from the actions of other agentss and the environment . Similarly , you might want to be able to attribute the source of this reward for various learning reasons , or for debugging purposes to find out the origin of your rewards . However , in thinking about reward origins , having all rewards emitted at once proves to be very confusing because rewards from different sources are all combined . Accessing this information via an API modeled after a POSG requires deviating from the model . This would come in the form of returning a 2D array of rewards instead of a list , which would be difficult to standardize and inconvenient for learning code to parse . A notable case where this caused an issue in practice is in the popular pursuit gridworld environment from Gupta et al . [ 2017 ] , shown in Figure 4 . In it , 8 red controllable pursuer must work together to surround and capture 30 randomly moving blue evaders . The action space of each pursuer is discrete ( cardinal directions or do nothing ) , and the observation space is a 7× 7 box centered around a pursuer ( depicted by the orange box ) . When an evader is surrounded on all sides by pursuers or the game boundaries , each contributing pursuer gets a reward of 5 . In pursuit , pursuers move first , and then evaders move randomly , before it ’ s determined if an evader is captured and rewards are emitted . Thus an evader that “ should have ” been captured is not actually captured . Having the evaders move second isn ’ t a bug , it ’ s just way of adding complexity to the classic genre of pursuer/evader multi-agent environments [ Vidal et al. , 2002 ] , and is representative of real problems . When pursuit is viewed as an AEC game , we ’ re forced to attribute rewards to individual steps , and the breakdown becomes pursuers receiving deterministic rewards from surrounding the evader , and then random reward due to the evader moving after . Removing this random component of the reward ( the part caused by the evaders action after the pursuers had already moved ) , should then lead to superior performance . In this case the problem was so innocuous that fixing it required switching two lines of code where their order made no obvious difference . We experimentally validate this performance improvement in Appendix A.1 , showing that on average this change resulted in up to a 22 % performance in the expected reward of a learned policy . Bugs of this family could easily happen in almost any MARL environment , and analyzing and preventing them is made much easier when using the POSG model . Because every agent ’ s rewards are summed together in the POSG model , this specific problem when looking at the code was extraordinarily non-obvious , whereas when forced to attribute the reward of individual agents this becomes clear . Moreover if an existing environment had this problem , by exposing the actual sources of rewards to learning code researchers are able to remove differing sources of reward to more easily find and remove bugs like this , and in principle learning algorithms could be developed that automatically differently weighted different sources of reward .

Summary: This paper introduces a PettingZoo, a python API similar to OpenAI gym for multi-agent reinforcement learning (MARL). The environment has been in use for several years, and has 62 built in environments, is supported by a number of libraries, and is already used by many other groups and students. A key contribution of the paper is explicating the Agent Environment Cycle (AEC) model that powers PettingZoo, and its key distinguishing features in comparison to alternative MARL models.