PettingZoo: Gym for Multi-Agent Reinforcement Learning
Authors: J K Terry
Benjamin J Black
Nathaniel Grammel
Mario Jayakumar
Ananth Hari
Ryan Sullivan
Luis Santos
Clemens Dieffendahl
Caroline Horsch
Rodrigo De Lazcano Perez-Vicente
Niall L Williams
Yashas Lokesh
Praveen Ravi
Venue: NeurIPS 2021 Poster (2021)
Keywords: Reinforcement Learning
Multi-agent Reinforcement Learning
Gym

Abstract: This paper introduces the PettingZoo library and the accompanying Agent Environment Cycle (``"AEC") games model. PettingZoo is a library of diverse sets of multi-agent environments with a universal, elegant Python API. PettingZoo was developed with the goal of accelerating research in Multi-Agent Reinforcement Learning (``"MARL"), by making work more interchangeable, accessible and reproducible akin to what OpenAI's Gym library did for single-agent reinforcement learning. PettingZoo's API, while inheriting many features of Gym, is unique amongst MARL APIs in that it's based around the novel AEC games model. We argue, in part through case studies on major problems in popular MARL environments, that the popular game models are poor conceptual models of the games commonly used with MARL, that they promote severe bugs that are hard to detect, and that the AEC games model addresses these problems.

PettingZoo: A Standard API for Multi-Agent Reinforcement Learning
K. Terry*t
Benjamin Black amin. blackOswarmlabs. com
Nathaniel Grammel ngrammel @umd. edu
j j.k. terry@swarmlabs........ com
Mario Jayakumar' Mariojay@uma edu
Ananth Hari² ahari1@ume 1arr10umd.om edu
Ryan Sullivan" f
ryan. sullivan@swarmlabs. com
Luis Santos lss@umd. edu
Rodrigo Perez rlazcano@umd.c edu
Caroline Horsch "
caroline www.hhh@swarlaaass.ooom com
Clemens Dieffendahl
Niall L. Williams niallw@umd edu
Yashas Lokesh
dieffendahle ewreeennnah@@camus..mmm tu- www.berlinnce de
yashloke@und edu
Praveen Ravit praviQund @umd. edu
Abstract
This paper introduces the PettingZoo library and the accompanying Agent Envi- ronment Cycle ("AEC") games model. PettingZoo a library of diverse sets multi-agent environments with universal, elegant Python API. PettingZoo developed with the goal of accelerating research in Multi-Agent Reinforcement Learning ("MARL"), by making work more interchangeable, accessible and producible akin to what OpenAl's Gym library did for single-agent reinforcement learning. PettingZoo's API, while inheriting many features of Gym, is unique amongst MARL APIs in that it's based around the novel AEC games model. We argue, part through case studies on major problems in popular MARL envi- ronments, that the popular game models are poor conceptual models of games commonly used in MARL and accordingly can promote confusing bugs that are hard to detect, and that the AEC games model addresses these problems.
Introduction
Multi-Agent Reinforcement Learning (MARL) has been behind many of the most publicized achieve- ments modern machine learning AlphaGio Zero [Silver et al. 2017], OpenAl Five [OpenAT
2018). AlphaStar [Vinyals et al., 2019 These achievements motivated a boom in MARL research, with Google Scholar indexing 9,480 new papers discussing multi-agent reinforcement learning in 2020 alone. Despite this boom, conducting research in MARL remains significant engineering
Swarm Labs
1 Department of Computer Science University of Maryland, College Park
DDepartment of Electrical and Computer Engineering University of Maryland, College Park
$ Department of Mechanical Engineering University of Maryland, College Park
laryland Rohotics Center I University of Maryland, College Park
|| Faculty Electrical Engineering and Computer Science Technical University of Berlin
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

challenge. A large part of this is because, unlike single agent reinforcement learning which has OpenAl's 's Gym, no de facto standard API exists in MARL for how agents interface with environments. This makes the reuse of existing learning code for new purposes require substantial effort, consuming researchers' time and preventing more thorough comparisons in research. This lack of a standardized API has also prevented the proliferation of learning libraries in MARL.. While massive number of Gym-based single-agent reinforcement learning libraries or code bases exist (as a rough measure 669 pip-installable packages depend on it at the time of writing GitHub (2021]), only 5 MARL libraries with large user bases exist (Lanctot et al. 2019 Weng et al. 2020 Liang et al. 2018 Samvelyan al. 2019 Nota 2020]. The proliferation of these Gym based learning libraries has proved essential to the adoption of applied RL fields like robotics or finance and without them the growth of applied
MARL is significantly greater challenge. Motivated by this, this paper introduces the PettingZoo library and API, which was created with the goal of making research in MARI. more accessible and serving as a multi-agent version of Gym.
Prior to PettingZoo, the numerous single-use MARL APIs almost exclusively inherited their design from the two most prominent mathematical models of games in the MARL literature-Partinn Observable Stochastic Games ("POSGs") and Extensive Form Games ("EFGs"). During our develop- ment, we discovered that these common models of games are not conceptually clear for multi-agent games implemented code and cannot form the basis of APIs that cleanly handle all types of multi-agent environments.
To solve this, we introduce a new formal model of games, Agent Environment Cycle ("AEC") games that serves as the basis of the PettingZoo API. We argue that this model is better conceptual fit for games implemented
code. and is uniquely suitable for general MARL APIs. We then prove that any AEC game can be represented by the standard POSG model, and that any POSG can be represented by an AEC game. To illustrate the importance of the AEC games model, this paper further covers two case studies of meaningful bugs in popular MARL implementations, In both cases, these bugs went unnoticed for a long time. Both stemmed from using confusing models games, and would have been made impossible by using an AEC games based API.
The PettingZoo library can be installed via pip install pettingzoo, the documentation is avail- able at https://wwwpptttiiggoo ml, and the repository is available at httth::/ / com/ Paraaa-FFommaatioo/Pettttooo
2 Background and Related Works
Here we briefly survey the state of modeling and APIS in MARL, beginning by briefly looking at Gym's API (Figure I). This API is the de facto standard in single agent reinforcement learning. has largely served as the basis for subsequent multi-agent APIs, and will be compared to later.
from ray. aay.rllib. www.ib..xaapple............ examplan, cnv wnv.e.vv.ll....m..... www.mmmlttsaaaoeo mllti_agont import tarr (altssggnttarrPolo()
Import gyn
env gyn make (xke(CCarPPolo-vo)) observation resat()
observation env resat: for in range(1000):
for in range(1000): action policy (observtiio))) observation, revard, done, info env env (tcptaiiioo)
actione policies(agente, obeervation) observation, revards, dones, infes anv anv. stt.sa(p(cctons)
Figure 1: An example of the basic usage of Gym
Figure 2: An example of the basic usage of RLlib
The Gym API is a fairly straightforward Python API that borrows from the POMDP conceptualization of RL. The API's simplicity and conceptual clarity has made highly influential, and it naturally accompanying the pervasive POMDP model that's used as the pervasive mental and mathematical model of reinforcement learning [Brockman al.) 2016). This makes it easier for anyone with an understanding of the RL framework to understand Gym's API in full.
2.1 Partially Observable Stochastic Games and RLJib
Multi-agent reinforcement learning does not have universal mental and mathematical model like the POMDP model in single-agent reinforcement learning. One of the most popular models the partially observable stochastic game ("POSG"). This model is very similar to, and strictly more

---

Dec-POMDPs [Bernstein et al. 2002). and Stochastic ("Markov") games [Shapley] 1953] In POSG, all agents step together, observe together, and are rewarded together. The full formal definition is presented in Apppeddix C.1
general than, multi-agent MDPs [Buutllirr 1996].
This model of simultaneous stepping naturally translates into Gym-like APIs, where the actions, observations, rewards, and so on are lists or dictionaries of individual values for agents. This design choice has become the standard for MARL outside of strictly turn-based games like poker, where simultaneous stepping would be poor conceptual fit (Lowe et al.
2017 [heng et al.) 2017 Cupta
al. 2017 Liu et al. 2019 Liang et al., 2018 Weng et al. 2020 One example of this is shown in
Figure with the multi agent API in RLlib [Liang et al. 2018), where agent-keyed dictionaries of
actions, observations and rewards are passed in simple extension of the Gym API.
This model has made much easier to apply single agent RL methods to multi-agent settings. However, there two immediate problems with this model:
1. Supporting strictly surn-based games like chess requires constantly passing dummy actions for non-acting agents (or using similar tricks).
2. Changing the number of agents for agent death or creation is very awkward, as learning code has cope with lists suddenly changing sizes.
2.2 OpenSpiel and Extensive Form Games
In the cases of strictly turn based games where POSG models are poorly suited (e.g. Chess), MARL researchers generally mathematically model the games as Extensive Form Games ("EFG"). The EFG represents games as tree, explicitly representing every possible sequence of actions as a root to leaf path in the tree. Stochastic aspects of a game (or MARL environment) are captured by adding a "Nature" player (sometimes also called "Chance") which takes actions according to some given probability distribution. For full definition of EFGs, refer the reader Osboree and Rubinstein [1994] or [Appendix C.2) OpenSpiel [Lanctot et al. 2019). a major library with large collection of classical board and card games for MARL bases their API off of the EFG paradigm, the API of which is shown in Figure 3
inport pyspiel import numpy up
gane pyspiel load_gane ('kuhn_pooeee state game ney set__inttall_sttttte
while not state. aaraiia_ttrmnnn terminal if state leste.ie,ae see_chance__oee :o)ce)
Step the stochastic environnent.
action_list, prob_list sip(*state (otioness))))
state state.apply.eee apply_action(up randon wttp:/ppp.aaaaocooooooaeeeeee choice(actionn,1iee.... list, p-prob_list)
else:
semple action for the agent
lsgal_actions state. state.logal..actonn
observations state obbervatioi. tenser()
action policies current_agent legoi_actions observations) caste.apply_eeee (ation(cioon)))))
revards state rewards()
Figure 3: An example of the basic usage of OpenSpiel
The EFG model has been successfully used for solving problems involving theory of mind with methods like game theoretic analysis and tree search. However, for application in general MARL problems, three immediate concerns arise with the EFG model:
1. The model, and the corresponding API, is very complex compared that of POSGs, and isn't suitable for beginners the way Gym is-this environment API is much more complicated than Gym's 's API or RLLib's POSG API for example. Furthermore, due to the complexity of the EFG model, reinforcement learning researchers don't 't ubiquitously use it as a mental model of games in the same way that they use the POSG or POMDP model.
2. The formal definition only includes rewards at the end of games, while reinforcement learning often requires frequent rewards. While this is possible to work around in the API implementation, it is not ideal.
3

---

3. The OpenSpiel API does not handle continuous actions (a common and important case in RL), though this was choice that is not inherent the EFG model.
It's also worth briefly noting that some simple strictly turn based games modeled with the single-agent Gym API, with the environment alternating which agent is controlled, (Ha 2020). This approach is unable reasonably scale beyond two agents due the difficulties of handling changes in agent order (e.g. Uno), agent death, and agent creation.
3 PettingZoo Design Goals
Our development of PettingZoo both general library and API centered around the following goals.
3.1 Be like Gym
In PettingZoo, wanted to leverage Gym's ubiquity, simplicity and universality. This created two concrete goals for us:
Make the API look and feel like Gym, and relatedly make the API pythonic and simple
Include numerous reference implementations of games with the main package
Reusing as many design metaphors from Gym as possible will help its massive existing user base to almost instantly understand PettingZoo's 's API. Similarly, for API to become standardized, must support large collection of useful environments to attract users and for adoption to begin, similar to what Gym did.
3.2 Be Universal API
there is to be a Gym-like API for MARL, it has to be able to support all use cases and types of environments. Accordingly, several technically difficult cases exist that have to be carefully considered:
Environments with large numbers of agents
Environments with agent death and creation
Enviromments where different agents can be chosen to participate in each episode
Learning methods that require access to specialty low level features
Two related softer design goals for universal design are ensuring the API is simple enough for beginners to easily use, and making the API easily changeable if the direction of research in the field dramatically changes.
Case Studies of Problems With The POSG Model in MARL
To supplement the description of the problems with the POSG models described in Section 2.1 overview problems with basing APIs around these models that could theoretically occur in software games, and then examine real cases of those problems occurring in popular MARL environments. We specifically focus on POSGs here because EFG based APIs are extraordinarily rare (OpenSpiel is the only major one), while POSG based ones are almost universal.
4.1 POSGs Don't 't Allow Access To Information You Should Have
Another problem with modeling environments using simultaneous actions in the POSG model is that all of an agent's rewards (from all sources) are summed and returned all at once. In multi-agent game though, this combined reward often the composite reward from the actions of other agentss and the environment. Similarly, you might want to be able to attribute the source of this reward for various learning reasons, or for debugging purposes to find out the origin of your rewards. However, in thinking about reward origins, having all rewards emitted at once proves to be very confusing

---

because rewards from different sources are all combined. Accessing this information via an API modeled after POSG requires deviating from the model. This would come the form of returning a 2DD array of rewards instead of list, which would be difficult to standardize and inconvenient for learning code to parse.
A notable case where this caused an issue in practice is in the popular pursuit gridworld environment from Gupta al. [2017). shown in Figure In red controllable pursuer must work together to surround and capture 30 randomly moving blue evaders. The action space of each pursuer is discrete (cardinal directions or do nothing), and the observation space is 7 7 box centered around pursuer (depicted by the orange box). When an evader is surrounded on all sides by pursuers or the game boundaries, each contributing pursuer gets reward of 5.
Figure 4: The pursuit environment from Gupta et al.
2017
In pursuit, pursuers move first, and then evaders move randomly, before it's determined an evader is captured and rewards are emitted. Thus an evader that "should have" been captured is not actually captured. Having the evaders move second isn't a bug, it's just way of adding complexity the classic genre of pursuer/evader multi-agent environments [Vidal al., 2002). and is representative of real problems. When pursuit is viewed as an ABC game, we re forced to attribute rewards to individual steps, and the breakdown becomes pursuers receiving deterministic rewards from surrounding the evader, and then random reward due to the evader moving after Removing this random component of the reward (the part caused by the evaders action after the pursuers had already moved), should then lead to superior performance. In this case the problem was so innocuous that fixing it required switching two lines of code where their order made obvious difference. We experimentally validate this performance improvement in Appendix A. showing that on average this change resulted in up to 22% performance in the expected reward of learned policy.
Bugs of this family could easily happen in almost any MARL environment, and analyzing and preventing them is made much easier when using the POSG model. Because every agent's rewards are summed together in the POSG model, this specific problem when looking at the code was extraordinarily non-obvious, whereas when forced to attribute the reward of individual agents this becomes clear. Moreover if an existing environment had this problem, by exposing the actual sources of rewards to learning code researchers are able to remove differing sources of reward to more casily find and remove bugs like this, and in principle learning algorithms could be developed that automatically differently weighted different sources of reward.
4.2 POSGs Based APIs Are Not Conceptually Clear For Games Implemented In Code
Introducing race conditions is very easy mistake make in MARL code in practice, and this occurs because simultaneous models of multi-agent games are not representative of how game code normally executes. This stems from a very common scenario in multi-agent environments where two agents are able to take conflicting actions (i.e. moving into the same space). This discrepancy has to be resolved by the environment (i.c. collision handling); which we call "tic-breaking."
Consider an environment with two agents, Alice and Bob, in which Alice steps first and bie-breaking biased in Alice's favor. If such an environment were assumed to have simultancous actions, then observations for both agents would be taken before either acted, causing the observation Bob acts on to no longer be an accurate representation of the environment if a conflict with biased tie- frebbraking occurs. For example, if both agents tried to step into the same square and Alice got the square because she was first the list, Bob's observation before acting was effectively inaccurate and the environment was not truly parallel. This behavior is a true race condition the result of stepping through the environment can inadvertently differ depending on the internal resolution order of agent actions.
5

---

In any environment that's even slightly complex, tremendous number of instances where tie tie-brea must be handled will typically occur. In any cases where a single one is missed, the environment will have race conditions that your code will attempt to learn. While finding these will always be important, a valuable tool to mitigate these possibilities is to use an API that treats each agent as acting sequentially, returning new observations afterwards. This entirely prevents the opportunity for introducing race conditions. Moreover, this entire problem stems from the fact that using APIs that model agents as updating sequentially for software MARL environments generally makes more conceptual sense than modeling the updates as simultaneous-
the authors of environments very complex parallelization, the environments will actually be updated one agent at time. It is worth mentioning that this race condition cannot occur in an environment simulated in the physical world with continuous time simulated environment with sufficient amount of observation delay (though most actively researched environment in MARL do not currently have any observation delay).
we go through a case study of a race condition like this happening in the open source implementation of the social sequential dilemma game environments (Vinitsky al. 2019]. These are popular multi-agent grid world environments intended to study emergent behaviors for various forms of resource management, and has imperfect tie-breaking in a case where two agents try to act on resources in the same grid while using simultaneous API. This bug in particular illustrates how extraordinarily difficult making all tie- breaking truly unbiased is in practice even for fairly simple environments. We defer this to the appendix explaining the specific origin requires large amount of exposition and diagrams about the rules of the environment.
In Anpppeddix A.
5 The Agent Environment Cycle Games Model
Motivated by the problems with applying the POSG and EFG models to MARL APIs, we developed the Agent Environment Cycle ("AEC") Game. In this model, agents sequentially see their observation, agents take actions, rewards are emitted from the other agents, and the next agent to act is chosen. This is effectively a sequentially stepping form of the POSG model.
Modeling multi-agent environments sequentially for APIs has numerous benefits:
It allows for clearer attribution of rewards to different origins, allowing for various learning improvements, as described in [Section 4.1
It prevents developers adding confusing and easy-to-introduce race conditions, as described in Section 4.2
It more closely models how computer games are executed in code, as described in Sec
tion 4.2
It formally allows for rewards after every step as is required in RL, but is not generally a part of the EFG model, as discussed in [Section 2.2]
It is simple enough to serve as a mental model, especially for beginners, unlike the EFG model as discussed in [Section 2.2) and illustrated in the definition in Appendiix C.2
Changing the number of agents for agent death creation is less awkward, as learning code does not have to account for lists constantly changing sizes, as discussed in (Sectiion 2.1
It is the least bad option for universal API, compared to simultaneous stepping, as alluded to in [Section 2.1 Simultaneous stepping requires the use of no-op actions if not all agents can act which are very difficult to deal with, whereas sequentially stepping agents that could all act simultancously and queuing up their actions is not especially inconvenient.
In In|Appendix C.3 we mathematically formalize the AEC games model, however understanding the formalism in full is not essential to understanding the paper. In Appenddix D we further prove that for every AEC game equivalent POSG exists and that for every POSG an equivalent ABC game exists. This shows that the AEC games model is as powerful a model as the most common current model of multi-agent environments.
One additional conceptual feature of the AEC games model exists that we have not previously discussed because it does not usually play a role in APIs (see [Section 6.4). In the AEC games model, we deviate from the POSG model by introducing the "environment" agent, which is analogous to the Nature agent from EFGs. When this agent acts in the model it indicates the updating of

---

the environment itself, realizing and reacting to submitting agent actions. This allows for a more comprehensive attribution rewards, causes of agent death, and discussion of games with strange updating rules and race conditions. An example of the transitions for Chess is shown in Figure 5 which serves as the inspiration for the name "agent environment eycle".
Player
Environment Step
Environment Step
Player
Figure 5: The AEC diagram of Chess
API Design
6.1
Basic API
The PettingZoo API is shown in Figure 6 and the strong similarities to the Gym API Figuure I) should be obvious each agent provides action step function and receives observation, reward, done, info as the return values. The observation and state spaces also use the the exact same space objects as Gym. The render and close methods also function identically to Gym's 's, showing a current visual frame representing the environment to the screen whenever called. The reset method similarly has identical function to Gym - resets the environment to a starting configuration after being played through. PettingZoo really only has two deviations from the regular Gym API the last and agent_itee -itr methods and the corresponding iteration logic.
from pettingeoo, butterfly Inport pistonball_v0
eay pistonball_vo.com (nv()) rav.reset()
for agent in amv.agent_iter env render()
observation, revard, done, info env. last() action poltcy(obaervation agent) (tcpppa env. env.close closc)
Figure 6: An example of the basic usage of Pettingzoo
6.2 The agent_iter Method
The agent_ited method is generator method of an environment that returns the next agent that the environment will be acting upon. Because the environment providing the next agent to act, this cleanly abstracts away any issues surrounding changing agent orders, agent generation, and agent death. This generation also parallels the functionality of the next agent function from the ABC games model. This method, combined with one agent acting at once, allows for the support every conceivable variation of the set of agents changing.
6.3
The last Method
An odd aspect of multi-agent environments is that from the perspective of one agent, the other agents are part of the environment. Whereas in the single agent case the observation given immediately, in the multi-agent case agent has to wait for all other agents to act before it's observation, reward, done and info can be fully determined. For this reason, these values are given by the last method, and they can then be passed into policy to choose action. Less robust
rewards can be
7

---

implementations would not allow for features like changing agent orders (like the reverse card in Uno).
6.4 Additional API Features
The agents attribute is list of all agents in the environment, as strings. The rewards, dones, infos attributes are agent-keyed dictionaries for each attribute (note that the rewards are the in- stantaneous ones resulting from the most recent action). These allow access to agent properties at all points on a trajectory, regardless of which is selected. The action_ observation_spea spacoonsssseeeeeeeee (agent) functions return the static action and observation spaces respectively for the agent given as an argument. The observe (agent) method provides the observation for a single agent by passing its name as an argument, which can be useful if you need to observe an agent in an unusual context. The state method is an optional method returns the global state of an environment, as is required for centralized critic methods. The egent_selecte method returns the agent that can currently be acted upon per agent_itez:
(agent) and
The motivation for allowing access to all these lower level pieces of information is to let researchers to attempt novel, unusual experiments. The space of multi-agent RL has not yet been comprehensively explored, and there are many perfectly plausible reasons you might want access to other agents rewards, observations, and so on. For an API to be universal in an emerging field, inherently has to allow access to all the information researchers could plausibly want. For this reason we allow access to a fairly straightforward set of lower level attributes and methods in addition to the standard higher level API. As we outline in (Section 6.5 we've structured PettingZoo in a way such that including these low-level features doesn't introduce engineering overhead in creating environments, as discussed further in the documentation website.
To handle environments where different agents can be present on each reset of an environment, PettingZoo has an optional possible _agents attribute which lists all the agents that might exist in an environment at any point. Environments which generate arbitrary numbers or types of agents will not define possible_agents e_agents list, requiring the user to check for new agents being instantiated as the environment runs. After resetting the environment, the agents attribute becomes accessible and lists all agents that are currently active. For similar reasons, hum_agentte rewards, dones, infos, and agent_ selection are not available until after a reset.
To handle cases where environments need to have environment agents per the formal AEC Games model, the standard is to put into the agents with the name env and have take None as it's action. We do not require this for all environments by default as it's s rarely used and makes the API more cumbersome, but this is an important feature for certain edge cases in research. This connects the formal model in that, when this feature is not used, the environment actor from the formal model and the agent actor that acted before are merged together.
6.5 Environment Creation and the Parallel API
PottingZoo environments actually only expose the reset, seed, step, observe, render, and close base methods and the agents, rewards, dones, infos, state and agent_ jite base attributes. These are then wrapped to add the last method. Only having environments implement primitive methods makes creating new environments simpler, and reduces code duplication. This has the useful side effect of allowing all PettingZoo environments to be easily changed to an alternative API by simply writing a new wrapper. We've ve actually already done this for the default environments and added additional "parallel API" to them that's almost identical to the RLlib POSG-based API via a wrapper. We added this secondary API because in environments with very large numbers of agents, this can improve runtime by reducing the number of Python function calls.
7 Default Environments
Similar to Gym's 's default environments, PettingZoo includes 63 environments. Half of the included environment classes (MPE, MAgent, and SISL), despite their popularity, existed as unmaintained "research grade" code, have not been available for installation via pip, and have required large amounts of maintenance to run at before our cleanup and maintainership. We additionally included multiplayer Atari games from Terry and Black 2020 Butterfly environments which original and
8

---

of our own creation, and popular classic board and card game environments. All default environments included are surveyed in depth in Appendix
Adoption
In it's relatively short lifespan, PettingZoo has already achieved a meaningful amount of adoption. It
supported by the following learning libraries: The Autonomous Learning Library [Nota] [020]]
Al-Traineree [aszzuk] 2020]. PyMARL (ongoing) (Samvelyan et al.) 2019). RI.ib [Liag et al.
2018). Stable Baselines [Hill et al.
2018 and Stable Baselines 3 |Raffin et al. 2019 similar
libraries such CleanRL [Huang et al. 2020 (through SuperSuit [Terry al. [2020b] and Tianshou
(ongoing) (Weng et al. 2020J. . Perhaps more significantly than any of this, PettingZoo is already
being used to teach in both graduate and undergraduate reinforcement learning classes all over the
world.
Conclusion
This paper introduces PettingZoo, Python library of many diverse multi-agent reinforcement learning environments under one simple API, akin to a multi-agent version of OpenAl's Gym library, and introduces the agent environment cycle game model of multi-agent games.
Given the importance of multi-agent reinforcement learning, we believe that PettingZoo capable of democratizing the field similar to what Gym previously did for single agent reinforcement learning, making it accessible to university scale research and to non-experts. As evidenced by it's early adoption into numerous MARL libraries and courses, PettingZoo is moving in the direction of accomplishing this goal.
We're aware of one notable limitation of the PettingZoo API. Games with significantly more than 10,000 agents (or potential agents) will have meaningful performance issues because you have to step each agent at once. Efficiently updating environments like this, and inferencing with the associated policies, requires true parallel support which almost certainly should be done a language other than Python. Because of this, we view this as practically acceptable limitation.
We see three directions for future work. The first is additions of more interesting environments under our API (possibly from the community, as has happened with Gym). The second direction we envision is a service allow different researchers' agents to play against each other in competitive games, leveraging the standardized API and environment set. Finally, we envision the development of procedurally generated multi-age agent environments to test how well methods generalize, akin the Gym procgen environments [Cobbe et al.) 2019].
Acknowledgements
Justin Terry was supported during part of this work by the QinetiQ Fundamental Machine Learning Fellowship. Thank you to Kyle Sang for their contributions to the documentation website. Thank you Rohan Potdar and Sang Hyun Son for their contributions to the Butterfly benchmarks. Thank you to Deepthi Raghunandan and Christian Clauss for their contributions to testing and continuous integration. Thank you to the PettingZoo community for the numerous bug reports and contributions to the package, especially Ross Allen and their group.
References
Nolan Bard, Jakob N. Foerster, Sarath Chandar, Neil Burch, Mare Lanctot, H. Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, Iain Dunning. Shibl Mourad, Hugo Larochelle, Mare G. Bellemare, and Michael Bowling. The hanabi challenge: A new frontier for Al research. CORR, abs/ 900.0000000 2019. http:/t //wxxvv org abs/ 00506
Daniel Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of Operations Reseurch, 27(4): 819-840. 2002. doi: 10. 27.
htRp:h/ts:///dtl org/10. 1287 /ww.//oor... 27
4. 4.819.. 297

---

Craig Boutilier. Planning, learning and coordination in multiagent decision processes. In Proceedings of the 6th conference on Theoretical aspects of rationality and knowledge, pages 195-210. Morgan Kaufmann Publishers Inc., 1996.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv: 01540, 2016.
Y. Chen, M. Zhou, Ying Wen, Y. Yang, Y. Su, W. Zhang, Dell Zhang. J. Wang, and Han Liu. Factorized q-learning for large-scale multi-agent systems. In DAI 19, 2019.
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. arXiv preprint arXiv: 1912.0/912,.0588.. 2019.
GitHub, openai/gym dependents. 2021.
URL https: httph:/ org/web/ 201166772445//ttss::/ttttt http://www.....ooooo//a/a//a///a h/gpe//a//ggymttttooto///eennssr deeendentt_type
cww/ com/openai.
Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer.
Cooperative multi-agent control using
deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems, pages 66-83. Springer, 2017.
David Ha. Slime volleyball gym environment.
https://github. hat.auu...oo/adrrmmrar....
slimevolleygym, 2020.
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. / com/
hill-a/stable sall-a//stalle-aaelinneee basebinseese ines] 2018.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. arXiv preprint arXiv: /803. 1803.00933, 2018.
Shengyi Huang, Rousslan Dossa, and Chang Ye. Cleanrl: High-quality single-iile implementation of deep reinforcement learning algorithms. https://githut com//vwxyzjn/cleanr] 2020.
Edward Hughes, Joel Z Leibo, Matthew Phillips, Karl Tuyls, Edgar Dueñez-Guzmann Antonio Gareía Castañeda, lain Dunning, Tina Zhu, Kevin McKee, Raphael Koster, al. Inequity aversion improves cooperation in intertemporal social dilemmas. In Advances in neural information processing systems, pages 3326-3336, 2018.
Mare Lanctot, Edward Lockhart, Jean -aptiste Lespiau, Vinícius Flores Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafici, Daniel Hennes, Dustin Morrill, Paul Muller, Timo Ewalds, Ryan Faulkner, János Kramár, Bart De Vylder, Brennan Saeta, James Bradbury, David Ding, Sebastian Borgeaud, Matthew Lai, Julian Schrittwieser, Thomas W. Anthony, Edward Hughes, Ivo Danihelka, and Jonah Ryan- -Dvis. Openspiel: A framework for reinforcement learning in games. CoRR, http//990.009533.. 2019. URL http: www.rxiv.org/abs/// abs 19t...//998909900
Dawid Laszuk. /ttaicaeeeeete////.ooooooooaaaar
2020.
Joel Z Leibo, Vinicius Zambaldi, Mare Lanctot, Janusz Marecki, and Thore Graepel. Multi-agent reinforcement learning in sequential social dilemmas. arXiv preprint arXiv:1702.03037, 2017.
Eric Liang, Richard Liaw, Philipp Moritz, Robert Nishihara, Roy Fox, Ken Goldberg, Joseph E Gonzalez, Michael I Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning, arXiv preprint 1rXiv:1712.095: 2017.
Eric Liang, Richard Liaw, Robert Nishihara. Philipp Moritz, Roy Fox, Ken Goldberg, Joseph E. Gonzalez, Michael I. Jordan, and Ion Stoica. RLlib: Abstractions for distributed reinforcement learning. In International Conference Machine Learning (ICML), 2018.
Siqi Liu, Guy Lever, Josh Merel, Saran Tunyasuvunakool, Nicolas Heess, and Thore Graepel. Emergent coordination through competition. CoRR, abs/1902.07151, 2019. http: lorg/abs, hagg/abs/100 /90p. 07151
10

---

Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-compettiie environments. Neural Information Processing Systems (NIPS), 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, nature, 51 518(7540):529-33 553-533,, 2015.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations, arXiv preprint vrXiv:170..049000 2017.
Chris Nota. The autonomous learning library.
https://githhb..commmpppn/
autonomous- wearoosss-eaaannnnnnee -iibrry 2020.
OpenAl. Openai five. https://blog opw.ai.enna.............. www./oo/ooaaaaec openai
wtt.:/oooooooaiiiaaaeaee five/ 2018.
Martin Osborne and Ariel Rubinstein. course in game theory. MIT press, 1994.
G. Palmer. Independent learning approaches: Overeoming multi-agent learning pathologies in team-games. 2020.
Stefanie Anna Baby Ling Li Ashwini Pokle. Analysis of emergent behavior in multi agent environ- ments using deep reinforcement learning. 2018.
Antonin Raffin. RI baselines3 700. ptttps://github.coo Sotpaaoooooo/LL--/////eaeeeelll
zzo 2020.
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah http:///gtthuu..ommmo///////aaaa
Dormann. Stable baselines3.
2019.
Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob N. Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. CoRR, abs/1902.04443, 2019. URL http:///rxxx org/ abs/1902 04043.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
L. S. Shapley. Stochastic games. Proceedings of the National Academy of Sciences, 19{1)):1095- 1100, 1953. ISSN 0027-8424. doi: (0.1073/pnas.33.10.10939
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittieeser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484448, 2016.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-335 2017.
Sriram Ganapathi Subramanian, P. Poupart, Matthew E. Taylor, and N. Hegde. Multi type mean field reinforcement learning. In AAMAS, 2020.
J K Terry, Nathaniel Grammel, Ananth Hari, Luis Santos, and Benjamin Black. Revisiting parameter sharing in multi-agent deep reinforcement learning. arXiv preprint arXiv:2005. 13625, 2020a.
Justin K Terry and Benjamin Black. Multiplayer support for the arcade learning environment. arXiv preprint arXiv:2009.093: 2020.
Justin K Tenry, Benjamin Black, and Ananth Hari. Supersuit: Simple microwrappers for reinforcement
learning environments. arXiv preprint (xXi::2008.08932,, 2020b.
Justin K Terry, Benjamin Black, and Ananth Hari. Supersuit: Simple microwrappers for reinforcement learning environments. arXiv preprint arXiv:2008.08932. 2020c.
Justin K Terry, Nathaniel Grammel, Ananth Hari, Luis Santos, and Benjamin Black. Revisiting parameter sharing in multi-agent deep reinforcement learning. arXiv preprint arXiv:2005. 13625, 2020d.
11

---

Gerald Tesauro. Temporal difference learning and td gammon. Commun ACM, 38(3):8 68. March 1995. ISSN 0001-0782.
doi: 10.1 10.11457203333.23333.. URL https:://oo..rrg//
203330. 203343
Rene Vidal, Omid Shakernia, H Jin Kim, David Hyunchul Shim, and Shankar Sastry. Probabilistic pursuit-evasion games: theory, implementation, and experimental evaluation. IEEE transactions robotics and automation, (855)662-6663 2002.
Eugene Vinitsky, Natasha Jaques, Joel Leibo, Antonio Castenada, and Edward Hughes. An open source implementation of sequential social dilemma games. https:///ithhb. com/
aaggaarrritttt//eppaarraaaaaaaaa stm.iiaalllllemmmaammmmeee////// 2019. GitHub repository.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3350 2019.
Jiayi Weng, Minghao Zhang, Alexis Dubureq, Kaichao You, Dong Yan, Hang Su, and Jun Zhu. Tianshou. https://githenn com/tthu-ml/tianshon 2020.
Daochen Zha, Kwei-Herng Lai, Yuanpu Cao, Songyi Huang, Ruzhe Wei, Junyu Guo, and Xia Hu. Rleard: A toolkit for reinforcement learning card games. arXiv preprint arXiv: 191.0499.. 2019.
Lianmin Zheng, Jiacheng Yang, Han Cai, Weinan Zhang, Jun Wang, and Yong Yu. Magent: A many-agent reinforcement learning platform for artificial collective intelligence. arXiv preprint arXiv:1712.00600, 2017. 2017.
12


Reviews:
[{"review_id": "vOjT-GimBj6", "reviewer": "Reviewer_yy96", "summary": "This paper presents the PettingZoo library containing a set of multiagent environments with a Gym-like interface to users. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: this paper is not scientifically novel. According to the claim, the main contribution is \"be GYM like for multiagent setting\". \nHowever, \"OpenAI Gym\" is an arxiv paper and has been published nowhere (as far as I know, please correct me if i am wrong). \nClarity: the writing of the paper easy to follow and clearn. It is more of an engineering document or tutorial than a scientific paper to me. \nOther: I would suggest the authors to expand the revisions to the ICML submission. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "ZIIPpAWjfWH", "reviewer": "Reviewer_KaoH", "summary": "This paper introduces a PettingZoo, a python API similar to OpenAI gym for multi-agent reinforcement learning (MARL). The environment has been in use for several years, and has 62 built in environments, is supported by a number of libraries, and is already used by many other groups and students. A key contribution of the paper is explicating the Agent Environment Cycle (AEC) model that powers PettingZoo, and its key distinguishing features in comparison to alternative MARL models. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## A principled and democratized framework for multi-agent reinforcement learning. \n\n### Strengths\n- Excellent library for multi-agent reinforcement learning. \n- The description of the AEC framework in comparison to other models is well constructed. \n- Number and description of environments and supporting packages is clear and impressive.\n\n### Weaknesses\n- Much of the comparison with other game models is handled in the text, and could be diagrammed more clearly, for instanced in Figure 5. A table could also be used to quickly lookup differences in these approaches, with appropriate subheadings (i.e. race conditions). \n- Any quantitative measures on users, implemented games or other comparisons with competing frameworks would be helpful, as part of the challenge in comparing PettingZoo with other libraries is that the arguments are somewhat qualitative. A table here could again help. \n\n\n### Correctness\nI believe the claims are correct.\n\n### Clarity\nThe manuscript is clear and easy to read. \n \n### Relation to prior work\nRelationship with other approaches is present and easy to parse. \n\n### Reproducibility\nThe code is well maintained and a popular repository on github. \n\n### Additional feedback, comments, suggestions for improvement and questions for the authors\nI understand the need to distinguish from past work, but the comparisons with POSG and EFL sometimes come across as a bit biased. For instance ‘ POSGs Based APIs Don’t Make Conceptual Sense For Games Implemented In Code’ mainly concerns race conditions.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "They could add some boilerplate on societal impacts. ", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "8W_SGozA6V3", "reviewer": "Reviewer_BcrF", "summary": "The paper introduces PettingZoo, a Python library for multi-agent RL tasks. It also discusses the API choices of said library, and how they might favoriably compare to other approaches available.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is highly readable and relatively clearly written, with only minor mistypes and glitches (examples: a missing \"s\" in \"observation[s]\" in Figure 2, a missing \"T\" in \"[T]he\" in line 97, a missing \"occur\" after \"typically\" in line 185). It presents the present state of the accompanying library, PettingZoo, relatively well. The main contribution of the submission itself, on top of the library presented, is a discussion of \"POSG\" and \"EFG\", and the introduction of \"AEC\". These acronyms are taken to mean different API choices for multi-agent RL environments. Specifically, \"POSG\" is used for the API model in which a vector of observations and rewards is returned by the task object, one for each agent, which in turn expects a vector of actions. The \"EFG\" model is used for the API of OpenSpiel, namely the tree-like structure more natural for board games, in which random effects of the environment (such as during a game of Poker or Blackjack) are represented as \"chance nodes\". The suggested \"AEC\" API model has the task define an order in which agents get to observe and act. That order is presented as a (potentially infinite) Python iterator object and is meant to represent situations in which the turn order changes during the game, as it would e.g. in the card game Uno.\n\nThe paper discusses a specific bug the authors ran into while using the \"POSG\" model and argues their model would help preventing that type of bug. On the face of it, this would seem like a worthwhile point, given the complexities of RL research at scale, especially in the multi-agent case. However, this reviewer wasn't entirely convinced by the presentation. Indeed, it seems the paper makes some rather strong claims without being able to give a reference as to why they would be true, and the central anecdote around the bug / race-condition discussed seems overly simplifying.\n\nExamples of statements that seem questionable in the generality in which they are stated:\n\n\"[popular previous APIs] promote severe bugs that are hard to detect\" (from the abstract, line 11): While that may be true in an extremely general sense in that all RL research easily suffers from bugs that could be described that way, the specific example discussed seems insufficient to illustrate this in the particular case the authors must have meant, see below.\n\n\"[Multi-agent RL] remains a significant engineering challenge. This is because, unlike single agent reinforcement learning which has OpenAI's Gym, no de facto standard API exists [...] for how agents interface with environments.\" (line 18): This is clearly an overstatement in both directions. Single-agent RL remains a significant engineering challenge despite OpenAI's useful library, and although PettingZoo sounds like a very useful contribution to the landscape, its authors can hardly believe it will make multi-agent RL trivial or even easy. It may perhaps make it simpler along some dimensions, but even that is probably better argued empirically, and at any rate not the obvious reading of this statement, although the authors would likely support that statement along these lines.\n\n\"[We] discovered [...] [other APIs] do not make good conceptual sense [...] and cannot form the basis of APIs that cleanly handle all types of multi-player environments\" (line 35): This is a rather strong statement that would require equally strong arguments. However, looking at the list of specific criticisms of previous APIs, we read (lines 72--): (1) Dummy actions are necessary and (2) changing numbers of agents requires learning code to deal with lists \"suddenly\" changing sizes. For the OpenSpiel API, the criticisms cited are (lines 89--): (1) It's too complex, especially for beginners or for RL researchers to \"ubiquitously use it as [their] mental model of games\", (2) the formal definition only has rewards at the end of the game -- which is followed by \"this is possible to work around\" (by changing that formal definition, perhaps?), but \"not ideal\", and finally (3) the lack of continuous action spaces -- followed by the admission that this, too, \"is not inherent in the [OpenSpiel API]\". This sauce is just a bit too weak for the strong praise the chef gave it.\n\nLet's now discuss the anecdotal evidence the submission relates around race-conditions in multi-agent RL. An example is given where two agents observe a state, each gets a chance to act on it and they chose to act in incompatible ways. \"Tie-breaking\" is then happening in the environment, e.g., by one agent being selected to be allowed to perform its action as intended, while the other agent might be denied, or have its action applied to the situation after the first agent got to modify the state with its action. The authors mention a situation where this tie-breaking was happening in a biased way and suggest that serializing the interaction with the environment would get around this issue. As indeed it would. However, the claim that this \"is more closed [sic!] models how computer games are executed in code\" (line 211) is naive at best. While this reviewer is no expert in game engine development, it seems more likely modern games are implemented around event loops interacting with a main rendering thread. That is, the state of the game might continuously change (units move, wind blows, bombs explode, etc), while the players have a chance to interact with that game in an asynchronous way, perhaps rate-limited to a certain number of actions per minute by either the system or natural constraints in hardware, network latency, or human motor control. When doing RL research on such games, this reality needs to be taken into account. And it is, one way or the other, in results like AlphaStar (which gets around asynchronous environments by trying to be fast enough at test time, and make the environment ignore the occasional action that can no longer be applied, like trying to construct a unit with a building that's already destroyed). The challenges of real-world interactions like robotics or self-driving cars are similar, only worse. Now, it might reasonably be argued that it's too much to ask from an API to cover chess, StarCraft, and Mars robots all at once. And indeed that might be so -- but that's not exactly an argument in favor of a single new standardized API.\n\nNone of this is to say that the library the authors developed doesn't have the potential to be extremely valuable to the community, and might well already have great positive impact. It would seem to this reviewer, however, that online documentation of their API and the design choices behind it is a better format (especially in view of potential future changes) than a research paper.\n\nAs a side note, the submission's supplementary material is insufficiently anonymized. This reviewer closed the relevant file right after observing this and believes no undue influence on the double-blinded process has occurred.\n\nUPDATE: Changed rating to 5: Marginally below the acceptance threshold based on author's response.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some limitations discussed below. No potential negative societal impact seems likely to incur as a result of this submission.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: I recommend this paper to be accepted. It provides a novel multi-agent reinforcement library with a corresponding games model. While the scientific novelty appears to be limited, the practical utility seems to be great for the community and such open-source libraries are explicitly encouraged in the Call for Papers. 
Decision: Accept (Poster)