PettingZoo: Gym for Multi-Agent Reinforcement Learning
Authors: J K Terry
Benjamin Black
Mario Jayakumar
Ananth Hari
Luis Santos
Clemens Dieffendahl
Niall L Williams
Yashas Lokesh
Ryan Sullivan
Caroline Horsch
Praveen Ravi
Venue:  (2021)
Keywords: Reinforcement Learning
Multi-agent Reinforcement Learning

Abstract: OpenAI's Gym library  contains a large, diverse set of environments that are useful benchmarks in reinforcement learning, under a single elegant Python API (with tools to develop new compliant environments) . The introduction of this library has proven a watershed moment for the reinforcement learning community, because it created an accessible set of benchmark environments that everyone could use (including wrapper important existing libraries), and because a standardized API let RL learning methods and environments from anywhere be trivially exchanged. This paper similarly introduces PettingZoo, a library of diverse set of multi-agent environments under a single elegant Python API, with tools to easily make new compliant environments.

Under review as a conference paper at ICLR 2021
PETTINGZo0: GYM FOR MULTI-AGENT REINFORCE- MENT LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
This paper introduces PettingZoo, a library of diverse sets of multi-agent environ- ments under single elegant Python API. PettingZoo was developed with the goal accelerating research in multi-agent reinforcement learning. by creating a set benchmark environments easily accessible to all researchers and standardized API for the field. This goal is inspired by what OpenAT's Gym library did for accelerat- ing research in single-agent reinforcement learning, and PettingZoo draws heavily from Gym in terms of API and user experience. PettingZoo is unique from other multi-agent environment libraries in that it's API based on the model of Agent Environment Cycle ("AEC") games, which allows for the sensible representation all varieties of games under one API for the first time. While retaining a very simple and Gym-like API, PettingZoo still allows access to low-level environment properties required by non-traditionll learning methods.
INTRODUCTION
Reinforcement Learning ("RL") considers learning policy - a function that takes in an observation from an environment and emits an action - that achieves the maximum expected discounted reward when acting in an environment, and it's capabilities have been one of the great success of modern machine learning. Multi-Agent Reinforcement Learning (MARL.) in particular has been behind many of the most publicized achievements of modern machine learning AlphaGo Zero (Silver ct al. 2017), OpenAl Five (OpenAl, 2018), AlphaStar (Vinyals et al., 2019) - and has seen boom recent years. However, popular benchmark environments locations (or made from scratch), are based around heterogeneous APIs, and are often in unmaintained states. Because of this, highly influential research in the field is generally restricted institutions with dedicated engineering teams, research into new methods generally aren't 't compared in like environments, and progress has been slow compared to single agent reinforcement learning (though this obviously cannot be attributed to benchmarks alone).
scattered across many different
Motivated by this, we introduce PettingZoo popular MARL environments under a single simple Python API similar to that of OpenAl's Gym library. It's available on PyPI and can be installed via pip install pettingzoo.
Python library collecting maintained versions of all
A TALE OF TOO MANY LIBRARIES
OpenAl Gym (Brockman et al. 2016) was introduced shortly after the potential of reinforcement
learning became widely known with Mnih et al. (013)). At the time, doing basic research in reinforcement learning was a large engincering challenge. The most popular set of environments were Atari games as part of the Arcade Learning Environment ("ALE") (Bellemare et al. 2013). The ALE originally was challenging compile and install, and had an involved API and later an unofficial fork with a Python wrapper (Goodrich] 2013). A scattering of other environments existed as independent projects, in various languages, all with unique APIs. This level of heterogeneity meant that reinforcement learning code had to be adapted every environment (including bridging programming languages). Accordingly, standardized reinforcement learning implementations weren't t possible, comparisons against wide variety of environments were very difficult, and doing simple research in reinforcement learning was generally restricted to organizations with software engineering divisions. Gym was created promote research in reinforcement learning by making comprehensive

---

Under review as a conference paper at ICLR 2021
benchmarking more accessible, by allowing algorithm reuse, and by letting average machine learning researchers access the environments. This last point was achieved by putting every environment that a researcher would want to benchmark with (at the time of creation) under simple API that anyone
could understand, in Python (which was just starting to be the lingua-de-france for machine learning). This lead to mass proliferation of reinforcement learning research (especially at smaller institutions),
many environments compliant with the API (Kidzinski et al., 2018 Leurent [0018 Zamora et al.
2016), and many RL libraries based around the API (Hill et al. 2018 Liang et al.
2018 Kuhnle
et al. 2017).
In the multiagent space, a similar level of fragmentation currently exists. Notable heterogenous sets of environments include OpenAl's Competitive MulliiAAent Environments for competitive robotic control (Bansal et al. 2017), Sequential Social Dilemma Games for games where cooperation is difficult in a game theoretic sense (Leibo et al.] (2017). RI.Card for various card games Zhaa et al. (2019), MAgent for huge numbers of agents (Zheng et al. 2017), Multi-Particle Environments ("MPE") for diverse agent roles (Mordatch and Abbeel 2017 Lowe et al., 2017), the Starcraft Multi-Agent Challenge (Samvelyan et al.) 2019), and dozens more.
RELATED WORKS
Two attempts at some level of unification in the multi-agent space have been made. The first is Open- Spiel, released by Deepmind in 2019 (Lanctot et al. 2019), which includes excellent implementations of 45 classic games under one sensible API. However, their framework is limited to supporting simple discrete games due to its modeling of games as trees (which is impractical to represent for more continuous environments such Atari). However, in the space of discrete games it has managed to encourage high quality and fair evaluations of general game solving methods.
The second is the multi-agent API of RLlib[Liang et al. 2018), ambitious distributed RL framework. While the API is powerful, it has very minimal feature support and cannot sensibly represent strictly turn based games (i.e. Hananbi or Go). However, we do maintain PettingZoo support within RLlib so that users can easily leverage the learning methods included on our environments.
4 DESIGN PHILOSOPHY
Simplicity and Similarity to Gym
The ability for the Gym API to be near instantly understood has been a large driving factor in it's widespread adoption. While multi-agent API will inherently add complexity, we wanted create similarly simple API, and one that would be instantly familiar to researchers who have worked with Gym.
Agent Environment Cycle Games Based API
Most environments have APIs that model agents as all stepping once (Lowe et al.
2017 Zheng
eet al. 2017 Gupta et al. 2017 Liu et al. 2019 Liang et al., (018)). based on the Partially Observable Stochastic Games (POSGs) model. turns out this easily results in bugs (Terry al. 2020b) and is undesirable for handling strictly turn-based games, like chess, since agents aren allowed to step simultaneously. We instead model our API after the new Agent Environment Cycle games model, an equivalent model where agents step sequentially. That is, an agent performs an action, the environment responds, the next agent acts, the environment responds again, and the cycle repeats. This model allows for the sensible interactions with both strictly turn based games like chess and games where agents truly step simultancously. A POSG can be easily converted to an equivalent sequential game by having each agent take step in cycle, and then updating the rewards and observations of the AEC game at the end of a cycle. For a formal proof of equivalence see (Terry et al. 2020b).
Variable Number of Agents
We designed our API to robustly support the widest range of multi-agent scenarios possible, including agent generation and death. No general API currently supports this, but it such an integral feature of so many environments that this support is essential
2

---

Under review as a conference paper at ICLR 2021
Sufficient Configurability
We wanted to make environments that are highly configurable by arguments the norm. In Gym, environments are generally not configurable, and arguments generation are not used at all. However, playing with various environment properties often highly desirable, and has accordingly been embraced by Gym environments outside the official library, this makes research easier and aids reproducibility. Accordingly, we tried to make every reasonable environment parameter an option for users in PettingZoo.
This notion of configuration extends beyond environment configuration to how learning methods interact with the environment. Due the wide diversity of optimizations and different strategies applied for MARL, we wanted our API to allow for low level access to rewards, observations, done states and other info, while still being very simple for normal applications. Cyclically expansive curriculum learning from [erry et al. (2020b) is good example of an interesting method that requires this sort of low level access.
Quality of Life Improvements
Being users of Gym ourselves, we sought to add several "quality of life" improvements in PettingZoo motivated by frustrations we faced as users. These are:
Comprehensive, production-gradd continuous integration testing. Testing in Gym is arguably lacking compared to other major libraries.
Tests of environments for API compliance and proper functionalityy, both for end users and for continuous integration testing of the library. We also provide detailed recommendations for better practices, inspired by the well liked messages of the Rust compiler.
Good error messages and warnings. When using Gym, triggering an error yields a trace back that needs to be slowly decoded to find the actual problem. We added speciality error messages and warnings for all common errors (that we'ne re aware of) to make development and debugging easier. This again inspired by the Rust compiler.
Detailed, comprehensive documentation. Documentation is a fundamental part of a user- friendly software library. Observation space, action space, reward schemes, and other notable environment details are something you generally need know to begin conducting even the most basic research with environment. One criticism of Gym is that almost all information only found in the source code, something especially problematic when working with sets of environments. To solve this in PettingZoo, we created user friendly wiki-styled website that clearly includes all relevant information for an environment, as well general information for sets of environments. Our website also includes details about tests, comprehensive API documentation, and so on. This is discussed further in section
API
5.1
MAIN API
Per our discussion above, we sought to create simple API that could encapsulate all games and be instantly understood to any Gym user, illustrated by comparing Figure [aad Figure 2) We use observation/action space objects from Gym, well the same seeding method because they well designed and familiar to researchers.
Figure 1: Basic Usage of Gym
import gym
env gym. CartPole-vO')
observation = env. () for in range (1000):
anv.render"
action policy (observation)
observation, zeward, done, info env,step (action)
env.elose
3

---

Under review as a conference paper at ICLR 2021
Figure 2: Basic Usage of PettingZoo
from pettingzoo .wwpingooo.uuutee. butterfly import pistonball_
env piistobbal__v..om env
env resete ()
for agent in env. www.agen__iter (1000):
env. revdee)der
observation, reward, done, info = env last ()
action = policy (observation, agent)
env. (action)
elveclose ()
5.2 PARALLEL API
In addition our main API, for certain environments we offer a separate API based off of the POSG model. It supports games and methods that assume this model very nicely. In style, it is very similar RLlib's multi-agent API (Liang al., 2018). accepting and returning dictionaries keyed on agent names. The primary motivation for including this secondary API is because in games where it's applicable, it can allow for parallelization features which can lead large performance improvements.
5.3 ADDITIONAL API FEATURES
While the API is elegant for simple cases, it's still able to handle important edge cases. This is achieved through lower level API calls for specific attributes: rewards emitted at any time are included for every agent the rewards dictionary attributes, and dones and infos dictionaries can similarly be used. We additionally have an observe (agent) function. render () and close () function identically to Gym for rendering environments being played by a policy. agents is a list of the names of all agents in the game, and due to considerations regarding variable numbers of agents poss bble_ageeee a gent an additional list attribute. Finally, abse rvat i ion_ion_ppaae opaces and act seiiiinncceeeeeeee are dictionaries of the Gym observation/action spaces for all agents in poss wessibbe__see cool__agentts
When an agent dies, its done is set to True, it will become the next selected agent to act, and the action taken is required to be None. After this dummy step is taken, the agent will be removed from agents s and all dictionary attributes. Note than a whole environment is done if and only if agents is empty. Agent generation can be implemented by simply adding an agent to the agents list and allowing its rewards and observations to be accessed.
Finally, environment configuration is handled by passing arguments to the env () constructor, a standard that all third-party Gym environments have adapted. These arguments fundamentally change environment behavior, such as the number of agents, the reward structure, and even its action and observation spaces. All environments PettingZoo allow for some degree of customization in this manner, with many allowing quite bit.
5.4 IMPLEMENTING AN ENVIRONMENT
Compliant environments inherit from a general class (AECEnv). To allow for sufficient flexibility, environments only expose lower level attributes (dictionaries of values for all agents - dones, infos, rewards) and an observe method that takes an agent. These are then wrapped to provide the more general functions you see above by the base class. This low level functionality allows for entirely new APIs to be efficiently added on top of PettingZoo environments should the need arise. We' ve done this ourselves with the secondary parallel POSG based API, which have wrappers that convert both and from the parallel and standard APIs.
ENVIRONMENTS
Similar to Gym, we wanted to include popular and interesting environments within one package, in easily usable format. Half of the environment classes include (MPF, MAgent, and SISI.),
4

---

Under review as a conference paper at ICLR 2021
Figure 3: Example Environments From Each Class
DYBO
DESO
0
A
(b) Butterfly: Pistonball
(a) Atari: Space Invaders
b
q
n
r
P
P
p
P
P
P
Q
P
PPPP
P
K
(c) Classic: Chess
(d) MAgent: Adversarial Pursuit
(f) SISL. Multiwalker
(e) MPE: Simple Adversary
despite their popularity, have previously only existed unmaintained "research grade" code, have not been available for installation via pip, have required large amounts of maintenance to run at all, and have required large amounts of debugging, code review, code cleanup and documentation to bring to a production-grade state. The Multi-Agent Atari and Butterfly classes are new environments that we believe pose important and novel challenges to multi multi-agent reinforcement learning. Finally, we include the Classic class classic board and card games popular within the RL literature.
Atari
Atari games represent the single most popular and iconic class of benchmarks in reinforcement learning. Recently, a multi-agent fork the Atari Learning Environment was created that allows programmatic control and reward collection of Atari's 's iconic multi-player games (Terry and Black 2020). As in the single player Atari environments, the observation is the rendered frame the game, which is shared between all agents, so there is partial observability. Most of these games have competitive or mixed reward structures, making them suitable for general study of adversarial and mixed reinforcement learning. In particular, Terry and Black (2000) categorizes the games into 7 different types: 1vl toumament games, mixed sum survival games (Space Invaders, shown in Figure 3al is example of this), competitive racing games, long term strategy games, 2v2 tournament
5

---

Under review as a conference paper at ICLR 2021
games, four- four-player free-for-all game and cooperative game. For easy ROM installation, AutoROM, a separate PyPI package, can be used to easily install the needed Atari ROMs in an automated manner.
Butterfly
Of all the environments included, the majority of them are competitive. We wanted to supplement this with set of interesting graphical cooperative environments. Pistonball, depicted in Figure 3b] where the pistons need to coordinate to move the ball to the left, while only being able to observe a local part of the screen, requires learning nontrivial emergent behavior and indirect communication to perform well. Knights Archers Zombies is a game in which players work together to defeat approaching zombies before they can reach the players. It is designed to be fast paced graphically interesting combat game with partial observability and heterogeneous agents, where achieving good performance requires extraordinarily high levels of agent coordination. Cooperative pong, where two dissimilar paddles work together to keep the ball in play as long as possible, was intended to be be very simple cooperative continuous control-type task, with heterogeneous agents. Prison was designed be the simplest possible game in MARL, and to be used as debugging tool. Prospector was included to intentionally a very challenging game for conventional methods-it has two classes of agents, with different goals, action spaces, and observation spaces (something many current cooperative MARL algorithms struggle with), and has very sparse rewards (something all RL algorithms struggle with). It is intended to be very difficult benchmark for MARL, in the same vein of Montezuma's ' Revenge.
Classic
Classical board and card games have long been some of the most popular environments in reinforce- ment learning (Tesauro 1995| Silver et al.) 2016; Bard et al.. (2019). We include all of the standard multiplayer games in RI. Card (Zha et al. (019): Dou Dizhu, Gin Rummy, Leduc Hold' em, Limit Texas Hold' em, Mahjong, No-limis Texas Hold em, and Uno. We additionally include all AlphaZero games, using the same observation and action spaces-Chess and Go. We finally included Backgammon, Connect Four, Checkers, Rock Paper Scissors, Rock Paper Scissors Lizard Spock, and Tic Tac Toe to add a diverse set of simple, popular games to allow for more robust benchmarking of RL methods.
MAgent
The MAgent library, from Zheng et al. (2017) was introduced as configurable and scalable envi- ronment that could support thousands of interactive agents. These environments have mostly been studied as a setting for emergent behavior (Pokle) 2018). heterogeneous agents (Subramanian et al. 2020p, and efficient learning methods with many agents (Chen et al.) 2019). We include number of preset configurations, for example the Adversarial Pursuit environment shown in Figure [3 We make a few changes to the preset configurations used in the original MAgent paper. The global "minimap" observations in the battle environment are turned off by default, requiring implicit communication between the agents for complex emergent behavior to occur. The rewards in Gather and Tiger- Deer also slightly changed to prevent emergent behavior from being direct result of the reward structure.
MPE
The Multi-Agent Particle Environments (MPE) were introduced as part of Mordatch and Abbeel (2117) and first released part of Lowe al. (2017). These are communication oriented environ- ments where particle agents can (sometimes) move, communicate, see each other, push each other around, and interact with fixed landmarks. Environments are cooperative, competitive, or require team play. They have been popular in research for general MARI. methods [owe et al. (2017). emergent communication (Mordatch and Abbeel 2017). team play (Palmer) (020), and much more. As part of their inclusion in PettingZoo, we converted the action spaces to discrete space which the Cartesian product of the movement and communication action possibilities. We also added comprehensive documentation, parameterized any local reward shaping (with the default setting being the same as in Lowe et al. (2017)), and made a single render window which captures all the activities of all agents (including communication), making it easier to visualize.
SISI.
We finally included the three cooperative environments introduced in Gupta et al. (017)): Pursuit, Waterworld, and Multiwalker. Pursuit is a standard pursuit-evasion game Vidal et al. (2002) where

---

Under review as a conference paper at ICLR 2021
pursuers and controlled in randomly generated map. Pursuer agents rewarded for capturing randomly generated evaders by surrounding them on all sides. Waterworld is continuous control game where the pursuing agents cooperatively hunt down food targets while trying to avoid poison targets. Multiwalker (Figure 3r) is a more challenging continuous control task that is based on Gym's Bipeda/Walker environment. In Multiwalker, package is placed three independently controlled robot legs, Each robot is given small positive reward for every unit of forward horizontal movement of the package, while they receive large penalty for dropping the package.
DOCUMENTATION
Documentation is fundamental part of user-friendly software library. There's tremendous amount of useful information about these environments, especially due to their diversity, so we sought create detailed documentation as possible, while designing it in way ensure it's still useful and approachable. PettingZoo includes comprehensive documentation for the API, the continuous integration tests, and each environment. A majority of popular libraries do not have extensive documentation. For example, OpenAl's popular Gym library only lists the observation space shape on each environment's s documentation page. PettingZoo's 's documentation thoroughly explains each environment's 's observation and action spaces, and includes relevant information to help researchers. The goal is to allow people to compare environments easily, and for developers to very rarely need to refer to source.
Our design for displaying so much information was inspired by Wikipedia's familiar and well-known layout. This is illustrated in Figure 4 All documentation is included in the supplemental materials to facilitate anonymous review.
Cn pare
TWe Maace
spper the boan Los ant vhow baspon
Colkal
Stmmro 2014
Osbes
Vikrs
April
Add
Figure 4: The beginning PettingZoo documentation for the Go environment, illustrating how we used the design metaphor of Wikipedia page to include large amount of detail in manner that isn't 't overwhelming
8 BASELINES
All environments implemented in PettingZoo include baselines to provide a general sense of the difficulty of the environment, and for something to initially compare against. We do this here for the
7

---

Under review as a conference paper at ICLR 2021
Butterfly environments that this library introduces for the first time; similar baselines exist in the papers introducing all other environments. We used parameter sharing (Terry et al. 2020c; Gupta et al. 2017) with Ape-X X DQN (Horgan et al. 2018), with RLLib (Liang al. 2018 Our results are shown in Figure 5) Preprocessing and hyperparameter details are included in Appendix A All preprocessing
done with the SuperSuit wrapper library (Terry et al.) 2020a), which has recently added support for PettingZoo based multi-agent environments based. Code for the environments, training logs, and saved policies available at httttts:/ggii com wtt./pttiiiggooopppp........ pett ingzoopaper
14400
1200
Average Tonl Reward
-05
Avenge: Tola:
1000
-75
600
400
-85
-80
12
2
12
14
Million Steps
Milliinn Steps
(a) Knights Archers Zombies
(c) Cooperative Pong
Average Tetal Rewar
10
14
12
14.
Million Steps
Million Stape
(b) Pistonball
(d) Prison
400
14 140
360
Vosa
200
180
14
Million Steps
(e) Prospector
Figure 5: Total reward when learning on each Butterfly environment via parameter shared Ape-X DQN (a-d) and parameter shared PPO (e).
CONCLUSION
This paper introduces Zoo,ingZoo, a Python library of many diverse multi-agent reinforcement learning environments under one simple API, akin multi-agent version of OpenAl's Gym library,
Reinforcement learning systems have two main components, the environment and the agent(s) that learn. Without standardized environment base, research progresses by designing and building both the environment and the agent (as has been the case for MARL). The main contribution of PettingZoo is that it enables more research which focuses agents by standardizing and democratizing the environments. We hope that this allows for research in multi-agent reinforcement learning to accelerate and flourish.
We' re aware of two notable limitations of PettingZoo. The first is that games with significantly more than 10,000 agents (or potential agents) will have meaningful performance issues. This arises
8

---

Under review as a conference paper at ICLR 2021
from needing to prespecify observation/action spaces and potential agent names. We view this practically acceptable limitation. The second notable limitation is that PettingZoo does not currently allow users to access the global environment state, a feature required by some centralized critic methods. We're actively working on supporting this via state () method.
We see three obvious directions for future work. The first is additions of more interesting environments under our API (possibly from by the community, as has happened with Gym). While we' 'e included large number of environments, there additional sets that would be valuable to include: the open-source implementations of social sequential dilemma games (Vinitsky et al. 2019), and the StarCraft 2 Multi-Agent Challenge ("SMAC") environments (Samvelyan al. 2019). The second direction we envision is service to allow different researchers agents to play against each other in competitive games, leveraging the standardized API and environment set. Finally, we envision the development of procedurally generated multi-agent environments to test how well methods generalize, akin to the Gym procgen environments. (Cobbe et al. 2019).
ACKNOWLEDGMENTS
Thank you to Deepthi Raghunandan and Kevin Hogan for many helpful discussions surrounding what testing should look like. Thank you to Nathaniel Grammel for many helpful discussions in the early planning stages of the project. Thank you to Ross Allen and his group for reporting numerous bugs.
REFERENCES
Trapit Bansal, Jakub Pachocki, Szymon Sidor, llya Sutskever, and Igor Mordatch. Emergent com- plexity via multi-agent competition. CoRR, abs/1710.03748, 2017. URL httt::t/txxxi..ooo
abs 1710. 03748
Nolan Bard, Jakob N. Foerster, Sarath Chandar, Neil Burch, Mare Lanctot, H. Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, lain Dunning, Shibl Mourad, Hugo Larochelle, Mare G. Bellemare, and Michael Bowling. The hanabi challenge: A new frontier for AI research. CoRR, abs/1902.00s5 2019. URL http::/arxiv..rttt wwtp:xiiirrg/as/////
1902. 60506
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:/606.015400 2016.
Y. Chen, M. Zhou, Ying Wen, Y. Yang, Y. Su, W. Zhang, Dell Zhang. J. Wang, and Han Liu. Factorized q g-learning for lcrge-scale multi-agent systems. In DAI '19, 2019.
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. arXiv preprint 1xXfn:1912.01158822000 2019.
Ben Goodrich. ale ale_ pythoooiitteeaceee......o.. interface] 2015. GitHub repository.
interface,
Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems, pages 66-83. 83. Springer, 2017.
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable basclines.
com/
hi 1 10l1-a/ae aa-aa/saa stwsssaallleeeeee basel nes 2018.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado van Hasselt, and David Silver. Distributed prioritized experience replay. CoRR, abs/ 0tbp//88330033,, 2018. URL http://arxx wtt.aawwww...ggga.oe
www./a/aaaa.... abcp///b//// wtw.bb/s/1133000900 00933

---

Under review as a conference paper at ICLR 2021
Lukasz Kidzitiski, Sharada P Mohanty, Carmichael Ong, Jennifer Hicks, Sean Francis, Sergey Levine, Marcel Salathé, and Scott Delp. Learning to run challenge: Synthesizing physiologically accurate motion using deep reinforcement learning. In Sergio Escalera and Markus Weimer, editors, NIPS 2017 Competition Book. Springer, Springer, 2018.
Alexander Kuhnle, Michael Schaarschmidt, and Kai Fricke. Tensorforce: a tensorflow li- brary for applied reinforcement learning. Web page, 2017. URL httttttss//gthhhtee com/
Mare Lanctot, Edward Lockhart, Jean-Baptieeeee Baptiste Lespiau, Vinícius Flores Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, Daniel Hennes, Dustin Morrill, Paul Muller, Timo Ewalds, Ryan Faulkner, János Kramár, Bart De Vylder, Brennan Saeta, James Bradbury, David Ding, Sebastian Borgeaud, Matthew Lai, Julian Schritwieser, Thomas W. Anthony, Edward Hughes, Ivo Danihelka, and Jonah Ryan-Davis. Openspiel: A framework for reinforcement learning in games. CoRR, abs/1908.09453, 2019. URL
09455
Joel Z. Leibo, Vinícius Flores Zambaldi, Mare Lanctot, Janusz Marecki, and Thore Graepel. Multi- agent reinforcement learning in sequential social dilemmas. CoRR, abs/1 0tss//772.03037,, 2017. URL
http://arxiv> http://xxww..orra/a//1.0..2....
Edouard Leurent. An enviromment for autonomous driving decision- decision-making. https://githus
www.a//euuuetttihhhaaaaeeeeee 2018.
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael I. Jordan, and Ion Stoica. RLlib: Abstractions for distributed reinforcement learning. In International Conference on Machine Learning (ICML), 2018.
Siqi Liu, Guy Lever, Josh Merel, Saran Tunyasuvunakool, Nicolas Heess, and Thore Graepel. Emergent coordination through competition. CoRR, abs/1902.07 151, 2019. URL http://
arxiv. wwx.v.org/abar /1002 07151
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed pooperatie-ccompeitie environments. Neural Information Processing Systems (NIPS), 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 8(7544):552-553 2015.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. arXiv preprint 075:: 2017.
OpenAI. Openai
2018.
G. Palmer. Independent learning approaches: Overcoming multi-agent learning pathologics in team-games. 2020.
Stefanie Anna Baby Ling Ashwini Pokle. Analysis emergent behavior in multi agent environ- ments using deep reinforcement learning. 2018.
Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob N. Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. CoRR, Abs/1992.004433, 2019. URL http: //tp://.xx.v.o////// abs/ 1902..0 04043
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche. Julian Schrittwieser, Joannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 489 489, 2016.
David Silver, Julian Schrittwieser. Karen Simonyan, Ioannis Antonoglou. Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-38 359, 2017.
10

---

Under review as a conference paper at ICLR 2021
Sriram Ganapathi Subramanian, P. Poupart, Matthew E. Taylor, and N. Hegde. Multi type field reinforcement learning. In AAMAS, 2020.
Justin K Terry and Benjamin Black. Multiplayer support for the arcade learning environment. arXiv preprint arXiv:2009. 200.000000 2020.
Justin Terry, Benjamin Black, and Ananth Hari. Supersuit: Simple microwrappers for reinforcement learning environments. arXiv preprint arXiv:2008.08922 2020a.
Justin K Terry, Nathaniel Grammel, Benjamin Black, Ananth Hari, Luis Santos, and Caroline Horsch. Agent environment cycle games. arXiv preprint arXiv:2009. 2020b.
Justin K Terry, Nathaniel Grammel, Ananth Hari, Luis Santos, and Benjamin Black. Revisiting parameter sharing in multi-agent deep reinforcement learning. arXiv preprint arXiv:2005. 13625, 2020c.
Gerald Tesauro. Temporal difference learning and td gammon. Commun ACM, 3883):8888888 68, March 1995. ISSN 0001-0782. doi: 10. 2021144/233330233444 URL httpp::////o 0ttp://///0...0//011115155 203330 203343
Rene Vidal, Omid Shakernia, H Jin Kim, David Hyunchul Shim, and Shankar Sastry. Probabilistic pursuit-evasioe games: theory, implementation, and experimental evaluation. IEEE transactions robotics and automation, 18(5):662-66)) 2002.
Eugene Vinitsky, Natasha Jaques, Joel Leibo, Antonio Castenada, and Edward Hughes. An /ttt://gttubb..../ com/ dilemma, dit.eeaaaaaaeeeeeeaaaaaeaaaaaeee 2019. GitHub repository.
open source implementation of sequential social dilemma games.
euugenevinttskyy http::skkyyyuuuuttaattaaee social
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-354 2019.
Iker Zamora, Nestor Gonzalez Lopez, Victor Mayoral Vilches, and Alejandro Hernandez Cordero. Extending the openai gym for robotics: toolkit for reinforcement learning using ros and gazebo. arXiv preprint 1wXvv:1608.05722 2016.
Daochen Zha, Kwei-Herng Lai, Yuanpu Cao, Songyi Huang, Ruzhe Wei, Junyu Guo, and Xia Hu. Rleard: toolkit for reinforcement learning card games. arXiv preprint arXiv: 1900.43376 2019.
Lianmin Zheng, Jiacheng Yang. Han Cai. Weinan Zhang, Jun Wang, and Yong Yu. Magent: A many-agent reinforcement learning platform for artificial collective intelligence. arXiv preprint 1rXiv: 1010000000, 2017.
A BASELINE EXPERIMENT HYPERPARAMETERS AND PREPROCESSING
All of the environments were preprocessed in the following way: observations were resized to 84x84 images with linear interpolation, converted grayscale, then normalized. This preprocessing was performed with SuperSuit (Terry et al. 2020a).
The graphically subtle environments (Knights Archers Zombies, Prospector and Cooperative Pong) had their observations processed with the RLlib default network: A convolutional layer with a 8x8 kernel, stride of 4, and 16 filters, followed by a convolutional layer with a 4x4 kernel, stride of 2, and 32 filters, followed by convolutional layer with 1 1x11 kernel, stride of 1, and 256 filters.
The graphically simple environments (Prison, Pistonball) were resized 32x32 and flattened in addition to the above preprocessing. The observation was processed with a network with two hidden linear layers, 400 and 300 neurons wide, respectively.
11

---

Under review as a conference paper at ICLR 2021
RL method
Hyperparameter
Value
ApeX-DQN
adam_epsilon
0.00015
buffer_size
400000
double_g
True
dueling
True
epsilon_timesteps
200000
final_epsilon
0.01
1.0
ganma
0.99
learning_starta
10000
1r
0.0001
n_step
3
num_attoms
1
num_envs_perr_wrker
4
num_gpus
1
num_workers
12
prioritized_repl..
True
priorit.lzed_replay_aallaa
0.5
prioritiizedrreeplyy_eeaa
0.4
2000000
rolloott_fragmentt__engttt
32
tacket_eettox___uccre_______
10000
=imesteps__er__ieratiioo
15000
train_batch_sizee
512
PPO
ganma
0.99
nun_envs_per_Morker
4
num_gpus
1
num_ num_workers
12
compress_obseervtiions
False
lambda
0.95
kl_coeff
0.5
alip_rewards
True
clip_param
0.1
vf_clip_param
10.0
entropy_coeff
0.01
train_batch_size
5000
sample_batch_size
25
sgd sine
256
num num_sgd_ites
100
batch_mode
truncate_ seiiesddes True
vf_share_layers
Table I: Hyperparameters for ApeX DQN and PPO each Butterfly environment.
12


Reviews:
[{"review_id": "iEa5FN3E1Kn", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n---\n\nThe paper describes a new framework, \"PettingZoo\", which is proposed to play a similar role for Multi-Agent RL research as OpenAI's Gym framework does for single-agent RL research. The paper describes various lessons learned from Gym and other frameworks, and how these were taken into account in PettingZoo's design and implementation.\n\nStrong Points\n---\n\n1) These types of frameworks for standardised, easy-to-use, and varied benchmarks are important contributions.\n2) Framework looks easy to use, familiar API for Gym users.\n3) Paper well-written, easy to follow.\n4) Nice detailed listing of all hyperparameters in Appendix for the included baseline results.\n\nWeak Points\n---\n\nThe paper could use a more thorough review of existing work and a more detailed comparison to make a stronger case for this framework satisfying a need in the research community that is not already satisfied. The primary point seems to be the different \"Agent Environment Cycle\" model from (Terry et al., 2020b). Since this is from a very new paper (this year), not widely-known already (to the best of my knowledge), and seemingly a primary core point of this paper's contribution, it would probably be useful to elaborate on exactly what this model is and how it's different from others a bit more. There is a very short description, but from that one I can't see how it's any different from Extensive Normal Form games (with imperfect information). If it is indeed very similar / identical to the standard Extensive Normal Form games model, it would probably also be worthwhile to discuss other frameworks that already use these kinds of models in related work. Especially OpenSpiel comes to mind, since it has a wide variety of Multi-Agent RL algorithms built-in (and many different kinds of games), but possibly even other general game playing systems may be worth comparing to (such as Polygames, Ludii, GDL, etc.). Many of these have different focuses, but may still be worth comparing to them.\n\nOverall Recommendation\n---\n\nCurrently I'm leaning towards marginally above the acceptance threshold, because I do see potential and feel like the contribution is interesting and there's nothing really \"wrong\" with the paper. More details (see above) could definitely be useful to add though.\n\nQuestions for Authors\n---\n\nCould you elaborate on how this Agent Environment Cycle games model is different from Extensive Normal Form games? I understand that this is not the contribution of this paper per se, but it does seem to be a core motivation point for why this new framework is better than existing ones?\n\nMinor Comments\n---\n\n- In abstract: \"wrapper\" --> \"wrappers for\" or \"wrapping\" (I guess)\n- Variable names should be kept consistent between python code in Figures 1 and 2. Now Figure 1 has \"obs\" but Figure 2 has \"observation\". Figure 2 also explicitly assigns the name \"agent\" to the variable it loops over, but this variable isn't used, so can just stick to \"_\" like in Figure 1 I suppose?\n- Section 4 starts naming a bunch of acronyms (MPE, MAgent, SISL), but I don't really know what any of those mean.\n- Inconsistent formatting for the \"Classic\" paragraph in Section 4 (text starts immediately after paragraph header instead of below it)", "extra_scores": {}, "extra_text": {}}, {"review_id": "YR4ha4qd3jm", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces yet another testing environment for RL, PettingZoo, as a logic evolution of OpenAI but, this time, for multi-agent RL systems, a feature not supported by the former.\n\nReasons to Accept:\n•\tVery nice engineering effort. The platform includes popular environments in a user-friendly way as well as detailed documentation and baselines for comparison.\nReasons to reject:\n•\tContents: I am confused by the nature of the work and assume it is meant as a demo. Actually, the paper reads like a product/company brochure, not as a scientific paper. This is mostly due PettingZoo is a great engineering effort (involving the use/incorporation of SW/packages from outside) with little science behind. Sections 2 to 4 gives an abstract overview of the main mechanisms in the API, environments and the interaction with the user, but all basically at the level of interaction at code level. The paper itself is not interesting from a research point of view, for me, because it does not provide any AI-like content apart from summarising the environments/APIs. I think sections 2-4 could have been compressed to one section introducing PettingZoo, after which an actual research contribution using the system could follow, to create a proper research paper. Further explanations, discussions and insights are thus needed (e.g., further comparisons with SOTA MARL approaches/models for different environments).\n•\tRelevance/Novelty: I also wonder whether the paper really represents a significant advance in the AI field, and I guess its relevance may be below the ECLR threshold. Even though I personally don’t see any particularly novel insight in this paper (it is a logic incremental evolution of OpenAI's Gym), the software coding/integration methodology followed is not wrong and the whole environment seem promising and may lead a mass proliferation of MARL research. And I happy to let the noisy process of science (and reviewing process) figure out the value here. I am fully open to change the score if the authors and the rest of reviewers convince me of the usefulness and relevancy of the contribution.\n•\tSuitability: Finally, I like the whole system (I actually think it is great and has great potential to be used as testbed for MARL system), but I feel this could have been better put forward at a dedicated workshop, as an overview, as an unpublished introduction paper, or as a white paper or technical report to briefly describe a system.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VV1liR3kBC", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Thanks for time and effort on writing the library.\n\nThe paper introduces an API and library for multi-agent reinforcement learning along with simple installation of a very diverse set of environments along. Each environment has clear documentation of inputs/outputs, etc. \n\nPros:\n- The majority of the paper is clear and easy to understand\n- Significant effort on the framework \n- Clean API\n- Integration of very diverse set of environments\n- The documentation of individual environments is great\n\nCons:\n- Abstract is very weak. OpenAI Gym is overemphasized, and PettingZoo is underemphasized. Make sure it highlights why PettingZoo is great\n- Baselines are weak. Ape-X is far from a good baseline in my opinion. There are simpler more data efficient agents available and also policy gradient based agents. (I realize that these are not the main points of the paper.)\n- The paper states it has been shown that AECs are equivalent to POSGs but there is no reference. I assume this is just executing the individual actions of the agents and then getting the observations of all of them instead of interleaving getting observations. Wrt. implementation, a POSG environment used as AEC's may be in invalid states when not all agents have executed an action?\n- Paper could be polished more in general.", "extra_scores": {}, "extra_text": {}}, {"review_id": "-WZ67UQc5Td", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Quality\nThe paper is overall a good quality but it has some deficiencies. The authors need to do a better job comparing PettingZoo to other alternatives. What's the selling point?\n\n### Clarity\nThe paper was straight forward and easy to read.\n\n### Originality\nWhile obviously taking inspiration from OpenAI Gym, others have also thought about MARL. However the drivers which led to the development of PettingZoo are original with a focus on quality of life improvements and configurability. These are all positives.\n\n### Significance\nThere is an obvious interest in MARL approaches and hence PettingZoo can be a potentially significant contribution\n\n## Comments on specific sections\n\n#### Abstract: \n80% of the abstract is talking about OpenAI Gym, not about PettingZoo. The last sentence then mentions PettingZoo almost in passing. I recommend you start with Petting Zoo and say that it was inspired by OpenAI Gym and focus your abstract on what your paper is about - Petting Zoo.\n\n#### Introduction\nI have a similar issue with the introduction. You spend a lot of time extolling the virtues of OpenAI Gym, which is great because I can see that Petting Zoo has taken inspiration from Gym, however this could be put into a background or related work section.\nYour introduction should focus on introducing PettingZoo, describing the scope of the paper, an overview of the contributions made and giving the reader an overview of what is coming in the rest of the paper.\n\n#### Background\nAs mentioned above, the paper is missing a dedicated background or related work. You mention RLlib as an alternative multi-agent RL library but you don't go into detail. How about other MARL libraries/frameworks? What is the difference between PettingZoo and these? Advantages/Disadvantages of other systems? Why not contribute to OpenAI Gym?\n\n\n#### Design Philosophy\nI liked the design philosophy described. This list can also be thought of as motivation or key development drivers. This potentially could be your development methodology. However what is missing is evaluation criteria. Once you've developed PettingZoo what are the measurable qualitative or quantitative criteria that you will/can use to evaluate if you have done a good job.\n\n#### API\nI feel you should have gone into more detail into your MARL API. I had a look at the code and there is a bit more of the API which would be good to talk about especially the parallel API. A design or block diagram showing the architecture of \n\n#### Environments\nIts good to see a comprehensive set of environments available with the PettingZoo library.\n\n#### Documentation: \nWhile this is good you probably don't need to spend 3/4 of a page on it.\n\n#### Baselines: \nThe baselines are good, but it would be nice to compare them to other MARL libraries as well.\n\n### Other Comments\nThe paper is missing a discussion/evaluation section where the authors critically evaluate the PettingZoo library.With some of the changes suggested the paper will be much stronger.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper represents the PettingZoo library of multi-agent environments, providing a common API and benchmark for multi-agent learning. The library has high potential for impact and is likely of interest to a wide range of people in the ICLR community. However, in its current form the paper could be significantly improved by actioning the many pieces of constructive feedback provided by all reviewers.

We have also been made aware of two highly related papers "Multiplayer Support for the Arcade Learning Environment" and "SuperSuit: Simple Microwrappers for Reinforcement Learning Environments." Together all three papers could be one comprehensive manuscript, but appear to have been unnecessarily split into three separate short papers.
Decision: Reject