Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a PettingZoo, a python API similar to OpenAI gym for multi-agent reinforcement learning (MARL). The environment has been in use for several years, and has 62 built in environments, is supported by a number of libraries, and is already used by many other groups and students. A key contribution of the paper is explicating the Agent Environment Cycle (AEC) model that powers PettingZoo, and its key distinguishing features in comparison to alternative MARL models. 

main_review: ## A principled and democratized framework for multi-agent reinforcement learning. 

### Strengths
- Excellent library for multi-agent reinforcement learning. 
- The description of the AEC framework in comparison to other models is well constructed. 
- Number and description of environments and supporting packages is clear and impressive.

### Weaknesses
- Much of the comparison with other game models is handled in the text, and could be diagrammed more clearly, for instanced in Figure 5. A table could also be used to quickly lookup differences in these approaches, with appropriate subheadings (i.e. race conditions). 
- Any quantitative measures on users, implemented games or other comparisons with competing frameworks would be helpful, as part of the challenge in comparing PettingZoo with other libraries is that the arguments are somewhat qualitative. A table here could again help. 


### Correctness
I believe the claims are correct.

### Clarity
The manuscript is clear and easy to read. 
 
### Relation to prior work
Relationship with other approaches is present and easy to parse. 

### Reproducibility
The code is well maintained and a popular repository on github. 

### Additional feedback, comments, suggestions for improvement and questions for the authors
I understand the need to distinguish from past work, but the comparisons with POSG and EFL sometimes come across as a bit biased. For instance ‘ POSGs Based APIs Don’t Make Conceptual Sense For Games Implemented In Code’ mainly concerns race conditions.


limitations_and_societal_impact: They could add some boilerplate on societal impacts.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents the PettingZoo library containing a set of multiagent environments with a Gym-like interface to users. 
main_review: Originality: this paper is not scientifically novel. According to the claim, the main contribution is "be GYM like for multiagent setting". 
However, "OpenAI Gym" is an arxiv paper and has been published nowhere (as far as I know, please correct me if i am wrong). 
Clarity: the writing of the paper easy to follow and clearn. It is more of an engineering document or tutorial than a scientific paper to me. 
Other: I would suggest the authors to expand the revisions to the ICML submission. 
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper introduces PettingZoo, a Python library for multi-agent RL tasks. It also discusses the API choices of said library, and how they might favoriably compare to other approaches available.
main_review: The paper is highly readable and relatively clearly written, with only minor mistypes and glitches (examples: a missing "s" in "observation[s]" in Figure 2, a missing "T" in "[T]he" in line 97, a missing "occur" after "typically" in line 185). It presents the present state of the accompanying library, PettingZoo, relatively well. The main contribution of the submission itself, on top of the library presented, is a discussion of "POSG" and "EFG", and the introduction of "AEC". These acronyms are taken to mean different API choices for multi-agent RL environments. Specifically, "POSG" is used for the API model in which a vector of observations and rewards is returned by the task object, one for each agent, which in turn expects a vector of actions. The "EFG" model is used for the API of OpenSpiel, namely the tree-like structure more natural for board games, in which random effects of the environment (such as during a game of Poker or Blackjack) are represented as "chance nodes". The suggested "AEC" API model has the task define an order in which agents get to observe and act. That order is presented as a (potentially infinite) Python iterator object and is meant to represent situations in which the turn order changes during the game, as it would e.g. in the card game Uno.

The paper discusses a specific bug the authors ran into while using the "POSG" model and argues their model would help preventing that type of bug. On the face of it, this would seem like a worthwhile point, given the complexities of RL research at scale, especially in the multi-agent case. However, this reviewer wasn't entirely convinced by the presentation. Indeed, it seems the paper makes some rather strong claims without being able to give a reference as to why they would be true, and the central anecdote around the bug / race-condition discussed seems overly simplifying.

Examples of statements that seem questionable in the generality in which they are stated:

"[popular previous APIs] promote severe bugs that are hard to detect" (from the abstract, line 11): While that may be true in an extremely general sense in that all RL research easily suffers from bugs that could be described that way, the specific example discussed seems insufficient to illustrate this in the particular case the authors must have meant, see below.

"[Multi-agent RL] remains a significant engineering challenge. This is because, unlike single agent reinforcement learning which has OpenAI's Gym, no de facto standard API exists [...] for how agents interface with environments." (line 18): This is clearly an overstatement in both directions. Single-agent RL remains a significant engineering challenge despite OpenAI's useful library, and although PettingZoo sounds like a very useful contribution to the landscape, its authors can hardly believe it will make multi-agent RL trivial or even easy. It may perhaps make it simpler along some dimensions, but even that is probably better argued empirically, and at any rate not the obvious reading of this statement, although the authors would likely support that statement along these lines.

"[We] discovered [...] [other APIs] do not make good conceptual sense [...] and cannot form the basis of APIs that cleanly handle all types of multi-player environments" (line 35): This is a rather strong statement that would require equally strong arguments. However, looking at the list of specific criticisms of previous APIs, we read (lines 72--): (1) Dummy actions are necessary and (2) changing numbers of agents requires learning code to deal with lists "suddenly" changing sizes. For the OpenSpiel API, the criticisms cited are (lines 89--): (1) It's too complex, especially for beginners or for RL researchers to "ubiquitously use it as [their] mental model of games", (2) the formal definition only has rewards at the end of the game -- which is followed by "this is possible to work around" (by changing that formal definition, perhaps?), but "not ideal", and finally (3) the lack of continuous action spaces -- followed by the admission that this, too, "is not inherent in the [OpenSpiel API]". This sauce is just a bit too weak for the strong praise the chef gave it.

Let's now discuss the anecdotal evidence the submission relates around race-conditions in multi-agent RL. An example is given where two agents observe a state, each gets a chance to act on it and they chose to act in incompatible ways. "Tie-breaking" is then happening in the environment, e.g., by one agent being selected to be allowed to perform its action as intended, while the other agent might be denied, or have its action applied to the situation after the first agent got to modify the state with its action. The authors mention a situation where this tie-breaking was happening in a biased way and suggest that serializing the interaction with the environment would get around this issue. As indeed it would. However, the claim that this "is more closed [sic!] models how computer games are executed in code" (line 211) is naive at best. While this reviewer is no expert in game engine development, it seems more likely modern games are implemented around event loops interacting with a main rendering thread. That is, the state of the game might continuously change (units move, wind blows, bombs explode, etc), while the players have a chance to interact with that game in an asynchronous way, perhaps rate-limited to a certain number of actions per minute by either the system or natural constraints in hardware, network latency, or human motor control. When doing RL research on such games, this reality needs to be taken into account. And it is, one way or the other, in results like AlphaStar (which gets around asynchronous environments by trying to be fast enough at test time, and make the environment ignore the occasional action that can no longer be applied, like trying to construct a unit with a building that's already destroyed). The challenges of real-world interactions like robotics or self-driving cars are similar, only worse. Now, it might reasonably be argued that it's too much to ask from an API to cover chess, StarCraft, and Mars robots all at once. And indeed that might be so -- but that's not exactly an argument in favor of a single new standardized API.

None of this is to say that the library the authors developed doesn't have the potential to be extremely valuable to the community, and might well already have great positive impact. It would seem to this reviewer, however, that online documentation of their API and the design choices behind it is a better format (especially in view of potential future changes) than a research paper.

As a side note, the submission's supplementary material is insufficiently anonymized. This reviewer closed the relevant file right after observing this and believes no undue influence on the double-blinded process has occurred.

UPDATE: Changed rating to 5: Marginally below the acceptance threshold based on author's response.
limitations_and_societal_impact: Some limitations discussed below. No potential negative societal impact seems likely to incur as a result of this submission.
review_title: null

Meta-review: I recommend this paper to be accepted. It provides a novel multi-agent reinforcement library with a corresponding games model. While the scientific novelty appears to be limited, the practical utility seems to be great for the community and such open-source libraries are explicitly encouraged in the Call for Papers. 
Decision: Accept (Poster)