Gymnasium: A Standard Interface for Reinforcement Learning Environments

## Abstract

Abstract Reinforcement Learning (RL) is a continuously growing field that has the potential to revolutionize many areas of artificial intelligence. However, despite its promise, RL research is often hindered by the lack of standardization in environment and algorithm implementations. This makes it difficult for researchers to compare and build upon each other’s work, slowing down progress in the field.
Gymnasium is an open-source library that provides a standard API for RL environments, aiming to tackle this issue. Gymnasium’s main feature is a set of abstractions that allow for wide interoperability between environments and training algorithms, making it easier for researchers to develop and test RL algorithms. In addition, Gymnasium provides a collection of easy-to-use environments, tools for easily customizing environments, and tools to ensure the reproducibility and robustness of RL research.
Through this unified framework, Gymnasium significantly streamlines the process of developing and testing RL algorithms, enabling researchers to focus more on innovation and less on implementation details. By providing a standardized platform for RL research, Gymnasium helps to drive forward the field of reinforcement learning and unlock its full potential. Gymnasium is available online at https://github.com/Farama-Foundation/Gymnasium .

## 1 Introduction

With the development of Deep Q-Networks (DQN) (Mnih et al., ), the field of Deep Reinforcement Learning (DRL) has gained significant popularity as a promising paradigm for developing autonomous AI agents.
Throughout the last decade, DRL-based approaches managed to achieve or exceed human performance in many popular games, such as Go (Silver et al., ), DoTA 2 (Berner et al., ) or Starcraft 2 (Vinyals et al., ).
During this time, OpenAI Gym (Brockman et al., ) emerged as the de facto standard open source API for DRL researchers.
Its simple structure and quality of life features made it possible to easily implement a custom environment that is compatible with existing algorithm implementations.
Gymnasium is the updated and maintained version of OpenAI Gym.
In this paper, we outline the main features of the library, the theoretical and practical considerations for its design, as well as our plans for future work.
We hope that this work removes barriers from DRL research and accelerates the development of safe, socially beneficial artificial intelligence.

## 2 Related Work

Due to its influential positioning, Gymnasium enjoys a wide ecosystem of compatible libraries. In this section, we describe some of the most prominent RL libraries that can be used with Gymnasium, as well as alternative API libraries.

### 2.1 Training Libraries

Stable Baselines3 (SB3) (Raffin et al., ) is a popular library providing a collection of state-of-the-art RL algorithms implemented in PyTorch. It builds upon the functionality of OpenAI Baselines (Dhariwal et al., ), aiming to deliver reliable and scalable implementations of algorithms like PPO, DQN, and SAC.

CleanRL (Huang et al., ) is designed to provide clean, minimalistic implementations of RL algorithms. It focuses on simplicity and transparency, making it easier for researchers to understand and experiment with different RL techniques.

Tianshou (Weng et al., ) is a versatile library for RL research that supports various training paradigms, including off-policy, on-policy, and multi-agent settings. It offers a modular design that allows users to easily customize and extend the library’s components.

Ray Rllib (Liang et al., ) is part of the Ray ecosystem and is known for its scalability and support for distributed RL training. Rllib provides a diverse range of algorithms and tools for both single-agent and multi-agent scenarios.

Dopamine (Castro et al., ) is a research framework developed by Google for experimenting with reinforcement learning algorithms. It is designed to provide a clean, minimalistic codebase that focuses on implementing and evaluating RL algorithms such as DQN and its variants. Dopamine emphasizes reproducibility and simplicity, offering well-structured, modular components that make it easy for researchers to implement and test new algorithms.

### 2.2 Alternative API Libraries

Several alternative libraries offer different approaches to defining and interacting with RL environments, providing valuable insights and capabilities beyond Gymnasium.

Dm_env (Muldal et al., ) is part of the DeepMind Control Suite and offers a lightweight API for RL environments. It emphasizes a clean, functional design and is used for evaluating algorithms in a controlled manner. While similar in some aspects to Gymnasium, dm_env focuses on providing a minimalistic API with a strong emphasis on performance and simplicity.

PettingZoo (Terry et al., ) is designed for multi-agent RL environments, offering a suite of environments where multiple agents can interact simultaneously. It builds on concepts from Gymnasium but extends its capabilities to support complex multi-agent scenarios, making it an important tool for research in cooperative and competitive settings.

OpenAI Gym (Brockman et al., ), the predecessor to Gymnasium, remains a widely used library in RL research. Gymnasium is built upon and extends the Gym API, retaining its core principles while introducing improvements and new features. Gym’s well-established framework continues to serve as a foundation for many RL environments and algorithms, reflecting its influence on the development of Gymnasium.

### 2.3 Other Tooling

Minari (Younis et al., ) defines a standardized format for offline RL datasets and provides a suite of tools for data management. These tools facilitate collection, ingestion, processing, and distribution of datasets. Additionally, Minari integrates with a cloud-based repository that hosts a variety of benchmark datasets, enhancing accessibility and reproducibility in offline RL research.

Metaworld (Yu et al., ) is a benchmark for meta-RL and multi-task learning. It contains 50 robotic manipulation tasks and allows for easy benchmarking of algorithms compatible with the Gymnasium interface.

Gymnasium Robotics (de Lazcano et al., ) is a collection of robotics environments, including maze path-finding and robot arm manipulation. It provides a multi-goal API compatible with Gymnasium, enabling support for algorithms like Hindsight Experience Replay  (Andrychowicz et al., ).

Atari Learning Environment (Bellemare et al., ) is a collection of environments based on classic Atari games. It uses an emulator of Atari 2600 to ensure full fidelity, and serves as a challenging and diverse testbed for RL algorithms.

## 3 Basic Library Structure

At its core, Gymnasium is a collection of interfaces tailored for usage in RL research, so that they can be reused by researchers and developers across various approaches and algorithms. In this section, we briefly describe the main abstractions and interfaces included in Gymnasium.

#### Environment

The central abstraction in Gymnasium is the Env. It represents an instantiation of an RL environment, and allows programmatic interactions with it. An Env roughly corresponds to a Partially Observable Markov Decision Process (POMDP) (Kaelbling et al., ), with some notable differences discussed in Section 4.1.

An Env is primarily defined by the following components:

- 1.
reset method
- 2.
step method
- 3.
Observation space
- 4.
Action space

The reset method sets the environment in an initial state, beginning an episode. The step method executes a selected action in the environment, and is the only mechanism of moving the simulation forward. The observation space defines the structure of observations that the environment emits, and the action space defines valid actions that can be used in step. For a detailed description of the Env class, we refer the reader to the
[documentation](https://gymnasium.farama.org/).

#### Vector Environment

A secondary, but arguably as important abstraction, is the VectorEnv . It represents a batch of identical, but independently running RL environments. In terms of POMDPs, they are identical POMDPs, but with independently sampled initial states, and executing potentially different actions. A VectorEnv has a structure analogous to that of an Env , with the key difference being that everything is batched across multiple independent environments. This is crucial for modern RL research, as it enables significant performance gains by parallelizing the environment execution, and the policy evaluation.

#### Space

Space is an abstraction shared between observation and action spaces. It roughly corresponds to a set, with some structure imposed on it to improve its utility for representing (observation and action) sets relevant to RL environments. We further describe Gymnasium spaces in Section 4.3.

#### Registry

Gymnasium provides a simple way to manage environments via a registry. Any environment can be registered, and then identified via a namespace, name, and a version number. The standard Gymnasium convention is that any changes to the environment that modify its behavior, should also result in incrementing the version number, ensuring reproducibility and reliability of RL research.

## 4 Novel features

Since its creation as a fork of Gym, we extended Gymnasium to include various new tools and features to simplify RL research and make it more reliable. In this section, we describe the main new additions.

### 4.1 Functional API and POMDPs

While the object-oriented Env class is the standard way of using Gymnasium, the library also provides FuncEnv, a secondary abstraction with a more functional approach. Beyond just enabling a different programming paradigm, this abstraction has two main advantages: it is more closely connected to the theoretical POMDP formalism, and it enables easy hardware acceleration of implemented environments by using libraries like Jax (Bradbury et al., ).

Similarly to Env, a FuncEnv is defined with several functions:

- 1.
initial function: Generates the initial state of the environment.
- 2.
observation function: Returns the observation for a given state in the environment.
- 3.
transition function: Computes the next state of the environment based on an action.
- 4.
reward function: Computes the reward for transitioning from one state to another given an action.
- 5.
terminal function: Determines if a state is terminal, indicating the end of an episode.

This is very close to the components of a POMDP. As a reminder, a POMDP is typically defined as a tuple $\mathcal{M}=(\mathcal{S},\mathcal{A},T,R,\Omega,O,\mu)$, where

- •
$\mathcal{S}$ is a set of states of the environment.
- •
$\mathcal{A}$ is a set of actions available to the agent.
- •
$T\colon\mathcal{S}\times\mathcal{A}\to\Delta\mathcal{S}$ is the environment transition function, representing its dynamics.
- •
$R\colon\mathcal{S}\times\mathcal{A}\times\mathcal{S}\to\mathbb{R}$ is the reward function which is used to define the agent’s task.
- •
$\Omega$ is a set of possible observations
- •
$O\colon S\to\Delta\Omega$ is the observation function mapping states to observations.
- •
$\mu\in\Delta\mathcal{S}$ is the initial state distribution.

In the usual object-oriented API, we cannot create a one-to-one connection
between the Env members and methods, and the POMDP components. Instead, the API combines some of the components:

- 1.
reset is equivalent to sampling $s\sim\mu$, and returning a sample from $O(s)$
- 2.
step takes an action $a\in\mathcal{A}$, computes the next state as $s^{\prime}\sim T(s,a)$, and then returns the new observation $o\sim O(s^{\prime})$ and the reward $r=R(s,a,s^{\prime})$
- 3.
The observation space is equivalent to $\Omega$
- 4.
The action space is equivalent to $\mathcal{A}$

In the case of the FuncEnv , the connection is much closer:

- 1.
initial function corresponds directly to sampling the initial state from $\mu$, i.e., $s\sim\mu$.
- 2.
observation function corresponds to the observation function $O(s)$, returning an observation $o\sim O(s)$ for a given state $s$.
- 3.
transition function directly implements the transition function $T(s,a)$, which computes the next state $s^{\prime}\sim T(s,a)$ given a state $s$ and action $a$.
- 4.
reward function implements the reward function $R(s,a,s^{\prime})$, which calculates the reward based on a given state $s$, action $a$, and the resulting next state $s^{\prime}$.

The one missing function, terminal, does not correspond to any element of the classical POMDP description, but is necessary for practical reasons. We expand on this in the following section.

### 4.2 Termination and Truncation

In RL theory, it is common to assume that actions can be executed in an environment indefinitely, without stopping. In practical work, this tends to be unrealistic, as researchers only have finite time to perform their experiments. To account for that, Gymnasium introduces the notions of episode termination and truncation. In OpenAI Gym, these two concepts were not clearly separated, so it is worth expanding on their utility and the distinction between them.

Termination is a signal that depends on the state reached after an environment transition. Typically, this represents either success or failure of the task that the RL agent is trying to achieve, but generally, it is any state-dependent reason to end an episode. After reaching a terminal state, step cannot be called again until the environment is reset. It is possible for an environment to not have any terminal states, as is common in theoretical RL problems. In the POMDP formalism, the typical way to represent episode termination is entering an absorbing state with 0 reward.

Truncation is similar to termination in the fact that it indicates the end of an episode, but instead of being state-based, it is time-based. It is roughly the equivalent of saying “The episode could keep going, but we ran out of time, so we decided to stop it here”. In Gymnasium, we provide the TimeLimit wrapper that is applied to environments by default, through which developers and users can define a number of steps after which the environment is guaranteed to finish via truncation.

While the difference between termination and truncation may seem minor, it has nontrivial implications for algorithm implementations. For example, in algorithms like REINFORCE (Sutton et al., ) and Proximal Policy Optimization (PPO) (Schulman et al., ), a key component is estimating the value of a given state, that is the expected discounted sum of rewards in the rest of the episode. Similarly, in value-based algorithms like Deep Q-Networks (DQN) (Mnih et al., ) or Soft Actor-Critic (Haarnoja et al., ), the central concept is that of a state-action value, which also depends on future rewards.

If an episode terminates, this means that there are no further rewards to be obtained. This means that in the value estimation process, the value after a terminal state is equal to 0. In contrast, if an episode is truncated, that means that the agent encountered a rather arbitrary cutoff. If not for the truncation, it could have kept acting and accumulating rewards, so in the value estimation algorithm, we use an estimate of the final state’s value.

For a practical example, consider a racing environment where the agent has to complete $n$ laps to finish the race as quickly as possible. If the final signal is that of termination, this means that the agent should optimize for that exact number of laps. If its trajectory at the end of the last lap means that it would crash shortly after passing the finish line, that is not a problem – that will never happen. Conversely, if the final signal is truncation, then the agent will be incentivized to act as if it would keep racing. This would result in decreasing the reward obtained in the original timeframe, but acting in a safer, more sustainable manner towards the end of the race.

### 4.3 Algebraic spaces

In Gymnasium, observation and action spaces can be broadly split into two categories – fundamental and composite. As the name suggests, composite spaces are composed of one or more subspaces, whereas fundamental spaces cannot be divided this way. There are the following fundamental spaces:

- 1.
Box – for multidimensional arrays typically containing real numbers
- 2.
Discrete – for single integers
- 3.
MultiBinary – for sequences of binary values
- 4.
MultiDiscrete – for sequences of integers
- 5.
Text – for strings

There are also the following composite spaces:

- 1.
Dict – for (Python) dictionaries of spaces
- 2.
Tuple – for tuples of spaces
- 3.
Sequence – for variable-length sequences of spaces
- 4.
Graph – for graphs of spaces (both nodes and edges)
- 5.
OneOf – for disjoint unions of spaces

Considering just the fundamental spaces, and the Tuple(^*^**Or, equivalently, Dict. The difference between the two is the customizability of dictionary keys for the sake of usability.) and OneOf composite spaces, we observe that Gymnasium spaces mirror the structure of Algebraic Data Types. We can take any collection of spaces and combine them into a Tuple to obtain a product type – an element of a Tuple space must contain an element from each of its subspaces. We can also combine them into a OneOf space, equivalent to a sum type, which contains an element from one of its subspaces, along with the index of that subspace.

### 4.4 Vectorization

Figure: (a) Individual environment steps per second. Higher is better.
Refer to caption: extracted/5987685/figures/cartpole_sps_mbp.png

Gymnasium puts an emphasis on treating vectorized environments as first class citizens, on par with individual environments. This is because vectorization is a common optimization technique in RL research, enabling significant performance gains without major changes to the algorithm implementation.

By default, Gymnasium supports two vectorization modes, as well as a custom mode that can be defined for each environment separately. SyncVectorEnv vectorizes arbitrary environments in the simplest way – it runs them in sequence and batches the results. AsyncVectorEnv is the second method of vectorizing arbitrary environments, by running each of them in its own subprocess.

The relative performance of Sync and Async vectorization depends on many factors, notably the complexity of the environment itself and the properties of the hardware that it is run on. For simple and computationally inexpensive environments like Cartpole, the overhead introduced by creating subprocesses in Async vectorization exceeds the computational needs of the environment. This means that Sync significantly outperforms it, and we can obtain even better performance by implementing a vectorized environment using NumPy vectorization. We show this in Figure 1.

Figure: (a) Individual environment steps per second, executed on a MacBook Pro. Higher is better.
Refer to caption: extracted/5987685/figures/lunar_sps_mbp.png

With more complex environments like the Lunar Lander, the picture gets more complicated. In this case, on a machine with enough memory, subprocess-based Async vectorization can provide a significant performance boost, as spawning new processes is not that impactful. On a weaker machine, the subprocess overhead remains impactful, resulting in Sync vectorization yielding better performance. We show this in Figure 2.

In conclusion, the choice of vectorization is far from a trivial one, and the VectorEnv abstraction in Gymnasium enables easy experimentation between the two built-in methods, as well as custom user-defined approaches. This allows every user to tailor their approach to obtain the best performance for their use case, without having to change the rest of their code.

## 5 Built-in environments

Gymnasium includes a collection of well-tested environments that are fully compliant with the API. Those can serve as references for new environment implementations, or as simple testing grounds for algorithm development. Here, we describe these environments.

#### Toy text

There are four toy text environments: Blackjack, Taxi, Cliff Walking and Frozen Lake. These are discrete environments that can be modeled as tabular MDPs, and as such can serve as testbeds for RL algorithms that do not rely on neural network for state representation.

#### Classic control

In the classic control suite, there are four environments: Cartpole, Acrobot, Mountain Car, and Pendulum. Each of them has a fairly simple physics simulation at its core, a continuous observation space, and either a discrete or continuous action space. These environments tend to serve well as quick and simple evaluations for new algorithm implementations. A notable exception is Mountain Car, which is often difficult to solve without a deliberate exploration mechanism.

#### Box2D

There are three environments using the open-source Box2D physics engine: Bipedal Walker, Car Racing, and Lunar Lander. These environments use a more complex physics simulation, including components like collision detection and contact forces (both of which are neglected in the classic control environments). These represent another step up in complexity, posing a non-negligible, but still very tractable challenge for RL algorithm implementations.

#### Mujoco

MuJoCo (Todorov et al., ) environments are a set of eleven physics-based continuous robot control environments. These environments simulate more complex physical interactions, including multi-body dynamics and contact forces, offering a more realistic and high-dimensional setting compared to the previously described environments. Each of these environments involves controlling a simulated agent to achieve tasks such as locomotion or balance in a continuous state and action space. The MuJoCo environments are often used as benchmarks for deep reinforcement learning algorithms due to their higher dimensionality, non-linear dynamics, and need for precise control.

## 6 Summary

This paper introduces Gymnasium, an open-source library offering a standardized API for RL environments. Building on OpenAI Gym, Gymnasium enhances interoperability between environments and algorithms, providing tools for customization, reproducibility, and robustness. It is compatible with a wide range of RL libraries and introduces various new features to accelerate RL research, such as an emphasis on vectorized environments, and an explicit interface for functional environments that can be hardware-accelerated. Gymnasium includes a suite of benchmark environments ranging from finite MDPs to MuJoCo simulations, streamlining RL algorithm development and evaluation, with the goal of accelerating advancements in safe and beneficial AI research.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 4
questions: There is still significant space available for additional results on other environments (which may be more complicated and thus show interesting ablations for synchronous and nonsynchronous methods) provided in Gymnasium. Why not provide additional examples of the work in the paper? There is plenty of room, and it will help show readers unfamiliar with your package about the library's use cases. See weaknesses for examples of papers with more extensive write-ups regarding libraries.

> We hope that this work removes barriers from DRL research and accelerates the development of safe, socially beneficial artificial intelligence.
How does this work do that? This claim is not defended by the experiments.
soundness: 2
strengths: Gym and now gymnasium is an essential part of reinforcement learning with many packages and works being built on top of or inspired by a similar methodology. The abstraction provided by the method is commonly used across reinforcement learning and is easy to use. The vectorization increases throughput of running an environment without significant overhead. The writing is fairly straightforward and it is easy to understand the methodology of the work. Without outside comparison, the results show that the vectorization is able to provide significant speedups. The use of vectorization is key to enable speed-up in performance.
summary: The paper attempts to address the lack of standardization in reinforcement learning environments. The paper’s library is able to be used in conjunction with a number of libraries, showing that it has been adopted across the field. The paper provides abstraction for environments through a set of fundamental spaces. The paper implements vectorization to enable parallelization across environments. The paper analyzes the increase in environment throughput with vectorization to show that this can provide substantial speedups to using the library.
weaknesses: Despite the use of the name recognition of the library, it is unclear if the improvements from OpenAI Gym to Gymnasium are comparable to the contributions from other RL libraries 2015-present. My main issue with the work is the lack of discussion and comparison with similar work. (In section 2 the paper details a number of comparable works that can use their framework but this is not sufficient to my point.) The paper should complement technical report details with meaningful comparisons to help researchers choose which library to use.

Reinforcement learning is going a massive change in environments due to parallelization but these benefits are from Jax and other GPU engines such as Madrona engine [1], not work created by this library so I do not think it is a fair claim that they can claim credit for those works. 

Other works have better vectorization such as Puffer [2] and I would expect a mention or comparison to similar libraries. The paper should explain why one should use Gymnasium versus competitors. It would be helpful to see comparisons across the same environments for each library.

RLlib [3], which is in the related work of the paper, shows substantial deployment and scalability for reinforcement learning on distributed training. 

Each of these papers regarding reinforcement learning libraries provide substantial merits for scaling reinforcement learning training. Though I think it is important to encourage RL infrastructure and environment development in reinforcement learning, it is unclear that a set of abstractions provided by Gymnasium have the same level of contribution as the other papers empirical efforts.



[1] Shacklett, B., Rosenzweig, L. G., Xie, Z., Sarkar, B., Szot, A., Wijmans, E., ... & Fatahalian, K. (2023). An extensible, data-oriented architecture for high-performance, many-world simulation. ACM Transactions on Graphics (TOG), 42(4), 1-13.
[2] Suarez, J. (2024). PufferLib: Making Reinforcement Learning Libraries and Environments Play Nice. arXiv preprint arXiv:2406.12905.
[3] Liang, E., Liaw, R., Nishihara, R., Moritz, P., Fox, R., Goldberg, K., ... & Stoica, I. (2018, July). RLlib: Abstractions for distributed reinforcement learning. In International conference on machine learning (pp. 3053-3062). PMLR.

---

confidence: 5
contribution: 4
initial_rating: 10
presentation: 4
questions: - Does Gymnasium include any tools or guidelines to help users choose the best vectorization mode (Sync vs. Async) based on their hardware?
- How does Gymnasium handle massively parallel or distributed training environments? Is there integration with cloud-based or cluster-based frameworks for larger-scale experiments?
soundness: 4
strengths: - By providing a consistent API, Gymnasium streamlines RL research, making it easier for researchers to compare and build upon previous work.
-The functional API enhances compatibility with theoretical frameworks like POMDPs and enables hardware-accelerated environments using libraries like JAX, which is beneficial for large-scale or computationally intensive applications.
- The built-in vectorization features (Sync and Async) support efficient parallelization of environments, which can significantly improve the performance of RL training processes.
- Gymnasium includes environments from basic MDPs to complex MuJoCo simulations, catering to various RL research needs and supporting a broad range of algorithms and approaches.
summary: The paper  introduces Gymnasium, a standardized API designed to improve interoperability and reproducibility in reinforcement learning (RL) research. Gymnasium, an update and extension of the OpenAI Gym, addresses issues such as inconsistent environment implementations and the need for easy experimentation and robust testing. Its main contributions include a unified API compatible with multiple RL libraries, support for both individual and vectorized environments, and novel features such as a functional API closely aligned with Partially Observable Markov Decision Processes (POMDPs), a structured approach to termination and truncation, and expanded support for algebraic spaces. Gymnasium also provides a suite of customizable benchmark environments that range from simple tabular MDPs to complex physical simulations using MuJoCo, helping to accelerate RL algorithm development and testing.
weaknesses: The choice between Sync and Async vectorization modes shows substantial performance variability depending on hardware, which could lead to inconsistencies across different systems or add complexity for users lacking high-performance resources.

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: I don't have any questions for the authors.
soundness: 4
strengths: The paper provides a well-written, comprehensive overview of Gymnasium, which would be of use to a newcomer to this API.

Gymnasium itself is an extremely valuable contribution to the RL community, as evidenced by its widespread adoption.
summary: The paper provides an overview of the widely-used Gymanasium open-source API for RL experiments. It describes the two approaches for interafcing to environments (standard Env class, and the FuncEnv functional API), discusses the difference between Termination and Truncation, the data types used for action and observation spaces, the role of VectorEnvs in enabling parallelisation of learning. It also reviews the built-in environments, related RL implementations and tool libraries, and alternative API libraries.
weaknesses: There is little original content in this paper as it is basically replicating the documentation available through the Gymnasium website (https://gymnasium.farama.org/index.html).

The decision of whether to accept this paper or not really comes down to a philosophical question around the purpose of ICLR. 

The Reviewer's Instructions ask us to consider the paper's value from the contribution of the ICLR community ("What is the significance of the work? Does it contribute new knowledge and sufficient value to the community? Note, this does not necessarily require state-of-the-art results. Submissions bring value to the ICLR community when they convincingly demonstrate new, relevant, impactful knowledge". From this perspective I don't think the paper has much value.  The vast majority of RL researchers that would attend ICLR would already be familiar with, and quite likely already using, Gymnasium. So there is little here that would be especially novel for attendees.

On the other hand we all know that publications at key venues like ICLR have considerable weight for career progression, and undoubtedly the authors deserve credit for their work in developing and maintaining this valuable resource. So there could be a case for acceptance on that basis.

This feels to me like a decision that perhaps needs to be made at a higher level in the conference hierarchy than reviewers?

My personal feeling is that this is perhaps the wrong venue for this work. If it is not accepted, I'd suggest the authors consider submitting to the Software Track of JMLR, as that is designed to support precisely this type of publication.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 2
questions: Based on the presented paper and the reported progress on GitHub, I have the following questions:

- Indeed, Gymnasium is a great successor to Gym. But it is undeniable that there are still a large number of projects using Gym due to various reasons like old environments or dependencies. How will Gymnasium promote further popularization in the community?
  
- On the other hand, how does Gymnasium achieve better backward compatibility with old code? A section about the code compatibility should be added.
  
- In Gymnasium, the `.reset()` is re-implemented, and the `.seed()` method is deprecated. I don't see the necessity for this change, could you elaborate it further?
  
- In Section 2.1, more new libraries should be included, such as d2rlpy [1] (for offline RL), TorchRL [2], and RLLTE [3]. Dopamine is a bit too old.
  
- I did't see the future work section, which is quite important for the paper.
  
- Code examples should be included, which also need to demonstrate the difference between Gym and Gymnasium.
  
- I strongly recommend the authors to make a chart to illustrate the structure and new logic of Gymnasium.
  
- For Section 5, using a table that illustrates the built-in envs and related official envs (e. g., MiniGrid and Procgen) would be better. The authors can also highlight the potential research targets of these envs, such as Procgen is procedurally-generated and MiniGrid is for exploration.

- The performance benchmarks focus on relatively simple environments such as **Cartpole** and **Lunar Lander**. It would better to include a broader range of environments, particularly more challenging and high-dimensional ones, to demonstrate Gymnasium's true potential.
  

I'm pleased to increase my rating if my questions are addressed.

References

[1] Seno T, Imai M. d3rlpy: An offline deep reinforcement learning library[J]. Journal of Machine Learning Research, 2022, 23(315): 1-20.

[2] Bou A, Bettini M, Dittert S, et al. TorchRL: A data-driven decision-making library for PyTorch[C]//The Twelfth International Conference on Learning Representations.

[3] Yuan M, Zhang Z, Xu Y, et al. RLLTE: Long-Term Evolution Project of Reinforcement Learning[J]. arXiv preprint arXiv:2309.16382, 2023.
soundness: 4
strengths: It is natural to compare Gymnasium and the OpenAI Gym to highlight its strengths:

- **Active developer community**. As a maintained fork, Gymnasium has gained 7.1k stars, 450+ issues and, 750+ PRs on GitHub, which demonstrates a really active community. This open-source collaboration will make Gymnasium a comprehensive library that aligns with the latest direction of RL.
  
- **New `.step()` logic**. Gymnasium introduces the notions of episode
  **termination** and **truncation**. **Termination** ends an episode based on the agent reaching a specific state, like winning or failing, signaling the task is complete. **Truncation** stops an episode after a set time or a number of steps, even if the task isn't finished, helping to limit long-running tasks for practical experiments. Both concepts are important for controlling episode length and ensuring efficient learning in RL.
  
- **More built-in environments**. As compared to Gym, Gymnasium provides more built-in environments that cover more research areas, such as offline RL and meta RL. Gymnasium's team also integrated many old and important environments, such as MiniGrid, Procgen, and Meta-World.
  
- **Performance optimization**. The library's emphasis on **vectorized environments** allows researchers to parallelize experiments and improve performance without major changes to algorithms. The performance comparisons presented in the paper show significant gains with proper vectorization modes.
  
- **Functional API**. The introduction of a *FuncEnv* offers flexibility in programming paradigms and is better aligned with the POMDPs. It also allows for efficient usage of hardware accelerators like JAX.
summary: This paper introduces Gymnasium, an open-source library that provides a standard API for building RL environments. As a maintained fork of OpenAI Gym, Gymnasium offers enhanced features and interoperability across RL environments and training algorithms. Gymnasium has significantly outperformed the Gym in terms of built-in environments, application ecosystem, and operating efficiency. There is no doubt that Gymnasium has become the new standard interface in the RL community, which is expected to continue to drive forward the field.
weaknesses: Please refer to the questions part.

Meta-review: This paper is a technical report of the well-known and widely-used Gymnasium benchmark for Reinforcement Learning (RL). Reviewers have unanimously praised the quality and relevance of the benchmark; however, some expressed concerns about the significance of this report and its overall quality.

Strengths
-----------
- Gymnasium is one of the most important benchmarks for RL;
- It is important that such benchmark is properly described in a technical report.

Weaknesses
--------------
- The report mostly contains technical details that are easily understandable from the online documentation, without providing additional relevant details;
- An extensive and comprehensive benchmarking of different algorithms in the Gymnasium environments is missing.
- The scope of this paper at this venue is questionable.

The paper has received positive scores, with all the Reviewers recognizing the importance of Gymnasium. However, some concerns about the impact and significance of this paper have been raised in the reviews. After engaging into discussion among them and with the meta-Reviewer, Reviewers have agreed that this work does not sufficiently suit this venue. More in detail, Reviewers and meta-Reviewer have agreed that this report does not provide a sufficient number of additional information to provide enough contribution to justify a publication. Moreover, the potential interest of attendees in this paper is likely to be significantly limited, given how well-known the library already is. Finally, the overall quality of the paper is not high, considering that the report mostly describes high-level design choices and technical details, lacking an extensive benchmarking and potential insightful additions, e.g., tutorials.

Based on this, I deem this paper not suitable for publication at this venue. I recommend the Authors considering other venues/journals highly focused on open-source software, e.g., JMLR MLOSS, NeurIPS Call for Dataset and Benchmarks.

Decision: Reject