Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks
Authors: Maxime Chevalier-Boisvert
Bolun Dai
Mark Towers
Rodrigo De Lazcano Perez-Vicente
Lucas Willems
Salem Lahlou
Suman Pal
Pablo Samuel Castro
J K Terry
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Reinforcement Learning
Simulation Environment

Abstract: We present the Minigrid and Miniworld libraries which provide a suite of goal-oriented 2D and 3D environments. The libraries were explicitly created with a minimalistic design paradigm to allow users to rapidly develop new environments for a wide range of research-specific needs. As a result, both have received widescale adoption by the RL community, facilitating research in a wide range of areas. In this paper, we outline the design philosophy, environment details, and their world generation API.  We also showcase the additional capabilities brought by the unified API between Minigrid and Miniworld through case studies on transfer learning (for both RL agents and humans) between the different observation spaces. The source code of Minigrid and Miniworld can be found at https://github.com/Farama-Foundation/Minigrid and https://github.com/Farama-Foundation/Miniworld along with their documentation at https://minigrid.farama.org/ and https://miniworld.farama.org/.

Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks
Maxime Chevalier-Boisvert
Bolun Dai
Mark Towers
Mila Québec AI Institute
New York University & Farama Foundation bolundai iondaa@@ny.eee edu
University of Southampton & Farama Foundation mt 70soton..ooooom uk
maximechhevllierbdgmall. com
Rodrigo de Lazcano
Lucas Willems
Salem Lahlou
Farama Foundation
Miple
Mila Québec Al Institute
rperravicentee@aaraaa.a org
lucas www.wememmiple.coo
lahlosal@mi la. quebec
Suman Pal
Pablo Samuel Castro
Jordan Terry
Telekinesis suman74950gmiil.com
Google DeepMind
Farama Foundation
pscêgoogle.co com
Swarm Labs jkterry@umd. edu
Abstract
We present the Minigrid and Miniworld libraries which provide a suite of goal- explicitly created with
oriented 2D and 3D environments. The libraries a minimalistic design paradigm to allow users to rapidly develop new environ- ments for wide range of research-specine ceived widescale adoption by the RL community, facilitating research in a wide range of areas. In this paper, outline the design philosophy, environment details, and their world generation API. We also showcase the additional capa- bilities brought by the unified API between Minigrid and Miniworld through case studies on transfer learning (for both RL agents and humans) between the different observation spaces. The source code of Minigrid and Miniworld can be found at https: httbbttttttttbet. oouttuuocomoaaaaaaoooooootttoooo weaiiiioo/iiiiiiii and https: //gi thub. http://aaaaaaaoooo Fondaaaatoooiiionnniiiiiioi
needs. As result, both have
along with their documentation at htttps:/mminge f farama. org/ and https: htt//mminnii htmm/miinwooroo ld. farama. org/.
Introduction
The capabilities of reinforcement learning (RL) agents have grown rapidly in recent years, in part thanks to the development of deep reinforcement learning (DRL) algorithms [32, 33J. This has been supported by suites of simulation environments such as OpenAl Gym [5] (now gymnasi um) and dm_ mo_cootrol [38] that provide common benchmarks for comparing algorithms. These libraries focus providing environments where the agent learns to control itself understand complex visual observations (c.g., swinging up a pendulum or playing video games) rather than logical reasoning or instruction following.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Minigrid
Miniworld
Figure 1: Example environments from Minigrid and Miniworld.
This paper outlines the Minigrid and Miniworld libraries for 2D and 3D goal-oriented environments which implement a suite of goal-oriented, navigation-based and instruction- -base environments. Furthermore, the two libraries have easily extendable environment API for implementing novel research-speciffc environments. Example environments for both libraries are shown in Figure In particular, Minigrid and Miniworld focus on providing users with the following features:
1. Easy installation process
The libraries maintain minimal list of dependencies such that a wide range of audiences can easily use the libraries.
2. Customizability Users can easily create new environments or add functionalities to existing environments.
3. Easy to visualization The enviromments can be viewed from the top down, making casier to visualize and understand the learned policy.
4. Scalable complexity A range of environments with different complexity is provided, which allows users to understand the limitations of the learned policy.
The libraries can be installed using Python's package manager PIP (pip install minigrid and pip install miniworld) with environment documentation and tutorials available at minigr id. farama. org and miniworld. farama. org, respectively.
Minigrid and Miniworld have already been used for developing new RL algorithms in number of areas, for example, safe RL [39], curiosity-driven exploration [24], and meta-learning [12]. Furthermore, research has built upon Minigrid for new environments, e.g. BabyAl [6] where environment is constructed and evaluated using natural language-based instructions. However, despite the popularity of the libraries, date, no academic paper has explained the design philosophy, environment API, provided case study for users.

---

Minigrid & Miniworld Libraries
In this section, we outline the design philosophy (Section 2.1), the environment specifications of Minigrid and Miniworld (Sections 2.2 and 2.3) along with the environment API (Section 2.4). Finally, we detail how published research has used both libraries to develop and evaluate novel Reinforcement Learning algorithms (Section 2.5).
The environments in the two libraries are obbsaallyobberrvable Markov Decision Processes (POMDP) [18]. These environments can be mathematically described by the tuple (A, O, T, R, 7) where X' represents state space, A the action space, the observation space, T X A + X transition function, R. X x A - IR the reward function, Q X' - O the observation function, and y E [0, 1) the discount factor.
2.1 Design Philosophy
Minigrid and Miniworld were originally created at Mila Québec Institute be primarily used by graduate students. Due to the variety in usages, customizability and simplicity were the highest priority allow as many users to use and understand the codebase. Gym's 's RL environment API [5] (now updated to Gymnasium due to Gym no longer being maintained) was selected to implement the libraries due to their popularity within machine learning and reinforcement learning communities. Example code for interacting with the environment is provided Listing 1.
support this, Python and
import gymnasium gym
load the environment in upper-left corner of Figure 1
env gym. ("MinnGriccBlocokkllloooccccci
render_mode-- "haimmmmmmmmm"na" "hmman")
observation, info = env reset (seed=42) for in (ange(1000).
# User-defined policy function
action policy (obyeroobsr)atton) observation, reward, terminated, truncated, info = env. step(action)
if terminated truncated: observation, info env. reset() env. clo.el
Listing 1: Code snippet for testing RL policy in Minigrid environment.
An additional core design point was to intentionally have as few external dependencies as possible, fewer dependencies make these packages easier install and less likely to break. As a result, Minigrid uses NumPy for the GridWorld backend along with the graphics to generate icons for each cell. Miniworld uses Pyglet for graphies with the environments being essentially 2.5D due the of flat floorplan, which allows for a number of simplifications compared to a true 3D engine. This allows the libraries relatively fast but more importantly enables users to understand the whole environment implementations and customize them for their own needs.
2.2 Minigrid Environments
Each Minigrid environment is 2D GridWorld made up of n x m tiles where each tile is either empty occupied object, e.g., wall, key, or goal. Using different tile configurations, tasks of varying complexity can be constructed. By default, the environments are deterministic with no randomness in the transition function (7).
By default, agent observations (O) are dictionary with three items: "image", "direction", and "mission" Example environments with their corresponding "image" and "mission" are provided

---

open the red door then the blue door
open the door
pick up the purple
Figure 2:
Example Minigrid environments with their mission instruction. For each of the environ- ments, the highlighted region indicates the partial observation received by the agent.
in Figure 2. The "image" observation top-down render of the agent's view which can be limited to a fixed distance or of the whole environment. The direction is an integer representing the direction the agent facing. The "sion"" a text-based instruction specifying the task to solve, The "mission" can change each environment's reset as the goal might change, for example, "Minigris MMingrridGGooooe d-GoToobjecc" has a mission of "go to the (color} [obj_type]" where color can be one of ["red", "green", "blue", "purple", "yellow", "grey"] and obj_type type can be one of ["key", "ball", "box"]. As a result, the instructions can be encoded as a one-hot vector. For more complex instructions, a language model is required interpret for RL agent.
The agents have discrete action space (A) of seven options representing ["turn left", "turn right", "move forward", "pickup", "drop", "toggle", "done"]. These actions are consis- tent across all environments however some actions might not produce any effect in certain states, e.g. "pickup" will not do anything if the agent is not next to object that can be picked up.
The default reward function (R) for the environment is sparse such that the reward is only non-zero when the mission is accomplished. Furthermore, the function can be easily customized for specific user needs through overwriting the environment's wewiGrridnvv_eeewwreeee function.
2.3 Miniworld Environments
Each Miniworld environment is 3D world that consists of connected rooms with objects inside (e.g. box, ball, or key). Like Minigrid, the worlds can be configured for various tasks with different goals and complexity.
For the agent, the observation space (O) is, by default, an RGB image of size 80 from the agent's perspective of the world. This image size can be modified by passing obs_wtt width and obs_ bos_height arguments to the environment constructor. Five of the images in Figure I are example observations (with the final being top-down view of the agent in environment). To act in the world, agents are provided with similar action space (AA) to Minigrid with an additional move-back action. Thus, there are in total eight discrete actions: ["turn left", "turn right", "move forward", "move back", "pickup", "drop", "toggle", "done"]. Like Minigrid, the default reward function is sparse with the agent only being rewarded when the agent completes the environment goal but be modified in custom environments.
2.4 Constructing and Extending Environments
In both libraries, the enviromments can be created using a small set functions. To demonstrate this feature, showcase two sample scripts used to define the simulation environment in Listing 2, one for Minigrid and one for Miniworld.
The structure of the environment generation function is the same for more complex see- narios, with few more helper functions. We have created tutorials for new environment creation: https: /tte:://nnnieree grid, farama. / //miniworld. faaaniaawwd....aaaa..mm.. farama. org/ /maim/a///// wwtp///entttteteeeeccce htt.a/////oooooeeeeerreeeee..
and https: respectively. Both libraries can be di- rectly integrated with existing RL libraries, e.g., Stable-Baselines? (SB3). Additionally, to aug- ment the libraries, we have created extra wrappers that customize the behavior of the libraries, such as adding stochastic actions and varying observation spaces, https: htt////miiigii// farama.

---

def _gn grienng width, height): """Minigrid Example"m
def _gen_world(self):
"""Miniworld Example
# Create an empty grid
# Create rectangular room self. add__reett_roommmiin.__a
self. self.grid = Grid(width, height) # Generate surrounding walls w.f.f.rrd..wall......
max_x-self.size min_z=0, max_z=self.size)
0, width,
height)
Place goal in random location
# Place goal
self fox. = self. haaffpplacceettttyl Box(color="red"
self. pou_obb(Goall] width 2,
height 2)
) Place agent in random location self. place_agent
Place agent in random location self. place_agent()
Listing 2: Code snippet for environment generation in Minigrid (left) and Miniworld (right).
org/api/wrapper/ and http:///twtttttoooooooo////oo//a miniworld/wrappers.py
2.5 Adoption
Since their creation, Minigrid and Miniworld have been widely adopted by the RL research community and used for various applications. Together, the two repositories have around 2400 stars and 620 forks on GitHub. We detail several instances where the two libraries have been utilized effectively.
Curriculum Learning: The two libraries provide programmatic approach to creating new envi- ronments on-the-fly, this functionality can utilized for automatic environment generation. For example, Dennis et al. [7] generated natural curriculum of increasingly complex environments and Parker-Holder et al. [26] harnessed the power of evolution a principled, regret-based curriculum.
The reward function in the two libraries is, by default, a sparse reward making them ideal candidates for developing new exploration techniques. Using Minigrid and Miniworld, Seo et al. [31] developed an exploration approach using state entropy as the extrinsic reward and Zhang et al. [40] proposed a simple yet effective exploration criterion by equally weighting the novel areas.
Exploration:
Meta Learning & Transfer Learning: Given the case creating new simulation environments, the two libraries have also been used in developing meta-learning and transfer-leaming algorithms. In Igl et al. [15], Minigrid has been used to develop regularization techniques to encourage agents to generalize new environments. In Liu et al. [21], Miniworld is used to develop
meta-learning approach that avoids local optima in end-to-end training, without sacrificing optimal exploration and Hutsebaut-Buye Buysse transfer learning.
[14] explored the of pre-trained fask-independent word embedding for
Recent work has also used both libraries number of other research topics, demonstrating how Minigrid and Miniworld prove useful for wide variety of domains. For example, Gumbsch et al. [9] leverage the partial observability in many of the Minigrid environments to develop POMDP planning algorithms; Zhou and Li [42] use Minigrid environments for inverse reinforcement learning tasks; and Zhao al. [41] customized Minigrid develop model-based RL algorithms.
3 Case Studies for Utilizing the Unified API
In this section, we provide two case studies to demonstrate the utility and ease use of Minigrid and Miniworld's 's unified API and hope to inspire future studies that span both environments. The first is on RI. agent transfer learning between different observation spaces of the two libraries. The second case study shows how human transfer learning can be conducted between different observation spaces.
5

---

Figure 3: Visualization of the miniworld wowworld-gooo wooorlddgotooo-eee (left) and waniigiddggtoobj--ee (right).
The miniwor Id -o-otoobjj-en- image shows both the top-down view and the agent view (top-right window). During training the agent only has access the agent view of the environment.
3.1 RL Agent Transfer Learning Between Different Observations Spaces
In this case study, we showcase the ability to transfer policies learned on Minigrid to Miniworld. We created two similar simulation environments in Minigrid and in Miniworld, where the task is to follow an instruction to go to an object, we call the two environments minigrid-gon sorriid-goooo-eeee and miniworld ld-gotoobj-en (screenshots of the two environments are shown in Figure 3). In both environments, the agent is given an instruction to "go to the (color) (object)" with the color and object being randomly selected from ["blue", "green", "grey", "purple", "red", "yellow"] and ["ball", "box". ", "key"), respectively.
When transferring the learned weights, one key question is which part of the agent's weights should be transferred. We first trained PPO agent [30] on minigrid--otoooj-enn and then we transferred the learned weights to the PPO agent for mission instruction encoder, an image encoder, an actor network, and a critic network. The policy transfer is made easy due to the unified APIs for Minigrid and Miniworld. We tested with 12 different weight transfer options, the results are given in Table
The PPO policy consists of
To measure the effectiveness of the transfer learning, we define the transfer improvement as Transfer_Learning AUC
Transfer__Imroveeeen
(1)
Miniworld_Learningg..o
where Transfer_Learning AUC represents the area under the curve (AUC) of the reward curve for the agent that is initialized with Minigrid learned weights, and www.lwowlddeeernnng.ooooo the AUC of the reward curve for the agent with randomly initialized weights. Both the transfer learning agent and the Miniworld learning agent are trained for 200k time steps. As Table shows: (1) the transfer learning behavior weights were not frozen; (2) transferring only the critic network and mission embeddings produces better results compared with also transferring the actor network weights.
represents
improved when the critic network and mission embedding
3.2 Transfer Learning Between Different Observations Spaces for 10 Human Subjects
In this case study, we show how can use the Minigrid and Miniworld libraries collect and visualize human data. We utilized two similar environments in the Minigrid and Miniworld libraries, where there are four rooms and the goal is reach target position denoted with a green box. in the least amount of steps. In both cases, the human subject has only partial observation of the environment. We performed two sets of experiments. The first set of experiments lets the subject collect experience in the Minigrid environment for 10 episodes, then transfers to the Miniworld environment, and plays for another episodes. The second set of experiments directly asks subject to play the Miniworld environment for 10 episodes. In both the Minigrid and Miniworld environments, the action space has dimension three with the actions: turn left, turn right, and go forward. To make the human experience more similar to the RL agent, randomly assign the

---

Non-Frozen Weights
Frozen Weights
Transferred Weights
Mean (%)
STD (%)
Mean (%)
STD (%)
0.089
6.760
-3.775
8.421
A
-8.881
14.399
-4.670
14.997
3.993
3.189
2.760
2.703
-20.668
13.915
-13.199
9.634
CM
3.207
2.808
0.001
3,957
ACM
-9.494
9.217
-30.958
16.540
Table I: Transfer improvement for the 12 sets of experiments. For the transferred weights, "M": represents mission embedding weights, "A": represents actor network weights, and "C": represents critic network weights. "Frozen Weights" refers to freezing the transferred weights, while "Non- Frozen Weights" refers to not freezing the transferred weights.
three actions to the 1-9 numbers keys the keyboard. The average rewards over 10 episodes on the Miniworld environments are shown Table and a sample subject trajectory is shown in Figure Similar to what is shown in Figure 4, we empirically observe an adaption phase to the random key assignment during the first episode for every human subject.
Minigrid
Miniworld
Directly Miniworld
Subject No.
Subject No.
Mean
STD
Mean
STD
0.93
0.04
6
0.89
0.04
0.84
0.28
7
0.91
0.04
0.74
0.37
8
0.94
0.04
0.89
0.04
9
0.82
0.28
0.93
0.05
10
0.94
0.04
Table Subject mean reward for the two sets of experiments. The "Minigrid + Miniworld" refers to the first set of experiments, and "Directly Miniworld" refers to the second set of experiments.
3.3 Implementation Details
In this section, discuss how the studies were implemented using the Minigrid and Miniworld librarics. The RL agent transfer learning case study represents custom experimental setting not natively supported by Minigrid, Miniworld, and SB3. Thus, on top of the two libraries, implemented the following functionalities:
created the minigrid-gotoobj-en (26 lines for gendg))d()) and minixorld-gotoobj-eev (19 lines for (agee__wwol()))) environments;
2. augmented minivorld-gotooojjeen with mission instructions (3 lines of code similar to the _gen_mission() from Minigrid);
3. created a custom wrapper for sinigrid-gotoobj-en to process the mission instructions (10 lines of code that are highly similar the angobsWrraper in Minigrid);
4. created custom feature extractor in SB3 for dunigrid-gotoobyy (23 lines mostly copied from the SB3 NatureCNN class);
5. created and trained PPO agent
using SB3 (6 lines);

---

Figure 4: Trajectories from one human subject when testing transferring experience Minigrid environments to Miniworld. The numbers correspond to the episode number.
6. transfer the learned policy from seiigrrdddgooooboo to miniworld-gotoobj-enn (14- 51 lines depending on the settings, can be reduced using a for loop).
In total, the implementation of this highly customized case study required 101 138 lines of code, highlighting the facility of use that our design provides. For the human transfer learning case study, we utilized two existing simulation environments, namely MiniGrid-FourRooms. FourRoom
and The human action input is achieved using the already existing manual control feature. To render the human trajectories, we altered the existing manual control feature and included data recording and data plotting functionalities. Although additional effort was required, given the Pythonic design of the two libraries, only required three hours of coding. detailed list of the required implementations is provided in the supplementary materials. For research with similar purposes, we plan to open-source our case study implementations as a separate codebase.
4 Related Works
Simulation libraries have been a crucial part of DRL research since the success playing Atari games using deep RL [25]. However, most simulation environments focus fixed, single tasks, such as swinging up a pendulum or making a humanoid stand up. Extending these environments to support custom objectives (e.g. "use the key open the door and then get the goal" is difficult).
In the robot learning community, there has been an increase in the number of benchmark simulation environments that focus goal-oriented tasks, notably the pixme environments [28] and the Franka kitchen environment [10]. Despite their popularity within the robot learning community, for RL research that solely focuses on the decision-making process, these robotic simulation benchmarks might not be the best experimental platform. One key issue is that these robot simulation benchmarks often utilize physics simulators, e.g., MuJoCo [37] and IssacSim [22), which are significantly more difficult to extend than Minigrid and Miniworld.
Historically, the RL community has made ample use of GridWorld environments for their research and education (notably by Sutton and Barto [35]). Given their wide usage, there have also been simulation libraries that focus 2D GridWorld-like environments. MazeBase [34] is simulation

---

library for GridWorld-like 2D games. However, because it is written in Lua and does not support the OpenAl gym API, it difficult to integrate with existing deep learning and DRL libraries (e.g.. PyTorch [27] and SB3 [29]). Griddly [2] is another library that provides GridWorld environments with a highly optimized and flexible game engine. Although Griddly provides more functionalities than Minigrid, it comes at the cost of higher complexity, making it difficult to understand and customize. DeepMind Lab2D [4] and Melting Pot [20] are also 2D GridWorld simulation environments. However, compared to Minigrid, they mainly focus multi-agent reinforcement learning tasks and do not support human language instructions. Crafter [13] and some of the Minigrid environments (e.g.. MiniGrid-Mun Room) are both designed to evaluate the generalization, exploration, and long teeg-term reasoning ability of RL agents. However, Crafter environments are more difficult compared to Minigrid environments, For RL researchers, Minigrid environments are better suited in the initial development phase of their algorithms, while Crafter environments can be used when the algorithm is more mature.
For Miniworld, the most relevant work ViZDoom. The ViZDoom research platform [19] a set of simulation environments based the popular first-person shooter (FPS) game Doom that enables RL agents to make tactical and strategic decisions, At high level, the type of environments that can be created are comparable in Miniworld and ViZDoom. However, ViZDoom uses custom- designed language to create new scenarios, while Miniworld uses a small set of Python functions for environment creation. This makes Miniworld easier to use for the RL research community, which is more familiar with Python. Nevertheless, ViZDoom does support shooting games, depth information, and audio which are not supported by Miniworld. Another 3D simulation library with similar purpose is DeepMind Lab [3], but given that the game engine is written in C and the levels are written using Lua, there is steep learning curve for customizing it. 3D simulation environments like Habitat 3D [23, 36] and Unity-basse simulation environments [17] also enable the agent to navigate and interact with its surroundings. Compared Miniworld, they can simulate more complex dynamics and are more photo-realistic. However, for target-reaching and object-collection tasks where the goal not deployment on embodied AI systems, Miniworld provides much simpler and lightweight simulation solution that enables faster iteration and evaluation of new research ideas. Open-ended ended 3D environments like Avalon [1], MineRL [11], Malmo [16], and MijeDojo [8] also have similar capabilities. But their focus is more on evaluating the RL agent's 's ability to generalize on wide
range of tasks, while Miniworld environments mostly focus on a single task.
5 Conclusion
The Minigrid and Miniworld libraries provide modular and customizable RL. environments for goal- oriented tasks. We detailed the design philosophy behind the two libraries and provided a walkthrough of their API along with research areas that already utilize the two libraries. In our case studies, we have shown the unified API among two libraries provides an easy way to study transfer learning between different observation spaces and human decision- decision-making. In future works, we plan to further develop the libraries' capabilities for human-in-the -oho---oo decision-making
The libraries have two main limitations, first, the environment creation process priori- tizes simplicity with minimal functions, which limits the type of environments that can be created. Second, the two libraries are implemented in Python, which makes them computationally slower than environments that utilize highly-optimized game engines in C++.
Limitations:
Societal Impact: Since the librarics have idcalized system dynamics, the learned policy might not be directly applicable to real-world applications without introducing safeguard mechanisms.
Acknowledgements
Minigrid and Miniworld were originally created as part of research work done at Mila Québec AI Institute. We thank Manuel Goulão for their contribution to the documentation website.

---

References
[1] J. Albbrecht, A. J. Fetterman, B. Fogelman, E. Kitanidis, B. Wróblewski, N. Seo, M. Rosenthal,
M. Knutins, Z. Polizzi. J. Simon, and K. Qiu. Avalon: A benchmark for RL generalization using procedurally generated worlds. In NeurIPS Datasets and Benchmarks Track, 2022.
[2] C. Bamford. Griddly: A platform for Al research in games. Software Impacts, 8: 8:100066, 2021.
[3] C. Beattic, J. Z. Leibo, D. Teplyashin. T. Ward, M. Wainwright, H. Küttler, A. Lefrancq.
S. Green, V. Valdés, A. Sadik, J. Schrittwieser, K. Anderson, S. York, M. Cant, A. Cain,
A. Bolton, S. Gaffney, H. King, D. Hassabis, S. Legg, and S. Petersen. DeepMind Lab. CoRR,
abs/1612.03801, 2016.
[4] C. Beattic. T. Köppe, E. A. Dućñeez-uzzáán and J. Z. Leibo. Deepmind lab2d. CoRR, 1.0020 2020.
[5] G. Brockman, V. Cheung, L. Pettersson, Schneider, J. Schulman, Tang, and W. Zaremba.
OpenAl Gym. CoRR, (tbs/1606.0110 005400 2016.
[6] M. Chevalier-Boisverr D. Bahdanau, S. Lahlou, L. Willems, C. Saharia, T. H. Nguyen, and Bengio. Alby A platform study the sample efficiency of grounded language learning. In Proceedings of International Conference on Learning Representations, New Orleans, LA, May 2019.
[7] M. Dennis, N. Jaques, Vinitsky, A. M. Bayen, S. Russell, A. Critch, and S. Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. In Proceedings of Advances Neural Information Processing Systems 33, Virtual, December 2020.
[8] L. Fan, G. Wang, Y. Jiang, A. Mandlekar, Y. Yang, H. Zhu, A. Tang, D.-A. Huang, Y. Zhu, and A. Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In Thirly-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[9] C. Gumbsch, M. V. Butz, and G. Martius. Sparsely changing latent states for prediction and planning in partially observable domains. In Advances in Neural Information Processing Systems 34, Virtual, pages 1751 1-7531, December 2021.
[10] A. Gupta, V. Kumar, C. Lynch. S. Levine, and K. Hausman. Relay Policy Learning: Solving long-horizon tasks via imitation and reinforcement learning. In Proceedings of the Conference on Robot Learning, Virtual, pages 1025-1037, October 2020.
[11] W. H. Guss, B. Houghton, N. Topin, P. Wang, C. Codel, M. Veloso, and R. Salakhutdinov. MineRL: A large-scale dataset of Minecraft demonstrations. Eightt International Joint Conference on Artificial Intelligence, 2019.
[12] R. L. Gutierrez and M. Leonetti. Information- -etooretcc task selection for meta--einforcement learning. In Proceedings of Advances in Neural Information Processing Systems 33, Virtual, December 2020.
[13] D. Hafner. Benchmarking the spectrum of agent capabilities. In The Tenth International Conference on Learning Representations, Virtual Event, April 2022.
[14] M. Hutsebaut-Buysse, K. Mets, and S. Latré. Pre-trained word embeddings for goal-conditional transfer learning in reinforcement learning. CoRR, abs/2007.00 096.,,9999999 2020.
[15] M. Igl, K. Ciosek, Y. Li, S. Tschiatschek, C. Zhang, S. Devlin, and K. Hofmann. Generalization in reinforcement learning with selective noise injection and information bottleneck. In Pro- ceedings of Advances in Neural Information P Processing Systems 32, Vancouver, Canada, pages 13956-13968, December 2019.
[16] M. Johnson, K. Hofmann. T. Hutton, and D. Bignell. The malmo platform for artificial intelligence experimentation. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, New York, NY, USA, pages 4246-4247, July 2016.
[17] A. Juliani, V. Berges, E. Vckay, Y. Gao, H. Henry, M. Mattar, and D. Lange. Unity: A general 2018.
platform for intelligent agents. CoRR,
[18] L. P. Kaelbling, M. I.. Littman, and A. R. Cassandra. Partially observable markov decision processes for artificial intelligence. KI-95; Advances in Artificial Intelligence, pages 1-17. Springer Berlin Heidelberg, 1995.
10

---

[19] M. Kempka, M. Wydmuch, G. Runc, J. Toczek, and W. Jaskowski. ViZDoom: A Doom-based Al research platform for visual reinforcement learning. In Proceedings of IEEE Conference on Computational Intelligence and Games, Santorini, Greece, pages 1-8. IEEE, September 2016.
[20] J. Z. Leibo, E. A. Duéñezz-Gzzmae A. Vezhnevets, J. P. Agapiou, P. Sunehag, R. Koster, J. Matyas, C. Beattic, I. Mordatch, and T. Graepel. Scalable evaluation of multi-agent rein- forcement learning with melting pot. In Proceedings of the 38th International Conference on Machine Learning, Virtual Event, volume 139, pages 6187-6199, July 2021.
[21] E. Z. Liu, A. Raghunathan, Liang, and C. Finn. Decoupling exploration and exploitation for meta-reiinooreement learning without sacrifices. In Proceedings of the 38th International Conference on Machine Learning, Virtual Event, volume 139, pages 6925-6935, July 2021.
[22] V. Makoviychuk, L. Wawrzyniak, Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. Isaac Gym: High performance GPU based physics simulation for robot learning. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, Virtual, December 2021.
[23] Manolis Savva*, Abhishek Kadian Oleksandr Maksymets*, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra. Habitat: A Platform for Embodied AI Research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[24] A. N. Mavor-Parker, K. A. Young, C. Barry, and L.. D. Griffin. How to stay curious while avoiding noisy tvs using aleatoric uncertainty estimation. In Proceedings of International Conference on Machine Learning, Baltimore, MD, volume 162, pages 15220-15240, July 2022.
(25) V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through
deep reinforcement learning. Nature, 51 518(7540):529-533 2015.
[26] J. Parker-Holder, M. Jiang, M. Dennis, M. Samvelyan, J. N. Foerster, E. Grefenstette, and T. Rocktäschel. Evolving curricula with regret-based environment design. In Proceedings of International Conference on Machine Learning, Baltimore, MD, volume 162, pages 17473- 17498, July 2022.
[27] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Z. Yang, Z. Vito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Proceedings of Advances in Neural Information Processing Systems 32, Vancouver, Canada, pages 8024-8033 8035, December 2019.
[28] I. Radosavovic, T. Xiao, S. James, P. Abbeel, J. Malik, and T. Darrell. Real-world robot learning with masked visual pre-training. In Proceedings of the Conference on Robot Learning, Auckland, New Zealand, pages 416-426, 2022.
[29] A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann. baslle- Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22: 268:1-268:8, 2021.
[30] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR, fbs/1707.006347 2017.
[31] Y. Seo, L. Chen, J. Shin, H. Lee, P. Abbeel, and K. Lee. State entropy maximization with random encoders for efficient exploration. In Proceedings of the 38th International Conference on Machine Learning, Virtual Event, volume 139, pages 9443-9454, July 2021.
[32] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser,
I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalch- brenner, I. Sutskever, T. P. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587): 484-489, 2016.
[33] D. Silver, T. Hubert, J. Schritwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. P. Lillicrap, K. Simonyan, and D. Hassabis. Mastering chess and shogi by self-play with general reinforcement learning algorithm. CoRR, abs/1712.01815, 2017.
11

---

[34] S. Sukhbaatar, A. Szlam, G. Synnaeve, S. Chintala, and R. Fergus. Mazebase: A sandbox for learning from games. CoRR, ab5/155 2015.
[35] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge,
MA, 1998.
[36] A. Szot, A. Clegg. E. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Maestre, M. Mukadam, D. Chaplot, O. Maksymets, A. Gokaslan, V. Vondrus, S. Dharur, F. Meier, W. Galuba, A. Chang,
Z. Kira, V. Koltun, J. Malik, M. Savva, and D. Batra. Habitat 2.0: Training home assistants to rearrange their habitat. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[37] E. Todorov, Erez, and Y. Tassa MuJoCo: A physies engine for model-based control. In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, Vilamoura, Portugal, pages 5026-5033, October 2012.
[38] S. Tunyasuvunakool, Muldal, Y. Doron, Liu, Bohez, J. Merel, T. Erez, T. Lillicrap. N. Heess, and Y. Tassa. dm_control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020.
[39] A. Wachi, Y. Wei, and Y. Safe policy optimization with local generalized linear function approximations. In Proceedings of Advances in Neural Information Processing Systems 34, Virtual, pages 20759-20771, December 2021.
[40] T. Zhang, H. Xu, X. Wang, Y. Wu, K. Keutzer, J. E. Gonzalez, and Y. Tian, Noveld: A simple yet effective exploration criterion. In Proceedings of Advances in Neural Information Processing Systems 34, Virtual, pages 25217-2551
December 2021.
[41] M. Zhao, Z. Liu, S. Luan, S. Zhang, D. Precup, and Y. Bengio. A consciousness-insee planning agent for model-based reinforcement learning. Advances Neural Information Processing Systems 34, Virtual, pages 1569-1581, December 2021.
[42] W. Zhou and W. Li. A hierarchical bayesian approach to inverse reinforcement learning with symbolic reward machines. In Proceedings of International Conference on Machine Learning, Baltimore, MD, USA, volume 162, pages 17159-2777,, July 2022.
12


Reviews:
[{"review_id": "glcD657fOW", "reviewer": "Reviewer_RPEh", "summary": "", "questions": "", "limitations": "I don't see any negative social impact. Only limitations are related to the basic nature of the environments: simplified world simulation.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "* Easy to use and update/reuse environments based on your research needs.\n* MiniGrid's simplicity and interpretability make it a great playground for benchmarking RL algorithms and understanding their behavior. \n* MiniWorld's realistic and visually immersive environments provide a more challenging and dynamic space for training agents in complex, real-world scenarios. \n* Minimalistic design approach is easy to understand and start using for everyone.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "mWUAsKCHuN", "reviewer": "Reviewer_UmzS", "summary": "", "questions": "", "limitations": "The authors have discussed the limitations of this work in their paper. See the \"Opportunities For Improvement\" for my suggestions.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The proposed libraries offer several key advantages:\n\n- Their minimalistic design paradigm makes these libraries user-friendly, streamlining both installation and customization.\n- They accommodate a broad range of research topics, including but not limited to, Curriculum Learning, Exploration, Meta Learning, and Transfer Learning.\n- The unified API between Minigrid and Miniworld enables transfer learning (for both RL agents and humans) between the different observation spaces. \n- These libraries are supported by comprehensive documentation and well-maintained open-source code.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "MfzzsZVhbL", "reviewer": "Reviewer_2jWC", "summary": "", "questions": "", "limitations": "The authors state clearly the limitation of the tool. It is more educational tool  than scientific benchmark. ", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The main strength of the tool lies in its simplicity. It adopts very few libraries. The user can train the tasks without any deep knowledge about RL. The ability to create new environments from the code is also interesting. I appreciate the formalisation of the relation among reward observation, state space and action space. The tool is very helpful for beginners in this area as everything runs out of the box.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "sPc5fWyFwS", "reviewer": "Reviewer_FdB5", "summary": "", "questions": "", "limitations": "1. It's not clear what you're trying to show by sharing the results of a transfer learning experiment with human subjects, and I don't think table 1 and 2 are meaningfully comparable.\n2. In this environment, it is necessary to show that customization will be useful in the future by providing suggestions for tasks or problems that do not exist or are rare.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. It's a learning environment that's already been utilized by many papers and has earned a number of Github Stars. I think this is because the environment is easy to install, the coding can be simplified, and it is highly customizable.\n2. This paper provides experimental evidence of the impact of each module through transfer learning experiments between different environments.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "yyNwUws7AA", "reviewer": "Reviewer_Semn", "summary": "", "questions": "", "limitations": "See above.", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The paper is well presented and easy to follow.\n- Code is open-sourced. Documents are comprehensive.\n- Two presented libraries are well-adopted by the community.\n- The philosophy of minimalistic design makes the proposed libraries easy to extend, as supported by the case study.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: MiniGrid & MiniWorld are environments released in 2018 that have been influential in RL research, with GitHubs having garnered 500+ citations in that time.  This paper is the much-belated first submission of these works to a conference, this time with their new maintainers, and with a unified API for the two environments and some experiments demonstrating domain transfer between the two.   

**Perceived Strengths:**
* The simplicity and interpretability of these environments were highlighted by reviewers, especially for their use in benchmarking & designing new algorithms. 
* Similarly ease of installation, comprehensive documentation and customizability of the environments were highlighted as positives by several reviewers.
* The extensive record of impact that these environments have had speaks for itself, especially in the field of curriculum design and others.

**Perceived Weaknesses:**
* One reviewer suggested this environment lacks features or complexity for research. Had this been an entirely new environment this might have been true, however other reviewerts argued that the simplicity was (and still is) a valuable feature of these environments.
* One reviewer questioned what improvements have been made to warrant a separate publication, while another suggested that they would prefer it if the work was more 'forward looking' highlighting new challenges. Authors have responded that this work has a unified API, that might be useful for transfer learning, and that this paper contains unmentioned details from the Github.

**AC View:**

This is an unusual paper to AC.  The core of this work in this submission was completed 5 years ago, and has already had a substantial impact on the field.  However, since it has never been officially published it is now presented here, with some modest adjustments, for acceptance in 2023. In these reviews we see that 'old' core of the work here is strongly praised by reviewers, while the response to 'new' experiments on transfer and unified API is somewhat cooler and more mixed.  While I am curious as to what motivated the submission of this widely-known work half a decade on, I must agree with the majority of reviewers in recognising the impact this environment has had on the field in advancing AI, even today.
Decision: Accept (Poster)