Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents a new benchmark on game-based RL environments that can be procedurally generated easily by users. The authors release a variety of tasks in the benchmark based on NetHack, making it possible to test various aspects of RL algorithms, such as navigation, skill acquisition and unsupervised environment design. The authors evaluated several baselines such as A2C, PPO, DQN, IMPALA and RND, on the proposed benchmark and show that there's room for improvement in future work.
strengths: The paper presents an easy-to-use RL benchmark that uses the popular OpenAI Gym interface. The environment can be created easily via either writing a description file or programmatically using Python. The proposed benchmark MiniHack also enables creation of many complex environments that can test the limit of RL algorithms in exploration, generalization and skill discovery. I think such a benchmark is of reasonable amount of significance to the RL community and also make RL environments more accessible to the community.
weaknesses: The proposed environment is still based on 2D games, which is less realistic and unclear if it would be of great impact on real-world RL applications. I'm also not sure if MiniHack would provide more interesting problems to study compared to existing benchmarks such as Atari. In general, it is unclear to me if the RL community requires more game-based benchmarks to test RL algorithms. I feel like testing RL algorithms in real-world situations are more appealing and important.

Furthermore, I think the evaluations of existing RL algorithms on MiniHack are not thorough. The authors should also evaluate more algorithms such as TD3, SAC, etc.
correctness: Yes
clarity: Yes
relation to prior work: Yes
additional feedback: see above.
review_title: Official Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper presents MiniHack, a package for easily designing new RL environments to study different challenges in RL. MiniHack is built upon their previous environment NetHack. The authors argue that the full game of NetHack is too challenging, and thus it is not suitable for answering specific research questions in isolation. Motivated by this, MiniHack allows creating different environments by modifying des-file, which is a domain-specific language to describe the environment. Then the authors create various environments as well as porting some exiting environments to MiniHack. Finally, the authors present benchmark results with IMPALA and RND, as well as some algorithms in RLlib.
strengths: 1.	MiniHack introduces many new environments and ways to create new environments. The description file in MiniHack seems very general and could be used to create various environments. I believe MiniGrid could help facilitate future RL research and would be of interest to many RL researchers.
2.	The interfaces seem to be easy to use. The code structure and the README are easy to follow. The authors interface MiniGrid with RLlib and Torchbeast, which could help users quickly run experiments.
weaknesses: 1.	MiniHack seems to be a very incremental enhancement to NetHack. Specifically, the codebase seems to generally follow NetHack. The only difference of this paper is to introduce des-file to customize the environment. It is unclear to me why this new feature cannot be directly integrated into the NetHack environment. While I like the motivation and the idea of MiniHack, this new feature seems to be not significant enough as a separate contribution.
2.	The benchmark experiments seem weak. Recently, many new algorithms are proposed to address procedurally-generated environments. Since MiniHack provides procedurally-generated environments, in addition to IMPALA and RND, I would expect some results using algorithms designed for procedurally-generated environments. The experiments could also be enhanced by trying different configurations, such as neural architectures.
correctness: No concern.
clarity: The paper is clear with detailed descriptions of the motivation, interfaces, etc.
relation to prior work: The authors have discussed various previous environments as well as sandboxes. While MiniHack is the not first sandbox, it is fast and can provide rich environmental dynamics.
additional feedback: See strengths and weaknesses.
review_title: Interesting work, but seems to be an incremental enhancement to NetHack environment

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper presents MiniHack, a framework for creating RL environments that leverages the tools and procedures of the NetHack game. MiniHack is motivated by the need of creating a continuum in the dataset space, which would allow to evaluate algorithms in scenarios of gradually increasing difficulty. Popular RL environments have been ported to MiniHack. The framework is tested in navigation and skill acquisition tasks using reasonable recent baselines, where it is shown how gradual changes in some data features change the difficulty of the scenarios, sometimes challenging existing baselines.
strengths: + The dataset is interesting and relevant. In my opinion, it fills an interesting gap in data generation for RL, showing a good balance between the capability of creating complex scenarios for several tasks, the effort needed to do it, and the amount of configurable features.
+ The documentation contains ample technical details and examples of use, which potentially will allow an easy use in the initial stages.
+ The authors made an effort to port existing scenarios, which will facilitate the adoption of the tool.
weaknesses: - The existing environments that have been ported are not identical. This is a limitation, as new results using MiniHack could not be comparable to existing ones. The authors should be more specific on this and state which tasks would be comparable with existing results and which ones not. Would there be any chance that the ported environments look identical?

- I think the evaluation methodology section, which is in the supplementary material right now, is quite relevant for the framework use. Wouldn't it be better to move it to the main paper?
correctness: The paper is correct as far as I checked. The framework and the explanations are sufficiently detailed. The baselines and metrics shown in the tests of the framework are correct.
clarity: The paper is well written and clear. As a personal opinion, the general tone is a bit enthusiastic (e.g., powerful is repeated in excess), I prefer manuscripts that are written in a more neutral tone.
relation to prior work: The previous work is nicely covered up to my knowledge.
additional feedback: In general, I have a positive view of this dataset. I would encourage the authors to work on the few weaknesses I found. Most importantly, if I am not wrong, there is not maintenance plan for MiniHack, and this is a critical aspect. The authors should design one carefully. The second critical aspect to ensure a wide adoption of the framework is its compatibility with previous scenarios. The authors should be more specific about the compatibility of the existing ones, and they should make an effort to make existing scenarios identical (if possible).
review_title: Interesting and useful dataset

Meta-review: The proposed environment to design RL problems shows to be very flexible useful. It can be a very valuable tool for the community.
Decision: Accept