Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper presents the NetHack Learning Dataset (NLD), which has 3 parts: i. 1.5 million human trajectories recorded from the NAO public NetHack server; ii. 100,000 trajectories from the symbolic bot winner of the NetHack challenge 2021; iii. code for users to apply these trajectories in a compressed format. To demonstrate the utility of NLD, the authors train and compare several algorithms spanning online RL, offline RL, imitation learning, and learning from observations only. 

strengths: 1. The paper presents a large-scale dataset for a fast and challenging RL environment (NetHack). The authors also provide high-performance code to load this dataset and train agents, which makes larger-scale experiments more accessible to those with smaller compute budgets. 


2. The dataset and code is well-documented. 

3. Baseline results cover a range of different approaches to train agents (online RL, offline RL, learning from observations only, combinations of offline and online learning). 

weaknesses: 1. Appendix G.6 mentions a SQL interface to consider subsets of the datasets, according to the metadata. Based on Table 2, the performance of learned policies is quite below the dataset average. 
I think the paper would benefit from additional investigation and discussion on why agents trained on NLD-AA (more narrow data distribution generated by symbolic bot, cleaner demonstrations) outperform those trained on NLD-NAO (human demonstrations). 
For instance, cutting NLD-NAO trajectories short so the length of the trajectories are similar to NLD-NAO to exclude the later-stage states of NetHack. From Figure 1, it seems that NLD-NAO trajectories may be several times longer than NLD-AA. 

2. Only 5 seeds are reported. In Figure 2, it seems that APPO methods have very high variance (blue and orange lines), and would likely benefit from running more seeds. The discussion would also benefit from exploring why APPO has higher variance b/w 0 and 600M steps, after which the variance suddenly drops.

correctness: To my knowledge, yes.

clarity: Yes, the paper is well-written and clear. I have a few minor comments in Additional Feedback which can be easily corrected.

relation to prior work: Yes, I think discussion of prior work is adequate. The paper compares NLD to related large-scale datasets for Starcraft, Dota, and MineRL, which require significantly more compute to use.

additional feedback: Minor comments:

L52-57: There is too much text formatting (bold / italics) that makes the paragraph difficult to read.

Figure 1 caption: Centered text alignment looks strange. 

Table 2: What is the difference b/w “-” and “(see col 1)”? Please clarify in the caption.

Figure 2: What does “- @” mean?

The paper uses both “BCO” and “BC(O)” to refer to Behavioral Cloning from Observations?
review_title: Official Review of Paper187 by Reviewer kge3

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper introduces a dataset called NLD, to be useful for imitation learning and offline RL study for Nethack Learning Environment (NLE) [Kuttler+, NeurIPS-20]. NLE (and potentially for a controlled version of NLE, MiniHack [Samvelyan+, NeurIPS-21]) is an OpenAI gym environment from the same author group, with a popular Rogue-like (or dungeon explorer-type) game, NetHack.

NLD-NAO collects 1.5M human game plays in a state (only) trajectories form with game play metadata. NLD-AA collects 100K plays by the winning bot (AutoAscend) from a NeurIPS-21 competition, in the state-action-score trajectory form.  NLE itself is updated (v0.9.0) to be compatible with NLD.

Experiments using the dataset is included.  Related dataset for similar benchmarks (StarCraft, Dota, and MineRL) is discussed.
strengths: NLD-NAO collects 1.5M human game plays in a state (only) trajectories form with game play metadata. NLD-AA collects 100K plays by the winning bot (AutoAscend) from a NeurIPS-21 competition, in the state-action-score trajectory form.  NLE itself seems to be updated (v0.9.0) to be compatible with NLD, containing TtyrecDataset in its python modules (nle.dataset).
weaknesses: It is essentially valuable that the work includes experiments for online RL, offline RL, imitation learning, and learning from demonstrations, which demonstrate the example usage and usefulness of the dataset.  Unfortunately, the process for reproducing experimental results shown in the paper is not clear from the main paper, supplemental material, or github (https://github.com/dungeonsdatasubmission/dungeonsdata-neurips2022/tree/a3f01c425f5d75ce9174b76105ac34a377c2df30/experiment_code).
correctness: TtyrecDataset is released as a part of the updated NLE (v0.9.0), not the dataset repository.  It is totally ok as the work but please make it clear and claim what is NLE update and how it is available as well as the dataset release.
clarity: It is well written to cover the details well in general.  Proof-reading is must, though.
relation to prior work: Comparing to related dataset for relevant benchmarks (or gym) of StarCraft, Dota, and MineRL is good and sufficient to this level, the reviewer believes.  Starcraft ii: A new challenge for reinforcement learning. ArXiv, abs/1708.04782, 2017.  Dota 2 with large scale deep reinforcement learning. ArXiv, abs/1912.06680, 2019. Minerl: A large-scale dataset of minecraft demonstrations. ArXiv, abs/1907.13440, 2019. 
additional feedback: Minor errors here and there.

For example,
Appendix p.2 564   before the an => before an
565 are may => may

Please go through proof-reading services.
review_title: Human and a champion bot play trajectories for Nethack Learning Environment.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces the Nethack Learning Dataset (NLD), which is a large dataset of demonstrations from the NetHack game. It has 2 partitions: NLD-NAO that contains 10 billion state-only trajectories and metadata from 1.5M human games, scraped from online web servers; and NLD-AA, a collection of 3 billion trajectories with complete state, acton, and scores, generated by a winning bot on the NetHack challenge. Python scripts are provided to load these datasets efficiently. Experimental results are demonstrated on the dataset with RL methods like APPO and DQN, and imitation learning methods like BC and BCO (clone from observations). 
strengths: * The paper is well-written and easy to follow.
* The human dataset is high quality, as it is scraped from actual game plays on the web servers. The scale is huge - 1.5M human trajectories with almost 10B transitions. This is much more than most other video game benchmark data, and is provided under the open-source GPL license.
* The NLD-AA dataset contains trajectories generated by a winning bot on NetHack. While not as good quality as human dataset, it has groundtruth actions to train inverse dynamics model and do direct behavior cloning. It is complementary to NLD-NAO and I am glad that the authors included both. 
* The experiments use off-the-shelf methods like A-PPO, DQN, CQL, etc. No novel algorithm is proposed, but these methods are standard enough to provide good baseline results for the dataset. 
* "Dungeon and Data" - great name! 
weaknesses: My main concern is about the benchmark's contribution to advances in the broader policy learning community. NetHack is a very niche domain. It is an ASCII-based game that does not make much sense to untrained eyes, and does not have any meaningful high-dimensional observation like 3D perception or other sensing modalities. In addition, the action space and world transition dynamics are quite simplistic. I am not convinced that the potential future algorithms developed on NetHack or NLD will be generally applicable to other embodied agent domains. As the authors pointed out, even a purely hard-coded agent called "AutoAscend" is able to achieve nontrivial performance on the benchmark. AutoAscend bot actually contributes the NLD-AA dataset with full state-action-score trajectories. 

I am not claiming that mastering NetHack is easy. But in contrast, it is extremely difficult to hard-code a robot agent in Habitat [1] or AI2Thor [2] from pixels alone even for the simplest tasks. While the authors discussed connection to robotics in L337-339, I am still doubtful of NetHack as an effective testbed for general-purpose embodied agent algorithms.

That being said, I'm still leaning towards acceptance for the large-scale dataset introduced in this paper, considering that much simpler Gridworld environments without datasets have been accepted at top conferences.

* [1] Habitat: A Platform for Embodied AI Research. Savva et al. 2019.
* [2] AI2-THOR: An Interactive 3D Environment for Visual AI. Kolve et al. 2019. 
correctness: As far as I am concerned, it is constructed correctly and contains enough details to reproduce. The experiments use standard methods like DQN, A-PPO, CQL, and BCO, and have sound details. 
clarity: The paper is clear to read and easy to follow. The details of the dataset and experiments are thoroughly explained. 
relation to prior work: The paper discusses prior works in depth on offline RL benchmarks and large-scale human demonstration datasets. In my opinion, it has discussed the differences in a satisfactory manner. 
additional feedback: Fig. 1 is a bit hard to read and could be improved. Maybe something like a violin plot can be better: https://en.wikipedia.org/wiki/Violin_plot
review_title: A large-scale dataset on a niche problem

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper introduces NLD, a large-scale dataset of demonstrations from NetHack. The dataset enables research in multiple areas, such as imitation learning and learning from both offline data and online interactions. Empirical results indicate that significant research advances are needed to leverage large-scale datasets to solve challenging decision-making problems fully.
strengths: 1.The dataset is combined with NLE, which provides potentially large insights for the study conducted in MDP problem. The proposed dataset enables computationally-accessible research in multiple areas including imitation learning, offline RL, learning from sequences of only observations, as well as combining learning from offline data with learning from online interactions.

2.The dataset has many properties of real-world domains such as partial observability, stochastic dynamics, sparse reward, long trajectories, rich environment, diverse behaviors, and a procedurally generated environment. Such properties allow the dataset to provide a more realistic evaluation environment, thereby making the evaluation of RL algorithms more reliable. 

3.NLD can enable agents learn from demonstrations containing only observations, learning from both static data and environment interaction. 

4.The proposed NLD is complete, which is decomposed of three components: NLD-NAO, NLD-AA, and TtyrecDataset. The significance of each component for dealing with decision-making problems is clearly explained in the paper. Also, the details of the released data, including the used raw features, file format, and scalability are described clearly. 

5.The dataset have been opensourced, and there is a detailed introduction for usage in the corresponding repository, thus facilitating researchers to quickly get started with the development and research of NLD.

6.NLD strikes a better balance between scale (i.e. a large number of diverse human demonstrations on a complex task) and efficiency (i.e. cheap to use and fast to run). 

7.Experimental results indicate that NLD poses a substantial challenge to state-of-the-art methods, as the NLE environment is highly stochastic, and partially observed.
weaknesses: 1.The authors should explain the relations between symbolic and RL in details, corresponding to the sentence in Section 2 "Symbolic bots decisively outperformed deep RL methods, with the best performing symbolic bots surpassing state-of-the-art deep RL methods by a factor of 5."

2.The paper should provide a quantitative metric to quantify data scale and efficiency of implementation in reinforcement learning between different datasets. Such metric could further demonstrate the significance of the NetHack dataset. Also, the authors need conduct experiments to compare different decision-making datasets in the area of randomness, magnitude of actions and state spaces, and partial observability. In other words, the paper needs to quantitatively emphasize the necessity and pioneering of NLD for existing research.

3.Section 5 does not fully explain the research significance of this dataset by only mentioning that there is room for improvement in symbolic, i.e., NAO-AA.
correctness: Yes, the dataset is constructed in a sound way
clarity: Yes, the paper is well written
relation to prior work: Yes, related work section clearly indicates the difference from prior work.

additional feedback: No.
review_title: A Large-Scale Dataset of Demonstrations from NetHack

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents NLD (NLD-NAO and NLD-AA), a large-scale dataset of NLE environment. NLD is easy to use (cheap but large-scale, pipeline code ready) and is hard for offline RL community. Several promising directions can utilize NLD, including learning from only observations.
strengths: 1. The author provides detailed descriptions and analysis of NLD-AA and NLD-NAO, including the performance distribution and other attributes (format, metadata, ...).
2. The author effectively shows the computation cost of this dataset backed with solid numbers (38TB -> 229GB), which is impressive!
3. The author illustrates the difficulty of solving this dataset and posts several promising research directions.
weaknesses: I did not personally find significant weaknesses.
correctness: All claims are addressed in this paper.
clarity: This paper is well written.
relation to prior work: Yes.
additional feedback: Small typo in appendix:

L540: The branch currently ~currently~ holds ...

A question to author:

Can we use two adjacent states in NLD-NAO to infer the action ((s, s') -> a) by heuristic methods?
review_title: NLD review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper proposes a dataset of trajectories gathered from the game of NetHack. It has human-generated trajectories (NLD-NAO), as well as ones generated by a symbolic agent that won the NetHack challenge at NeurIPS 2021 (NLD-AA). The paper describes the dataset, its analysis in terms of the game coverage, as well as experimental results for popular RL methods. These include online RL, as well as offline RL and learning from demonstrations using the dataset NLD-AA. The results show the large gap between the current RL methods and the symbolic agent, not to mention the human players.


Minor: You refer to Table 4 multiple times, was that supposed to be Table 2? 
strengths: The dataset can be useful for research in offline RL, but also possibly of interest to the broader RL community. The efficient implementation is definitely a plus, given the computational hunger of current RL methods.
weaknesses: As the paper indicates, the symbolic method used to gather the data can be characterized as slightly above "Beginner" level. This data is static, in the sense that it is gathered once and is not updated, with larger parts of the state space explored. Thus, at some point, the dataset will become obsolete. Even if that day may be not too close, it will come. Therefore, it would be beneficial to have a sort of symbolic agent that can search the state space and update the dataset with additional trajectories. Obviously, no significant portion of the entire state space can be kept offline, but maybe a symbolic agent would be able to generate some portions on the fly. 
correctness: Not relevant.

clarity: The paper is very readable. No clarity issues.

relation to prior work: Related work section clearly indicates the difference from prior work.

additional feedback: Questions to the authors:
1. What's the size of the state space of NLE? 
2. What is the number of unique states along the trajectories in your database?
3. Is there a high-level symbolic data used by the symbolic agent that can be gathered from the dataset? Factored state representation? High-level actions and their models (e.g., PDDL models)?
review_title: Potentially useful dataset

Meta-review: metareview: 
This paper presents the NetHack Learning Dataset (NLD), which has 3 parts: i. 1.5 million human trajectories recorded from the NAO public NetHack server; ii. 100,000 trajectories from the symbolic bot winner of the NetHack challenge 2021; iii. code for users to apply these trajectories in a compressed format. To demonstrate the utility of NLD, the authors train and compare several algorithms spanning online RL, offline RL, imitation learning, and learning from observations only.

weakness:
- A niche domain: "NetHack is a very niche domain. It is an ASCII-based game that does not make much sense to untrained eyes, and does not have any meaningful high-dimensional observation like 3D perception or other sensing modalities"
- quantitative metric to quantify data scale and efficiency of implementation
- This data is static, in the sense that it is gathered once and is not updated, with larger parts of the state space explored.

Some of these points were addressed in the rebuttal, while the challenge of a static dataset is deferred to future work. 
Broadly there is agreement among the reviewers, and the ACs that this is a useful benchmark for the community. 

AC would request the authors to carefully integrate all the feedback in the updated manuscript as well as any leftover comments added as clarifications in the appendix. 


confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept