SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Katakomba:
Tools and Benchmarks for Data-Driven NetHack
Vladislav Kurenkov*
Alexander Nikulin*
NetHack is known as the frontier of reinforcement learning research where learning-based methods still need to catch up to rule-based solutions. One of the promising directions for a breakthrough is using pre-collected datasets similar to recent developments in robotics, recommender systems, and more under the umbrella of offline reinforcement learning (ORL). Recently, a large-scale NetHack dataset was released; while it was a necessary step forward, it has yet to gain wide adoption in the ORL community. In this work, we argue that there are three major obstacles for adoption: resource-wise, implementation-wise, and benchmark-wise. To address them, we develop an open-source library² that provides workflow fundamentals familiar to the ORL community: pre-defined D4RL-style tasks, uncluttered baseline implementations, and reliable evaluation tools with accompanying config and logsynced to the cloud.
Reinforcement learning (Arulkumaran et al., 2017; Levine et al., 2020) led to remarkable progress in the decision-making problems in recent years: robotics (Smith et al., 2022; Kumar et al., 2021), recommender systems (Chen et al., 2022a), traffic control (Chen et al., 2020), energy management (Yu et al., 2019), combinatorial optimization (Mazyavkina et al., 2021), and videogames (Baker et al., 2022; Hafner et al., 2023). Many of those are considered solved or close to solved problems, and the arguably next untapped frontier for RL algorithms is the NetHack (Küttler et al., 2020).
NetHack<sup>3</sup> is considered one of the most challenging games for humans, even more to learning agents. This game requires strong generalization capabilities, as the levels are procedurally generated with each reset, and extraordinary memory capacity, as the episodes may last for 100k steps requiring to remember what happened a thousand steps before to act optimally. Moreover, the high level of uncertainty and its dependence on the initial character configuration further hardens the problem. Indeed, at the moment, the best agent, AutoAscend (Hambro et al., 2022a), is essentially rule-based, and yet reinforcement learning agents cannot even come close in terms of the game score or levels reached. While there were various attempts to advance online reinforcement learning agents, this task seems out of reach for them.
* Contributed equally.
Source code: https://github.com/corl-team/katakomba
For a brief introduction to the game, we recommend excellent game wiki, as well as the original publication by Kuttler et al. (2020), which introduced the NetHack environment.
Recently, a promising direction was outlined – re-using large-scale datasets of human/bot demonstrations to bootstrap RL agents, either in the pre-training phase or in the fine-tuning stage (Hambro et al., 2022b; Kumar et al., 2022b). This research direction holds great promise to advance the community's progress in solving the NetHack and attacking the weak sides of the RL agents. While the proposed tools and datasets are necessary, their implementations, support, and ease of adoption still need to be improved. What is essential, the use of NetHack promises the democratization of research efforts, providing way more efficient tools compared to both StarCraft (Vinyals et al., 2019) and Minecraft (Guss et al., 2019) learning environments. However, while the open-sourced datasets are a significant first step, the provided tools are very far from being readily adopted by other researchers (Section 2).
In this work, we aim to address this gap and provide a set of instruments that are conducive to future research endeavors, smooth the learning curve, and define a precise set of tasks to be evaluated against when training offline RL agents on the NetHack dataset, where contributions are as follows:
- D4RL-Style Benchmark A set of small-scale datasets based on the large-scale data source (Hambro et al., 2022b) for faster experimentation, including several data loaders for different speed-memory tradeoffs with a familiar interface to the ORL community alike Fu et al. (2020).
- Clean Recurrent Offline RL Baselines Straightforward implementations of popular offline RL baselines augmented with recurrence: BC (Michie et al., 1990), CQL (Kumar et al., 2020), AWAC (Nair et al., 2020), REM (Agarwal et al., 2020), IQL (Kostrikov et al., 2021). Each implementation is separated into single files akin to Huang et al. (2021); Tarasov et al. (2022) to smooth the learning curve.
- Evaluation Guideline We use reliable evaluation tools for comparing offline RL agents in the natural presence of high variability. For this, we employ recent RLiable tools (Agarwal et al., 2021) that avoid comparing against mean values (which are especially deceptive in the case of the NetHack environment). In order to facilitate the comparisons, we also release the raw results we obtained in our experiments so that future work may re-use them for a fair comparison.
- Open-Sourced Training Logs and Configs Moreover, we keep a public track of our experiments: including config, implementations, and learning dynamics using Weights&Biases (Biewald, 2020), so that it is transparent for inspection and reproduction.
We believe our work is a logical next step for advancing reinforcement learning in the NetHack environment, providing a conducive set of tools, algorithms, and utilities for experimenting with offline RL algorithms. Moreover, as we plan to support the benchmark, we intend to extend it with offline-to-online tools and benchmarks. By no means this work serves as a converged fixed point. We view this as a starting step in building reliable tools for solving NetHack with prior data and commit to further support and development.
2 What Hinders the Adoption of Data-Driven NetHack?
In this section, we describe the main obstacles and motivate our work. We outline our experience with the recently released dataset (Hambro et al., 2022b), accompanying tools and divide the main issues into three categories: implementational, resources, and benchmark-wise. These are needed to illustrate that the initial release of the dataset is a significant step. However, it is only the first step, and further development is needed, which we will address in further sections.
Implementation-wise First, when trying to reproduce the experiments, we found that installing the released tools is error-prone: where the provided Dockerfile was not operable and required fixing the moolib library with a CMake modification, which is probably undesirable for broader adoption of practitioners. Second, when we attempted to reproduce the scores reported in Hambro et al. (2022b), the only configuration file provided was for the IQL algorithm<sup>4</sup> and its modification with hyperparameters for other algorithms from the paper required additional effort to assign proper
4 Consider line 38 for an example of hyperparameters ambiguity: https://github.com/dungeondatasubmission/dungeondata-neurips2022/blob/main/experiment_code/hackrl/dqn_ttyrec_config.yaml. When changing the crop dimensions, the implementation crashes.
values. Consequently, the reproduction did not result near what was reported in the paper (note that a similar pattern was observed in Piterbarg et al. (2023)). While we do not claim that the reproduction is impossible, we could not find the proper fix in a reasonable amount of time and consider this as another obstacle in adoption – practitioners should either have access to the original training logs or be able to replicate the results reliably.
Beyond the replication frictions, another issue lies in the design of implementations. Offline and offline-to-online settings are interleaved within just one file (1500 lines of code), where the algorithms are realized using the framework for distributed reinforcement learning not typical for the ORL community (Mella et al., 2022). Consequently, understanding the underlying logic is hard, and some performance-related implementation tricks can be hard to spot. For example, q-learning-based algorithms do not use the last element of the sequence, i.e., every sequence_length game tuple is not used for training (but only for bootstrapping). While all of these may seem like minor issues, their sum brings a significant burden upon practitioners interested in the data-driven NetHack research.
Resource-wise The great addition to the released datasets was an accompanying data loader that operates over compressed game episodes (we refer to it as TTYRec throughout the text). While the Hambro et al. (2022b) demonstrated that the proposed loader could operate at approximately 5 hours per epoch, we observed a slower rate when adapting it for recurrent algorithms. The main issue lies in the underlying format where the data is fetched the following way  $-(s_{t},a_{t},r_{t - 1})^{5}$ . While this may seem non-significant, it results in the need to fix the tuples sequence to obtain a suitable format  $(s_t,a_t,s_{t + 1},r_t)$ . To better demonstrate the impact of this problem, consider Table 1, where we test the original data loader and the one with the fix, observing a significant increase in the loading time as both batch size and sequence length increase. While this can be avoided with the original loader by simply discarding each sequence_length element (as done in the original work), the problem persists as the original implementations do not work without reward shaping with potentials requiring a scan over the sequence, not to mention that such data rejection is not justified beyond performance reasons. As an additional but important issue to resource-constrained practitioners, one must first download the entire 90GB AA dataset, even if they aim to work on a subset of data.
Table 1: Loading times for different storage formats averaged over 500 iterations. The right part of the table, with HDF5 columns, depicts the loading time for datasets repacked within Katakomba. Note that the transitions are in the proper format for them by design.
Variants	TTYRec	TTYRec, Proper Tuples	HDF5 (Memmap)	HDF5 (RAM)	HDF5 (Compressed)
batch_size=64, seq_len=16	15ms	17ms	2ms	1ms	516ms
batch_size=256, seq_len=32	74ms	132ms	12ms	9ms	2.39s
batch_size=256, seq_len=64	255ms	372ms	18ms	14ms	2.42s
Benchmark-wise Arguably, one of the driving forces in Deep Learning and Reinforcement Learning, in general, is a well-defined set of tasks. To demonstrate how the proposed large-scale dataset could be utilized for tasks definition, Hambro et al. (2022b) described two settings: learning on the whole dataset for all role-race-alignment-sex combinations and learning on the subset of data for the Monk-Human-Neutral setting. While this is a good entry point and could be of great use to practitioners interested in large-scale data regimes, there was no detailed proposal on how one should further define tasks which is indeed an open question in the NetHack learning community (Küttler et al., 2020). Moreover, the original raw large-scale dataset requires practitioners to manually define SQL queries for extracting the data of interest, which is flexible but could be an overkill.
More importantly, the proposed comparison metrics were mean and median statistics of average episode returns over training seeds. One may argue that the median is a robust statistic. However, in this case, it is a median of average episode returns over training seeds and not of the whole set of evaluation episodes. It is well known in the RL community (Agarwal et al., 2021) that those are not reliable due to the highly-variable nature of RL agents. As of the NetHack, this further amplifies by the extremely-variable nature of the game itself as both demonstrated in Kuttler et al. (2020); Hambro et al. (2022b). Therefore, to reduce the noise in the progress of RL agents in this domain, one would need a different evaluation toolset that is better suited for it.
5 Notation standard for the RL community, where  $s$  is a state,  $a$  is an action, and  $r$  is a reward.
3 Katakomba
Given the issues described in the previous section, we are ready to present Katakomba – a set of tools, benchmarks, and memory-based offline RL baselines. In this section, we gradually introduce the components of the library with accompanying code examples to better demonstrate the ease of use. All the numbers and performance metrics discussed in the text were measured on the same set of hardware: 14CPU, 128GB RAM, NVMe, 1xA100; for more details, please, see Appendix D.
3.1 Benchmark
Listing 1: Usage example: Tasks are defined via the character field that is then used by the offline wrapper for dataset loading and score normalization.
from katakomba.env import NetHackChallenge   
from katakomba.env import OfflineNetHackChallengeWrapper   
# The task is specified using the character field   
env  $=$  NetHackChallenge( character  $\equiv$  "mon-hum-neu", observation_keys  $\coloneqq$  ["tty_chars","ttycolors","tty_cursor"]   
# A convenient wrapper that provides interfaces for   
# dataset loading and score normalization   
env  $=$  OfflineNetHackChallengeWrapper(env)
Decomposition In our benchmark, we re-use the dataset collected by the AutoAscend bot (Hambro et al., 2022b,a). While this bot is highly-capable, it still is considered an early-game contender because it can not descend further than two levels in half of the episodes. As the dataset becomes an early game bridgehead, we divide the tasks based on the character configurations: role, race, and alignment. In the NetHack community, these are known to be the most important (and even having a dramatic effect), requiring to utilize varying abilities of each role or race<sup>6</sup>. Overall, in opposition to merging all combinations, this decomposition allows more focus on the characters' gameplay and the size reduction one needs to download for both playing-around and committed research, as we will describe further.
Categories This results in 38 datasets in total. However, it may not be possible for researchers to examine each of them as the training times can be an obstacle. To this end, we further divide the tasks into three categories: Base, Extended, and Complete. We separate each category based on the wisdom of the NetHack community, i.e., that roles have a more substantial effect on the gameplay than race, and race has more effect than alignment. The datasets and categories are listed in Table 2. Base tasks consist of all possible roles for the human race; we choose a neutral alignment where possible; otherwise, we pick the alternative one (for humans, if there is no neutral alignment, there is only one alternative). We include all other role-race combinations for Extended tasks. For Complete, we add tasks that were not included in the Base and Extended categories. Note that these three categories combined cover all of the allowed role-race-alignment combinations.
Data Selection As demonstrated in the previous section, the original TTYRec dataset is actually slower than expected when it is used for learning agents. Moreover, one may be unable to download the 90GB dataset. Therefore, we take a different path by subsampling and repacking the original dataset task-wise, averaging 680 episodes and 1.3GB per task. The subsampling procedure is stratified by the game score (for more details, please see the script<sup>7</sup>). This allows for more versatility: one can download the needed datasets on demand as in D4RL (Fu et al., 2020); furthermore, this permits us to address the rolling issue as we repacked the transition tuples in the way suitable to typical ORL pipeline as a part of the release. To ensure the reliable accessibility of the data, we host it on the HuggingFace Hub (Engstrom et al., 2020).
$^{6}$ https://nethackwiki.com/wiki/Player  
 $^{7}$ https://github.com/corl-team/katakomba/blob/main/scripts/generate_small_dataset.py
Table 2: Katakomba tasks. We split the large-scale AutoAscend dataset into 38 problems dividing them into three categories. The splits are justified by the early-game nature of the AutoAscend data, where roles have a higher impact on the gameplay, then races, and then alignments. Note that the whole benchmark covers all possible role-race-alignment combinations.
Tasks	# Transitions	Median Turns	Median Score	Median Deathlvl	Size (GB)	Compressed Size (GB)
Base (Role-Centric)	-	-	-	-	-	-
arc-hum-neu	24527163	32858.0	4802.5	2.0	94.5	1.3
bar-hum-neu	26266771	35716.0	11964.0	4.0	101.1	1.7
cav-hum-neu	21674680	30361.0	8152.0	4.0	83.5	1.3
hea-hum-neu	14473997	18051.0	2043.0	1.0	55.7	0.8
kni-hum-law	22287283	28246.0	6305.0	3.0	85.8	1.5
mon-hum-neu	33741542	42400.0	11356.0	4.0	129.9	2.1
pri-hum-neu	18376473	26796.5	5366.5	2.0	70.8	1.1
ran-hum-neu	17625493	25354.0	6168.0	2.0	67.9	1.0
rog-hum-cha	14284927	19334.0	3005.5	1.0	55.0	0.8
sam-hum-law	22422537	32951.0	7850.0	4.0	86.3	1.3
tou-hum-neu	13376498	17955.5	2554.5	1.0	51.5	0.8
val-hum-neu	27784788	35250.0	11402.5	4.0	107.0	1.8
wiz-hum-neu	14343449	19808.5	3132.5	1.0	55.2	0.8
Extended (Race-Centric)	-	-	-	-	-	-
pri-elf-cha	18796560	26909.5	4718.5	2.0	72.4	1.1
ran-elf-cha	18238686	26607.0	7583.0	4.0	70.2	1.1
wiz-elf-cha	15277820	19512.0	2988.5	1.0	58.8	0.9
arc-dwa-law	25100788	34669.0	4026.0	1.0	96.7	1.5
cav-dwa-law	22871890	32261.0	7158.0	3.0	88.1	1.5
val-dwa-law	32787658	33973.0	8652.5	3.0	126.6	2.5
arc-gno-neu	24144048	34432.0	4077.5	1.0	93.0	1.4
cav-gno-neu	21624779	29860.0	6446.0	3.0	83.3	1.4
hea-gno-neu	14884704	18518.0	1980.5	1.0	57.3	0.9
ran-gno-neu	17571659	25970.0	5326.0	2.0	67.7	1.1
wiz-gno-neu	14193637	19206.0	2736.0	1.0	54.7	0.9
bar-orc-cha	27826356	39291.0	10499.0	4.0	107.2	1.8
ran-orc-cha	18127448	26707.0	5460.0	2.0	69.8	1.1
rog-orc-cha	16674806	22351.0	3103.0	1.0	64.2	1.0
wiz-orc-cha	15994150	22570.5	3241.5	1.0	61.6	1.0
Complete (Alignment-Centric)	-	-	-	-	-	-
arc-hum-law	23422383	31446.0	4188.0	1.0	90.2	1.3
cav-hum-law	22328494	31039.0	8174.0	4.0	86.0	1.3
mon-hum-law	30782317	39647.0	10855.0	4.0	118.5	1.9
pri-hum-law	18298816	27192.0	4833.0	1.0	70.5	1.1
val-hum-law	30171035	34570.5	9707.0	4.0	116.2	2.1
bar-hum-cha	25362111	35925.0	12574.0	5.0	97.7	1.6
mon-hum-cha	33662420	41730.5	11418.0	4.0	129.6	2.1
pri-hum-cha	18667816	28204.5	5847.0	2.0	71.9	1.1
ran-hum-cha	16999630	24698.5	6236.0	2.0	65.6	1.0
wiz-hum-cha	14635591	20257.0	3294.0	1.0	56.4	0.9
3.2 Data Loaders for Speed-Memory Tradeoff
As we outlined in Section 2, while the original large-scale dataset is well-compressed, its iteration speed integrated with proper sequential loaders is still an obstacle for fast experimentation loop. Moreover, it requires at least one full download for the entire large-scale dataset, which may not be suitable for resource-constrained scenarios. In Katakomba, we address this by providing three different loaders trading-off memory and speed allowed by the deliberate task division. In each of them, the compressed task's dataset is automatically downloaded and decompressed if one aims for speed. Moreover, we provide a Python interface for an automatic clean-up if one does not want to store the decompressed dataset beyond the code execution.
HDF5, In-Memory (Decompressed, RAM) This is the fastest format that relies on the decompression of the dataset into the RAM, which may require from 51GB to 129GB depending on the dataset listed in Table 2. This option is quite demanding regarding the memory but comes with the advantage of the fastest data access (see Table 1).
HDF5, Memory Map (Decompressed, Disk) This option is a middle-ground that allows the datasets to be efficiently utilized without requiring large RAM. When one specifies this preference, the dataset will be decompressed on one's hard drive, and the consequent reads will be conducted from it. The advantage of this approach compared to the loader from Hambro et al. (2022b) is that there is no need to decompress on the fly, and this is done precisely once at the start of the training process (taking from three to ten minutes on average).
Listing 2: Usage example: Katakomba provides different loaders that can be chosen depending on one's speed-memory tradeoff.
# Decompress the dataset into RAM  
dataset = env.get_dataset(mode="in_memory")  
# Decompression on-read  
dataset = env.get_dataset(mode="compressed")  
# The original dataloader introduced in Hambro et al, 2022  
dataset = env.get_dataset(scale="big")  
# Decompress the dataset on disk  
dataset = env.get_dataset(mode="memmap")  
# If you want to delete the decompressed dataset  
# This will not affect the compressed version  
dataset.close()
HDF5, In-Disk (Compressed, Disk) This mode is the most cheap one but slow. Essentially, the dataset is read on from disk and decompressed on-the-fly. We found this useful for debugging purposes where one does not need the whole training process to be run.
TTYRec, In-Disk (Compressed, Disk) In case one finds the original approach to data loading more suitable, we also provide a convenient interface that wraps the source large-scale dataset and loader. One can also set it up in more detail using keyword arguments from the original TTYRec data loader. However, this option comes with the downsides described in Section 2, i.e., slower iteration speed and the need to download the 90GB dataset at least once.
In addition to the dataset interfaces, we also provide an implementation of a replay buffer suitable for sequential offline RL algorithms for bootstrapping practitioners with ready-to-go tools.
3.3 Evaluation Methodology under High Variability
In Hambro et al. (2022b), authors used an average episode return across seeds when comparing baselines. While this is a standard practice in the RL and ORL communities, it was recently shown to be unreliable (Agarwal et al., 2021) as the algorithms are known to be highly variable in performance. This problem amplifies even more for NetHack, where the score distribution is typically quite wide for humans and bots (Küttler et al., 2020).
To address this, we argue that the evaluation toolbox from Agarwal et al. (2021) is more appropriate and suggest using it when comparing NetHack learning agents. We use these tools for two dimensions: in-game score and death level. The first dimension corresponds to what one typically optimizes with ORL agents but is considered a proxy metric (Küttler et al., 2020). While the latter lower bounds the early-game progress and is more indicative of the game completion.
Furthermore, similar to Fu et al. (2020), we also suggest reporting normalized scores to capture how far one is from the AutoAscend bot. We use mean scores per dataset as a normalization factor and rescale to [0, 100] range after normalization, similar to D4RL (Fu et al., 2020). This functionality is also provided as a part of the offline wrapper for the NetHackChallenge environment. Please refer to Appendix D for precise values.
4 Benchmarking Recurrent Offline RL Algorithms
Algorithms For our benchmark, we rely on the following set of algorithms: Behavioral Cloning (BC) (Michie et al., 1990), Implicit Q-Learning (IQL) (Kostrikov et al., 2021), Conservative Q-Learning (CQL) (Kumar et al., 2020), Randomized Ensemble Mixture (REM) (Agarwal et al., 2020), and Advantage-Weighted Actor-Critic (AWAC) (Nair et al., 2020). These are known as either the most competitive in the continuous control setting (Tarasov et al., 2022) or were shown to be competitive in the discrete control (Agarwal et al., 2020; Kumar et al., 2022a). Similar to Hambro et al. (2022a,b), we build upon Chaotic-Dwarven-GPT-5 architecture that converts the tty observation into an image
(a) Bootstrapped point estimates. It can be seen that except REM, which failed, all algorithms perform about the same across all metrics and are generally far behind the AutoAscend algorithm, which mean scores per dataset were used for normalization.
(b) Performance profiles. Notably, around  $5\%$  of episodes can surpass the normalized score of 20.
(c) Probability of improvement of BC to other algorithms.
Figure 1: Normalized performance under the Katakomba benchmark for all 38 datasets. Each algorithm was run for three seeds and evaluated over 50 episodes resulting in 5700 points for constructing these graphs. As one can see, there is not much improvement beyond naive behavioral cloning.