Katakomba: Tools and Benchmarks for Data-Driven NetHack
Authors: Vladislav Kurenkov
Alexander Nikulin
Denis Tarasov
Sergey Kolesnikov
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Offline Reinforcement Learning
NetHack

Abstract: NetHack is known as the frontier of reinforcement learning research where learning-based methods still need to catch up to rule-based solutions. One of the promising directions for a breakthrough is using pre-collected datasets similar to recent developments in robotics, recommender systems, and more under the umbrella of offline reinforcement learning (ORL). Recently, a large-scale NetHack dataset was released; while it was a necessary step forward, it has yet to gain wide adoption in the ORL community. In this work, we argue that there are three major obstacles for adoption: tool-wise, implementation-wise, and benchmark-wise. To address them, we develop an open-source library that provides workflow fundamentals familiar to the ORL community: pre-defined D4RL-style tasks, uncluttered baseline implementations, and reliable evaluation tools with accompanying configs and logs synced to the cloud.

Katakomba: Tools and Benchmarks for Data-Driven NetHack
Vladislav Kurenkov Tinkoff v. wwrrenooo@tinkoofffoooooooooooo. ai
Alexander Nikulin" Tinkoff a. p p.mikuulin@tinkoff...mmm ai
Denis Tarasov Tinkoff den. tarasssrsoovt..off.....m ai
Sergey Kolesnikov Tinkoff s. s.s. ..s.koleemikov@tinkoff
Abstract
NetHack is known as the frontier of reinforcement learning research where learning- based methods still need catch up to rule-based solutions. One of the promising directions for a breakthrough is using pre-collected datasets similar to recent developments in robotics, recommender systems, and more under the umbrella of offline reinforcement learning (ORL). Recently, large-scale NetHack dataset was released; while it was a necessary step forward, has yet to gain wide adoption the ORL community. In this work, we argue that there are three major obstacles for adoption:
resource-wise, implementation-wise wise. and benchmark-wisg them, we develop open-source library that provides workflow fundamentals familiar to the ORL community: pre-defined D4RL-style tasks, uncluttered baseline implementations, and reliable evaluation tools with accompanying configs and logs synced to the cloud.
To address
Introduction
Reinforcement learning (Arulkumaran et al.) 2017 Levine et al.
2020) led to remarkable progress
in the decision-making problems in recent years: roboties (Smith al. 2022 Kumar al. 2021).
recommender systems (Chen et al. 2022a), traffic control (Chen et al.) 2020 energy management
(Yu et al. 2019), combinatorial optimization (Mazyavkina et al. [021] I), and videogames (Baker et al.) 2022 Hafer et al., 2023). Many of those are considered solved or close to solved problems, and the
arguably next untapped frontier for RL algorithms is the NetHack (Küttler et al. 2020).
2 is considered one of the most challenging games for humans, even more to learning agents. This game requires strong generalization capabilities, as the levels are procedurally generated with each reset, and extraordinary memory capacity, as the episodes may last for 100k steps requiring to remember what happened thousand steps before act optimally. Moreover, the high level of uncertainty and its dependence on the initial character configuration further hardens the problem. Indeed, at the moment, the best agent, AutoAscend (Hambro et al.) 2022a), essentially rule-based, and yet reinforcement learning agents cannot even come close terms of the game score or levels reached. While there various attempts to advance online reinforcement learning agents, this task seems out of reach for them.
Contributed equally.
"Source code: hattps://github www.///oo-tee///eeeeeeeeeeeeeeee
FFor a brief introduction to the game, we recommend Excellentggans wiki, as well as the original publication by Küttler et al. (2020), which introduced the NetHack environment.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Recently, a promising direction was outlined - re-using large-scale datasets of human/bot demonstra- tions bootstrap RL agents, either the pre-training phase or in the fine-tuning stage (Hambro et al.) Kumar et al. 20226). This research direction holds great promise to advance the community's s progress solving the NetHack and attacking the weak sides of the RI. agents. While the proposed tools and datasets are necessary, their implementations, support, and ease of adoption still need to be improved. What is essential, the use of NetHack promises the democratization of research efforts, providing way more efficient tools compared to both StarCraft (Vinyals et al., 2019) and Minecraft (Gus et al. 2019) learning environments. However, while the open-sourced datasets are a significant first step, the provided tools very far from being readily adopted by other researchers (Section 2)
2022b
In this work, we aim to address this gap and provide set of instruments that are conducive to future research endeavors, smooth the learning curve, and define precise set of tasks to be evaluated against when training offlin RI. agents on the NetHack dataset, where contributions are as follows:
D4RL-Style Benchmark A set of small-scale datasets based on the large-scale data source (Hambro et al., 2022b) for faster experimentation, including several data loaders for different speed-memory tradeoffs with a familiar interface to the ORL community alike Fu et al. (2020).
Clean Recurrent Offline RL Baselines Straightforward implementations of popular offline RL baselines augmented with recurrence: BC (Michie et al. 1990), CQL (Kumar et al. 2020 AWAC (Nair et al. 2020), REM (Agarwal et al. 2020), IQL (Kostrikov et al. (0211) Each implementation is separated into single files akin to Huang et al. (2021); Tarasov et al.
2022 smooth the learning curve.
Evaluation Guideline We use reliable evaluation tools for comparing offline RL agents in the natural presence of high variability. For this, we employ recent Rabae tools (Agarwal jt al.) 2021) ) that avoid comparing against mean values (which are especially deceptive in the case of the NetHack environment). In order to facilitate the comparisons, we also release the raw results we obtained in our experiments that future work may re-use them for fair comparison.
Open-Sourced Training Logs and Configs Moreover, we keep public track of our experi- ments: including configs, implementations, and learning dynamics using Weights&Biases (Biewald, 2020), so that it is transparent for inspection and reproduction.
We believe our work is a logical next step for advancing reinforcement learning in the NetHack environment, providing a conducive set of tools, algorithms, and utilities for experimenting with offline RL algorithms. Moreover, as we plan support the benchmark, we intend to extend it with oflline-o-oolliee tools and benchmarks. By no means this work serves as a converged fixed point. We view this as a starting step in building reliable tools for solving NetHack with prior data and commit to further support and development.
2 What Hinders the Adoption of Data-Driven NetHack?
In this section, we describe the main obstacles and motivate our work. We outline our experience with the recently released dataset (Hambro et al.) 2022b), accompanying tools and divide the main issues into three categories: implementational, resources, and benchmark wise. These are needed to illustrate that the initial release of the dataset is significant step. However, it is only the first step, and further development is needed, which we will address in further sections.
Implementation-wian First, when trying to reproduce the experiments, we found that installing the released tools is error-prone: where the provided Dockerfile was not operable and required fixing the moolib library with a CMake modification, which is probably undesirable for broader adoption of practitioners. Second, when we attempted to reproduce the scores reported in Hambro al. (202bb), the only configuration file provided was for the IQL. algorithn and its modification with hyperparameters for other algorithms from the paper required additional effort to assign proper
*Consider line 38 for an example of hyperparameters ambiguity: Hunggoonsdatasubbisssiooo daw.aioonnnngnnaaaaaaaaa neur wauurrps222//00 httpppp22222//aaaaiiii expprimeet......m cotee//oooooeeeeeee bodde/hackr// sop__ttyrec_confae yaml When changing the crop dimensions, the implementation crashes.
https: htpp:://githus com/

---

values. Consequently, the reproduction did not result near what was reported in the paper (note that a similar pattern was observed in Piterbarg et al. (2023). ) While we do not claim that the reproduction is impossible, we could not find the proper fix in reasonable amount of time and consider this as another obstacle adoption practitioners should either have access to the original training logs or be able to replicate the results reliably.
Beyond the replication frictions, another issue lies in the design of implementations. Offline and oflline-to-onlne settings are interleaved within just one file (1500 lines of code), where the algorithms are realized using the framework for distributed reinforcement learning not typical for the ORL community (Mella et al., 2022). Consequently, understanding the underlying logic is hard, and some performance-relate implementation tricks can be hard to spot. For example, q-learning- -baed algorithms do not use the last element of the sequence, i.c., every sequence length game tuple is not used for training (but only for bootstrapping). While all of these may seem like minor issues, their sum brings a significant burden upon practitioners interested in the data-driven NetHack research.
Resource-wise The great addition to the released datasets was an accompanying data loader that operates over compressed game episodes (we refer to it as TTYRec throughout the text). While the Hambro et al. (2022b) demonstrated that the proposed loader could operate at approximately 5 hours per epoch, we observed slower rate when adapting for recurrent algorithms. The main issue lies in the underlying format where the data is fetched the following way - ($), at - While this may seem non- signifiin,,,,,,, it results in the need to fix the tuples sequence to obtain suitable format (st, az $1+1,7:) 7t). To better demonstrate the impact of this problem. consider Table D where we test the original data loader and the one with the fix, observing a significant increase in the loading time as both batch size and sequence length increase. While this can be avoided with the original loader by simply discarding each sequence _engthh element (as done in the original work), the problem persists as the original implementations do not work without reward shaping with potentials requiring a scan over the sequence, not to mention that such data rejection is not justified beyond performance reasons. As an additional but important issue to essurce-constraing download the entire 90GB AA dataset, even if they aim to work on a subset of data.
practitioners, one must first
Table 1: Loading times for different storage formats averaged over 500 iterations. The right part of the table, with HDF5 columns, depicts the loading time for datasets repacked within Katakomba. Note that the transitions are in the proper format for them by design.
Variants
TTYRec
TeYRec Proper Tuples
HDF5 (Memmap)
HDES (RAM)
HDF5 (Compressed)
batch, sine__, seq, len=16
15ms
17ms
2ms
Ims
516ms
hatch_ 25ze=256, seg len=33
74ms
132ms
12ms
9ms
2. 2.39s
batch_size=26,, len
255ms
372ms
18ms
14ms
2.42s
Benchmark-wise Arguably, one of the driving forces in Deep Learning and Reinforcement Learn- ing, in general, is a well-defined set of tasks. To demonstrate how the proposed large-scale dataset could be utilized for tasks definition, HHmmro et al. (2022b) described two settings: learning the whole dataset for all role-race- eee-racc-aligmmen--se combinations and learning on the subset of data for the Mlonk-Human-De practitioners interested in large-scale data regimes, there was no detailed proposal on how one should further define tasks which is indeed an open question in the NetHack learning community (Küttler let al. 2020). Moreover, the original raw large-scale dataset requires practitioners to manually define SQL queries for extracting the data of interest, which is flexible but could be an overkill.
setting. While this is good entry point and could be of great use to
More importantly, the proposed comparison metrics were mean and median statistics of average episode returns over training seeds. One may argue that the median is a robust statistic. However, in this case, it is a median of average episode returns over training seeds and not of the whole set of evaluation episodes. It is well known in the RL community (Agarwal al. 2021) that those not reliable due to the highly-variable nature of RL agents. As of the NetHack, this further amplifies by the extremely-variable nature of the game itself as both demonstrated in Küttler et al. (2020): HHambro et al. 20226). Therefore, to reduce the noise in the progress of RL. agents in this domain, one would need a different evaluation toolset that better suited for it.
NNottion standard for the RL community, where is state, is an action, and is reward.
3

---

3 Katakomba
Given the issues described in the previous section, we are ready to present Katakomba a set of tools, benchmarks, and memory-based offline RL baselines. In this section, we gradually introduce the components of the library with accompanying code examples to better demonstrate the case use. All the numbers and performance metrics discussed in the text were measured on the same set of hardware: 14CPU, 128GB RAM, NVMe, IxA100; for more details, please, see Appendix D
3.1 Benchmark
Listing 1: Usage example: Tasks defined via the character field that is then used by the offline wrapper for dataset loading and score normalization.
from katakomba. env import MettHackChallenge
from katakomba. env import ofrineeleeieecchaleeegllaaaa
The task is specified using the character field NetHHackChaleenget character=" hum-neu". neu" iiservation_keyse ("y
"tty_colors", "tty_cursor"]
)
convenient wrapper that provides interfaces for dataset loading and score normalization
env = aaffinnneetaaaccchlleeeeerrrrpp (env)
Decomposition In our benchmark, re-use the dataset collected by the AutoAscend bot (Hambro et al., [))2b While this bot is highly-capable, it still is considered an early-game contender because can not descend further than two levels in half of the episodes. As the dataset becomes an early game bridgehead, we divide the tasks based on the character configurations: role, race, and alignment. In the NetHack community, these known be the most important (and even having dramatic effect), requiring to utilize varying abilities of each role or race Overall, in opposition to merging all combinations, this decomposition allows more focus the characters' gameplay and the size reduction one needs to download for both playing-arowe around and committed research, as we will describe further.
Categories This results in 38 datasets in total. However, it may not be possible for researchers to examine each of them as the training times can be an obstacle. To this end, we further divide the tasks into three categories: Base, Extended, and Complete. We separate each category based on the wisdom of the NetHack community, i.e., that roles have a more substantial effect on the gameplay than race, and race has more effect than alignment. The datasets and categories are listed in Table Base tasks consist of all possible roles for the human race; we choose a neutral alignment where possible; otherwise, we pick the alternative one (for humans, there no neutral alignment, there only one alternative). We include all other role-race combinations for Extended tasks. For Complete. add tasks that were not included in the Base and Extended categories. Note that these three categories combined cover all of the allowed role-race-alle race-alignment combinations.
Data Selection As demonstrated in the previous section, the original TTYRec dataset is actually slower than expected when it is used for learning agents. Moreover, one may be unable to download the 90GB dataset. Therefore, we take a different path by subsampling and repacking the original dataset task wise, averaging 680 episodes and 1.3GB per task. The subsampling procedure is stratified by the game score (for more details, please see the scripf"). This allows for more versatility: one can download the needed datasets demand as in D4RL. (Fu al. 2020); furthermore, this permits us to address the rolling issue as we repacked the transition tuples in the way suitable to typical ORL pipeline part of the release. To ensure the reliable accessibility of the data, host it the HuggingFace Hub (Engstrom et al. 2020).
https://ethaaeiii..co////////
ott. dataset.
py

---

Table Katakomba tasks. We split the large-scate scale AutoAsceed dataset into 38 problems dividing them into three categories. The splits are justified by the early-game nature of the AstoAscend data, where roles have a higher impact on the gameplay, then races, and then alignments. Note that the whole benchmark covers all possible role-race-aligmmen combinations.
Trausitions
Median Turns
Median Score
Median Deathlvi
(GB)
Compressed Size
Base (Role-Centrie)
are-hum-nccu
24527163
32858.0
4802.5
2.0
94.5
1.3
bur bum neu
26266771
35716.0
11964.0
4.0
101.1
1.7
cay-hum-neu
21674680
303610 0
8152.0
4.0
83.5
1.3
hes-buni-neu
14473997
18051.0
2043.0
1.0
55.7
0.8
kni bum-lay
22287283
28246.0
6305,0
3.0
85.8
1.5
33741542
42400.0
11356.0
4.0
129.9
2.1
pri-humm-neu
18376473
26796.5
5366.5
2.0
70.8
1.1
tan-bumt-eeu
17625493
25354.0
6168.0
2.0
67.9
1.0
rog- hum
14284927
sam- hum-law
224225:37
329510
7850.0
4.0
86.3
1.3
lou-bunt-ccu
13376498
17955.5
val- hum neu
27784788
35250.0
11402.5
4.0
107.0
1.8
wiz-hum--ne
14343449
19808.5
3132.5
1.0
55.2
0.8
Extended (Race-Centric)
pri-elf-cha
18796560
26909.5
4718.5
2.0
72.4
1.1
ran-elf-cha
18238686
26617.0
7583.0
4.0
70.2
wiz-2lf-cha
15277820
195120
2968..5 5
1.0
38.8
0.9
are law-da--law
25100788
34669.0
4026.0
1.0
96.7
1.5
cav- sav-dea--lw
7158.00
88.1
1.5
val-dwa-low
32787658
33973
8652.5
3.0
126.6
2.5
24141048
34432.0
4077.5
1.0
93.0
1.4
cav- sav-gno-nes -nen
21624779
29860.0
6446.0
3.0
83.3
1.4
hea- hea-gno-neu
14884704
18518.0
1.0
52.3
0.9
ran-gnn-ncu
17571659
25970.0
5326.0
2.0
67.7
1.1
gno nou
14193637
19206.0
bar ore cha
27826356
39291.0
10499.0
4.0
107.2
1.8
18127448
26707.0
5460.0
2.0
69.8
1.1
rog-are-clia
16674806
22351 O
3103.0
1.0
64.2
1.0
wiz are cha
15994150
22570.5
3241.5
1.0
61.6
1.0
Complete (Alignment-Cuutee
ane hum--um--a low
23422383
31446.0
4188.0
1.0
90.2
1.3
cav-tuum-bar law
223284994
31039.0
mon hum Jaw
30782317
39647.0
10855.0
4.0
118.5
1.9
pri-hum-Iare law
182988 16
27192.0
4833.0
1.0
70.5
1.1
val-hum law
30171035
34570.5
bar- har-ham-cho
25362111
35925.0
12574.0
5.0
97.7
1.6
mon-hun--cla
33662420
41730.5
11418.0
4.0
129.6
2.1
pri-hum-shhe
28204_5
2.0
ran-hum-chhu
16999630
24698.5 5
6236.0
2.0
65.6
1.0
wiz-hum-chaa
14635591
20257.0
3294,0
1.0
56.4
0.9
3.2
Data Loaders for Speed-Memory Tradeoff
As we outlined in Section 2 while the original large-scale dataset is well-compressed, its iteration speed integrated with proper sequential loaders is still an obstacle for fast experimentation loop. Moreover, it requires least one full download for the entire large-scale dataset, which may not be suitable for essource-onssrrinee scenarios. In Katakomba, we address this by providing three different loaders trading-off memory and speed allowed by the deliberate task division. In each of them, the compressed task's ' dataset is automatically downloaded and decompressed if one aims for speed. Moreover, we provide Python interface for an automatic clean-up one does not want to store the decompressed dataset beyond the code execution.
HDF5, In-Memory (Decompressed,, RAM) This is the fastest format that relies on the decompres- sion of the dataset into the RAM, which may require from GIGB to 129GB depending on the dataset listed in T This option quite demanding regarding the memory but comes with the advantage of the fastest data access (see Table IT.
HDF5, Memory Map (Decompressed, Disk) This option is middle middle-ground that allows the datasets to be efficiently utilized without requiring large RAM. When one specifies this preference, the dataset will be decompressed on one's hard drive, and the consequent reads will conducted from it. The advantage of this approach compared to the loader from Hambro et al. (2022b) is that there is no need to decompress on the fly, and this is done precisely once at the start of the training process (taking from three to ten minutes average).
5

---

Listing 2: Usage example: Katakomba provides different loaders that can be chosen depending one's speed-memory tradeoff.
Decompress the dataset into RAM dataset env nav.get_dataae (mode inn in__mmmry""
Decompression on-read
dataset get dataset (mode=" compresee))))
The original dataloader introduced in Hambro et al, 2022 dataset = env datastt (scale="big")
Decompress the dataset on disk dataset env get_dataset (mode=" "mmmemmmmmm)p))
If you want to delete the decompressed dataset This will not affect the compressed version dataset. ()
HDF5, In-Disk (Compressed, Disk) This mode is the most cheap one but slow. Essentially, dataset read on from disk and decompressed on-the-fly. We found this useful for debugging purposes where one does not need the whole training process to be run.
TTYRec, In-Disk (Compressed, Disk) In case one finds the original approach to data loading more suitable, also provide convenient interface that wraps the source large-scale dataset and loader. One can also set it up in more detail using keyword arguments from the original TTYRec data loader. However, this option comes with the downsides described in Section 2 i.e., slower iteration speed and the need to download the 90GB dataset at least once.
In addition to the dataset interfaces, also provide implementation of replay buffer suitable for sequential offline RL algorithms for bootstrapping practitioners with ready-to-go tools.
3.3 Evaluation Methodology under High Variability
In Hambro et al. (2022b), authors used average episode return across seeds when comparing baselines. While this a standard practice in the RL and ORL communities, it was recently shown to be unreliable (Agarwal al. 2021) the algorithms
known be highly variable in performance. This problem amplifies even more for NetHack, where the score distribution is typically quite wide for humans and bots (Küttler al. 2020).
To address this, we argue that the evaluation toolbox from Agarwal et al. 2021 more appropriate and suggest using it when comparing NetHack learning agents. We use these tools for two dimensions: in-game score and death level. The first dimension corresponds to what one typically optimizes with ORL agents but is considered proxy metric (Küttler et al. 2020). While the latter lower bounds the early-game progress and is more indicative of the game completion.
Furthermore, similar to Fu et al. (2020), we also suggest reporting normalized scores to capture how far one from the AutoAscend bot. We use mean scores per dataset as a normalization factor and rescale to [0, 100] range after normalization, similar to D4RI. (Fu et al. 2020). This functionality is also provided as part of the offline wrapper for the NetHackChallenge environment. Please refer to Appendix D for precise values.
4 Benchmarking Recurrent Offline RL Algorithms
Algorithms For our benchmark, we rely on the following set of algorithms: Behavioral Cloning (BC) (Michie al., 1990), Implicit Q-Learning (IQL) (Kostrikov al.) (02T), Conservative Q- Learning (CQL) (Kumar et al. (020), Randomized Ensemble Mixture (REM) (Agarwal et al. 2020). and kdvantage-Weightted Actor- Critic (AWAC) (Nair et al. (020)) These are known as either the most
competitive in the continuous control setting Tarasovv et al. 2022) or were shown to be competitive
the discrete control (Agarwal al. 2020 Kumar al. 2002a). Similar to Hambro et al. (2022a b).
we build upon Chaotic- Seseri-DDaavve--GPP architecture that converts the tty observation into image

---

Median
IQM
Mean
Optimality Gap
IQL-
AWAC
CQL
BC
3.0
94
Normalized Score
(a) Bootstrapped point estimates. can be seen except REM, which failed, algorithms perform about the same across all metrics and are generally far behind the AutoAscend algorithm, which mean scores per dataset were used for normalization.
1.00
Fraction of runs with score >
0.75
Algorithm x
Algorithm Y
BC
REM
BC
IQL
0.25
BC
AWAC
0.10
BC
CQL
0.00
40
100
0.4
0.5
0.6
0.7
0.8 0.9 1.0
Normalized Score (v)
(b) Performance profiles, Notably, around 5% of episodes surpass the normalized score of 20.
(c) Probability of improvement of BC to other algorithms.
Figure 1: Normalized performance under the Katakomba benchmark for all 38 datasets. Each algorithm was run for three seeds and evaluated over 50 episodes resulting 5700 points for constructing these graphs. As one can see, there is not much improvement beyond naive behavioral cloning.
and then feeds it into the CNN layers followed by the LSTM (Hochreiter & Schmidhuber 1997). Ultimately, we test five common ORL algorithms augmented with recurrence. To the best of our knowledge, there do not implement the AWAC algorithm.
other open-sourced alternatives beyond the Hambro et al. (2022a) that also
Experimental Setup We train every algorithm for 500k gradient steps, resulting in around 20 epochs per dataset. We report the scores of the last trained policy over 50 evaluation episodes as standard the ORL community. Important highlight that while this amount may seem small for NetHack, this number is adequate when used in conjunction with stratified datasets and RLiable evaluation tools due to the underlying bootstrapping mechanism. For specific hyperparameters used, please either see Appendix B or the configuration files provided in the code repository.
Replicability and Reproducibility Statement To ensure that our experiments' results are replicable and can easily be reproduced and inspected, we rely on the Weights& Biases (Biewald (2020). In the provided code repository, one find all the logs, including configs, dependencies, code snapshots, hyperparameters, system variables, hardware specifications, and more. Moreover, to reduce the efforts of interested parties required for inspection, structurize the results using the Weights& Biases public reports.
Results The outcomes are twofold. First, the results achieved are similar the already observed by the Piterbarg et al. (2023) and Hambro et al. (2022b), but on larger number of offline RL algorithms tested. As shown Figure I [and Figure 2 all algorithms were unable to replicate the scores of the AutoAscend bot, reaching normalized scores below 6.0 on average and not progressing beyond the first level on the majority of training runs. Notably, only 5% of the episodes resulted in normalized score of around 20.0 (Figure Tb). Moreover, REM has not been able to achieve even the non-zero normalized score. Second, perhaps surprisingly, the only algorithm that does not rely any way on policy constraints is also the only algorithm that completely failed. This, and also the high correlation
7

---

in the performance profiles (Figure lb). gives us a hint that all other methods showing non-zero results actually rely primarily on behavioral cloning in one form or another, such as advantage weighted regression in IQI. and AWAC or KI.-divergence as in CQL. Indeed, the most successful hyperparameters our experiments proved to be those that significantly increase the weight of losses that encourage similarity to the behavioral policy (see Taalee[0ji in the Appendix E). Furthermore, as shown in the Figure Ind and Figure [I Behavioral Cloning algorithm is not worse than all the other more sophisticated offline RL algorithms. Thus, NetHack remains major challenge for offline RL algorithms, and Katakomba can serve as a starting point and testbed for offline RL research this direction. For graphs stratified by the Base, Extended, and Complete categories, Appendix F
Median
IQM
Mean
Optimality Gap
REM IQL
AWAC: CQL
1.000 1.008 1.016 1.024
0.96 0.99 1.02
1.05
1.01
1.02
1.03 98.97 98.98 98.99 99.00
Death Level
(a) Bootstrapped point estimates.
0.100
80
Fract on of runs with Scora >TT
COL
AAAC
ICL R2M
Algorithm X
Algorithm
BC
REM
0.050
IQL
0.025
BC
AWAC
BC
CQLL
0.000
1.00
1.25
1.50
2.25 Death Level (T)
2.50 2.75
3.00
0.4
0.5
0.6 0.7
0.8 0.9 1.0
PIX
(b) Performance proñles. Notably, only 2.5% of episodes progress to the sccond level.
(c) Probability of improvement of BC to other algorithms.
Figure 2: Death level under the Katakomba benchmark for all 38 datasets. Each algorithm was run for three seeds and evaluated over 50 episodes resulting in 570 points for constructing these graphs. As one can clearly see, there is not much improvement beyond the naive behavioral cloning.
Related Work
Offfine RL In recent years, there was considerable interest of the reinforcement learning community
in the offline setting, which resulted in numerous and diverse algorithms specifically tailored for this
Chen et al., 2021 Fujimoto & Gu 2021 An et al.) 202 Nikulin et al.) 2023; Tarasov et al. 2023). The core idea behind most of
setup (Nair et al.] 2020 Kumar et al.) 2020; Kostrikov et al. 2021
them is to ensure the resulting policy stays close to the behavioral one. This could be achieved via
different ways: penalizing value function (Kumar al. 2020 An et al. 2021 Nikulin et al. 2023),
constraining the policy outputs (Fujimoto & Gu 2021 Tarasov et al. 2023) or even training directly in the conservative latent space (Zhou et al.) 2020 Chen et al. 2022b Akimov et al.) 2022). Due to the menchmark-centricin of the RL field, most of the proposed ORL algorithms are for continuous
control with a few exceptions (Agarwal al. 2020 Kumar al. 2020 2022a). The de-facto standard benchmark is the D4RL (Fu et al. 2020). which provides a suite of datasets focused on continuous control with proprioceptive states under different regimes, such sparse-reware
low-data regimes.
Also, few benchmarks move the focus from proprioceptive states to images or other more complex entities QQn al. 2022
Lu et al. 2022
Agarwal al. 2020).
RL for NetHack NetHack as a testbed for RL agents was introduced in Küttler al. (2020). To further advance the RL agents in this domain, the NetHack Challenge Competition (NHC) was held
8

---

(Hambro et al. 2022a) that resulted in two of the most performant agents learning -ased Chaotic- Dwarven-GPT-5, and a rule-based AutoAscend (AA). Notably, the latter outperformed learning- based agents by wide margin. Consequently, this solution was used to collect the large-scale NetHack Learning Dataset (Hambro et al., 2022b). The closest concurrent work by PPterbargg et al. (2023) the authors released another AA dataset but accompanied with the hierarchical labels, which arise due to the nature the AutoAscend bot, demonstrating their usefulness in cloning the AA bot. However, in contrast to our work, Piterbarg et al. (2023) focuses on the large-scale setup similar to Hambro ct al. (202bb).
6 Discussion, Limitations, and Future Work
In this work, we focused on building reliable tools and benchmarks for offline RL setting using the recently released AutoAscend large-scale dataset (Hambro et al. (0226). While this does not cover the whole spectrum of NetHack's community interests, such as offline-to-oning regime or learning from human demonstrations, we believe our effort is helpful in establishing reliable and open instruments for data-driven NetHack. Moreover, our contributions could be of interest to the part of the ORL community studying discrete control, memory, and adaptivity (Ghosh et al. 2022).
Given our results and experience with the NetHack Learning Environment, believe fruitful future research may lie along the following directions: finding a better state-encoder, as the current one presents bottleneck in both efficiency (rendering is expensive) and locality (only the small part of the terminal is used). Another interesting research direction would be to assess recently appeared recurrence mechanisms such as Linear Recurrent Unit (Orvieto et al. 2023). which might also speed up the training process without hurting the performance. Also, as the interest in generalization properties will appear, it would be a great addition include more datasets that will provide metadata on the seeds used for data generation, as it will allow to assess trained agents on both seen and unseen seeds quantify the generalization gap more systematically.
Overall, we firmly believe that NetHack provides a nice playground for investigating how to build next generation of reinforcement learning agents using prior data that would encompass stronger generalization and memory capabilities. To this end, we plan to continuously maintain the benchmark, accompanying tools, and curate new datasets if considered useful for further advancements.
References
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective offline reinforcement learning. In International Conference on Machine Learning, pp. 104-114 14. PMLR, 2020.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304 29320, 2021.
Dmitriy Akimov, Vladislav Kurenkov, Alexander Nikulin, Denis Tarasov, and Sergey Kolesnikov. Let offline rl flow: Training conservative agents in the latent space of normalizing flows. arXiv preprint Xrxi::2211.1100 2022.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-bart offline reinforcement learning with diversified g-ensemble. Advances in neural information processing systems, 744447367477 2021.
Kai Arulkumaran, Mare Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. A brief 2017.
survey of deep reinforcement learning. arXiv preprint
Bowen Baker, Iige Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639-24654, 2022.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https:: // //www.wandb com/ Software available from wandb.com.

---

Chacha Chen, Hua Wei, Nan Xu, Guanjie Zheng. Ming Yang, Yuanhao Xiong, Kai Xu, and Zhenhui Li. Toward a thousand lights: Decentralized deep reinforcement learning for large-scale traffic signal control. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 3414-3421, 2020.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097, 2021.
Minmin Chen, Can Xu, Vince Gatto, Devanshu Jain, Aviral Kumar, and Ed H. Chi. Off-policy actor-critic for recommender systems. Proceedings of the 16th ACM Conference on Recommender Systems, 2022a.
Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Yuan Gao, Jianhao Wang, Wenzhe Li, Bin Liang, Chelsea Finn, and Chongjie Zhang. Latent-variable wwwaattge-weigheee policy optimization for offline rl. arXiv preprint arXiv:2203.08949 2022b.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, L. Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study ppo and trpo. In ICLR, 2020.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. Advances in neural information processing systems, 14-20132-20145. 2021.
Dibya Ghosh, Anurag Ajay, Pulkit Agrawal, and Sergey Levine. Offline rl policies should be trained to be adaptive. In International Conference on Machine Learning, pp. 7513-7530. PMLR, 2022.
William H Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. Minerl: a large-scale dataset of minecraft demonstrations. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 2442-2448, 2019.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104 2023.
Eric Hambro, Sharada Mohanty, Dmitrii Babaev, Minwoo Byeon, Dipam Chakraborty, Edward Grefenstette, Minqi Jiang, Jo Daejin, Anssi Kanervisto, Jongmin Kim, et al. Insights from the neurips 2021 nethack challenge. NeurlPS 2021 Competitions and Demonstrations Track, pp. 41-52. PMLR. 2022a.
Eric Hambro, Roberta Raileanu, Danielle Rothermel, Vegard Mella, Tim Rocktäschel, Heinrich Kuttler, and Naila Murray. Dungeons and data: A large-scale nethack dataset. In Thitth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022b.
Sepp Hochreiter and Jürgen Schmidhuber, Long short-term memory. Neural computation, 9:1735-80, 12 1997. doi: 10. 16.2ne2o 1997.9.8.1 1735..
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, and Jeff Braga. Cleanrl: High- quality single-file implementations of deep reinforcement learning algorithms. arXiv preprint arXiv:2/ /..-8// 2021.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 14Xiv:1412.6980 2014.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:21 11X:::2110.06166) 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179-1199 1191, 2020.
10

---

Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine. A workflow for offline model-free robotic reinforcement learning. In Sth Annual Conference on Robot Learning. 2021.
URL htttss://ppenevveieie http://wwww....oooomoooo///////a
Aviral Kumar, Rishabh Agarwal, Xinyang Geng, G. Tucker, and Sergey Levine. Offline q-learning on diverse multi-task data both scales and generalizes. ArXiv, abs/2211.15144, 2022a.
Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline q learning on diverse multi-task data both scales and generalizes. arXiv preprini 1rX1v:2211.15144, 2022b.
Heinrich Küttler, Nantas Nardelli, Alexander Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. Advances in Neural Information Processing Systems, 33:7671-7684, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint wrXiv:2005.0/644 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Cong Lu, Philip J. Ball, Tim G. Rudner, Jack Parker-Holder, Michael A. Osbome, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations, 2022.
Nina Mazyavkina, Sergey Sviridov, Sergei Ivanov, and Evgeny Burnaev. Reinforcement learning for combinatorial optimization: A survey. Computers & Operations Research, 134: 105:000, 2021.
Vegard Mella, Eric Hambro, Danielle Rothermel, and Heinrich Küttler. moolib: A Platform for Distributed RL. 2022. h/tpp:////// http:///ww...ccccooooooeererea/o
Donald Michie, Michael Bain, and J Hayes Micche. Cognitive models from subcognitive skills. IEE control engineering series, 44:71-99, 1990.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09350 2020.
Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov. Anti-exploration by random network distillation. arXiv preprint 1xXiv:2301.16611 2023.
Antonio Orvicto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcchre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences, 2023.
Ulyana Piterbarg, Lerrel Pinto, and Rob Fergus. Nethack hard to hack. arXiv preprint 1924: 2023.
Rong-Jun Qin, Xingyuan Zhang, Songyi Gao, Xiong-Hui Chen, Zewen Li, Weinan Zhang, and Yang Yu. Neorl: A near real-world benchmark for offline reinforcement learning. Advances in Neural Information Processing Systems, 55:24753-247655 2022.
Laura Smith, llya Kostrikov, and Sergey Levine. A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning, August 2022.
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. CORL: Research-oriented deep offline reinforcement learning library. In 3rd Offline RI. Workshop: Offline RL. as " "Launchpadd, 2022. httl/htttp::/openevviie hew.:
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. arXiv preprint arXiv:2305.0983: 2023.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft ii using multi-agent reinforeement learning. Nature, 575(7782):3503354 2019.
11

---

Liang Yu, Weiwei Xie, Di Xie. Yulong Zou, Dengyin Zhang, Zhixin Sun, Linghua Zhang, Yue Zhang. and Tao Jiang. Deep reinforcement learning for smart home energy management. IEEE Internet of Things Journal, 7(4):2751-2762, 2019.
Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforce- ment learning. In Conference on Robot Learning, 2020.
12

---

A What Is Inside the Datasets?
Every dataset is repacked into HDF5 files similar Fu et al. (2020). The data keys are described in Table 3 along to the typical (st, dz, ft, d,) tuples, the metadata is also provided as the datasets' attributes with comprehensive information about specific trajectories similar to Hambro al. (202bb). The re-packing script is provided at pestts://gitoee
com/ corl- team// katakomba/
tree/mmain/scrrpps//geee wwwp:pppttgeeeeeeeeeeeeeeeeeeeee smalsssallll.mm dataset. py
The re- re-packed datasets constitute of transformed data from Hambro et al. (0222b) Dissimilar the the large scale dataset, the repacked data now in the format familiar the ORL practitioners. We also save the metadata for each trajectory, for comprehensive description, please, see Appendix F in Hambro et al. (002bb).
Table 3:
Name
Type
Shape
Description
tty_chars
np. uint8
[B, H. W]
8;: The on-screen characters (default screen size 80
24).
tty_colors
np. int8
[B, T. H. W]
8;: The on-screen colors for each character.
tty_cursor
up. int16
[B, T. 2]
Sy: The coordinates of the on-screen cursor.
actions
np. uinuint8
[B, T]
ay: The NLE actions the player made in response to
the St.
rewards
np.int:i. int32
[B, T]
ry: The difference between in-game scores at states and 8t-1. Note that this was used in all implemen- tations of the algorithms provided in Hambro et al.) (2022b). We also found that without this reward- shaping, all offline RL algorithms failed completely. de: An indicator whether the current state is the last one the trajectory.
dones
mp. uint8
[B. T]
B
License
Our codebase and repacked datasets are released under the NETHACK GENERAL PUBLIC LI- CENSE. The original NetHack Learning environment (Küttler et al., 2020) and large-scale datasets
(Hambro et al. 2022b) also released under NETHACK GENERAL PUBLIC LICENSE.
C General Ethic Conduct and Potential Negative Societal Impact
To the best of our knowledge, our work does not present any direct potential negative societal impact.
As of the general ethic conduct, we believe that the most relevant issue to be discussed is the "Consent to use or share the data". Our work largely built upon both the NetHack Learning Environment (Küttler et al., 2020) and the coresponding large-scale dataset (Hambro et al. 2022b), and as already described the Appendix B both are distributed under the NETHACK GENERAL PUBLIC LICENSE that explicitly allows for re-usage and re-distribution.
13

---

D Resources and Statistics
We used 64 separated computational nodes with IxA100, 14CPU, 128GB RAM, and the NVMc as long-term storage for all our experiments. All the values reported in the paper were also obtained under this configuration. One can also find more detailed information inside the Weights& Biases logs in the code repository.
Table 4: Scores used for Normalization. You can also find them at https: com/ corl- team/ wataaaaabaabllboaaaaaaaaaaaaaaae httpi////oooomannaaaaaaaaaa http:///mmmmaaaaaaaaaaaaa/al//// see Table 2 in the main text.
py For other statistics, please.
Tasks
Minimum Score
Maximum Score
Mean Score
Base (Rolc-Centric)
arc-hum-neu
0.0
138103.0
6636.44
bar-hum-neu
0.0
292342.0
17836.68
cav-hum-new
0.0
258978.0
12113.87
hea-hum-neu
0.0
64337.0
4068.27
kni-hum-law
0.0
419154.0
14137.06
mon-hum-neu
0.0
171224.0
17456.05
pri-hum-neu
0.0
114269.0
7732.69
ran-hum-neu
0.0
54874.0
8067.99
rog-humn hog-hum-cha
0,0
68628.0
4818.20
sam-hum-late
0.0
155163.0
11009.36
tou- tou-hum-neu
0.0
59484.0
4211.47
val-
ne--hum-neu
16.0
313858.0
18624.77
wiz-hum-neu
0.0
71709.0
5323.48
Extended (Race-Centric)
pri-elf-cha
0.0
83744.0
7109.35
ran-elf-cha
0.0
66690.0
9014.18
wiz-elf-cha
0.0
71664.0
5005.16
arc-dwa-lawe
0.0
83496.00
5445.69
cav-dwa-law
0.0
161682.0
11893.48
val-dwa-law
0.0
136591.0
23473.61
are-gno-neu
0.0
10004.00
5316.57
cav-gno-neu
0.0
142460.0
10083.06
hea-gno-neu
0.0
69566.0
3783.93
ran-gno-neu
0.0
58137.0
6965.04
wiz-gno-neu
0.0
37376.0
4317.51
bar-orc-cha
0.0
164296.0
17594.38
ran-ore-cha
3.0
69244.0
7608.48
rog-ore-cha
0.0
54892.0
4897.69
wiz-orc-cha.
0.0
40871.0
5016.74
Complete (Alntreent-Centric)
arc-hum-lave
2.0
84823.0
5826.35
cav-hum-lan
0.0
156966.0
12462.82
mon-hum-law
7.0
190783.0
16091.57
pri-hum-law
0.0
99250.0
6847.99
val-hum-law
0.0
428274.0
26103.03
bar-hum-cha
0.0
164446.0
18228.11
non-huu--hha
0.0
223997.0
18353.30
pri-hum-chan
0.0
58367.0
8262.56
ran-hum-cha
3.0
62599.0
8378.50
wiz-hum-chh
0.0
55185.0
5316.82
14

---

E Hyperparameters
For all algorithms, hyperparameters have been reused from previous work whenever possible. For BC, CQL. and IQL reference values, see Appendix 1.4 in the Hambro et al. (2022b). For AWAC, hyperparameters from IQL reused due the very similar policy updating scheme. For REM, hyperparameters were taken from the original work (see Agarwal et al. (2020).
As in Hambro et al. (2022b), and in contrast to the original CQL implementation, TD loss by the a coefficient instead of the CQL loss, as we observed better results with such scheme. We performed search for [0.0001, 0.0005, 0.001, 0.01, 0.05, 0.1, 0.5, 1.0) with best value a = 0.0001.
multiply the
Table 5: BC hyperparameters.
Parameter
Value
optimizer
AdamW (Kingma & Ba 2014
Loshchilov & Hutter 2017)
training iterations
500000
batch size
64
sequence length
16
learning
3e-4
weight decay
0.0
state encoder
Chaotic- Seree--aaaee---P--555 GPT-5(Hambro Hambro al.
2022a
LSTM hidden dim
2048
LSTM layers
LSTM dropout
0.0
use previous action
True
Table 6: CQL hyperparameters. Note that our implementation, the a coefficient multiplies the TD
loss.
Parameter
Value
optimizer
AdamW (Kingma & Ba 2014
Loshchilov & Hutter 2017)
training iterations
500000
batch size
64
sequence length
16
learning rate
3e-4
weight decay
0.0
state encoder
Chaotic- Seaiee-DwaaveeGPP--Aaaaa Hambro al. 2022a b
LSTM hidden dim
2048
LSTM layers
LSTM dropout
0.0
use previous action
True
tau (T)
5e-3
gamma (7)
0.999
reward clip range
[-10.0, 10.0]
alpha (a)
le-4
15

---

Table IQL hyperparameters.
Parameter
Value
optimizer
AdamW W (Kingma & Ba 2014
Loshchilov & Hutter 2017)
training iterations
500000
batch size
64
sequence length
16
learning rate
3e-4
weight decay
0.0
state encoder
Chaotie-Dwarven-GGP-S// Hammro et al.
2022a b)
LSTM hidden dim
2048
LSTM layers
LSTM dropout
0.0
use previous action
True
tau (7)
5e-3
gamma (?)
0.999
reward clip range
[-10.0, 10.0]
expectile
0.8
temperature
1.0
advantage clip max
100
Table 8: WWA hyperparameers.............
Parameter
Value
optimizer
AdamW (Kingma & Ba 2014
Loshchilov & Hutter 2017
training iterations
500000
batch size
64
sequence length
16
learning
3e-4
weight decay
0.0
state encoder
Chaotie-Dwarven-GGPT5 Hambro al. 2022a b)
LSTM hidden dim
2048
LSTM layers
LSTM dropout
0.0
use previous action
True
tau (7)
5e-3
gamma (?)
0.999
reward clip range
[-10.0, 10.0]
temperature
1.0
advantage clip max
100
Table 9: REM hyperparameters.
Parameter
Value
optimizer
AdamW (Kingma & Ba 2014
Loshchilov & Hutter 2017)
training iterations
500000
batch size
64
sequence length
learning rate
3e-4
weight decay
0.0
state encoder
Hambrr al. 2022a
LSTM hidden dim
2048
LSTM layers
2
LSTM dropout
0.0
previous action
True
tau (T)
5e-3
gamma (7)
0.999
reward clip range
[-10.0, 10.0]
ensemble heads
200.0
16

---

Table 10: The effect of CQL policy constraint strength on the performance. Results, which are averaged across 3 seeds, are sorted based on the final unnormalized game score.
Alpha (ax)
Return
0.0001
526.72 + 71.37
0.0005
396.50 39.35
0.001
395.56 + 139.92
0,05
226.55 + 196.45
0.01
32.42 + 43.69
0.1
14.04 + 10.29
0.5
0.00 0.00
1.0
17

---

Results per Benchmark Categories
In this section, we report the results stratified by the introduced categories. If is willing to inspect specific datasets, we organized all training logs into Weights&Biasse, public reports, found at httpss//wandb... ai/tlab/ httpaa/a//eeeaaaa////ge/e/e//eee /etetaa/e/a/eeetrtt
Note that one can find all the evaluation scores (for more than one checkpoint) within the runs and them for any evaluation tools of interest. Also, we provide convenient scripts for con- structing RLiable (Agarwal et al. 2021) graphs based on the provided runs that can be config- ured for one's purposes well (see https://gitthbb
scripts/t1iable_reprrs peppottttpperrrrpp../
Median
IQM
Mean
Optimality Gap
REM
JQL
AWAC
CQL
0
2
6
0.0
1.5
3.0
2
94
96
98
100
Normalized Score
(a) Bootstrapped point estimates.
1.00
8C
Fraction of runs with score >
COL
Algorithm x
Algorithm Y
10)
REM
0.50
BC
IQL
BC
AWAC
0.25
BC
CQL
0.10
0.00
0.4
0.5
0.6 0.7
0.8
0.9 1.0
PIX
Y)
0
20
40
80
100
Normalized Score (t)
(c) Probability of improvement of BC to other algorithms.
(b) Performance profiles.
Figure 3: Normalized performance under the Katakomba benchmark for Base datasets. Each algorithm was run for three seeds and evaluated over 50 episodes resulting 1950 points for constructing these graphs.
18

---

Median
IQM
Mean
Optimality Gap
REM
IQL
AWAC
COL
BC:
0.0
2.5
5.0
7.5
0.0
1.5
3.0
4.5
98
100
Normalized Score
(a) Bootstrapped point estimates.
1.00
Fracton runs with score >t
AWMC
Algorithm
Algorithm
0.75
RCM
BC
|
REE
IQL
BC
AWAC
BC
CQL
0.10
0.00
1.0
PIX Y)
Normalized Score
(c) Probability of improvement of BC to other algorithms.
(b) Performance profiles.
Figure 4: Normalized performance under the Katakomba benchmark for Extended datasets. Each algorithm run for three seeds and evaluated over 50 episodes resulting in 2250 points for constructing these graphs.
Median
IQM
Mean
Optimality Gap
REM-
IQL
AWAC
COL
BC
0.0
1.5
3.0
4.5
0.0
1.5
3.0
4.5
94.5
96.0
97.5 99.0
Normalized Score
(a) Bootstrapped point estimates.
1.00
Fraction of runs with score >
CDL
Algorithm X
Algorithm Y
0.75
REEM
0.50
BC
IQL
BC
AWAC
CQL
0.10
0.4 0.5
0.6 0.7
0.8 0.9 1.0
0
40 Normalized Score (t)
80
100
(c) Probability of improvement of BC to other algorithms.
(b) Performance profiles.
Figure 5: Normalized performance under the Katakomba benchmark for Complete datasets. Each
algorithm was run for three seeds and evaluated over 50 episodes resulting 1500 points for constructing these graphs.
19

---

Median
IQM
Mean
Optimality Gap
REM
IQL
AWAC
COL
BC
1.00
1.01
1.02
1.03
0.96
0.99
1.02
1.05
1.01
1.02
1.03
98.97
98.98 98.99 99.00
Death Level
(a) Bootstrapped point estimates.
0.100
BC
Fraction of runns with score >T
COL AAAC
Algorithm
Algorithm
101
0.075
BC
-RM
0.050
BC
IQL
AWAC
0.025
BC
CQL
0.000
0.4
0.5
0.6
0.7
0.8
0.9
1.0
P(X > Y)
1:00
1.25
1.50
1.75
2,00
2,25
2:50
2.75
3,00
Death
(c) Probability improvement to other algorithms.
(b) Performance profiles.
Figure 6: Death levels under the Katakomba benchmark for Base datasets. Each algorithm was run for three seeds and evaluated over 50 episodes resulting in 1950 points for constructing these graphs
Median
IQM
Mean
Optimality
REM-
IQL
AWAC
COL
1.00
1.01
1.02 1.03
0.96
0.99
1.02
1.09.000
1.01
1.02
Death Level
(a) Bootstrapped point estimates.
0.100
8C
Fraction of runs with Score >T
AMAC
CCI
Algorithm
Algorithm
0.075
10L
BC
REM
IQL
0.050
BC
AWAC
0.025
BC
CQL
0.000
0.4
0.5
0.7
0.8
1.0
P(X > Y)
1:00
1.25
1.50
1.75
2.00
2.25 (T)
2.50
2.75
3.00
(c) Probability of improvement BC to other algorithms.
(b) Performance profiles.
Figure Death level under the Katakomba benchmark for Extended datasets. Each algorithm was run for three seeds and evaluated over 50 episodes resulting in 2250 points for constructing these graphs.
20

---

Median
IQM
Mean
Optimality Gap
REM
IQL AWAC COL BC
1.005
1.020
1.035
0.96
0.99
1.02
1.05
1.005
1.020 1.035
98.965 98.980 98.995
Death Level
(a) Bootstrapped point estimates.
BC
Fraction of runs with score >
COL AMAL
Algorithm
Algorithm
ICL REM
BC
REM
0.050
IQL
BC
AWAC
0.025
CQL
0.000
0.4
1.0
1:00
2.25 Death Level (x)
PIX
(c) Probability of improvement of BC to other algorithms.
(b) Performance profiles.
Figure 8: Death levels under the Katakomba benchmark for Complete datasets. Each algorithm was run for three seeds and evaluated over 50 episodes resulting in 1500 points for constructing these graphs.
Median
IQM
Mean
Optimality Gap
REM
AWAC:
COL
BC
300
0
100
Score
(a) Bootstrapped point estimates.
1.00
Fraction of ruus with score T
Algorithm
Algorithm
0.75
REM
BC
REM
0.50
BC
AWAC
BC
CQL
0.00
0.8
PIX Y)
2000
4000
Score (t)
(c) Probability of improvement BC to other algorithms.
(b) Performance profiles.
Figure 9: Unnormalized in- ma-game score under the Katakomba benchmark for Base datasets. Each
algorithm run for three seeds and evaluated over 50 episodes resulting 1950 points for constructing these graphs.
21

---

Median
IQM
Mean
Optimality Gap
REM
IQL:
AWAC
CQL
BC
150
300
450
0
100
200
0
150
300
450
25
50
75
100
Score
(a) Bootstrapped point estimates.
1.00
Fraction of with score I
Algorithm
Algorithm
0.75
BC
|
REM
0.50
BC.
BC
AWAC
CQL
0.00
0.4
1.0
PIX
2000
Score (1)
(c) Probability of improvement of BC to other algorithms.
(b) Performance profiles.
Figure 10: Unnormalized in-game score under the Katakomba benchmark for Extended datasets. Each algorithm was for three seeds and evaluated over 50 episodes resulting in 2250 points for constructing these graphs.
Median
Mean
Optimality Gap
REM
IQL
AWAC
COL
BC
0
200
400
600
150
300
200
400
600
25
50
100
Score
(a) Bootstrapped point estimates.
1.00
Fraction of runs with score >
COL
Algorithm X
Algorithm
0.75
BC
RREM
0.50
BC
IQL
BC
AWAC
CQL
0.00
0.4
0.5
0.6
0.7
0.8 0.9 1.0
Score (T)
(c) Probability of improvement of BC to other algorithms.
(b) Performance profiles.
Figure 11: Unnormalized in-game score under the Katakomba benchmark for Complete datasets.
Each algorithm was run for three seeds and evaluated over 50 episodes resulting in 1500 points for constructing these graphs.
22


Reviews:
[{"review_id": "LhxPEirZWB", "reviewer": "Reviewer_obG3", "summary": "", "questions": "", "limitations": "n/a", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper investigates an interesting problem and provides a comprehensive solution. It seems that the library would benefit a large audience of researchers interested in offline reinforcement learning research.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "tT5LksjYdK", "reviewer": "Reviewer_ZoWq", "summary": "", "questions": "", "limitations": "The author rarely discussed the limitations of the dataset. As I have mentioned previously, the possible game states covered by the dataset might be limited due to the use of rule-based agents. Meanwhile, the dataset can be more thoroughly investigated under more sophisticated benchmark experiment settings.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. A more refined categorization in terms of game features;\n2. Efficient dataloader facilitating research for different purposes;", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "y3d5zoSRLg", "reviewer": "Reviewer_yKAo", "summary": "", "questions": "", "limitations": "The limitations are well discussed in Section 6.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "* This paper identifies the current challenges in adopting the large-scale NetHack dataset in offline RL research, and makes engineering efforts to address them. It perfectly fits the theme of Dataset & Benchmark track.\n* The authors make significant contributions in making the large-scale NetHack dataset more readily adoptable, including compiling a new set of smaller datasets, creating more proper and faster data-loading pipelines, implementing several offline RL baselines, and evaluating those methods on data-driven NetHack tasks.\n* NetHack is a challenging game that requires strong generalization abilities and extraordinary memory capacity, serving as a good testbed for building intelligent agents. The proposed benchmark will help promote research on data-driven NetHack and be of great practical value to the RL community.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "6lNp2nMpTa", "reviewer": "Reviewer_iC1Z", "summary": "", "questions": "", "limitations": "The limitations discuss weaknesses of the existing ORL implementations as they fail to achieve much success on the task. It would help to consider other limitations of the tooling itself:\n- How well can the tooling support scaling to more compute?\n- What can the tooling do to support gathering new datasets and sharing them?\n- Are there opportunities to provide tools and training for sub-tasks? For example, completing certain in-game objectives or pre-training a state encoder.\n\n\nThe authors note no substantial negative societal implications of the work. This makes sense for providing better tooling to use datasets derived from bots playing a video game with only vague parallels to real life activities.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "# significance\n\nModest. Provides a new set of tooling around existing datasets. The tooling and best practices they support around benchmarking are welcome additions, particularly given the clear difficulty of the task for existing RL algorithms. Tooling has incremental value to accelerate research on this task and others could contribute more data to use with these tools.\n\n# relevance\n\nModest. The ORL community is primarily interested in robotics at this time. NetHack has appeal as a task for game playing AI and so will appeal to that group. The limited success of RL in NetHack domain makes this a promising addition to help accelerate work in the game playing subfield. That said, the intersection of ORL and NetHack narrows the audience compared to alternative multi-task benchmarks of problems with large existing research communities.\n\n\n# quality\n\nGood. The empirical evaluations support two key points:\n1) The new data loader can be much faster than the previous data loader. (Even with data modified to the standard RL form)\n2) Baseline implementations of ORL algorithms do poorly on these tasks.\n\n\n# ethical and social implications\n\nGood. Nothing stands out. The dataset is for a video game with few direct real-world parallels and gathered from bots. The tooling simplifies interaction with this data, so there are no obvious novel social or ethical implications.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper presents a library to help practitioners looking to do research in Offline RL in NetHack.  Building on a large-scale dataset presented last years datasets & benchmarks, the authors implement a library that specifically aims to help Offfline RL researchers get up and running with ‘nld_aa’.  This includes adding benchmarked offline RL implementations, wandb runs and new dataloaders for a ‘split up’ dataset.


**Perceived Strengths:**
* The code and implementations appears to be high quality and genuinely of use to those looking to do Offline RL on NetHack. Several important algorithms are implemented, along with dataloaders that are more natural to q-value based methods (eg s,a,r,s’ tuples vs s,a,r tuples).
* The benchmarks appear to be reproducible with wandb traces, and good RLiable estimates.

**Perceived Weaknesses:**

* There is a repeated suggestion that is only a minor amount of novelty in this work.  The data is repackaged from the original, new implementations of old algorithms are presented. The authors respond that that their clean implementations and better reproducibility are easier for researchers looking to get started offline RL. 


**AC View:**

At surface-level the contributions of this work looks minor: repackaging old data; reimplementing cleaner and reproducible benchmark algorithms; writing new dataloaders; and adding more robust evaluations.  However, as pointed out by one reviewer, this is all the unloved work that leads to good datasets and benchmarks, that benefits the community.  However, often such work is done iteratively, contributing to the original open-source repository and helping to improve the software, and it is reasonable to question whether such contributions are worthy of a separate paper and package. In this case I believe there is a distinguishing factor in the specific targeting of the small scale offline RL setting (over the large-scale one presented in the original dataset), most noticed by splitting up and rigorously benchmarking different algorithms on different datasets.  As such I agree with the majority of reviewers and recommend this benchmark for inclusion.
Decision: Accept (Poster)