SlowGuess/NeurIPS_2023_OCR huggingface dataset:
and then feeds it into the CNN layers followed by the LSTM (Hochreiter & Schmidhuber, 1997). Ultimately, we test five common ORL algorithms augmented with recurrence. To the best of our knowledge, there are no other open-sourced alternatives beyond the Hambro et al. (2022a) that also do not implement the AWAC algorithm.
Experimental Setup We train every algorithm for 500k gradient steps, resulting in around 20 epochs per dataset. We report the scores of the last trained policy over 50 evaluation episodes as standard in the ORL community. Important to highlight that while this amount may seem small for NetHack, this number is adequate when used in conjunction with stratified datasets and RLiable evaluation tools due to the underlying bootstrapping mechanism. For specific hyperparameters used, please either see Appendix E or the configuration files provided in the code repository.
Replicability and Reproducibility Statement To ensure that our experiments' results are replicable and can easily be reproduced and inspected, we rely on the Weights&Biases (Biewald, 2020). In the provided code repository, one can find all the logs, including config, dependencies, code snapshots, hyperparameters, system variables, hardware specifications, and more. Moreover, to reduce the efforts of interested parties required for inspection, we structurize the results using the Weights&Biases public reports.
Results The outcomes are twofold. First, the results achieved are similar to the already observed by the Piterbarg et al. (2023) and Hambro et al. (2022b), but on a larger number of offline RL algorithms tested. As shown in Figure 1 and Figure 2, all algorithms were unable to replicate the scores of the AutoAscend bot, reaching normalized scores below 6.0 on average and not progressing beyond the first level on the majority of training runs. Notably, only  $5\%$  of the episodes resulted in a normalized score of around 20.0 (Figure 1b). Moreover, REM has not been able to achieve even the non-zero normalized score. Second, perhaps surprisingly, the only algorithm that does not rely in any way on policy constraints is also the only algorithm that completely failed. This, and also the high correlation
in the performance profiles (Figure 1b), gives us a hint that all other methods showing non-zero results actually rely primarily on behavioral cloning in one form or another, such as advantage weighted regression in IQL and AWAC or KL-divergence as in CQL. Indeed, the most successful hyperparameters in our experiments proved to be those that significantly increase the weight of losses that encourage similarity to the behavioral policy (see Table 10 in the Appendix E). Furthermore, as shown in the Figure 1c and Figure 1c Behavioral Cloning algorithm is not worse than all the other more sophisticated offline RL algorithms. Thus, NetHack remains a major challenge for offline RL algorithms, and Katakomba can serve as a starting point and testbed for offline RL research in this direction. For graphs stratified by the Base, Extended, and Complete categories, see Appendix F.
Death Level
(a) Bootstrapped point estimates.
(b) Performance profiles. Notably, only  $2.5\%$  of episodes progress to the second level.
Figure 2: Death level under the Katakomba benchmark for all 38 datasets. Each algorithm was run for three seeds and evaluated over 50 episodes resulting in 570 points for constructing these graphs. As one can clearly see, there is not much improvement beyond the naive behavioral cloning.
Algorithm X
(c) Probability of improvement of BC to other algorithms.
Offline RL In recent years, there was considerable interest of the reinforcement learning community in the offline setting, which resulted in numerous and diverse algorithms specifically tailored for this setup (Nair et al., 2020; Kumar et al., 2020; Kostrikov et al., 2021; Chen et al., 2021; Fujimoto & Gu, 2021; An et al., 2021; Nikulin et al., 2023; Tarasov et al., 2023). The core idea behind most of them is to ensure the resulting policy stays close to the behavioral one. This could be achieved via different ways: penalizing value function (Kumar et al., 2020; An et al., 2021; Nikulin et al., 2023), constraining the policy outputs (Fujimoto & Gu, 2021; Tarasov et al., 2023), or even training directly in the conservative latent space (Zhou et al., 2020; Chen et al., 2022b; Akimov et al., 2022). Due to the benchmark-centricity of the RL field, most of the proposed ORL algorithms are for continuous control with a few exceptions (Agarwal et al., 2020; Kumar et al., 2020, 2022a). The de-facto standard benchmark is the D4RL (Fu et al., 2020), which provides a suite of datasets focused on continuous control with proprioceptive states under different regimes, such as sparse-reward or low-data regimes. Also, few benchmarks move the focus from proprioceptive states to images or other more complex entities (Qin et al., 2022; Lu et al., 2022; Agarwal et al., 2020).
RL for NetHack NetHack as a testbed for RL agents was introduced in Kuttler et al. (2020). To further advance the RL agents in this domain, the NetHack Challenge Competition (NHC) was held
(Hambro et al., 2022a) that resulted in two of the most performant agents – learning-based Chaotic-Dwarven-GPT-5, and a rule-based AutoAscend (AA). Notably, the latter outperformed learning-based agents by a wide margin. Consequently, this solution was used to collect the large-scale NetHack Learning Dataset (Hambro et al., 2022b). The closest concurrent work is by Piterbarg et al. (2023) – the authors released another AA dataset but accompanied with the hierarchical labels, which arise due to the nature of the AutoAscend bot, demonstrating their usefulness in cloning the AA bot. However, in contrast to our work, Piterbarg et al. (2023) focuses on the large-scale setup similar to Hambro et al. (2022b).
6 Discussion, Limitations, and Future Work
In this work, we focused on building reliable tools and benchmarks for offline RL setting using the recently released AutoAscend large-scale dataset (Hambro et al., 2022b). While this does not cover the whole spectrum of NetHack's community interests, such as offline-to-online regime or learning from human demonstrations, we believe our effort is helpful in establishing reliable and open instruments for data-driven NetHack. Moreover, our contributions could be of interest to the part of the ORL community studying discrete control, memory, and adaptivity (Ghosh et al., 2022).
Given our results and experience with the NetHack Learning Environment, we believe fruitful future research may lie along the following directions: finding a better state-encoder, as the current one presents a bottleneck in both efficiency (rendering is expensive) and locality (only the small part of the terminal is used). Another interesting research direction would be to assess recently appeared recurrence mechanisms such as Linear Recurrent Unit (Orvieto et al., 2023), which might also speed up the training process without hurting the performance. Also, as the interest in generalization properties will appear, it would be a great addition to include more datasets that will provide metadata on the seeds used for data generation, as it will allow to assess trained agents on both seen and unseen seeds to quantify the generalization gap more systematically.
Overall, we firmly believe that NetHack provides a nice playground for investigating how to build a next generation of reinforcement learning agents using prior data that would encompass stronger generalization and memory capabilities. To this end, we plan to continuously maintain the benchmark, accompanying tools, and curate new datasets if considered useful for further advancements.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pp. 104-114. PMLR, 2020.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-29320, 2021.
Dmitriy Akimov, Vladislav Kurenkov, Alexander Nikulin, Denis Tarasov, and Sergey Kolesnikov. Let offline rl flow: Training conservative agents in the latent space of normalizing flows. arXiv preprint arXiv:2211.11096, 2022.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.
Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. A brief survey of deep reinforcement learning. arXiv preprint arXiv:1708.05866, 2017.
Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639-24654, 2022.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https://www.wandb.com/. Software available from wandb.com.
Liang Yu, Weiwei Xie, Di Xie, Yulong Zou, Dengyin Zhang, Zhixin Sun, Linghua Zhang, Yue Zhang, and Tao Jiang. Deep reinforcement learning for smart home energy management. IEEE Internet of Things Journal, 7(4):2751-2762, 2019.
Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforcement learning. In Conference on Robot Learning, 2020.
A What Is Inside the Datasets?
Every dataset is repacked into HDF5 files similar to Fu et al. (2020). The data keys are described in Table 3; along to the typical  $(s_t, a_t, r_t, d_t)$  tuples, the metadata is also provided as the datasets' attributes with a comprehensive information about specific trajectories similar to Hambro et al. (2022b). The re-packing script is provided at https://github.com/corl-team/katakomba/tree/main/scripts/generate_small_dataset.py.
Table 3: The re-packed datasets constitute of transformed data from Hambro et al. (2022b). Dissimilar the the large scale dataset, the repacked data is now in the format familiar to the ORL practitioners. We also save the metadata for each trajectory, for a comprehensive description, please, see Appendix F in Hambro et al. (2022b).
Name	Type	Shape	Description
tty_chars	np uint8	[B, T, H, W]	st: The on-screen characters (default screen size 80 x 24).
ttycolors	np.int8	[B, T, H, W]	st: The on-screen colors for each character.
tty_cursor	np.int16	[B, T, 2]	st: The coordinates of the on-screen cursor.
actions	np uint8	[B, T]	at: The NLE actions the player made in response to the st.
rewards	np.int32	[B, T]	rt: The difference between in-game scores at states st and st-1. Note that this was used in all implementations of the algorithms provided in Hambro et al. (2022b). We also found that without this reward-shaping, all offline RL algorithms failed completely.
done	np uint8	[B, T]	dt: An indicator whether the current state is the last one in the trajectory.
B License
Our codebase and repacked datasets are released under the NETHACK GENERAL PUBLIC LICENSE. The original NetHack Learning environment (Küttler et al., 2020) and large-scale datasets (Hambro et al., 2022b) are also released under NETHACK GENERAL PUBLIC LICENSE.
C General Ethic Conduct and Potential Negative Societal Impact
To the best of our knowledge, our work does not present any direct potential negative societal impact.
As of the general ethic conduct, we believe that the most relevant issue to be discussed is the "Consent to use or share the data". Our work is largely built upon both the NetHack Learning Environment (Küttler et al., 2020) and the corresponding large-scale dataset (Hambro et al., 2022b), and as already described in the Appendix B both are distributed under the NETHACK GENERAL PUBLIC LICENSE that explicitly allows for re-usage and re-distribution.
D Resources and Statistics
We used 64 separated computational nodes with 1xA100, 14CPU, 128GB RAM, and the NVMe as long-term storage for all our experiments. All the values reported in the paper were also obtained under this configuration. One can also find more detailed information inside the Weights&Biases logs in the code repository.
Table 4: Scores used for Normalization. You can also find them at https://github.com/corl-team/katakomba/blob/main/katakomba/util/scores.py. For other statistics, please, see Table 2 in the main text.
Tasks	Minimum Score	Maximum Score	Mean Score
Base (Role-Centric)	-	-	-
arc-hum-neu	0.0	138103.0	6636.44
bar-hum-neu	0.0	292342.0	17836.68
cav-hum-neu	0.0	258978.0	12113.87
hea-hum-neu	0.0	64337.0	4068.27
kni-hum-law	0.0	419154.0	14137.06
mon-hum-neu	0.0	171224.0	17456.05
pri-hum-neu	0.0	114269.0	7732.69
ran-hum-neu	0.0	54874.0	8067.99
rog-hum-cha	0.0	68628.0	4818.20
sam-hum-law	0.0	155163.0	11009.36
tou-hum-neu	0.0	59484.0	4211.47
val-hum-neu	16.0	313858.0	18624.77
wiz-hum-neu	0.0	71709.0	5323.48
Extended (Race-Centric)	-	-	-
pri-elf-cha	0.0	83744.0	7109.35
ran-elf-cha	0.0	66690.0	9014.18
wiz-elf-cha	0.0	71664.0	5005.16
arc-dwa-law	0.0	83496.00	5445.69
cav-dwa-law	0.0	161682.0	11893.48
val-dwa-law	0.0	1136591.0	23473.61
arc-gno-neu	0.0	110054.0	5316.57
cav-gno-neu	0.0	142460.0	10083.06
hea-gno-neu	0.0	69566.0	3783.93
ran-gno-neu	0.0	58137.0	6965.04
wiz-gno-neu	0.0	37376.0	4317.51
bar-orc-cha	0.0	164296.0	17594.38
ran-orc-cha	3.0	69244.0	7608.48
rog-orc-cha	0.0	54892.0	4897.69
wiz-orc-cha	0.0	40871.0	5016.74
Complete (Alignment-Centric)	-	-	-
arc-hum-law	2.0	84823.0	5826.35
cav-hum-law	0.0	156966.0	12462.82
mon-hum-law	7.0	190783.0	16091.57
pri-hum-law	0.0	99250.0	6847.99
val-hum-law	0.0	428274.0	26103.03
bar-hum-cha	0.0	164446.0	18228.11
mon-hum-cha	0.0	223997.0	18353.30
pri-hum-cha	0.0	58367.0	8262.56
ran-hum-cha	3.0	62599.0	8378.50
wiz-hum-cha	0.0	55185.0	5316.82
For all algorithms, hyperparameters have been reused from previous work whenever possible. For BC, CQL, and IQL reference values, see Appendix I.4 in the Hambro et al. (2022b). For AWAC, hyperparameters from IQL were reused due to the very similar policy updating scheme. For REM, hyperparameters were taken from the original work (see Agarwal et al. (2020)).
As in Hambro et al. (2022b), and in contrast to the original CQL implementation, we multiply the TD loss by the  $\alpha$  coefficient instead of the CQL loss, as we observed better results with such a scheme. We performed a search for  $\alpha \in [0.0001, 0.0005, 0.001, 0.01, 0.05, 0.1, 0.5, 1.0]$  with best value  $\alpha = 0.0001$ .
Table 5: BC hyperparameters.
Parameter	Value
optimizer	AdamW (Kingma & Ba, 2014; Loshchilov & Hutter, 2017)
training iterations	500000
batch size	64
sequence length	16
learning rate	3e-4
weight decay	0.0
state encoder	Chaotic-Dwarven-GPT-5(Hambro et al., 2022a,b)
LSTM hidden dim	2048
LSTM layers	2
LSTM dropout	0.0
use previous action	True
Table 6: CQL hyperparameters. Note that in our implementation, the  $\alpha$  coefficient multiplies the TD loss.
Parameter	Value
optimizer	AdamW (Kingma & Ba, 2014; Loshchilov & Hutter, 2017)
training iterations	500000
batch size	64
sequence length	16
learning rate	3e-4
weight decay	0.0
state encoder	Chaotic-Dwarven-GPT-5(Hambro et al., 2022a,b)
LSTM hidden dim	2048
LSTM layers	2
LSTM dropout	0.0
use previous action	True
tau (τ)	5e-3
gamma (γ)	0.999
reward clip range	[-10.0, 10.0]
alpha (α)	1e-4
Table 7: IQL hyperparameters.
Parameter	Value
optimizer	AdamW (Kingma & Ba, 2014; Loshchilov & Hutter, 2017)
training iterations	500000
batch size	64
sequence length	16
learning rate	3e-4
weight decay	0.0
state encoder	Chaotic-Dwarven-GPT-5(Hambro et al., 2022a,b)
LSTM hidden dim	2048
LSTM layers	2
LSTM dropout	0.0
use previous action	True
tau (τ)	5e-3
gamma (γ)	0.999
reward clip range	[-10.0, 10.0]
expectile	0.8
temperature	1.0
advantage clip max	100
Table 8: AWAC hyperparameters.
Parameter	Value
optimizer	AdamW (Kingma & Ba, 2014; Loshchilov & Hutter, 2017)
training iterations	500000
batch size	64
sequence length	16
learning rate	3e-4
weight decay	0.0
state encoder	Chaotic-Dwarven-GPT-5(Hambro et al., 2022a,b)
LSTM hidden dim	2048
LSTM layers	2
LSTM dropout	0.0
use previous action	True
tau (τ)	5e-3
gamma (γ)	0.999
reward clip range	[-10.0, 10.0]
temperature	1.0
advantage clip max	100
Table 9: REM hyperparameters.
Parameter	Value
optimizer	AdamW (Kingma & Ba, 2014; Loshchilov & Hutter, 2017)
training iterations	500000
batch size	64
sequence length	16
learning rate	3e-4
weight decay	0.0
state encoder	Chaotic-Dwarven-GPT-5(Hambro et al., 2022a,b)
LSTM hidden dim	2048
LSTM layers	2
LSTM dropout	0.0
use previous action	True
tau (τ)	5e-3
gamma (γ)	0.999
reward clip range	[-10.0, 10.0]
ensemble heads	200.0
Table 10: The effect of CQL policy constraint strength on the performance. Results, which are averaged across 3 seeds, are sorted based on the final unnormalized game score.
Alpha (α)	Return
0.0001	526.72 ± 71.37
0.0005	396.50 ± 39.35
0.001	395.56 ± 139.92
0.05	226.55 ± 196.45
0.01	32.42 ± 43.69
0.1	14.04 ± 10.29
0.5	0.00 ± 0.00
1.0	0.59 ± 0.45
F Results per Benchmark Categories
In this section, we report the results stratified by the introduced categories. If one is willing to inspect specific datasets, we organized all training logs into Weights&Biases public reports, found at https://wandb.ai/tlab/NetHack/reports.
Note that one can find all the evaluation scores (for more than one checkpoint) within the runs and use them for any evaluation tools of interest. Also, we provide convenient scripts for constructing RLiable (Agarwal et al., 2021) graphs based on the provided runs that can be configured for one's purposes as well (see https://github.com/crl-team/katakomba/tree/main/scripts/rliable_report.py).
(a) Bootstrapped point estimates.
(b) Performance profiles.
(c) Probability of improvement of BC to other algorithms.
Figure 3: Normalized performance under the Katakomba benchmark for Base datasets. Each algorithm was run for three seeds and evaluated over 50 episodes resulting in 1950 points for constructing these graphs.
(a) Bootstrapped point estimates.
(b) Performance profiles.
(c) Probability of improvement of BC to other algorithms.
Figure 4: Normalized performance under the Katakomba benchmark for Extended datasets. Each algorithm was run for three seeds and evaluated over 50 episodes resulting in 2250 points for constructing these graphs.
(a) Bootstrapped point estimates.
(b) Performance profiles.
Figure 5: Normalized performance under the Katakomba benchmark for Complete datasets. Each algorithm was run for three seeds and evaluated over 50 episodes resulting in 1500 points for constructing these graphs.
(a) Bootstrapped point estimates.
(b) Performance profiles.
Figure 6: Death levels under the Katakomba benchmark for Base datasets. Each algorithm was run for three seeds and evaluated over 50 episodes resulting in 1950 points for constructing these graphs.
(a) Bootstrapped point estimates.
(b) Performance profiles.
Figure 7: Death level under the Katakomba benchmark for Extended datasets. Each algorithm was run for three seeds and evaluated over 50 episodes resulting in 2250 points for constructing these graphs.
(a) Bootstrapped point estimates.
(b) Performance profiles.
Figure 8: Death levels under the Katakomba benchmark for Complete datasets. Each algorithm was run for three seeds and evaluated over 50 episodes resulting in 1500 points for constructing these graphs.
(a) Bootstrapped point estimates.
(b) Performance profiles.
Figure 9: Unnormalized in-game score under the Katakomba benchmark for Base datasets. Each algorithm was run for three seeds and evaluated over 50 episodes resulting in 1950 points for constructing these graphs.
(a) Bootstrapped point estimates.
(b) Performance profiles.
Figure 10: Unnormalized in-game score under the Katakomba benchmark for Extended datasets. Each algorithm was run for three seeds and evaluated over 50 episodes resulting in 2250 points for constructing these graphs.
(a) Bootstrapped point estimates.
(b) Performance profiles.
Figure 11: Unnormalized in-game score under the Katakomba benchmark for Complete datasets. Each algorithm was run for three seeds and evaluated over 50 episodes resulting in 1500 points for constructing these graphs.